these are being repeated as well for a single task (this is training a t5_model with transformers): (edited)
Seems like someone is storing lots of files with torch.save that ClearML automatically logs.
You can disable the autolog:task = Task.init(..., auto_connect_frameworks={'pytorch': False})
AgitatedDove14 - yeah wanted to see what’s happening before disabling as I wasn’t sure if this is what’s expected.
AgitatedDove14 - i am disabling pytorch like above but still see auto models . I even see model import when running evaluation from a loaded model
Also is there a way to disable pytorch like this from clearml.conf?
Does a pipeline step behave differently?
Are you disabling it in the pipeline step ?
(disabling it for the pipeline Task has no effect on the pipeline steps themselves)
OK i found what’s happening:
I had an additional Task.init() - just the blank one, to get the project name. Adding the disable to that as well fixed the issue
` if project_name is None and Task.current_task() is not None:
project_name = Task.current_task().get_project_name()
if project_name is None and not Task.running_locally():
task = Task.init()
project_name = task.get_project_name() `
Maybe two thing here:
If Task.init() is called in an already running task, don’t reset auto_connect_frameworks? (if i am understanding the behaviour right) Option to disable these in the clearml.conf
If Task.init() is called in an already running task, don’t reset auto_connect_frameworks? (if i am understanding the behaviour right)
Hmm we might need to somehow store the state of it ...
Option to disable these in the clearml.conf
I think this will be to general, as this is code specific , no?
Yeah concerns make sense.
The underlying root issue is unnecessary models being added or at least what I think are unnecessary and even happening when you load a model to test.
Do people use ClearML with huggingface transformers? The code is std transformers code.
Will create an issue.
Do people use ClearML with huggingface transformers? The code is std transformers code.
I believe they do 🙂
There is no real way to differentiate between, "storing model" using torch.save and storing configuration ...
Can you point me at relevant code in ClearML for the autoconnect so that I can understand exactly what's happening