give me a minute to test

Oh dear, I think your theory might be correct, and this is just the mongo preallocating storage.
Which means the entire /opt/trains just disappeared

Eg, i'm creating a task using 

clearml.Task.create

 , often it doesn't properly get the git diff correctly,

ShakyJellyfish91 Task.create does not store any "git diff" automatically, is there a reason not to use Task.init ?

Hi ElegantCoyote26 , yes I did 🙂
It seems cometml puts their default callback logger for you, that's it.

Hi WittyOwl57
I think what happens is it auto-logs the joblib load/save calls, these calls track models used/created by the code, and attach them to the model repository representing these model.
I'm assuming there are multiple load/save , and there are multiple model instances pointing to the same local file "file:///tmp/..." . The earning basically says it is re-registering existing models.
Make sense ?

Hi TrickyRaccoon92
If you are reporting to tensor-board, then "iteration" equals step. Is this the case?

Seems like credentials error
Do you have everything setup correctly in your ~/clearml.conf ?

Actually scikit implies joblib 🙂 (so you should use scikit, anyhow I'll make sure we add joblib as it is more explicit)

JitteryCoyote63 Should be quite safe, there is no major change that I'm aware of on the ClearML side that can effect it.
That said, wait for after the weekend, we are releasing a new ClearML package, I remember there was something with the model logging, it might not directly have something to do with ignite, but worth testing on the latest version.

Hi @<1545216070686609408:profile|EnthusiasticCow4>
The auto detection of clearml is based on the actual imported packages, not the requirements.txt of your entire python environment. This is why some of them are missing.
That said you can always manually add them

Task.add_requirements("hydra-colorlog") # optional add version="1.2.0"
task = Task.init(...)

(notice to call before Task.init)

yes, looks like. Is it possible?

Sounds odd...
Whats the exact project/task name?
And what is the output_uri?

SubstantialElk6 on the client side?

my pleasure

Run clearml-agent and enqueue the pipeline ? What am i missing?

(I am not an expert on UI to be honest)

Same here 🙂 lol

we can implement this externally

What do you mean by that?

Hi @<1730033904972206080:profile|FantasticSeaurchin8>
You mean in the UI , or when reporting on the SDK?

Any chance there is an env variable you set to get 1.5.0rc0? Because this is the version that is being used

And do you need to run your code inside a docker, or is venv enough ?

but now since 

Task.current_task()

 doesn't work on the pipeline object we have a serious problem

How is that possible ?
Is there a small toy code that can reproduce it ?

TenseOstrich47 as long as on the machine running the agent has credentials to your ECR, when the agent will run Any docker container, it will able to pull it. There is no need to manually change anything, notice the Task itself contains the name of the image it will use

You can only edit I on the cloned (copy of) the original experiment. Make sense?

BTW, we figure out that  

'

  is belong the echo

yep, when seeing the full command it is apparent

pip freezeworks ?

Where do you have your Task.init ?

Hi RoundSeahorse20
Try the following , let me know if it worked.
clear_logger = logging.getLogger('clearml.metrics') clear_logger.setLevel(logging.ERROR)

Okay fixed, you will be able to override it with output_uri=False (which is ignored on remote execution if you have a project default or Task output uri set in the UI).
Make sense ?

Thanks DeliciousBluewhale87 ! greatly appreciated 🙂

Actually it cannot be differed, long story short when the agent is running the same code we have to verify and pass arguments at import time. I have to wonder, I'm expecting the env variables to be preset (I.e previously set for the entire environment) how come they are manually set inside the code (and wouldn't that break when running with an agent)?

If I call explicitly 

task.get_logger().report_scalar("test", str(parse_args.local_rank), 1., 0)

 , this will log as expected one value per process, so reporting works

JitteryCoyote63 and do prints get logged as well (from all processes) ?

his means that you guys internally catch the argparser object somehow right?

Correct 🙂 this is how you get the type checking casting abilities, and a few other perks