🙂

` param = {'arg': value}
task.connect(param, section='new section')

create pipeline here

pipeline `

trains-agent runs a container from that image, then clones ...

That is correct

I'd like the base_docker_image to not only be defined at runtime

I see, may I ask why not just build it once, push it into artifactory and then have trains-agent use it? (it will be much faster)

However, it's very interesting why ability to cache the step impacts artifacts behavior

From you log:

videos_df = StorageManager.download_file(videos_df)

Seems like "videos_df" is the DataFrame, why are you trying to download the DataFrame ? I would expect to try and download the pandas file, not a DataFrame object

yes that makes send, I think what happened is one of the processes completed the Task (i.e. closed it) before the others did, and so they threw exception.

I switched to have all tasks in a separate process

I think that's probably the best (performance wise as well), nice!

GreasyPenguin14 whats the clearml version you are using, OS & Python ?
Notice this happens on the "connect_configuration" that seems to be called after the Task was closed, could that be the case ?

Hmm let me check something

MiniatureHawk42 could you re-test with the latest from github?
pip install -U git+

Hmm let me check something

Yes I think you are correct

Would be very cool if you could include this use case!

I totally think we should, any chance you can open an Issue, so this feature is not lost?

JitteryCoyote63 did you add the bash script here: https://github.com/allegroai/trains-agent/blob/master/docs/trains.conf#L99

btw: both should work fine

Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/build/80754af9/attrs_1604765588209/work'Seems like pip failed creating a folder
Could it be you are out of space ?

We are planning an RC later this week, I'll make sure this fix is part of it

Ohh I see, makes total sense. I'm assuming the code base itself can do both 🙂

If you are using user/pass for the git (i.e. not ssh key) we are not passing it to the pip install (and come to think about it, we probably should?!)

PompousBeetle71 is this ArgParser argument or a connected dictionary ?

PompousBeetle71 BTW: if you remove the type=str from the argparse, it will do what you want, None will stay None (instead of ''), all other values will be of type str as this is always the default 🙂

PompousBeetle71 quick question, will you ever want to pass an empty string ? reason for asking is that it is either one or the other, there is no way for Trains to actually differentiate (from the web UI, perspective this is just an empty string field...)

Hmm... That's what happens with the exception of None/'' if type is str... There is no way to differentiate in the UI.
This is why we opted for type=str will "cast" everything to str so you always get str, while not specifying a type will leave the variable as is... If you have an idea on how to support both, feel free to suggest 🙂

ReassuredTiger98 when you look for task "dca2e3ded7fc4c28b342f912395ab9bc" there are no artifacts ?
Could you add some prints? this should have worked...

Hi ReassuredTiger98
Could you add some print ? before / after the artifact upload?
Also what's the clearml version you are using ?

Hi StraightCoral86

When I run an experiment using 

Task.create()

 ,

Use Task.init 🙂
Task.create is meant to create an extranl Task (i.e. Job) ins the system, Not to auto-gernerate a job from the running code. Make sense ?

Plan is to have it out in the next couple of weeks.
Together with a major update in v0.16

Hi SlimyRat21 :

Tool that will help me track and manage the different configs and simulation logs across different runs and versions of the simulation.

Definitely covered by Trains, it also does that with very little code changes (if any) to your current code base

Tool that will help me gather and compare the results from specific simulation runs

Same as above 🙂

Do you you have an experience or tips on using trains for non-ML before investing time into this and seeing...

UnevenDolphin73 I have a suspicion we have a few terms mixed:
hyperparameters :
These are essentially key/value.
when you call Task. connect (dict_with_params), clearml will flatten the dict and you end up with key/value
configuration objects :
These are actually blobs of text, the UI will show as is
When you call my_local_file=Task. connect_configuration (name, "path/to/config/file")
The entire Content of the config file is stored on the Task object itself.

Back to the use case, instead ...

Yes... I think that this might be a bit much automagic even for clearml 😄

Could be nice to write some automation

PompousBeetle71 If this is argparser and the type is defined, the trains-agent will pass the equivalent in the same type, with str that amounts to '' . make sense ?