Yes

I cannot execute step 4 because I can't get past step 3. Does that make sense?

I hope this helps

I had to manually create a dump for the mongo data and import it into 4.4. I was just referring to adding a note to the documentation for other users.

I posted the https://stackoverflow.com/questions/64636294/trains-reusing-previous-task-id/64636297#64636297 on stackoverflow with the answer :)

hmm, ok

mongo 4.4 image does not launch a container if the data in mongo dir is for previous versions. We should add that comment in the documentation

Oh ok. That's great. I will try it.

I'm getting the same error when I followed the instructions to the letter.

Here is one line from the mongo docker output
"This version of MongoDB is too recent to start up on the existing data files. Try MongoDB 4.2 or earlier."

There was some complication during the upgrade so I had to resort to the manual process.

I have now been able to upgrade by dumping the mongodb data and restoring it independently.

Yes the 'training' is my main code. You can think of it has launching a job (training or inference). My main code launches multiple jobs using multiprocessing. Each job is a seprate task for clearml that gets logged. Does that make sense?

Thanks for the tip with the config file. I have reverted back to 0.17.4 but will try this.

ok

Ok, So Git credentials are present at two locations - 1) outside the agent config and 2) inside it. I updated credentials at both locations and now I'm seeing agent.git_user = <username> in the dump, but I still have the same issue.
` # Set GIT user/pass credentials

leave blank for GIT SSH credentials ...

Thanks

Haha.. that would be a problem then!

sounds good

Hi AgitatedDove14 , I'll wait for you to reply on Github before I add my comments to these points.

Wait but that will skip all the assertion checks that I have in my code?!

Hi AgitatedDove14 , yes, I was able to change the color from the UI. But this may be less than ideal for the following use case.

A model is an ensemble of say 10 models. Each member of the ensemble generates two train-validation curves. So for 1 model, I will have 20 plots. There are two problems with the current setup:

Manually changing colors of all the plots is not feasible The default color scheme is not consistent and changes randomly with every run
It would be nice if I can control t...

Great, thanks!

Steps 1 and 2 basically copy mongo 3.6 data into a new dir mongo_4 but mongo image of version 4.4 does not accept that data. So I had to perform the following steps:

Launch docker container with mongo=3.6 dump data using mongo dump Launch docker container with mongo=4.4 and empty mongo_4 data dir Restore the dump data using mongo restore
This made sure the data is now compatible with mongo 4.0 or greater

Hmm, ok. Yes that would make it easier.

From architectural point of view - say I know I'll be running the experiment on a trains-agent , when I initialize and execute the experiment locally, how hard would it be to instead send all the execution details and env to the trains agent and run it directly there? Can the configuration be packaged when we initialize the Task? Does the question make sense?

Haha.. ok, good to know

I'm using docker to run the experiment. Could it be that the config in the docker container doesn't have the git credentials?

2. interesting error, maybe we can revert to "thread mode" if running under a daemon. (I have to admit, I'm not sure why python has this limitation, let me check it...)

Yes, I'm not sure either. I have banged my head against the wall in trying to have multiple level of subprocesses, but it gets too complicated with python. Let me know what you find out

ok, thanks

This is happening manually. I am not using agent yet

Got it. I haven't tried setting up trains-agent yet so I don't know much about the overhead of launching the agent. I'd imagine if it has to create the full environment (installing requirements, etc), the overhead might not be that low. But as I'm reading, it looks like I can use a docker image with the full env. Is my understanding correct?

Got it. That makes sense. Thanks!