ok

Yes, I am using multiprocessing.Pool to launch each job

The second subprocess is by design. It becomes the primary process when clearml does not use multiprocessing. I hope I'm not confusing you further

Yes the 'training' is my main code. You can think of it has launching a job (training or inference). My main code launches multiple jobs using multiprocessing. Each job is a seprate task for clearml that gets logged. Does that make sense?

Haha.. that would be a problem then!

I hope this helps

Ok, great. Thanks

ok, thanks

I had to manually create a dump for the mongo data and import it into 4.4. I was just referring to adding a note to the documentation for other users.

mongo 4.4 image does not launch a container if the data in mongo dir is for previous versions. We should add that comment in the documentation

Yes, this is what I'm looking for. Excellent! Thank you!

hmm, ok

I used trains-agent init to create the config file

Yes, I am using Pool. Here is what I think is happening. clearml launches a subprocess which I assume is a daemonic process. That process in-turn launches a subprocess for training which causes the error I mentioned

Hi AgitatedDove14 , yes, I was able to change the color from the UI. But this may be less than ideal for the following use case.

A model is an ensemble of say 10 models. Each member of the ensemble generates two train-validation curves. So for 1 model, I will have 20 plots. There are two problems with the current setup:

Manually changing colors of all the plots is not feasible The default color scheme is not consistent and changes randomly with every run
It would be nice if I can control t...

Hi AppetizingMouse58

Yes, I tried to perform steps 3-10, however step 3 raised an error because data files for mongo were incompatible between 3.6 and >4.0

The docker container in step 3 does not run because of the incompatibility

Great, thanks!

I was getting the error in step number 3

Wait but that will skip all the assertion checks that I have in my code?!

2. interesting error, maybe we can revert to "thread mode" if running under a daemon. (I have to admit, I'm not sure why python has this limitation, let me check it...)

Yes, I'm not sure either. I have banged my head against the wall in trying to have multiple level of subprocesses, but it gets too complicated with python. Let me know what you find out

Thanks AgitatedDove14

Haha.. ok, good to know

Steps 1 and 2 basically copy mongo 3.6 data into a new dir mongo_4 but mongo image of version 4.4 does not accept that data. So I had to perform the following steps:

Launch docker container with mongo=3.6 dump data using mongo dump Launch docker container with mongo=4.4 and empty mongo_4 data dir Restore the dump data using mongo restore
This made sure the data is now compatible with mongo 4.0 or greater

SuccessfulKoala55
For security reasons I don't want to have my password written out in a file. I'm trying to use https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token (PAT) from Github but I get authentication error. Is there an issue using PAT?

Hi AgitatedDove14 Thanks for checking. I would like to compare several experiments (plots, hyperparams, etc), so it would have to been nice to do it in the UI. I have to search through the long list right now. With python, I can only do few of the things that I intend to do. Is this something that might be added in the future?

sounds good

fatal: could not read Username for ' ': terminal prompts disabled error: Could not fetch originWhy is trains-agent trying read from terminal prompt instead of trains.conf ?

Yes

Got it. That makes sense. Thanks!