Hmm, ok. Yes that would make it easier.

From architectural point of view - say I know I'll be running the experiment on a trains-agent , when I initialize and execute the experiment locally, how hard would it be to instead send all the execution details and env to the trains agent and run it directly there? Can the configuration be packaged when we initialize the Task? Does the question make sense?

Great, thanks!

fatal: could not read Username for ' ': terminal prompts disabled error: Could not fetch originWhy is trains-agent trying read from terminal prompt instead of trains.conf ?

but when I try to clone the repo directly, PAT works.

SuccessfulKoala55 Yes, I am using the --docker flag.

You are right about the Keyring. Once I make sure credentials are stored in a secure way, it works as expected. Thanks :)

I used trains-agent init to create the config file

That makes sense. The configuration file is located at ~/trains.conf which I believe is the default location.

No I can't see my username printed out in the dump

I'm using docker to run the experiment. Could it be that the config in the docker container doesn't have the git credentials?

Oh ok. That's great. I will try it.

This is happening manually. I am not using agent yet

ok

2. interesting error, maybe we can revert to "thread mode" if running under a daemon. (I have to admit, I'm not sure why python has this limitation, let me check it...)

Yes, I'm not sure either. I have banged my head against the wall in trying to have multiple level of subprocesses, but it gets too complicated with python. Let me know what you find out

Thanks for the tip with the config file. I have reverted back to 0.17.4 but will try this.

Yes, I am using multiprocessing.Pool to launch each job

Haha.. ok, good to know

Haha.. that would be a problem then!

ok, thanks

You will need to habe multiple 

trains-agent

s  but they will be sharing the same queue (i.e. pulling jobs from the same queue the HPO process is pushing to)
Make sense ?

Hmm. So say I have a parameter NUM_PARALLEL_EXECUTIONS , I can programmatically launch that many trains-agent for every optimization run?!

Yes, I will do that.

Hi AgitatedDove14 , I'll wait for you to reply on Github before I add my comments to these points.

The docker container in step 3 does not run because of the incompatibility

Yes, I tried to run steps 1,2,3,4 in order but got stuck at 3

I'm getting the same error when I followed the instructions to the letter.

Here is one line from the mongo docker output
"This version of MongoDB is too recent to start up on the existing data files. Try MongoDB 4.2 or earlier."

Steps 1 and 2 on this https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_mongo44_migration/ say to backup opt/clearml/data/mongo and uncompress into /opt/clearml/data/mongo_4 . Isn't it just copying the old data files?

Thanks

hmm, ok

Hi AppetizingMouse58

Yes, I tried to perform steps 3-10, however step 3 raised an error because data files for mongo were incompatible between 3.6 and >4.0

I had to manually create a dump for the mongo data and import it into 4.4. I was just referring to adding a note to the documentation for other users.

I hope this helps

Yes