Oh ok. That's great. I will try it.

Hmm, ok. Yes that would make it easier.

From architectural point of view - say I know I'll be running the experiment on a trains-agent , when I initialize and execute the experiment locally, how hard would it be to instead send all the execution details and env to the trains agent and run it directly there? Can the configuration be packaged when we initialize the Task? Does the question make sense?

Haha.. that would be a problem then!

You mean without manually executing it once ?

Yes. Just as it would be executed in trains

ok

Thanks for the tip with the config file. I have reverted back to 0.17.4 but will try this.

What would be the query ? Are you reporting 100+ diff scalars ?

At the moment I am not reporting any scalars related to inference. I'm only reporting data related to training a model. But I would like to report records that result from an inference process. For example the record would contain key_1, key_2, datetime, pred_1, pred_2 ... pred_n. I would have about 20 scalars if each of these fields is reported as a scalar.

The query can be a simple filtering criteria matching some keys ...

This is happening manually. I am not using agent yet

Hi AgitatedDove14 , yes, I was able to change the color from the UI. But this may be less than ideal for the following use case.

A model is an ensemble of say 10 models. Each member of the ensemble generates two train-validation curves. So for 1 model, I will have 20 plots. There are two problems with the current setup:

Manually changing colors of all the plots is not feasible The default color scheme is not consistent and changes randomly with every run
It would be nice if I can control t...

Yes, this is what I'm looking for. Excellent! Thank you!

fatal: could not read Username for ' ': terminal prompts disabled error: Could not fetch originWhy is trains-agent trying read from terminal prompt instead of trains.conf ?

I used trains-agent init to create the config file

I'm using docker to run the experiment. Could it be that the config in the docker container doesn't have the git credentials?

Wait but that will skip all the assertion checks that I have in my code?!

Yes, I am using multiprocessing.Pool to launch each job

SuccessfulKoala55 Yes, I am using the --docker flag.

You are right about the Keyring. Once I make sure credentials are stored in a secure way, it works as expected. Thanks :)

Yes

Steps 1 and 2 basically copy mongo 3.6 data into a new dir mongo_4 but mongo image of version 4.4 does not accept that data. So I had to perform the following steps:

Launch docker container with mongo=3.6 dump data using mongo dump Launch docker container with mongo=4.4 and empty mongo_4 data dir Restore the dump data using mongo restore
This made sure the data is now compatible with mongo 4.0 or greater

mongo 4.4 image does not launch a container if the data in mongo dir is for previous versions. We should add that comment in the documentation

Hi AppetizingMouse58

Yes, I tried to perform steps 3-10, however step 3 raised an error because data files for mongo were incompatible between 3.6 and >4.0

Thanks

I was getting the error in step number 3

I'm getting the same error when I followed the instructions to the letter.

Here is one line from the mongo docker output
"This version of MongoDB is too recent to start up on the existing data files. Try MongoDB 4.2 or earlier."

I cannot execute step 4 because I can't get past step 3. Does that make sense?

The docker container in step 3 does not run because of the incompatibility

That makes sense. The configuration file is located at ~/trains.conf which I believe is the default location.

No I can't see my username printed out in the dump

Got it. That makes sense. Thanks!

2. interesting error, maybe we can revert to "thread mode" if running under a daemon. (I have to admit, I'm not sure why python has this limitation, let me check it...)

Yes, I'm not sure either. I have banged my head against the wall in trying to have multiple level of subprocesses, but it gets too complicated with python. Let me know what you find out

ok, thanks

The second subprocess is by design. It becomes the primary process when clearml does not use multiprocessing. I hope I'm not confusing you further