no, we've only been using metric and model logging so far

everything is working as expected

we're using EC2

nope

yeah, that's exactly what I'm looking to right now 😃

works like a charm! you guys are the best, as always =)

nope, didn't work =(

https://github.com/allegroai/clearml/issues/496

not quite. for example, I’m not sure which info is stored in Elastic and which is in MongoDB

got it, thanks!

not necessarily, there are rare cases when container keeps running after experiment is stopped or aborted

will do!

I guess I could manually explore different containers and their content 😃 as far as I remember, I had to update Elastic records when we moved to the new cloud provider in order to update model URLs

on the side note, is there any way to automatically give more meaningful names to the running docker containers?

done!

I’ll let you know 😃

we're using os.getenv in the script to get a value for these secrets

will it pass variables to the training containers?

1.2.3

yes

it works, but it's not very helpful since everybody can see a secret in logs:
Executing: ['docker', 'run', '-t', '--gpus', '"device=0"', '-e', 'DB_PASSWORD=password']

right now we can pass github secrets to the clearml agent training containers ( CLEARML_AGENT_GIT_PASS) to install private repos

we need a way to pass secrets to access our database with annotations

parents and children. maybe tags, maybe separate tab or section, idk. I wonder if anyone else is interested in this functionality, for us this is a very common case

agent.hide_docker_command_env_vars.extra_keys: ["DB_PASSWORD=password"]

like this? or ["DB_PASSWORD", "password"]

I guess I could edit docker-compose.yaml

thank you

tags are somewhat fine for this, I guess, but there will be too many of them eventually, and they do not reflect sequential nature of the experiments

hard to say, maybe just “related experiments” in experiment info would be enough. I’ll think about it

this is probably what I need, thanks. I'll check if it works

nice!

that's right
for example, there are tasks A, B, C
we run multiple experiments for A, finetune some of them in separate tasks, then choose one or more best checkpoints, run some experiments for task B, choose the best experiment, and finally run task C

so we get a chain of tasks: A - A-ft - B- C

ClearML pipeline doesn't quite work here because we would like to analyze results of each step before starting next task

but it would be great to see predecessors of each experiment in the chain