this would be great. I could just then pass it as a hyperparameter

nice! exactly what I need, thank you!

hmmm allegroai/trains:latest whatever it is

LOL
wow 😃
I was trying to find how to create a queue using CLI 😃

it might be that there is not enough space on our SSD, experiments cache a lot of preprocessed data during the first epoch...

kind of

if you click on the experiment name here, you get 404 because link looks like this:
https://DOMAIN/projects/PROJECT_ID/EXPERIMENT_ID
when it should look like this:
https://DOMAIN/projects/PROJECT_ID/experiments/EXPERIMENT_ID

another stupid question - what is the proper way to delete a worker? so far I've been using pgrep to find the relevant PID 😃

we are working on the medical projects, so probably images of the different body parts 😃

it will probably screw up my resource monitoring plots, but well, who cares 😃

wow, thanks, just updated our server!
can't seem to find these metrics snapshot plots =) how do I plot one?

in the docker image

our GPUs are 48GB, so it's quite wasteful to only run one job per GPU
yeah, I'm aware of that, I would have to make sure they don't fail to infamous CUDA out of memory, but still

thanks! I need to read all parts of documentation really carefully =) for some reason, couldn't find this section

will do

will do, thanks

okay, what do I do if it IS installed?

I'm not sure since names of these parameters do not match with boto3 names, and num_download_attempt is passed https://github.com/allegroai/clearml/blob/3d3a835435cc2f01ff19fe0a58a8d7db10fd2de2/clearml/storage/helper.py#L1439 as container.config.retries

yeap, it works 😍

yeah, that sounds right! thanks, will try

yeah, it works for the new projects and for the old projects that have already had a description

oh wow, I didn't see delete_artifacts_and_models option

I guess we'll have to manually find old artifacts that are related to already deleted tasks

we already have cleanup service set up and running, so we should be good from now on

two more questions about cleanup if you don't mind:
what if for some old tasks I get WARNING:root:Could not delete Task ID=a0908784a2a942c3812f947ec1f32c9f, 'Task' object has no attribute 'delete'? What's the best way of cleaning them? What is the recommended way of providing S3 credentials to cleanup task?

thanks!

we're using the latest version of clearml, clearml agent and clearml server, but we've been using trains/clearml for 2.5 years, so there are some old tasks left, I guess 😃

I guess, this could overcomplicate ui, I don't see a good solution yet.

as a quick hack, we can just use separate name (eg "best_val_roc_auc") for all metric values for the current best checkpoint. then we can just add columns with the last value of this metric

not necessarily, there are rare cases when container keeps running after experiment is stopped or aborted

will do!

that's right, I have 4 GPUs and 4 workers. but what if I want to run two jobs simultaneously at the same GPU

hard to say, maybe just “related experiments” in experiment info would be enough. I’ll think about it