sorry that I keep bothering you, I love ClearML and try to promote it whenever I can, but this thing is a real pain in the ass 😃

give me a sec

awesome!

just DMed you a screenshot where you can see a part of the token

we often do ablation studies with more than 50 experiments, and it was very convenient to compare their dynamics at the different epochs

I added the link just in case anyway 😃

also, is there any way to install a repo that we clone as a package. we often use absolute imports and do "pip install -e ." to utilize it
sorry there are so many questions, we just really want to migrate to trains-agent)

we've already restarted everything, so I don't have any logs on hands right now. I'll let you know if we face any problems 😃 slack bot works though! 🎉

I use the following Base Docker image setting: my-docker-hub/my-repo:latest -v /data/project/data:/data

self-hosted ClearML server 1.2.0
SDK version 1.1.6

I updated S3 credentials, I'll check if they work later

it doesn't explain inability to delete logged images and texts though

I'm so happy to see that this problem has been finally solved!

after the very first click, there is a popup with credentials request. nothing happens after that

isn't this parameter related to communication with ClearML Server? I'm trying to make sure that checkpoint will be downloaded from AWS S3 even if there are temporary connection problems

there's https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig parameter in boto3, but I'm not sure if there's an easy way to pass this parameter to StorageManager

1.3.0

network logs

https://allegro.ai/clearml/docs/docs/examples/frameworks/pytorch/notebooks/table/tabular_training_pipeline.html

yeah, backups take much longer, and we had to increase our EC2 instance volume size twice because of these indices

got it, thanks, will try to delete older ones

I guess I could edit docker-compose.yaml

yes

it works, but it's not very helpful since everybody can see a secret in logs:
Executing: ['docker', 'run', '-t', '--gpus', '"device=0"', '-e', 'DB_PASSWORD=password']

we're using os.getenv in the script to get a value for these secrets

1.2.3

nice!

agent.hide_docker_command_env_vars.extra_keys: ["DB_PASSWORD=password"]

like this? or ["DB_PASSWORD", "password"]

ah, I see, I still keep it in agent.extra_docker_arguments

works like a charm!

Error
Failed to get Scalar Charts

done!

sounds like an overkill for this problem, but I don’t see any other pretty solution 😃

thanks! we copy S3 URLs quite often. I know that it’s better to avoid double spaces in task names, but shit happens 😄