agent.hide_docker_command_env_vars.extra_keys: ["DB_PASSWORD=password"]

like this? or ["DB_PASSWORD", "password"]

ah, I see, I still keep it in agent.extra_docker_arguments

nice, thanks

thanks! this bug and cloning problem seem to be fixed

okay, I will try it with port

maybe I should use explicit reporting instead of Tensorboard

yeah, I am aware of trains-agent, we are planning to start using it soon, but still, copying original training command would be useful

okay, I will open an issue

I guess I could manually explore different containers and their content 😃 as far as I remember, I had to update Elastic records when we moved to the new cloud provider in order to update model URLs

same here, no live logs

give me a sec

task = Task.get_task(task_id = args.task_id)
task.mark_started()
task.set_parameters_as_dict(
{
"General": {
"checkpoint_file": model.url,
"restart_optimizer": False,
}
}
)
task.set_initial_iteration(0)
task.mark_stopped()
Task.enqueue(task = task, queue_name = task.data.execution.queue)

yes. we upload artifacts to Yandex.Cloud S3 using ClearML. we set " s3://storage.yandexcloud.net/clearml-models " as output uri parameter and add this section to the config:
{
host: " http://storage.yandexcloud.net "
key: "KEY"
secret:"SECRET_KEY",
secure: true
}

this works like a charm. but download button in UI is not working

not sure what you mean. I used to do task.set_initial_iteration(task.get_last_iteration()) in the task resuming script, but in the training code I explicitly pass global_step=epoch to the TensorBoard writer

yeah, we've used pipelines in other scenarios. might be a good fit here. thanks!

I added the link just in case anyway 😃

also, is there any way to install a repo that we clone as a package. we often use absolute imports and do "pip install -e ." to utilize it
sorry there are so many questions, we just really want to migrate to trains-agent)

yeah, server (1.0.0) and client (1.0.1)

exactly

yes

do you have any idea why cleanup task keeps failing then (it used to work before the update)

for me, increasing shm-size usually helps. what does this RC fix?

does this mean that setting initial iteration to 0 should help?

okay, what do I do if it IS installed?

in the docker image

on the side note, is there any way to automatically give more meaningful names to the running docker containers?

this is how the interface looks

everything is working as expected

that was tough but I finally manage to make it working! thanks a lot for your help, I definitely wouldn't be able to do it without you =)

the only problem that I still encounter is that sometimes there are random errors in the beginning of the runs, especiialy when I enqueue multiple experiments at the same time (I have 4 workers for 4 GPUs).
for example, this
from torch.utils.tensorboard import SummaryWrite
writer = SummaryWriter()
sometimes randomly leads to FileNotFoundError: [Errno...

got it, thanks!

it’s a pretty standard pytorch train/eval loop, using pytorch dataloader and https://docs.monai.io/en/stable/_modules/monai/data/dataset.html