Some more context: the second experiment finished and now, in the UI, in workers&queues tab, I see randomly
trains-agent-1 | - | - | - | ... (refresh page) trains-agent-1 | long-experiment | 12h | 72000 |

AgitatedDove14 Yes with the command you shared I can now ssh again manually to the agent, but I still clearml-agent will raise the same error

Basically what I did is:
` if task_name is not None:
project_name = parent_task.get_project_name()
task = Task.get_task(project_name, task_name)
if task is not None:
return task

Otherwise here I create the Task `

No 🙂

Could be, but not sure -> from 0.16.2 to 0.16.3

But I see in the agent logs:
Executing: ['docker', 'run', '-t', '--gpus', '"device=0"', ...

to pass secrets to each experiment

Still investigating, task.data.last_iteration is correct (equal to engine.state["iteration"] ) when I resume the training

Alright I have a followup question then: I used the param --user-folder “~/projects/my-project”, but any change I do is not reflected in this folder. I guess I am in the docker space, but this folder is not linked to my the folder on the machine. Is it possible to do so?

by mistake I have two agents started in one machine

SuccessfulKoala55 Here is the trains-elastic error

https://github.com/allegroai/clearml-agent.git@f019905720529acbd316bd39b67c5ab0c02fcd55 to be exact

Thanks! I will investigate further, I am thinking that the AWS instance might have been stuck for an unknown reason (becoming unhealthy)

CostlyOstrich36 yes, when I scroll up, a new events.get_task_log is fired and the response doesn’t contain any log (but it should)

The host is accessible, I can ping it and even run curl " http://internal-aws-host-name:9200/_cat/shards " and get results from the local machine

I assume you’re using a self-hosted server?

Yes

Still failing with the same error 😞

0.17.1

Will it freeze/crash/break/stop the ongoing experiments?

as it's also based on pytorch-ignite!

I am not sure to understand, what is the link with pytorch-ignite?

We're in the brainstorming phase of what are the best approaches to integrate, we might pick your brain later on

Awesome, I'd be happy to help!

Hi SuccessfulKoala55 , Yes it’s for the same host/bucket - I’ll try with a different browser

we use task.models[] 🙂

 you mean “docker” was not installed and it did not throw an error ?

Yes docker was not installed in the machine

Yes you must make sure the docker can mount a persistent folder for you to work on.

Ok, it would be nice to have a --user-folder-mounted that do the linking automatically

on a p3.2xlarge instance

TimelyPenguin76 , no, I’ve only set the sdk.aws.s3.region = eu-central-1 param

What happens is different error but it was so weird that I thought it was related to the version installed

Restarting the server ( docker-compose down then docker-compose up ) solved the problem 😌 All experiments are back

SuccessfulKoala55 I want to avoid writing creds in plain text in the config file

I am using pip as a package manager, but i start the trains-agent inside a conda env 😄

There is no need to add creds on the machine, since the EC2 instance has an attached IAM profile that grants access to s3. Boto3 is able retrieve the files from the s3 bucket