WOW.. Thanks 💯

Hi CumbersomeCormorant74 ,
This is a server we installed.
The server version is: 0.17
We checked with Chrome, and FireFox

Thanks, ophir

I am sure you add this timeout for a reason.

Probably since increasing the timeout can affect other functionality. .

Am I wrong?

hey, I test it, it looks it works, still it takes much time (mainly in the second run of the code, it part of my eval process)

Yes this is what we are doing 👍

it took around 50 min but it now finish the backup..Thanks

Does it possible to know in advance where the Agent will clone the code?
Or running a link command just before the execution of the code?

Ok looks It is starting the training...
Thanks 💯

Thanks, will try and let you know 😊

So for now I am leaving this issue...
Thanks a lot 🙏 🙌

Hi, AgitatedDove14 Thanks for the answer.

I think the upload reporting (files over 5mb) was added post 0.17 version,

That what I thought...

I think it can be helpful to add it to the conf since 5MB is really small and my files are ~300MB, meaning 60 messages for each upload.
Another option is maybe to configure it as Task.init() parameter

I think both are OK 🙂

I just need it to ran the docker and run the command inside it no?

I did it just because FAIR did it in detectron2 Dockerfile

Regarding of moving the fileserver to S3, what is the best way to move the old data to S3 ?

I think if I will move all the /opt/trains/data/fileserver to s3,
the trains-server will not know that right?

Thanks, I will make sure that all the python packages install as root..
And will let you know if it works

So I ask my boss and DevOps and they say for now we can use the root user inside the docker image...

The hang is still happening in trains==0.15.2rc0

It is now stacking after:
` 2021-03-09 14:54:07
task 609a976a889748d6a6e4baf360ef93b4 pulled from 8e47f5b0694e426e814f0855186f560e by worker ov-01:gpu1
2021-03-09 14:54:08
running Task 609a976a889748d6a6e4baf360ef93b4 inside default docker image: MyDockerImage:v0
2021-03-09 14:54:08
Executing: ['docker', 'run', '-t', '--gpus', '"device=1"', '-e', 'CLEARML_WORKER_ID=ov-01:gpu1', '-e', 'CLEARML_DOCKER_IMAGE=MyDockerImage:v0', '-v', '/tmp/.clearml_agent.jvxowhq4.cfg:/root/clearml.conf', '-v', '/...

Hi DeterminedCrab71 .
Thanks 🙂

Thanks AgitatedDove14 ,
I need to check with my boss that it is OK to share more code, will let you know..

But I will give 0.16 a try when it will release.
🙏

For now we are using AWS batch for running those experiments.
because like this we don`t have to hold machines who waits for the jobs

Thanks I will upgrade the server for now and will let you know

I am trying to reproduce it with little example

how long? 😅
I am now stuck in
Copying index events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b
for more then 40 min 😥

yes it looks like this.. I just wanted to understand if it is should be so slow.. or I did something wrong

the elastic folder is around 30GB

Sure, love to do it when I have more time 🙂

Ohh I understood, so can you give me a short explanation on how to change the meta data?

Thanks I will upgrade my instance type and the add more workers. where I need to configure it?

OHH nice, I thought that it just some kind of job queue on up and running machines