AgitatedDove14 Yes exactly, I tried the fix suggested in the github issue urllib3>=1.25.4 and the ImportError disappeared 🙂

Hi AgitatedDove14 , that’s super exciting news! 🤩 🚀
Regarding the two outstanding points:
In my case, I’d maintain a client python package that takes care of the pre/post processing of each request, so that I only send the raw data to the inference service and I post process the raw output of the model returned by the inference service. But I understand why it might be desirable for the users to have these steps happening on the server. What is challenging in this context? Defining how t...

Ok, now I would like to copy from one machine to another via scp, so I copied the whole /opt/trains/data folder, but I got the following errors:

Could be, but not sure -> from 0.16.2 to 0.16.3

now I can do nvcc --version and I get
Cuda compilation tools, release 10.1, V10.1.243

Thanks! I will investigate further, I am thinking that the AWS instance might have been stuck for an unknown reason (becoming unhealthy)

Thanks a lot, I will play with that!

No worries! I asked more to be informed, I don't have a real use-case behind. This means that you guys internally catch the argparser object somehow right? Because you could also simply use sys argv to find the parameters, right?

Hi AgitatedDove14 , How should we proceed to fix this bug? Should I open an issue in github? Should I try to make a minimal reproducible example? It’s blocking me atm

-> seems to run properly now

PS: in the new env, I’v set num_replicas: 0, so I’m only talking about primary shards…

ha nice thanks 🙏

I am happy if I can be of any help to fix that 😄

Yea, the config is not appearing in the webUI anymore with this method 😞

No I agree, it’s probably not worth it

clearml 1.1.1 works

awesome thanks!

ClearML has a task.set_initial_iteration , I used it as such:
checkpoint = torch.load(checkpoint_fp, map_location="cuda:0") Checkpoint.load_objects(to_load=self.to_save, checkpoint=checkpoint) task.set_initial_iteration(engine.state.iteration)But still the same issue, I am not sure whether I use it correctly and if it’s a bug or not, AgitatedDove14 ? (I am using clearml 1.0.4rc1, clearml-agent 1.0.0)

Yes that’s correct - the weird thing is that the error shows the right detected region

and the agent says agent.cudnn_version = 0

And I do that each time I want to create a subtask. This way I am sure to retrieve the task if it already exists

it should return the task regardless if it is complete or not

yes

AgitatedDove14 Is it fixed with trains-server 0.15.1?

Is there any channel where we can see when new self hosted server version are published?

See my answer in the issue - I am not using docker

thanks Jake, good to know!

This is consistent: Each time I send a new task on the default queue, if trains-agent-1 has only one task running (the long one), it will pick another one. If I add one more experiment in the queue at that point (trains-agent-1 running two experiments at the same time), that experiment will stay in queue (trains-agent-2 and trains-agent-3 will not pick it because they also are running experiments)

on a p3.2xlarge instance

venv mode