Hi AgitatedDove14 , that’s super exciting news! 🤩 🚀
Regarding the two outstanding points:
In my case, I’d maintain a client python package that takes care of the pre/post processing of each request, so that I only send the raw data to the inference service and I post process the raw output of the model returned by the inference service. But I understand why it might be desirable for the users to have these steps happening on the server. What is challenging in this context? Defining how t...

I am already trying with latest of pip 😞

Ok, now I would like to copy from one machine to another via scp, so I copied the whole /opt/trains/data folder, but I got the following errors:

Could be, but not sure -> from 0.16.2 to 0.16.3

now I can do nvcc --version and I get
Cuda compilation tools, release 10.1, V10.1.243

Thanks! I will investigate further, I am thinking that the AWS instance might have been stuck for an unknown reason (becoming unhealthy)

Thanks a lot, I will play with that!

No worries! I asked more to be informed, I don't have a real use-case behind. This means that you guys internally catch the argparser object somehow right? Because you could also simply use sys argv to find the parameters, right?

Hi AgitatedDove14 , How should we proceed to fix this bug? Should I open an issue in github? Should I try to make a minimal reproducible example? It’s blocking me atm

(by console you mean in the dashboard right? or the terminal?)

-> seems to run properly now

Note: I can verify that post_packages is well picked up by the trains-agent, since in the experiment log I see:
agent.package_manager.type = pip agent.package_manager.pip_version = \=\=20.2.3 agent.package_manager.system_site_packages = true agent.package_manager.force_upgrade = false agent.package_manager.post_packages.0 = PyJWT\=\=1.7.1

PS: in the new env, I’v set num_replicas: 0, so I’m only talking about primary shards…

ha nice thanks 🙏

I am happy if I can be of any help to fix that 😄

Yea, the config is not appearing in the webUI anymore with this method 😞

No I agree, it’s probably not worth it

clearml 1.1.1 works

I could delete the files manually with sudo rm (sudo is required, otherwise I get Permission Denied )

awesome thanks!

ClearML has a task.set_initial_iteration , I used it as such:
checkpoint = torch.load(checkpoint_fp, map_location="cuda:0") Checkpoint.load_objects(to_load=self.to_save, checkpoint=checkpoint) task.set_initial_iteration(engine.state.iteration)But still the same issue, I am not sure whether I use it correctly and if it’s a bug or not, AgitatedDove14 ? (I am using clearml 1.0.4rc1, clearml-agent 1.0.0)

Yes that’s correct - the weird thing is that the error shows the right detected region

and the agent says agent.cudnn_version = 0

And I do that each time I want to create a subtask. This way I am sure to retrieve the task if it already exists

it should return the task regardless if it is complete or not

yes

AgitatedDove14 Is it fixed with trains-server 0.15.1?

Is there any channel where we can see when new self hosted server version are published?

See my answer in the issue - I am not using docker

thanks Jake, good to know!