Shards that I can see are using a lot of disk space are

• events-training_stats_scalar
• events-log
• And then various worker_stats_*

CostlyOstrich36 thanks for getting back to me!

yes!

That's great! Please can you let me know how to do it/how to set the default files server?

However it would be advisable to also add the following argument to your code :

That's useful thanks, I didn't know about this kwarg

task.get_parameters and task.get_parameters_as_dict have the keyword argument cast which attempts to convert values back to their original type, but interestingly doesn't seem to work for properties:

` task = Task.init()
task.set_user_properties(x=5)
task.connect({"a":5})
task.get_parameters_as_dict(cast=True)

{'General': {'a': 5}, 'properties': {'x': '5'}} Hopefully would be a relatively easy extension of get_user_properties ` !

FrothyDog40 https://github.com/allegroai/clearml-docs/pull/235

Yeah it's strange isn't it!

https://github.com/allegroai/clearml/pull/737

Thank you 🙂

Yep can do 👍

Tasks are running locally and recording to our self deployed server, no output in my task log that indicates an issue. This is all of the console output:

2023-01-09 12:53:22 ClearML Task: created new task id=7f94e231d8a04a8c9592026dea89463a ClearML results page: 2023-01-09 12:53:24 ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoringAre there any logs in the server I can check? The server is running v1.3.1 and the issue I’m see is with version 1....

Updating the server has solved the issue 👍

I think a note about the fileserver should be added to the https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_security page!

OK that's great, thanks for the info SuccessfulKoala55 👍

it might be an issue in the UI due to this unconventional address or network settings

I think this is related to an https://github.com/allegroai/clearml-server/issues/112#issue-1149080358 that seems to be a reoccurring issue across many different setups

Amazing, task.__dict__ is just what I was looking for!

connect_configuration seems to take about the same amount of time unfortunately!

Thanks @<1523701087100473344:profile|SuccessfulKoala55> , I’ve taken a look and is this force merging you’re referring to? Do you know how often ES is configured to merge in clearml server?

Hi CostlyOstrich36 , thanks for getting back to me!

I want to launch multiple tasks from one python process to be run by multiple agents simultaneously.

My current process for launching one task remotely is to use task.execute_remotely , and then I separately spin up a VM and execute a ClearML agent on that VM with the task ID.

Ideally, I would like to create multiple tasks in this way - so do Task.init(…) , set up some configuration, and then task.execute_remotely in a l...

Will do! What’s the process for adding task.reset to the public API, just adding it to the docs?

I think you should open a github feature request since there is currently no way to do this via UI

Will do. Is there a way to do it no via the UI? E.g. in the server configuration (I'm running a self hosted server)?

I guess two more straightforward questions:

Could it be made possible for task.execute_remotely(clone=False, exit_process=False) to not raise an exception? Im happy work on a PR if this would be possible Is there any issue to having task.reset() in the public API/is there any potential issues with using?

Perhaps linked to https://github.com/allegroai/clearml-server/issues/112 ?

Is the GCP disk image released for it? I get access denied with this link: https://storage.googleapis.com/allegro-files/clearml-server/clearml-server-1-3-0.tar.gz
Cheers!

Thanks CumbersomeCormorant74

Could well be the same as https://github.com/allegroai/clearml-server/issues/112 which is also discussed https://clearml.slack.com/archives/CTK20V944/p1648547056095859 🙂

Ah right, nice! I didn’t think it was as I couldn’t see it in the Task reference , should it be there too?

CostlyOstrich36 I use the GCP disk image to launch a Compute Engine instance which sits behind an HTTP load balancer

CumbersomeCormorant74 just to confirm in my case the file's aren't actually deleted - I have to manually delete them from the fileserver via a terminal

That said, maybe the connect dict is not the best solution for thousand key dictionary

Seems like it isn't haha!
What is the difference with connect_configuration ? The nice thing about it not being an artifact is that we can use the gui to see which hashes have changed (which admittedly when there are a few thousand is tricky anyway)

No worries, thanks for sorting it! 🙂

Awesome, thanks SuccessfulKoala55 .