Last but not least - can I cancel the offline zip creation if I'm not interested in it 🤔
EDIT: I see not, guess one has to patch ZipFile ...

https://clear.ml/docs/latest/docs/references/sdk/services_monitor

Then you can run this as a task, see also this example https://clear.ml/docs/latest/docs/guides/services/slack_alerts

After the task was initialized? 🤔

Thanks LazyFish41 , that explains it 👍

I think you're interested in the Monitor class:)

Using the PipelineController with add_function_step

Thanks David! I appreciate that, it would be very nice to have a consistent pattern in this!

It misses the repository information of course, but the 'configuration/Args' were logged. So something weird in identifying the repository

I'm trying to decide if ClearML is a good use case for my team 🙂
Right now we're not looking for a complete overhaul into new tools, just some enhancements (specifically, model repository, data versioning).

We've been burnt by DVC and the likes before, so I'm trying to minimize the pain for my team before we set out to explore ClearML.

Yeah, and just thinking out loud what I like about the numpy/pandas documentation

Or is just integrated in the ClearML slack space and for some reason it's showing the clearml address then?

We load the endpoint (and S3 credentials) from a .env file, so they're not immediately available at the time of from clearml import Task .
It's a convenience thing, rather than exporting many environment variables that are tied together.

Hi SuccessfulKoala55 !
Could you elaborate on how best to delete these from the database?

Another example - trying to validate dataset interactions ends with

` else:
self._created_task = True
dataset_project, parent_project = self._build_hidden_project_name(dataset_project, dataset_name)
task = Task.create(
project_name=dataset_project, task_name=dataset_name, task_type=Task.TaskTypes.data_processing)
if bool(Session.check_min_api_server_version(Dataset.__min_api_version)):
get_or_create_proje...

If everything is managed with a git repo, does this also mean PRs will have a messy metadata file attached to them?

I think so, it was just missing from the official documentation 🙂 Thanks!

On it! Should I include the additional user filters described above?

packages an entire folder as zip

What if I have multiple files that are not in the same folder? (That is the current use-case)

It otherwise makes sense I think 🙂
Our workaround now for using a Dataset as we do, is to store the dataset ID as a configuration parameter, so it's always included too 😉

i.e. It does not process tasks on its own?

Sorry, found it on my end!

Another side effect btw is that some of our log files (we add a file handler to the logger) end up at 0 bytes. This specifically happens with Ray and ClearML and does not reproduce locally

I’ll give the create_function_task one more try 🤔

Adding bucket = clearml in aws.s3.credentials did not help either

The documentation is messy, I’ve complained about it the in the past too 🙈

Coming back to this; ClearML prints a lot of error messages in local tests, supposedly because the output streams are not directly available:
` --- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.10/logging/init.py", line 1103, in emit
stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
File "/home/idan/CC/git/ds-platform/.venv/lib/python3.10/site-packages/clearml/task.py", line 3504, in _at_exit
self.__shutdown...

And last but not least, for dictionary for example, it would be really cool if one could do:
my_config = task.connect_configuration(my_config, name=name) my_other_config = task.connect_configuration(my_other_config, name=other_name) my_other_config['bar'] = my_config # Creates the link automatically between the dictionaries

I am; it seems like maybe a couple of hours?

Yeah I managed to work around those former two, mostly by using Task.create instead of Task.init . It's actually the whole bunch of daemons running in the background that takes a long time, not the zipping.

Regarding the second - I'm not doing anything per se. I'm running in offline mode and I'm trying to create a dataset, and this is the error I get...
There is a data object it, but there is no script object attached to it (presumably again because of pytest?)

21s is just ridiculous, it's scanning the entire file system starting at /

Also I appreciate the time youre taking to answer AgitatedDove14 and CostlyOstrich36 , I know Fridays are not working days in Israel, so thank you 🙂