I restarted the cleanup service. Now I get some messages like this:

2021-07-16 12:39:46,736 - clearml.storage - ERROR - Failed creating storage object file:// Reason: 'NoneType' object has no attribute 'replace'
2021-07-16 12:39:46,736 - clearml.Task - ERROR - Failed deleting None: 'NoneType' object has no attribute 'delete'
WARNING:root:Could not delete Task ID=eb11c92928af477e9e732d0cad47a57e, sequence item 0: expected str instance, NoneType found

any idea?

Some of the experiments are done on a GCP instance instead of the local server on which we also run ClearML. The experiments running on GCP report to the same local clearml server, but the IP address for clearml configured on the GCP instance is different (and then forwarded). Is this the problem?

Yes I see:

"The default location for output models and other artifacts. If True is passed, the default files_server will be used for model storage. In the default location, ClearML creates a subfolder for the output. The subfolder structure is the following: <output destination name> / <project name> / <task name>.<Task ID>"

So it makes a folder in the output destination <project_name>/<task name>.<Task ID>. It is not possible to specify the full output destination right?

yep

Okay, I am working with medical images. And when running a testing script I want to save the predictions (also big medical images of another modality). What happens when I do logger.upload_artifact(..). Then a file is copied to this folder?

Old legacy code that has its own folder structure per experiment. I can also do it the other way around. Does task.get_output_destination() return the folder including project name and <task_name>.<task_id>?

I see, however it looks like medical images are not supported. We use nifty images, except for an 3D array the image also contains voxel spacing, and origin and direction in a world frame

I see task.get_output_destination() returns a url like http://localhost:8081 . Is it possible to get the folder with the artifacts/models?

I see, will it be possible in the future to directly write custom/not supported formats to the folder? Because we are working with very big files, having them stored at multiple locations is something we try to avoid

Okay, so I have to first save the generated image somewhere and then with logger.report_media it is copied to the folder?

It is for storing the predictions a trained model makes, so two different models do create slightly different images

Yes, now I new unique folder is created per experiment where the predictions are saved. That works. The only thing is that now there is the folder that clearml makes for an experiment and the folder that saves the resuts. So two folders with artifacts per experiment. I was wondering if there was a more efficient solution and if it could be combined.

It is the folder the clearml creates and the folder we create ourself to store the predictions

This was with using one task in a multiprocessing.pool and the next one in the main process. I switched to have all tasks in a separate process via ProcessPoolExecutor and now it runs fine 👍 (version 0.17.5)

Thanks ClearML team, is there an example of the metric snapshot plot in the project overview UI available in the demo dashboard?

Okay finalize works. I was looking here: https://github.com/allegroai/clearml/blob/master/docs/datasets.md

yes, I wanted the confirmation that this is also a good solution for datasets with medical images

I see this indeed when I create a new project with an empty description. Is this also possible for older project created before clearml 1.0? For these projects this button is not there

Thanks !

Nested in the UI is not possible I think? I mean that it is possible to start the subtask while the main task is still active. Maybe I should just try

Yes that is the easiest solution, I will create the main task when the others are closed

done, see https://github.com/allegroai/clearml/issues/489

how do I install the RC?

` Backend TkAgg is interactive backend. Turning interactive mode on.
Traceback (most recent call last):
File "/snap/pycharm-community/226/plugins/python-ce/helpers/pydev/_pydevd_bundle/pydevd_frame.py", line 747, in trace_dispatch
self.do_wait_suspend(thread, frame, event, arg)
File "/snap/pycharm-community/226/plugins/python-ce/helpers/pydev/_pydevd_bundle/pydevd_frame.py", line 144, in do_wait_suspend
self._args[0].do_wait_suspend(*args, **kwargs)
File "/snap/pycharm-community...

it is the same with rc4. Under the variables tab it keeps hanging on 'collecting data...' OS: Ubuntu 18.04, PyCharm CE 2020.3

upgrading to? 2020.3.3 is the latest version? https://www.jetbrains.com/pycharm/download/other.html

This is the full log

using auto_connect_frameworks={'pytorch': False} now

Yes, it did I think that makes sense

In the process MyProcess other processes are created via a ProcessPoolExecutor. In these processes calls to logger.report_matplotlib_figure are made, but I get the same issue when I remove these calls.

It looks like I don't have hanging issues when I use mp.set_start_method('spawn') at the top of the script.

I don't have a fully reproducilble example that I can share, sorry for that