Hi SpotlessLeopard9

I got many tasks that were just hang at the end of the script without ...

I remember this exact issue was fixed with 1.1.5rc0, see here:
https://clearml.slack.com/archives/CTK20V944/p1634910855059900

Can you verify with the latest RC?
pip install clearml==1.1.5rc3

how can I for example convert it back to a pandas dataframe?

You can always report csv file with report_media as well, or if this is not for debugging maybe an artifact ?

It seems to follow a structure specific to clearml,

Actually plotly.js 🙂

We use nifty images, except for an 3D array the image also contains voxel spacing, and origin and direction in a world frame

Yep, make sense ... you can just upload them as debug samples from local files.
I guess the main difference is the context, debug samples (used for debugging) vs artifacts (might be useful from other Tasks / context)
https://github.com/allegroai/clearml/blob/6b9297660e0ed83a77bce3da2fab384c552206fd/examples/reporting/image_reporting.py#L36

It is the folder the clearml creates and the folder we create ourself to store the predictions

I see... If that is the case, the only solution I can think of is manually uploading the files with StorageManager(...) then get the url, and register it as debug_media or artifact:
logger.report_media("image", "type a", iteration=iteration, url=" ") task.upload_artifact('a link', artifact_object=' ')

BTW: GreasyPenguin14 you can also upload them as debug samples (when setting the output_uri, the debug samples will be uploaded to the same destination)
https://github.com/allegroai/clearml/blob/6b9297660e0ed83a77bce3da2fab384c552206fd/examples/reporting/image_reporting.py#L21

JitteryCoyote63 see here https://stackoverflow.com/questions/55385900/pip3-setup-py-install-requires-pep-508-git-url-for-private-repo bottom line, you have to add package@ before the link, but if you do that and the package is already installed it will not install using the git repo, this is an issue with pip. I think that since the agent installs everything from scratch it should work for you. Wdyt?

So two folders with artifacts per experiment. I was wondering if there was a more efficient solution and if it could be combined.

Not sure I follow, is two subfolders for two different things are not they it is supposed to be ?

GreasyPenguin14 you mean the artifacts/models ?

Task.init(..., output_uri='s3://...')

Because we are working with very big files, having them stored at multiple locations is something we try to avoid

Just so I better understand, is this for storing files as part of a dataset, or as debug samples ?
In other words can two diff processes create the exact same file (image) ?

It is for storing the predictions a trained model makes, so two different models do create slightly different images

That actually makes sense.
So how would you create exactly the same file (i.e. why do you need to manually control the upload folder, wouldn't creating a new unique folder suffice ?)

And your ~/clearml,conf ?

Hi GreasyPenguin14

However the cleanup service is also running in a docker container. How is it possible that the cleanup service has access and can remove these model checkpoints?

The easiest solution is to launch the cleanup script with a mount point from the storage directory, to inside the container ( -v <host_folder>:<container_folder> )
The other option, which clearml version 1.0 and above supports, is using the Task.delete, that now supports deleting the artifacts and mod...

parser.add_argument( "--dataset_mean", type

=

float, nargs

=

"+", default

=

0.5)

I think providing nargs='+ ' assumes the type is a list. nonetheless we should be able to support it. Could you please add a GitHub issue so we do not forget ?

on the side note, is there any way to automatically give more meaningful names to the running docker containers?

What do you mean by that? running where? and where will you see them ?

FYI: if you need to query stuff you can always look directly in the RestAPI:
https://github.com/allegroai/clearml/blob/master/clearml/backend_api/services/v2_9/projects.py
https://allegro.ai/clearml/docs/rst/references/clearml_api_ref/index.html

BTW: get_tasks has project_name argument, I would just use it 🙂

Task.get_projects() 🙂

Seems like a Task contained an invalid artifact link.
I wouldn't sweat over it, it basically a warning that it could not locate the actual file to delete (albeit an ugly warning 🙂 )
I think AnxiousSeal95 would know when will the new version be ready.
regardless, is it actually deleting old Tasks ?

Hi GreasyPenguin14
It looks like you are trying to delete a Task that does not exist
Any chance the cleanup service is misconfigured (i.e. accessing the incorrect server) ?

So you want these two on two different graphs ?

Is there any way to make that increment from last run?

pipeline_task = Task.clone("pipeline_id_here", name="new execution run here") 
Task.enqueue(pipeline_task, queue_name="services")

wdyt?

What do you mean by "tag" / "sub-tags"?

You can do:
task = Task.get_task(task_id='uuid_of_experiment')
task.get_logger().report_scalar(...)

Now the only question is who will create the initial Task, so that the others can report to it. Do you have like a "master" process ?

Now, when I add delta to calculate the variation of this: error: bad_data: 1:110: parse error: ranges only allowed for vector selectors

This means your avg is already a scalar (i.e. not a vector) which means you can (as you said) have the alert based on that

Before this line, call Task.init

No. since you are using Pool. there is no need to call task init again. Just call it once before you create the Pool, then when you want to use it, just do task = Task.current_task()

fyi: hot fix for 1.3.0 (smoothing graphs) was just released see v1.3.1

I am actually considering rolling back to 1.1.0,

Can you share why?
JitteryCoyote63 notice from the release notes of 1.2:

Important Note!
This release requires a MongoDB migration from previous versions. Please see

for more information.

I'm not sure you can downgrade that easily ...

JitteryCoyote63 I remember something with "!" in the name or maybe "/" in the name that might cause this behavior. May I suggest checking with clearml-server 1.3 ?

@<1523710674990010368:profile|GreasyPenguin14> what do you mean "but I do I get the... " ?
Configuring git user/pass will allow you to launch Tasks from private repositories on the services queue (the agent is part of the docker-compose).
That said, this is not a must, worst case you'll get an error when git fails to clone your repo :)