Nevermind, all I need is to use Task.get_task() with the id of the dataset, since the ID was re-used.

I'd still be interested in knowing how to retrieve the task_id of a dataset if reuse_task_id was set to false.

Here is an updated and improved version.

if anyone can tell me on how to improve the cookie situation, I'd be grateful

For anyone else interested in this, I wrote a little script which pulls all the data from a given project, seems to work well enough

How can I adjust the parameter overrides from tasks spawned by the hyperparameter optimizer?

My template task has some environment depending parameters that I would like to clear for the newly spawned tasks, as the function that is run for each tasks handles the environment already.

According to None I am supposed to install

libgl1

I changed my clearml.conf to include that installation for the task container started by the agent.

Will see if it helps in a minute

None

None of these submodules are required for the tasks, they are there for a different part of the project dealing with data generation.

So even having them fetched (even when cached) is quite the delay on the actual task.

The code that is run in regards to clearml is really small.

If there's no mechanism on side of clearml, I might consider just putting that codebase into it's own submodule, making it a different repo without knowledge of the others.

That is unfortunate.

None

I cleared the vcs cache manually already, it results in the same behaviour illustrated above
(allthough the logs show that it used the cache, I had another run without cache - but don't have the logs from that)

Hi @<1523701087100473344:profile|SuccessfulKoala55>

I am using 1.8.0 for the clearml-agent.

Attached is the logfile.

If it were possible to override the checkout behaviour I would ignore all submodules anyways, but in the documentation of clearml.conf as well as the pipeline decorator I couldn't find anything that would allow me doing that.

Any help would be greatly appreciated

Figured it out, I installed clearml[gs] but since I don't need that I removed it. it's gone now.

This is the full log of the task.

I am trying to run HPO.

Sure can do

Here is the latest version with all issues ironed out.

Maybe it has something to do with my general environment? I am running on WSL2 in debian

just to give an idea about the scale of the problem on my side.

These are the submodules...

`Fetching submodule lvgl_ui_generator
2024-04-27 20:45:34
Fetching submodule lvgl_ui_generator/lv_drivers
Fetching submodule lvgl_ui_generator/lvgl
Fetching submodule lvgl_ui_generator_v2
Fetching submodule lvgl_ui_generator_v2/lv_micropython
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/asf4
2024-04-27 20:45:40
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/axtls
Fetch...

If there's source URLs in the plots of the task, how can I authenticate against ClearML to properly download them?

Or is there some SDK way to download them?

Here are the codefiles for my pipelines.

They do not work yet, I am struggling with the pipeline stuff quite a bit.

But both pipelines always give these warnings.

I am getting the same when starting regular tasks.
I think it has something to do with my paramaters, which contain an environment variable which contains a list of datasets

I noticed that it's actually independent of the pipelines

It happens on all of my pipeline run attempts and there's nothing more that gives insight.

As an example:

python src/train.py 
ClearML Task: created new task id=102a4f25c5ac4972abd41f1d0b6b9708
ClearML results page: 

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal...

Alright cool!
I will check it out and let you know what it was.

Yea, I get that.. But it's really hard to tell what's causing it due to the "<unknown>"

I don’t know what would cause slowness

Alright, good to know.

My experiments are all using YOLOv8 and they contain the data from what is gathered there automatically