There has been a restart of my machine in the mean time :man-shrugging:
So only the matrix knows now I guess..

Hi @<1523701087100473344:profile|SuccessfulKoala55>

I am using 1.8.0 for the clearml-agent.

Attached is the logfile.

Hey. I should have closed this..

The thing that I was looking for is called set_parameter on the task.
The HPO uses a task I created previously and I had trouble with that, since it contained a path, which wasn't available on the colab instance.
I fixed my code, so it always updates this parameter depending on the environment.

It was less of an HPO issue, more of a programming failure on the function, which didn't properly update the parameter, even though I thought it should.

Alright, good to know.

The installed packages of the task say this:

# Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]

PyYAML == 6.0.1
clearml == 1.15.1
google google_api_core
google_cloud_storage == 2.16.0
ultralytics == 8.2.2

I do not know where the google_api_core comes from and I'd like to remove it.

A minimal illustration of the problem:

If I run model.tune(...) from ultralytics, then it automatically will track each iteration in ClearML and each iteration will be its own task (as it should be, given that the parameters change)

But the actual tune result will not be stored in a ClearML task, since I believe there is no integration on ultralytics side to do so.

If I create a task myself which then performs model.tune(...) it will get immediately overridden by the parameters fro...

Here is an updated and improved version.

if anyone can tell me on how to improve the cookie situation, I'd be grateful

Maybe it has something to do with my general environment? I am running on WSL2 in debian

This is the full log of the task.

I am trying to run HPO.

If it were possible to override the checkout behaviour I would ignore all submodules anyways, but in the documentation of clearml.conf as well as the pipeline decorator I couldn't find anything that would allow me doing that.

If there's source URLs in the plots of the task, how can I authenticate against ClearML to properly download them?

Or is there some SDK way to download them?

None

Sure can do

Here are the codefiles for my pipelines.

They do not work yet, I am struggling with the pipeline stuff quite a bit.

But both pipelines always give these warnings.

Any help would be greatly appreciated

None

I am getting the same when starting regular tasks.
I think it has something to do with my paramaters, which contain an environment variable which contains a list of datasets

Back when I wrote this, I thought HPO does something magical for overwriting the general args of the task when cloning.
Turns out it just was my code that was missing a more explicit set_parameter for this environment path.

How can I adjust the parameter overrides from tasks spawned by the hyperparameter optimizer?

My template task has some environment depending parameters that I would like to clear for the newly spawned tasks, as the function that is run for each tasks handles the environment already.

I don’t know what would cause slowness

My experiments are all using YOLOv8 and they contain the data from what is gathered there automatically

I ‘ m using the app.clearml server

Hem, yeah might be the case.

It comes from the PipelineDecorator.pipeline I assume or from PipelineDecorator.component

Alright cool!
I will check it out and let you know what it was.

Nevermind, all I need is to use Task.get_task() with the id of the dataset, since the ID was re-used.

I'd still be interested in knowing how to retrieve the task_id of a dataset if reuse_task_id was set to false.

I see

just to give an idea about the scale of the problem on my side.

These are the submodules...

`Fetching submodule lvgl_ui_generator
2024-04-27 20:45:34
Fetching submodule lvgl_ui_generator/lv_drivers
Fetching submodule lvgl_ui_generator/lvgl
Fetching submodule lvgl_ui_generator_v2
Fetching submodule lvgl_ui_generator_v2/lv_micropython
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/asf4
2024-04-27 20:45:40
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/axtls
Fetch...

You mean a seperate branch to work in without the submodules linked?

Not really sure how I'd go about doing that.

I'd be more happy with an option to say 'pull_submodules=False'

The code that is run in regards to clearml is really small.

If there's no mechanism on side of clearml, I might consider just putting that codebase into it's own submodule, making it a different repo without knowledge of the others.