What’s considered large in that case?

None

I ‘ m using the app.clearml server

I don’t know what would cause slowness

Well.. I'll guess I'll do the workaround then of putting the main code into a submodule and have everything run from there

My experiments are all using YOLOv8 and they contain the data from what is gathered there automatically

The code that is run in regards to clearml is really small.

If there's no mechanism on side of clearml, I might consider just putting that codebase into it's own submodule, making it a different repo without knowledge of the others.

That is unfortunate.

Sure can do

Hi @<1523701087100473344:profile|SuccessfulKoala55>

I am using 1.8.0 for the clearml-agent.

Attached is the logfile.

If it were possible to override the checkout behaviour I would ignore all submodules anyways, but in the documentation of clearml.conf as well as the pipeline decorator I couldn't find anything that would allow me doing that.

just to give an idea about the scale of the problem on my side.

These are the submodules...

`Fetching submodule lvgl_ui_generator
2024-04-27 20:45:34
Fetching submodule lvgl_ui_generator/lv_drivers
Fetching submodule lvgl_ui_generator/lvgl
Fetching submodule lvgl_ui_generator_v2
Fetching submodule lvgl_ui_generator_v2/lv_micropython
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/asf4
2024-04-27 20:45:40
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/axtls
Fetch...

None of these submodules are required for the tasks, they are there for a different part of the project dealing with data generation.

So even having them fetched (even when cached) is quite the delay on the actual task.

This is the full log of the task.

I am trying to run HPO.

You mean a seperate branch to work in without the submodules linked?

Not really sure how I'd go about doing that.

I'd be more happy with an option to say 'pull_submodules=False'

None

Alright, good to know.

How can I adjust the parameter overrides from tasks spawned by the hyperparameter optimizer?

My template task has some environment depending parameters that I would like to clear for the newly spawned tasks, as the function that is run for each tasks handles the environment already.

This here.. I know how to get the source code info, but it doesn't include the commit ID. And I also cannot access the uncommitted changes.

Hem, yeah might be the case.

Hey. I should have closed this..

The thing that I was looking for is called set_parameter on the task.
The HPO uses a task I created previously and I had trouble with that, since it contained a path, which wasn't available on the colab instance.
I fixed my code, so it always updates this parameter depending on the environment.

It was less of an HPO issue, more of a programming failure on the function, which didn't properly update the parameter, even though I thought it should.

According to None I am supposed to install

libgl1

I changed my clearml.conf to include that installation for the task container started by the agent.

Will see if it helps in a minute

Maybe it has something to do with my general environment? I am running on WSL2 in debian

Here are the codefiles for my pipelines.

They do not work yet, I am struggling with the pipeline stuff quite a bit.

But both pipelines always give these warnings.

When developing I use the poetry environment, but in the queues I let clearML handle the installation via pip.

There is no need to use poetry if the task is a one-time thing

Yup.

I really don't know what it's about.

It doesn't affect the process. Everything seems to run fine.

If the warnings would provide a bit more info I could maybe pinpoint it better, but it's really all I got

What's wdyt?

Nevermind, all I need is to use Task.get_task() with the id of the dataset, since the ID was re-used.

I'd still be interested in knowing how to retrieve the task_id of a dataset if reuse_task_id was set to false.

Not that I would know of..

I attached the possible problematic argument.
The strings are just regular string, nothing fancy there.

args :
{'epochs': 3, 'imgsz': 480, 'data': '/home/rini-debian/git-stash/lvgl-ui-detector/datasets/ui_randoms.yaml'}

model_variant :
yolov8n

dataset_id :
50e10f640d7548458d9c38ab9aac571b

The installed packages of the task say this:

# Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]

PyYAML == 6.0.1
clearml == 1.15.1
google google_api_core
google_cloud_storage == 2.16.0
ultralytics == 8.2.2

I do not know where the google_api_core comes from and I'd like to remove it.