None

Here is the latest version with all issues ironed out.

You mean a seperate branch to work in without the submodules linked?

Not really sure how I'd go about doing that.

I'd be more happy with an option to say 'pull_submodules=False'

On another attempt with a cleaned repository (no dirty commits) I get the same result, even though it states that it got a new commit id, so I'm at a loss at what is actually going wrong here:

`Using cached repository in "/root/.clearml/vcs-cache/lvgl-ui-detector.git.7c8ae2688810ceed26c1ebcc1e911cf2/lvgl-ui-detector.git"
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 8 (delta 4), reused 7 ...

That is unfortunate.

just to give an idea about the scale of the problem on my side.

These are the submodules...

`Fetching submodule lvgl_ui_generator
2024-04-27 20:45:34
Fetching submodule lvgl_ui_generator/lv_drivers
Fetching submodule lvgl_ui_generator/lvgl
Fetching submodule lvgl_ui_generator_v2
Fetching submodule lvgl_ui_generator_v2/lv_micropython
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/asf4
2024-04-27 20:45:40
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/axtls
Fetch...

If there's some or any mechanism that would allow me to constrain what the task sees, it would really help me alot.

What's wdyt?

Yea, but even though it's cached, it takes quite a long time, because my project has really alot of submodules, due to the submodules having their own submodules as well.

I don't really understand why fetching the submodules is the default.

When developing I use the poetry environment, but in the queues I let clearML handle the installation via pip.

There is no need to use poetry if the task is a one-time thing

The code that is run in regards to clearml is really small.

If there's no mechanism on side of clearml, I might consider just putting that codebase into it's own submodule, making it a different repo without knowledge of the others.

This is the full log of the task.

I am trying to run HPO.

The installed packages of the task say this:

# Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]

PyYAML == 6.0.1
clearml == 1.15.1
google google_api_core
google_cloud_storage == 2.16.0
ultralytics == 8.2.2

I do not know where the google_api_core comes from and I'd like to remove it.

I noticed poetry can be a problem in my run.
Not specifically due to the cache, but due to the installation of much more packages than the runtime might need.

When using regular pip, it will use the requirements list determined by ClearML to install necessary packages, which usually already excludes all dev-tools and similar.

I am not sure if poetry uses the cache properly, but I can't verify either atm.

A minimal illustration of the problem:

If I run model.tune(...) from ultralytics, then it automatically will track each iteration in ClearML and each iteration will be its own task (as it should be, given that the parameters change)

But the actual tune result will not be stored in a ClearML task, since I believe there is no integration on ultralytics side to do so.

If I create a task myself which then performs model.tune(...) it will get immediately overridden by the parameters fro...

Here is an updated and improved version.

if anyone can tell me on how to improve the cookie situation, I'd be grateful

Well.. I'll guess I'll do the workaround then of putting the main code into a submodule and have everything run from there

Back when I wrote this, I thought HPO does something magical for overwriting the general args of the task when cloning.
Turns out it just was my code that was missing a more explicit set_parameter for this environment path.

Not that I would know of..

I attached the possible problematic argument.
The strings are just regular string, nothing fancy there.

args :
{'epochs': 3, 'imgsz': 480, 'data': '/home/rini-debian/git-stash/lvgl-ui-detector/datasets/ui_randoms.yaml'}

model_variant :
yolov8n

dataset_id :
50e10f640d7548458d9c38ab9aac571b

Alright cool!
I will check it out and let you know what it was.

Is there some verbose mode I could run it with?

I have the strong suspicion it is somewhat related to my parameters of the function or generally the hyperparameters gathered by the task automatically.

1.15.1

Yup.

I really don't know what it's about.

It doesn't affect the process. Everything seems to run fine.

If the warnings would provide a bit more info I could maybe pinpoint it better, but it's really all I got

Here are the codefiles for my pipelines.

They do not work yet, I am struggling with the pipeline stuff quite a bit.

But both pipelines always give these warnings.

None of these submodules are required for the tasks, they are there for a different part of the project dealing with data generation.

So even having them fetched (even when cached) is quite the delay on the actual task.

None

Figured it out, I installed clearml[gs] but since I don't need that I removed it. it's gone now.

Hi @<1523701087100473344:profile|SuccessfulKoala55>

I am using 1.8.0 for the clearml-agent.

Attached is the logfile.

Hey. I should have closed this..

The thing that I was looking for is called set_parameter on the task.
The HPO uses a task I created previously and I had trouble with that, since it contained a path, which wasn't available on the colab instance.
I fixed my code, so it always updates this parameter depending on the environment.

It was less of an HPO issue, more of a programming failure on the function, which didn't properly update the parameter, even though I thought it should.