There has been a restart of my machine in the mean time :man-shrugging:
So only the matrix knows now I guess..

I have the strong suspicion it is somewhat related to my parameters of the function or generally the hyperparameters gathered by the task automatically.

Yea, but even though it's cached, it takes quite a long time, because my project has really alot of submodules, due to the submodules having their own submodules as well.

I don't really understand why fetching the submodules is the default.

Alright cool!
I will check it out and let you know what it was.

Back when I wrote this, I thought HPO does something magical for overwriting the general args of the task when cloning.
Turns out it just was my code that was missing a more explicit set_parameter for this environment path.

I noticed that it's actually independent of the pipelines

What's wdyt?

I am getting the same when starting regular tasks.
I think it has something to do with my paramaters, which contain an environment variable which contains a list of datasets

Is there some verbose mode I could run it with?

It happens on all of my pipeline run attempts and there's nothing more that gives insight.

As an example:

python src/train.py 
ClearML Task: created new task id=102a4f25c5ac4972abd41f1d0b6b9708
ClearML results page: 

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal...

Hem, yeah might be the case.

Thanks!

Hey. I should have closed this..

The thing that I was looking for is called set_parameter on the task.
The HPO uses a task I created previously and I had trouble with that, since it contained a path, which wasn't available on the colab instance.
I fixed my code, so it always updates this parameter depending on the environment.

It was less of an HPO issue, more of a programming failure on the function, which didn't properly update the parameter, even though I thought it should.

Alright, good to know.

For anyone else interested in this, I wrote a little script which pulls all the data from a given project, seems to work well enough

I see

Here is an updated and improved version.

if anyone can tell me on how to improve the cookie situation, I'd be grateful

If there's some or any mechanism that would allow me to constrain what the task sees, it would really help me alot.

Here is the latest version with all issues ironed out.

On another attempt with a cleaned repository (no dirty commits) I get the same result, even though it states that it got a new commit id, so I'm at a loss at what is actually going wrong here:

`Using cached repository in "/root/.clearml/vcs-cache/lvgl-ui-detector.git.7c8ae2688810ceed26c1ebcc1e911cf2/lvgl-ui-detector.git"
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 8 (delta 4), reused 7 ...

Sure can do

I ‘ m using the app.clearml server

I noticed poetry can be a problem in my run.
Not specifically due to the cache, but due to the installation of much more packages than the runtime might need.

When using regular pip, it will use the requirements list determined by ClearML to install necessary packages, which usually already excludes all dev-tools and similar.

I am not sure if poetry uses the cache properly, but I can't verify either atm.

None

None

Figured it out, I installed clearml[gs] but since I don't need that I removed it. it's gone now.

Here is the code doing the reporting:

def capture_design(design_folder: str):
    import subprocess, os, shutil
    from clearml import Task
    print(f"Capturing designs from {design_folder}...")
    task = Task.current_task()
    logger = task.get_logger()
    design_files = [f for f in os.listdir(design_folder) if os.path.isfile(os.path.join(design_folder, f))]
    if len(design_files) == 0:
        print(f"No design files found in {design_folder}")
        return
    widgets = {}
  ...

The installed packages of the task say this:

# Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]

PyYAML == 6.0.1
clearml == 1.15.1
google google_api_core
google_cloud_storage == 2.16.0
ultralytics == 8.2.2

I do not know where the google_api_core comes from and I'd like to remove it.