Yea, I get that.. But it's really hard to tell what's causing it due to the "<unknown>"

There has been a restart of my machine in the mean time :man-shrugging:
So only the matrix knows now I guess..

This function shows the same behaviour once the task gets initialized:

# Training helper functions
def prepare_training(env: dict, model_variant: str, dataset_id: str, args: dict, project: str = "LVGL UI Detector"):
    from clearml import Task, Dataset
    import os
    print(f"Training {model_variant} on dataset: {dataset_id}")
    # Fetch dataset YAML
    env['FILES'][dataset_id] = Dataset.get(dataset_id).list_files("*.yaml")
    # Download & modify dataset
    env['DIRS']['target'] = download_dataset(env, dataset_id)
    dataset_file = os.path.join(env['DIRS']['target'], env['FILES'][dataset_id][0])
    dataset_content = fix_dataset_path(dataset_file, env['DIRS']['target'])
    args['data'] = os.path.join(env['DIRS']['target'], env['FILES'][dataset_id][0])
    # Create a ClearML Task
    task = Task.init(
        project_name="LVGL UI Detector",
        task_name=f"Train {model_variant} ({env['DATASETS'][dataset_id]['name']})",
        task_type=Task.TaskTypes.training
    )
    task.connect(args)
    # Log "model_variant" parameter to task
    task.set_parameter("model_variant", model_variant)
    task.set_parameter("dataset", dataset_id)
    task.set_parameter("General/data", args['data'])
    task.connect_configuration(name="Dataset YAML", configuration=args['data'])
    task.connect_configuration(name="Dataset Content", configuration=dataset_content)
    return task.id

It happens on all of my pipeline run attempts and there's nothing more that gives insight.

As an example:

python src/train.py 
ClearML Task: created new task id=102a4f25c5ac4972abd41f1d0b6b9708
ClearML results page: 

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

ClearML pipeline page: 

... and then the pipeline just continues ...

Here are the codefiles for my pipelines.

They do not work yet, I am struggling with the pipeline stuff quite a bit.

But both pipelines always give these warnings.

Maybe it has something to do with my general environment? I am running on WSL2 in debian

Hi @<1694157594333024256:profile|DisturbedParrot38>
Could you attach a full log? This is quite cryptic and does not ring a bell

Let me check if we can reproduce it

Thanks! Let me check if we can reproduce it. BTW what's your clearml package version?

1.15.1

I noticed that it's actually independent of the pipelines

I have the strong suspicion it is somewhat related to my parameters of the function or generally the hyperparameters gathered by the task automatically.

Wait even without the pipeline decorator this function creates the warning?

Is there some verbose mode I could run it with?

Is this from the pipeline logic? Or a component?

No idea what's going on now, but I cannot reproduce the behaviour either.. also tried my old code posted here, but the warning doesn't pop up anymore.

I will inform once it pops again and will use the provided traceback function then.

I have a slight suspicion that it was indeed environment based on my local machine, but I have no idea what is the trigger for that.

It may or may not be related to this

2024-04-29 23:38:25,932 - clearml.Task - WARNING - Parameters must be of builtin type (Args/env/ENV[_Environ], Args/env/DATASETS/d1c4ec5c8f7e4492ab7b2a8d0b584c88/created[datetime], Args/env/DATASETS/e7f05a856ca34cd88940c4b8774f4c45/created[datetime], Args/env/DATASETS/50e10f640d7548458d9c38ab9aac571b/created[datetime], Args/env/DATASETS/e113a14fec574025a15a5c9868746c4b/created[datetime], Args/env/DATASETS/c520737c392c47608ffd52f33fc593f6/created[datetime], Args/env/DATASETS/12f2a7ee7a7f45b8866d2c5f75e92413/created[datetime], Args/env/DATASETS/640fc2e67b3b4b498fe6d9b1da764911/created[datetime], Args/env/DATASETS/cef7f5e021484b089a5eccf0c014d150/created[datetime], Args/env/DATASETS/28b1c17f1fef47a793f9ac086b2d6b10/created[datetime], Args/env/DATASETS/af33141c123843d4bf0c50db76635bba/created[datetime], Args/env/DATASETS/2b561087cf974658b576fbd17a87bc87/created[datetime], Args/env/DATASETS/366058db107e485ca273a11601e68e7a/created[datetime], Args/env/DATASETS/f4d7bdcb58534f1ebbc5c5578b6801a5/created[datetime], Args/env/DATASETS/90648781cf4f4848a2a66ca572e1edf5/created[datetime], Args/env/DATASETS/a4b6cc7161334df1ad94143a819842d6/created[datetime], Args/env/DATASETS/853a677148c5417e8cc003d6761236e3/created[datetime], Args/env/DATASETS/b4f89b652b1940848f186c5ab9e2de3f/created[datetime], Args/env/DATASETS/231b526d445b416fa487fc902a1ebb0e/created[datetime], Args/env/DATASETS/55a19d9bd15c4c67a6931e3eba7b454a/created[datetime], Args/env/DATASETS/e8289fd5259c4e78830fd2e2d88a6d51/created[datetime], Args/env/DATASETS/55fba0728c3d4af1953fa0fe04b30ce6/created[datetime], Args/env/DATASETS/abc76956f1de41b386fa805e8858c884/created[datetime], Args/env/DATASETS/326ea43fc9ff4e6db5209218b6020ca6/created[datetime], Args/env/DATASETS/172affc0c4bb4a369e75f685782b5ad8/created[datetime])

But that's not used on my side anymore.

It's a weird warning and now it's a weird unreproducible warning, even though I had it on each trial run earlier today.

I am getting the same when starting regular tasks.
I think it has something to do with my paramaters, which contain an environment variable which contains a list of datasets

Hi @<1694157594333024256:profile|DisturbedParrot38> ! We weren't able to reproduce, but you could find the source of the warning by appending the following code at the top of your script:

import traceback
import warnings
import sys

def warn_with_traceback(message, category, filename, lineno, file=None, line=None):
    log = file if hasattr(file,'write') else sys.stderr
    traceback.print_stack(file=log)
    log.write(warnings.formatwarning(message, category, filename, lineno, line))

warnings.showwarning = warn_with_traceback

you should (hopefully) see a traceback

https://www.geeksforgeeks.org/invalid-decimal-literal-in-python/

This is the warning hence my question

Hmm yeah I think that makes sense. Can you post here the arguments?
I'm assuming you have something like '1.23a' in the arguments?

Not that I would know of..

I attached the possible problematic argument.
The strings are just regular string, nothing fancy there.

args :
{'epochs': 3, 'imgsz': 480, 'data': '/home/rini-debian/git-stash/lvgl-ui-detector/datasets/ui_randoms.yaml'}

model_variant :
yolov8n

dataset_id :
50e10f640d7548458d9c38ab9aac571b

It comes from the PipelineDecorator.pipeline I assume or from PipelineDecorator.component

Alright cool!
I will check it out and let you know what it was.

Yup.

I really don't know what it's about.

It doesn't affect the process. Everything seems to run fine.

If the warnings would provide a bit more info I could maybe pinpoint it better, but it's really all I got