Hi @<1694157594333024256:profile|DisturbedParrot38>
Could you attach a full log? This is quite cryptic and does not ring a bell

It happens on all of my pipeline run attempts and there's nothing more that gives insight.

As an example:

python src/train.py 
ClearML Task: created new task id=102a4f25c5ac4972abd41f1d0b6b9708
ClearML results page: 

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

ClearML pipeline page: 

... and then the pipeline just continues ...

Is this from the pipeline logic? Or a component?

Here are the codefiles for my pipelines.

They do not work yet, I am struggling with the pipeline stuff quite a bit.

But both pipelines always give these warnings.

It comes from the PipelineDecorator.pipeline I assume or from PipelineDecorator.component

Thanks! Let me check if we can reproduce it. BTW what's your clearml package version?

1.15.1

I noticed that it's actually independent of the pipelines

I am getting the same when starting regular tasks.
I think it has something to do with my paramaters, which contain an environment variable which contains a list of datasets

This function shows the same behaviour once the task gets initialized:

# Training helper functions
def prepare_training(env: dict, model_variant: str, dataset_id: str, args: dict, project: str = "LVGL UI Detector"):
    from clearml import Task, Dataset
    import os
    print(f"Training {model_variant} on dataset: {dataset_id}")
    # Fetch dataset YAML
    env['FILES'][dataset_id] = Dataset.get(dataset_id).list_files("*.yaml")
    # Download & modify dataset
    env['DIRS']['target'] = download_dataset(env, dataset_id)
    dataset_file = os.path.join(env['DIRS']['target'], env['FILES'][dataset_id][0])
    dataset_content = fix_dataset_path(dataset_file, env['DIRS']['target'])
    args['data'] = os.path.join(env['DIRS']['target'], env['FILES'][dataset_id][0])
    # Create a ClearML Task
    task = Task.init(
        project_name="LVGL UI Detector",
        task_name=f"Train {model_variant} ({env['DATASETS'][dataset_id]['name']})",
        task_type=Task.TaskTypes.training
    )
    task.connect(args)
    # Log "model_variant" parameter to task
    task.set_parameter("model_variant", model_variant)
    task.set_parameter("dataset", dataset_id)
    task.set_parameter("General/data", args['data'])
    task.connect_configuration(name="Dataset YAML", configuration=args['data'])
    task.connect_configuration(name="Dataset Content", configuration=dataset_content)
    return task.id

Wait even without the pipeline decorator this function creates the warning?

Yup.

I really don't know what it's about.

It doesn't affect the process. Everything seems to run fine.

If the warnings would provide a bit more info I could maybe pinpoint it better, but it's really all I got

Is there some verbose mode I could run it with?

I have the strong suspicion it is somewhat related to my parameters of the function or generally the hyperparameters gathered by the task automatically.

Hmm yeah I think that makes sense. Can you post here the arguments?
I'm assuming you have something like '1.23a' in the arguments?

Not that I would know of..

I attached the possible problematic argument.
The strings are just regular string, nothing fancy there.

args :
{'epochs': 3, 'imgsz': 480, 'data': '/home/rini-debian/git-stash/lvgl-ui-detector/datasets/ui_randoms.yaml'}

model_variant :
yolov8n

dataset_id :
50e10f640d7548458d9c38ab9aac571b

Maybe it has something to do with my general environment? I am running on WSL2 in debian

https://www.geeksforgeeks.org/invalid-decimal-literal-in-python/

This is the warning hence my question

Yea, I get that.. But it's really hard to tell what's causing it due to the "<unknown>"

Let me check if we can reproduce it

Hi @<1694157594333024256:profile|DisturbedParrot38> ! We weren't able to reproduce, but you could find the source of the warning by appending the following code at the top of your script:

import traceback
import warnings
import sys

def warn_with_traceback(message, category, filename, lineno, file=None, line=None):
    log = file if hasattr(file,'write') else sys.stderr
    traceback.print_stack(file=log)
    log.write(warnings.formatwarning(message, category, filename, lineno, line))

warnings.showwarning = warn_with_traceback

you should (hopefully) see a traceback

Alright cool!
I will check it out and let you know what it was.

No idea what's going on now, but I cannot reproduce the behaviour either.. also tried my old code posted here, but the warning doesn't pop up anymore.

I will inform once it pops again and will use the provided traceback function then.

I have a slight suspicion that it was indeed environment based on my local machine, but I have no idea what is the trigger for that.

It may or may not be related to this

2024-04-29 23:38:25,932 - clearml.Task - WARNING - Parameters must be of builtin type (Args/env/ENV[_Environ], Args/env/DATASETS/d1c4ec5c8f7e4492ab7b2a8d0b584c88/created[datetime], Args/env/DATASETS/e7f05a856ca34cd88940c4b8774f4c45/created[datetime], Args/env/DATASETS/50e10f640d7548458d9c38ab9aac571b/created[datetime], Args/env/DATASETS/e113a14fec574025a15a5c9868746c4b/created[datetime], Args/env/DATASETS/c520737c392c47608ffd52f33fc593f6/created[datetime], Args/env/DATASETS/12f2a7ee7a7f45b8866d2c5f75e92413/created[datetime], Args/env/DATASETS/640fc2e67b3b4b498fe6d9b1da764911/created[datetime], Args/env/DATASETS/cef7f5e021484b089a5eccf0c014d150/created[datetime], Args/env/DATASETS/28b1c17f1fef47a793f9ac086b2d6b10/created[datetime], Args/env/DATASETS/af33141c123843d4bf0c50db76635bba/created[datetime], Args/env/DATASETS/2b561087cf974658b576fbd17a87bc87/created[datetime], Args/env/DATASETS/366058db107e485ca273a11601e68e7a/created[datetime], Args/env/DATASETS/f4d7bdcb58534f1ebbc5c5578b6801a5/created[datetime], Args/env/DATASETS/90648781cf4f4848a2a66ca572e1edf5/created[datetime], Args/env/DATASETS/a4b6cc7161334df1ad94143a819842d6/created[datetime], Args/env/DATASETS/853a677148c5417e8cc003d6761236e3/created[datetime], Args/env/DATASETS/b4f89b652b1940848f186c5ab9e2de3f/created[datetime], Args/env/DATASETS/231b526d445b416fa487fc902a1ebb0e/created[datetime], Args/env/DATASETS/55a19d9bd15c4c67a6931e3eba7b454a/created[datetime], Args/env/DATASETS/e8289fd5259c4e78830fd2e2d88a6d51/created[datetime], Args/env/DATASETS/55fba0728c3d4af1953fa0fe04b30ce6/created[datetime], Args/env/DATASETS/abc76956f1de41b386fa805e8858c884/created[datetime], Args/env/DATASETS/326ea43fc9ff4e6db5209218b6020ca6/created[datetime], Args/env/DATASETS/172affc0c4bb4a369e75f685782b5ad8/created[datetime])

But that's not used on my side anymore.

It's a weird warning and now it's a weird unreproducible warning, even though I had it on each trial run earlier today.

There has been a restart of my machine in the mean time :man-shrugging:
So only the matrix knows now I guess..