Maybe it has something to do with my general environment? I am running on WSL2 in debian

Wait even without the pipeline decorator this function creates the warning?

Here are the codefiles for my pipelines.

They do not work yet, I am struggling with the pipeline stuff quite a bit.

But both pipelines always give these warnings.

I am getting the same when starting regular tasks.
I think it has something to do with my paramaters, which contain an environment variable which contains a list of datasets

I noticed that it's actually independent of the pipelines

Hmm yeah I think that makes sense. Can you post here the arguments?
I'm assuming you have something like '1.23a' in the arguments?

It happens on all of my pipeline run attempts and there's nothing more that gives insight.

As an example:

python src/train.py 
ClearML Task: created new task id=102a4f25c5ac4972abd41f1d0b6b9708
ClearML results page: 

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

<unknown>:1: SyntaxWarning:

invalid decimal literal

ClearML pipeline page: 

... and then the pipeline just continues ...

Alright cool!
I will check it out and let you know what it was.

Yea, I get that.. But it's really hard to tell what's causing it due to the "<unknown>"

https://www.geeksforgeeks.org/invalid-decimal-literal-in-python/

This is the warning hence my question

This function shows the same behaviour once the task gets initialized:

# Training helper functions
def prepare_training(env: dict, model_variant: str, dataset_id: str, args: dict, project: str = "LVGL UI Detector"):
    from clearml import Task, Dataset
    import os
    print(f"Training {model_variant} on dataset: {dataset_id}")
    # Fetch dataset YAML
    env['FILES'][dataset_id] = Dataset.get(dataset_id).list_files("*.yaml")
    # Download & modify dataset
    env['DIRS']['target'] = download_dataset(env, dataset_id)
    dataset_file = os.path.join(env['DIRS']['target'], env['FILES'][dataset_id][0])
    dataset_content = fix_dataset_path(dataset_file, env['DIRS']['target'])
    args['data'] = os.path.join(env['DIRS']['target'], env['FILES'][dataset_id][0])
    # Create a ClearML Task
    task = Task.init(
        project_name="LVGL UI Detector",
        task_name=f"Train {model_variant} ({env['DATASETS'][dataset_id]['name']})",
        task_type=Task.TaskTypes.training
    )
    task.connect(args)
    # Log "model_variant" parameter to task
    task.set_parameter("model_variant", model_variant)
    task.set_parameter("dataset", dataset_id)
    task.set_parameter("General/data", args['data'])
    task.connect_configuration(name="Dataset YAML", configuration=args['data'])
    task.connect_configuration(name="Dataset Content", configuration=dataset_content)
    return task.id

It comes from the PipelineDecorator.pipeline I assume or from PipelineDecorator.component

Let me check if we can reproduce it

No idea what's going on now, but I cannot reproduce the behaviour either.. also tried my old code posted here, but the warning doesn't pop up anymore.

I will inform once it pops again and will use the provided traceback function then.

I have a slight suspicion that it was indeed environment based on my local machine, but I have no idea what is the trigger for that.

It may or may not be related to this

2024-04-29 23:38:25,932 - clearml.Task - WARNING - Parameters must be of builtin type (Args/env/ENV[_Environ], Args/env/DATASETS/d1c4ec5c8f7e4492ab7b2a8d0b584c88/created[datetime], Args/env/DATASETS/e7f05a856ca34cd88940c4b8774f4c45/created[datetime], Args/env/DATASETS/50e10f640d7548458d9c38ab9aac571b/created[datetime], Args/env/DATASETS/e113a14fec574025a15a5c9868746c4b/created[datetime], Args/env/DATASETS/c520737c392c47608ffd52f33fc593f6/created[datetime], Args/env/DATASETS/12f2a7ee7a7f45b8866d2c5f75e92413/created[datetime], Args/env/DATASETS/640fc2e67b3b4b498fe6d9b1da764911/created[datetime], Args/env/DATASETS/cef7f5e021484b089a5eccf0c014d150/created[datetime], Args/env/DATASETS/28b1c17f1fef47a793f9ac086b2d6b10/created[datetime], Args/env/DATASETS/af33141c123843d4bf0c50db76635bba/created[datetime], Args/env/DATASETS/2b561087cf974658b576fbd17a87bc87/created[datetime], Args/env/DATASETS/366058db107e485ca273a11601e68e7a/created[datetime], Args/env/DATASETS/f4d7bdcb58534f1ebbc5c5578b6801a5/created[datetime], Args/env/DATASETS/90648781cf4f4848a2a66ca572e1edf5/created[datetime], Args/env/DATASETS/a4b6cc7161334df1ad94143a819842d6/created[datetime], Args/env/DATASETS/853a677148c5417e8cc003d6761236e3/created[datetime], Args/env/DATASETS/b4f89b652b1940848f186c5ab9e2de3f/created[datetime], Args/env/DATASETS/231b526d445b416fa487fc902a1ebb0e/created[datetime], Args/env/DATASETS/55a19d9bd15c4c67a6931e3eba7b454a/created[datetime], Args/env/DATASETS/e8289fd5259c4e78830fd2e2d88a6d51/created[datetime], Args/env/DATASETS/55fba0728c3d4af1953fa0fe04b30ce6/created[datetime], Args/env/DATASETS/abc76956f1de41b386fa805e8858c884/created[datetime], Args/env/DATASETS/326ea43fc9ff4e6db5209218b6020ca6/created[datetime], Args/env/DATASETS/172affc0c4bb4a369e75f685782b5ad8/created[datetime])

But that's not used on my side anymore.

It's a weird warning and now it's a weird unreproducible warning, even though I had it on each trial run earlier today.

Yup.

I really don't know what it's about.

It doesn't affect the process. Everything seems to run fine.

If the warnings would provide a bit more info I could maybe pinpoint it better, but it's really all I got

Is there some verbose mode I could run it with?

Thanks! Let me check if we can reproduce it. BTW what's your clearml package version?

Hi @<1694157594333024256:profile|DisturbedParrot38>
Could you attach a full log? This is quite cryptic and does not ring a bell

Is this from the pipeline logic? Or a component?

1.15.1

Not that I would know of..

I attached the possible problematic argument.
The strings are just regular string, nothing fancy there.

args :
{'epochs': 3, 'imgsz': 480, 'data': '/home/rini-debian/git-stash/lvgl-ui-detector/datasets/ui_randoms.yaml'}

model_variant :
yolov8n

dataset_id :
50e10f640d7548458d9c38ab9aac571b

I have the strong suspicion it is somewhat related to my parameters of the function or generally the hyperparameters gathered by the task automatically.

Hi @<1694157594333024256:profile|DisturbedParrot38> ! We weren't able to reproduce, but you could find the source of the warning by appending the following code at the top of your script:

import traceback
import warnings
import sys

def warn_with_traceback(message, category, filename, lineno, file=None, line=None):
    log = file if hasattr(file,'write') else sys.stderr
    traceback.print_stack(file=log)
    log.write(warnings.formatwarning(message, category, filename, lineno, line))

warnings.showwarning = warn_with_traceback

you should (hopefully) see a traceback

There has been a restart of my machine in the mean time :man-shrugging:
So only the matrix knows now I guess..