Maybe it has something to do with my general environment? I am running on WSL2 in debian
Yea, I get that.. But it's really hard to tell what's causing it due to the "<unknown>"
Hi @<1694157594333024256:profile|DisturbedParrot38> ! We weren't able to reproduce, but you could find the source of the warning by appending the following code at the top of your script:
import traceback
import warnings
import sys
def warn_with_traceback(message, category, filename, lineno, file=None, line=None):
log = file if hasattr(file,'write') else sys.stderr
traceback.print_stack(file=log)
log.write(warnings.formatwarning(message, category, filename, lineno, line))
warnings.showwarning = warn_with_traceback
you should (hopefully) see a traceback
This function shows the same behaviour once the task gets initialized:
# Training helper functions
def prepare_training(env: dict, model_variant: str, dataset_id: str, args: dict, project: str = "LVGL UI Detector"):
from clearml import Task, Dataset
import os
print(f"Training {model_variant} on dataset: {dataset_id}")
# Fetch dataset YAML
env['FILES'][dataset_id] = Dataset.get(dataset_id).list_files("*.yaml")
# Download & modify dataset
env['DIRS']['target'] = download_dataset(env, dataset_id)
dataset_file = os.path.join(env['DIRS']['target'], env['FILES'][dataset_id][0])
dataset_content = fix_dataset_path(dataset_file, env['DIRS']['target'])
args['data'] = os.path.join(env['DIRS']['target'], env['FILES'][dataset_id][0])
# Create a ClearML Task
task = Task.init(
project_name="LVGL UI Detector",
task_name=f"Train {model_variant} ({env['DATASETS'][dataset_id]['name']})",
task_type=Task.TaskTypes.training
)
task.connect(args)
# Log "model_variant" parameter to task
task.set_parameter("model_variant", model_variant)
task.set_parameter("dataset", dataset_id)
task.set_parameter("General/data", args['data'])
task.connect_configuration(name="Dataset YAML", configuration=args['data'])
task.connect_configuration(name="Dataset Content", configuration=dataset_content)
return task.id
I noticed that it's actually independent of the pipelines
I have the strong suspicion it is somewhat related to my parameters of the function or generally the hyperparameters gathered by the task automatically.
Is there some verbose mode I could run it with?
There has been a restart of my machine in the mean time :man-shrugging:
So only the matrix knows now I guess..
No idea what's going on now, but I cannot reproduce the behaviour either.. also tried my old code posted here, but the warning doesn't pop up anymore.
I will inform once it pops again and will use the provided traceback function then.
I have a slight suspicion that it was indeed environment based on my local machine, but I have no idea what is the trigger for that.
It may or may not be related to this
2024-04-29 23:38:25,932 - clearml.Task - WARNING - Parameters must be of builtin type (Args/env/ENV[_Environ], Args/env/DATASETS/d1c4ec5c8f7e4492ab7b2a8d0b584c88/created[datetime], Args/env/DATASETS/e7f05a856ca34cd88940c4b8774f4c45/created[datetime], Args/env/DATASETS/50e10f640d7548458d9c38ab9aac571b/created[datetime], Args/env/DATASETS/e113a14fec574025a15a5c9868746c4b/created[datetime], Args/env/DATASETS/c520737c392c47608ffd52f33fc593f6/created[datetime], Args/env/DATASETS/12f2a7ee7a7f45b8866d2c5f75e92413/created[datetime], Args/env/DATASETS/640fc2e67b3b4b498fe6d9b1da764911/created[datetime], Args/env/DATASETS/cef7f5e021484b089a5eccf0c014d150/created[datetime], Args/env/DATASETS/28b1c17f1fef47a793f9ac086b2d6b10/created[datetime], Args/env/DATASETS/af33141c123843d4bf0c50db76635bba/created[datetime], Args/env/DATASETS/2b561087cf974658b576fbd17a87bc87/created[datetime], Args/env/DATASETS/366058db107e485ca273a11601e68e7a/created[datetime], Args/env/DATASETS/f4d7bdcb58534f1ebbc5c5578b6801a5/created[datetime], Args/env/DATASETS/90648781cf4f4848a2a66ca572e1edf5/created[datetime], Args/env/DATASETS/a4b6cc7161334df1ad94143a819842d6/created[datetime], Args/env/DATASETS/853a677148c5417e8cc003d6761236e3/created[datetime], Args/env/DATASETS/b4f89b652b1940848f186c5ab9e2de3f/created[datetime], Args/env/DATASETS/231b526d445b416fa487fc902a1ebb0e/created[datetime], Args/env/DATASETS/55a19d9bd15c4c67a6931e3eba7b454a/created[datetime], Args/env/DATASETS/e8289fd5259c4e78830fd2e2d88a6d51/created[datetime], Args/env/DATASETS/55fba0728c3d4af1953fa0fe04b30ce6/created[datetime], Args/env/DATASETS/abc76956f1de41b386fa805e8858c884/created[datetime], Args/env/DATASETS/326ea43fc9ff4e6db5209218b6020ca6/created[datetime], Args/env/DATASETS/172affc0c4bb4a369e75f685782b5ad8/created[datetime])
But that's not used on my side anymore.
It's a weird warning and now it's a weird unreproducible warning, even though I had it on each trial run earlier today.
Thanks! Let me check if we can reproduce it. BTW what's your clearml package version?
Wait even without the pipeline decorator this function creates the warning?
It happens on all of my pipeline run attempts and there's nothing more that gives insight.
As an example:
python src/train.py
ClearML Task: created new task id=102a4f25c5ac4972abd41f1d0b6b9708
ClearML results page:
<unknown>:1: SyntaxWarning:
invalid decimal literal
<unknown>:1: SyntaxWarning:
invalid decimal literal
<unknown>:1: SyntaxWarning:
invalid decimal literal
<unknown>:1: SyntaxWarning:
invalid decimal literal
<unknown>:1: SyntaxWarning:
invalid decimal literal
<unknown>:1: SyntaxWarning:
invalid decimal literal
ClearML pipeline page:
... and then the pipeline just continues ...
Hi @<1694157594333024256:profile|DisturbedParrot38>
Could you attach a full log? This is quite cryptic and does not ring a bell
It comes from the PipelineDecorator.pipeline I assume or from PipelineDecorator.component
Not that I would know of..
I attached the possible problematic argument.
The strings are just regular string, nothing fancy there.
args
:{'epochs': 3, 'imgsz': 480, 'data': '/home/rini-debian/git-stash/lvgl-ui-detector/datasets/ui_randoms.yaml'}
model_variant
:yolov8n
dataset_id
:50e10f640d7548458d9c38ab9aac571b
I am getting the same when starting regular tasks.
I think it has something to do with my paramaters, which contain an environment variable which contains a list of datasets
Alright cool!
I will check it out and let you know what it was.
https://www.geeksforgeeks.org/invalid-decimal-literal-in-python/
This is the warning hence my question
Here are the codefiles for my pipelines.
They do not work yet, I am struggling with the pipeline stuff quite a bit.
But both pipelines always give these warnings.
Hmm yeah I think that makes sense. Can you post here the arguments?
I'm assuming you have something like '1.23a' in the arguments?
Yup.
I really don't know what it's about.
It doesn't affect the process. Everything seems to run fine.
If the warnings would provide a bit more info I could maybe pinpoint it better, but it's really all I got
Is this from the pipeline logic? Or a component?