Damn it, you're right 😅
# Allow ClearML access to the training args and allow it to override the arguments for remote execution
args_class = type(training_args)
args, changed_keys = cast_keys_to_string(training_args.to_dict())
Task.current_task().connect(args)
training_args = args_class(**cast_keys_back(args, changed_keys)[0])
An update: using your code (the snippet above) I was getting no scalars when simply installing ultralytics and clearml packages using pip. Because indeed tensorboard is not installed. When I do install tensorboard, I get metrics in like normal, so I can't seem to reproduce the issue when tensorboard is correctly installed. That said, maybe we should look at not having this dependency 🤔
Would you mind posting a pip freeze of your environment that you're using to run yolo?
Also, this might be a little stupid sorry, but your torch save command saves the model in the current folder, whereas you give clearml the 'model_folder/model' path instead. Could it be that the path is just incorrect?
Hey CheekyFox58 like Martin said, it should at least work locally. If not, can you give us some more details on what exactly the werid behaviour is?
It depends on how complex your configuration is, but if config elements are all that will change between versions (i.e. not the code itself) then you could consider using parameter overrides.
A ClearML Task can have a number of "hyperparameters" attached to it. But once that task is cloned and in draft mode, one can EDIT these parameters and change them. If then the task is queued, the new parameters will be injected into the code itself.
A pipeline is no different, it can have pipeline par...
No worries! And thanks for putting in the time.
For the record, this is a minimal reproducible example:
Local folder structure:
` ├── remove_folder
│ ├── batch_0
│ │ ├── file_0_0.txt
│ │ ├── file_0_1.txt
│ │ ├── file_0_2.txt
│ │ ├── file_0_3.txt
│ │ ├── file_0_4.txt
│ │ ├── file_0_5.txt
│ │ ├── file_0_6.txt
│ │ ├── file_0_7.txt
│ │ ├── file_0_8.txt
│ │ └── file_0_9.txt
│ └── batch_1
│ ├── file_1_0.txt
│ ├── file_1_1.txt
│ ├── file_1_2.txt
│ ├── file_1_3.txt
│ ├── fi...
Yeah, I do the same thing all the time. You can limit the amount of tasks that are kept in HPO with the save_top_k_tasks_only parameter and you can create subprojects by simply using a slash in the name 🙂 https://clear.ml/docs/latest/docs/fundamentals/projects#creating-subprojects
Like Nathan said, custom engines are a TODO, but for your second question, you can add that API request in the model preprocessing, which is a function you can define yourself! It will be ran every time a request comes in and you can do whatever you want in it and change the incoming data however you wish 🙂
example: https://github.com/allegroai/clearml-serving/blob/main/examples/keras/preprocess.py
Isitdown seems to be reporting it as up. Any issues with other websites?
That makes sense! Maybe something like dataset querying as is used in the clearml hyperdatasets might be useful here? Basically you'd query your dataset to only include sample you want and have the query itself be a hyperparameter in your experiment?
Just to be sure I understand you correctly: you're saving/dumping an sklearn model in the clearml experiment manager, then want to serve it using clearml serving, but you do not wish to specify the model input and ouput shapes in the CLI?
Could you tell us what browser (version) you're using, maybe we can recreate ourselves 🙂
That wasn't my intention! Not a dumb question, just a logical one 😄
Hi ThoughtfulGrasshopper59 !
You're right, we should probably add the convenient allow_archived function in .get_task s () as well.
That said, for now this can be a workaround:
` from clearml import Task
print([task.name for task in Task.get_tasks(
project_name="TAO Toolkit ClearML Demo",
task_filter=dict(system_tags=['archived'])
)]) Specifically task_filter=dict(system_tags=['archived']) ` should be what you need.
FierceHamster54 I saw you saying the YOLOv5 project and name are hardcoded in there. Fixed that for ya 😉 https://github.com/ultralytics/yolov5/pull/10100
Pipelines! 😄
ClearML allows you to create pipelines, with each step either being created from code or from pre-existing tasks. Each task btw. can have a custom docker container assigned that it should be run inside of, so it should fit nicely with your workflow!
Youtube videos:
https://www.youtube.com/watch?v=prZ_eiv_y3c
https://www.youtube.com/watch?v=UVBk337xzZo
Relevant Documentation:
https://clear.ml/docs/latest/docs/pipelines/
Custom docker container per task:
https://...
Hi William!
1 So if I understand correctly, you want to get an artifact from another task into your preprocessing.
You can do this using the Task.get_task() call. So imagine your anomaly detection task is called anomaly_detection it produces an artifact called my_anomaly_artifact and is located in the my_project project you can do:
` from clearml import Task
anomaly_task = Task.get_task(project_name='my_project', task_name='anomaly_detection')
treshold = anomaly_ta...
Hi ExuberantParrot61 ! Can you try using a wildcard? E.g. ds.remove_files(dataset_path='folder_to_delete/*')
If you didn't use git, then clearML saves your .py script completely in the uncommited changes section like you say. You should be able to just copy paste it to get the code. In what format are your uncommited changes logged? Can you paste a screenshot or paste the contents of uncommitted changes ?
In order to prevent these kinds of collisions it's always necessary to provide a parent dataset ID at creation time, so it's very clear which dataset and updated one is based on. If multiple of them happen at the same time, they won't know of each other and both use the same dataset as the parent. This will lead to 2 new versions based on the same parent dataset, but not sharing data with each other. If that happens, you could create a 3rd dataset (potentially automatically) that can have bot...
Hi LackadaisicalDove24 !
Does this happen with every csv file? If so, I can reproduce it to check if it is a bug 🙂
Hi! Have you tried adding custom metrics to the experiment table itself? You can add any scalar as a column in the experiment list, it does not have color formatting, but it might be more like what you want in contrast to the compare functionality 🙂
Don't paste your API keys! 🙈
1 Can you give a little more explanation about your usecase? It seems I don't fully understand yet. So you have multiple endpoints, but always the same preprocessing script to go with it? And you need to gather a different threshold for each of the models?
2 Not completely sure of this, but I think an AMD APU simply won't work. ClearML serving is using triton as inference engine for GPU based models and that is written by nvidia, specifically for nvidia hardware. I don't think triton will ...
Wait is it possible to do what i'm doing but with just one big Dataset object or something?
Don't know if that's possible yet, but maybe something like the proposed querying could help here?