Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Profile picture
NuttyCamel41
Moderator
12 Questions, 42 Answers
  Active since 18 January 2023
  Last activity 9 months ago

Reputation

0

Badges 1

42 × Eureka!
0 Votes
4 Answers
2K Views
0 Votes 4 Answers 2K Views
Hi all! Does anyone know a solution to my issue with deploying models saved on azure on the clearml-serving docker container?
2 years ago
0 Votes
0 Answers
1K Views
0 Votes 0 Answers 1K Views
Hi all, I am trying to add a model to my recently setup k8s self-hosted clearml-serving. The command looks like this and worked whith my previous docker setu...
11 months ago
0 Votes
13 Answers
869 Views
0 Votes 13 Answers 869 Views
Hi there! Can anybody help me with specifying the 'platform' for a model in clearml-serving. I am using the k8s clearml-serving setup (version 1.3.1). I alre...
9 months ago
0 Votes
8 Answers
1K Views
0 Votes 8 Answers 1K Views
one year ago
0 Votes
2 Answers
1K Views
0 Votes 2 Answers 1K Views
Hi all, I again have a problem which was already reported here. It seems like the agent ignores the reqirements even when I am explicitely adding them by Tas...
2 years ago
0 Votes
6 Answers
932 Views
0 Votes 6 Answers 932 Views
11 months ago
0 Votes
3 Answers
1K Views
0 Votes 3 Answers 1K Views
2 years ago
0 Votes
3 Answers
904 Views
0 Votes 3 Answers 904 Views
Hi all! I am in the process of setting up clearml-serving on my kubernetes cluster using the provided helm charts. Currently I am stuck with running the cont...
11 months ago
0 Votes
6 Answers
1K Views
0 Votes 6 Answers 1K Views
Hi there! :) I have an issue regarding the get_local_copy(..) function of the Model class. Whenever this function is called in the course of a hyperparameter...
one year ago
0 Votes
5 Answers
2K Views
0 Votes 5 Answers 2K Views
Hi all, I have the same problem as stated in this Thread. The file encoding of all files in my project is utf-8 and I already set the environment variable PY...
2 years ago
0 Votes
22 Answers
1K Views
0 Votes 22 Answers 1K Views
Hi all! I recently started working with clearML serving. I got this example working https://github.com/allegroai/clearml-serving/tree/main/examples/pytorch a...
2 years ago
0 Votes
6 Answers
1K Views
0 Votes 6 Answers 1K Views
2 years ago
0 Hi There! :) I Have An Issue Regarding The

This is the hpo task log with the azure-storage-blob in version 12.14.1

one year ago
0 Hi There! :) I Have An Issue Regarding The

Yes, I also find that very weird... I start the hyperparameter optimization via python code using the HyperParameterOptimizer class of clearml. Which configurations are you explicitely interested in?
When comparing the logs of the two hpo tasks it seems like no logs of the subtasks are getting to the hpo task. So maybe this is the reason for the infinitely long running subtask? But what does the azure package have to do with that?

one year ago
0 Hey All, I Want To Purchase The Pro Version Of Clearml But Would Like To Have A Better Understanding Of The Metric Events And Api Calls That Are Performed When Using Clearml-Serving. For Example: I Have No Understanding Which Docker Container Calls The Ap

Hi @<1523701205467926528:profile|AgitatedDove14> , thanks for your answer!
I reached over 1M API calls in about one week using clearml-serving on one machine with only calling the deployed model a few hundred times for testing purpose. So I wanted to dig a little bit deeper on that. Thanks for the channel suggestion, I will repost my question there. :)

2 years ago
0 Hi There! :) I Have An Issue Regarding The

Hi @<1523701070390366208:profile|CostlyOstrich36> , thanks for your answer! I just updated the 'azure_storage_blob' package to the newest version and got some strange behaviour. When running the BOHB hyperparameter optimization, there is only one job executed and not stopped. I aborted the job after 3500 epochs because I set the the max_iteration_per_job parameter to 1000 and the job seems to run infinitely long. I just downgraded the package back to version 12.14.1 and everything works as b...

one year ago
0 Hi There! :) I Have An Issue Regarding The

This is the log of the hpo task with the newest azure azure-storage-blob version

one year ago
0 Hi All! I Recently Started Working With Clearml Serving. I Got This Example Working

Ok, I have some weird update... I shut down and restarted the docker container just to get fresh logs and now I am getting the following error message by clearml-serving-triton
` clearml-serving-triton | clearml-serving - Nvidia Triton Engine Controller
clearml-serving-triton | Warning: more than one valid Controller Tasks found, using Task ID=433aa14db3f545ad852ddf846e25dcf0
clearml-serving-triton | ClearML Task: overwriting (reusing) task id=350a5a919ff648148a3de4483878...

2 years ago
0 Hi All! I Recently Started Working With Clearml Serving. I Got This Example Working

Hi ExasperatedCrab78 , thanks for your answer! In fact I used your recommended format for passing input and output size before and changed it in my debugging process. I have just tried again but got the same error message.
Also thanks for the hint to check the log for warnings I wil do this in a moment.

2 years ago
0 Hi There! I Want To Set Up Clearml On A Kubernetes Cluster. The Whole Setup Including Authentication Of Fixed Users Seems To Work Fine. I Can Log In On The Webapp And Generate Credentials For Connecting My Local Clearml Installation. The Clearml-Init Also

The clearml-data call results in these two lines in the ingress logs. Is that sufficient or would you like to have a larger section of the log?

2024/03/26 16:07:10 [warn] 2879#2879: *1151249 upstream sent duplicate header line: "server: clearml", previous value: "Server: Werkzeug/3.0.1 Python/3.9.18", ignored while reading response header from upstream, client: ***.***.***.22, server: api.clearml.****.com, request: "GET /auth.login HTTP/1.1", upstream: "
", host: "api.clearm...
11 months ago
0 Hi All, I Have The Same Problem As Stated In This Thread. The File Encoding Of All Files In My Project Is Utf-8 And I Already Set The Environment Variable

Full log:
` Current configuration (clearml_agent v1.5.1, location: C:/Users/USER~1/AppData/Local/Temp/.clearml_agent.g6ysfs_g.cfg):

agent.worker_id = HPZBook:0
agent.worker_name = HPZBook
agent.force_git_ssh_protocol = false
agent.python_binary =
agent.package_manager.type = pip
agent.package_manager.pip_version.0 = <20.2 ; python_version < '3.10'
agent.package_manager.pip_version.1 = <22.3 ; python_version >= '3.10'
agent.package_manager.system_site_packages = false
...

2 years ago
0 Hi All! I Was Just Wondering What Is The Best Way To Log Additional Information? Right Now I'M Only Printing It To The Console, But That'S Not The Most Pleasant Way To Retrieve The Information Later On. As Far As I Can See, The 'Logger.Report_Text(...)' M

Hello CostlyOstrich36 , thanks for your question. At the moment I am training a MLP for a regression problem and in one case I want to store the number of neurons per layer. Note that in my case it is not a hyperparameter because I calculate the number of neurons based on the number of layers and the number of model parameters. Another case is that I want to store some local paths where the models are stored, since I currently don't have any remote storage set up for my models.

2 years ago
0 Hi There! Can Anybody Help Me With Specifying The 'Platform' For A Model In Clearml-Serving. I Am Using The K8S Clearml-Serving Setup (Version 1.3.1). I Already Tried A Bunch Of Variants Like

Hi @<1523701205467926528:profile|AgitatedDove14> thanks for your hint! I already convert it to torch script using tracing. Everything around the model should be fine, since it already worked with the docker clearml-serving setup.
I think the real issue is that I am not able to specify a platform for the model, as the error above tells me that no platform is given no matter how I try to pass it.

9 months ago
0 Hi All! I Am In The Process Of Setting Up Clearml-Serving On My Kubernetes Cluster Using The Provided Helm Charts. Currently I Am Stuck With Running The Control Task. When I Call

Hi @<1523701070390366208:profile|CostlyOstrich36> , of course! Here it is (with blurred urls, paths and account names)

11 months ago
0 Hi There :) Can Anybody Tell Me What The Best Practice Is For Performing A Normalization In The Preprocess.Py Script Used By Clearml-Serving? Currently I Use A Sklearn Minmaxscaler Which Is Loaded And Applied Before And After The Data Is Send To The Model

Hi @<1523701205467926528:profile|AgitatedDove14> thanks for your answer! 🙂 I think my case is a bit different. I do not want to load a custom model but I want to load a custom object used for preprocessing. So I think the load method would not fit, as the local_file_name parameter I get in the load function would lead to the model file. And as far as I can see there is no mechanism installed to load other objects than the model file inside the Preprocess class, right?

one year ago
0 Hi There :) Can Anybody Tell Me What The Best Practice Is For Performing A Normalization In The Preprocess.Py Script Used By Clearml-Serving? Currently I Use A Sklearn Minmaxscaler Which Is Loaded And Applied Before And After The Data Is Send To The Model

Hi @<1523701205467926528:profile|AgitatedDove14> , that is an interesting idea! But wouldn't it be better to load the model in the load() function, so that the model doesn't have to be loaded again with every request? Or is there kind of internal link that when the load() method is implemented it is expected that there was a custom model loaded and applied in the process() function?

one year ago
0 Hi There :) Can Anybody Tell Me What The Best Practice Is For Performing A Normalization In The Preprocess.Py Script Used By Clearml-Serving? Currently I Use A Sklearn Minmaxscaler Which Is Loaded And Applied Before And After The Data Is Send To The Model

Hi @<1523701205467926528:profile|AgitatedDove14> , I serialized a sklearn MinMaxScaler object which I created on the training data using pickle. So when serving the model I would like to load that pickle file in the preprocess script such that I can perform the same normalization as done during training. Unless there is a better practice applying the same normalization during training and serving time.

one year ago
0 Hi All! I Recently Started Working With Clearml Serving. I Got This Example Working

Ok, I have found the issue. 🙌 When I try to serve a model which is saved on azure (generated by Task.init(..., output_uri='azure://...') ) I get the poll failed for model directory 'test_model_pytorch': failed to open text file for read /models/test_model_pytorch/config.pbtxt: No such file or directory error. A model which was saved on the clearML server (generated by Task.init(..., output_uri=True) ) can be served without any problems.
For now I am not sure why th...

2 years ago
0 Hi All, I Have The Same Problem As Stated In This Thread. The File Encoding Of All Files In My Project Is Utf-8 And I Already Set The Environment Variable

Thank you, I did not think about that. It helped a lot! I found out that the problem causing the unicode error was, that only the 'python' command was set up on my windows machine, but not the 'python3' command. This was the exact error for documentation:

DEBUG:clearml_agent.commands.worker:Searching for python3
Traceback (most recent call last):
  File "C:\Users\User\venvs\clearml\lib\site-packages\clearml_agent\helper\process.py", line 204, in normalize_exception
    yield
  File "C:\...
2 years ago
0 Hi All! I Recently Started Working With Clearml Serving. I Got This Example Working

Ok, so I killed all docker containers (the proposal by chatgpt did not work for me, but your commands did). The result is that we have one less warning. The warning clearml-serving-triton | Warning: more than one valid Controller Tasks found, using Task ID=4709b0b383a04bb1a033e99fd325dcbf seems to be solved. All remaining errors come up in the clearml-serving-triton service and this is the log I get

CLEARML_SERVING_TASK_ID=9309c20af9244d919b0f063642198c57
CLEARML_TRITON_POLL...
2 years ago
0 Hi All, I Have The Same Problem As Stated In This Thread. The File Encoding Of All Files In My Project Is Utf-8 And I Already Set The Environment Variable

Yes I am running the agent by calling clearml-agent daemon --queue default in my virtual environment on my local computer.

2 years ago
0 Hi All! I Recently Started Working With Clearml Serving. I Got This Example Working

By the way, the example which worked for me in the beginning also produces the same error now poll failed for model directory 'test_model_pytorch': failed to open text file for read /models/test_model_pytorch/config.pbtxt: No such file or directory . So there really seems to be something wrong with the docker containers.

2 years ago
0 Hi All! I Recently Started Working With Clearml Serving. I Got This Example Working

I got it working!! For now I am not sure what did the trick because I tried a bunch of different things. But I will try to reproduce it and come back to this thread for other users facing this problem. So big thanks for your help, @<1523701118159294464:profile|ExasperatedCrab78> !

2 years ago
0 Hi All! I Recently Started Working With Clearml Serving. I Got This Example Working

You're very welcome, thank you again for the great support. :)) I followed the instructions of the clearml-serving README on github None . There is one section called 'Optional: advanced setup - S3/GS/Azure access'. Maybe the syntax could be added there? I also saw the additonal link to configure the storage access, but this site focuses on setting up the clearml.conf and I was not sure how and if I could transfer it to the docker .env-file.
A...

2 years ago
0 Hi All! I Was Just Wondering What Is The Best Way To Log Additional Information? Right Now I'M Only Printing It To The Console, But That'S Not The Most Pleasant Way To Retrieve The Information Later On. As Far As I Can See, The 'Logger.Report_Text(...)' M

Hi @<1523701323046850560:profile|OutrageousSheep60> , thanks for your message as well. So far I have actually been using these exact functions until I noticed the following: when I run a task with these calls, everything works as expected. However, if I do a hyperparameter tuning and change some of the hyperparameters so that the additional information that is not a hyperparameter also changes, they are not adjusted. For better understanding again my concrete example: I have 3 parameters/inf...

2 years ago
0 Hi All! I Was Just Wondering What Is The Best Way To Log Additional Information? Right Now I'M Only Printing It To The Console, But That'S Not The Most Pleasant Way To Retrieve The Information Later On. As Far As I Can See, The 'Logger.Report_Text(...)' M

Hi @<1523701087100473344:profile|SuccessfulKoala55> , thanks for your message! 🙂 I am aware that the console is also logged on the server, but I somehow find it not optimal to look for relevant information in the console log and would like to place the information in a more structured way.

2 years ago
0 Hi All, I Again Have A Problem Which Was Already Reported Here. It Seems Like The Agent Ignores The Reqirements Even When I Am Explicitely Adding Them By Task.Add_Requirements(...). I Would Really Appreciate Some Help As I Am New To Clearml.

Hi @<1523701435869433856:profile|SmugDolphin23> , thanks for your question. For now I just deleted the requirements.txt and let ClearML track the requirements automatically and it works. For long term I would still like to use a requirements.txt, so I will come back to this topic a little later.

2 years ago
0 Hi There! I Want To Set Up Clearml On A Kubernetes Cluster. The Whole Setup Including Authentication Of Fixed Users Seems To Work Fine. I Can Log In On The Webapp And Generate Credentials For Connecting My Local Clearml Installation. The Clearml-Init Also

Hi @<1523701827080556544:profile|JuicyFox94> I figured out what the problem is! For some recent experimentation I set an acces_key and secret_key as environment variables in my os. When I deleted them everything worked fine so the environment variables overwrote the keys given by the clearml.conf. Is that the desired default behaviour?
And just one tip for everbody having similar problems: Switch to using the SDK instead of the CLI for better debugging. This helped me to find the cause of m...

11 months ago
0 Hi There! Can Anybody Help Me With Specifying The 'Platform' For A Model In Clearml-Serving. I Am Using The K8S Clearml-Serving Setup (Version 1.3.1). I Already Tried A Bunch Of Variants Like

Hi @<1523701205467926528:profile|AgitatedDove14> , thanks for coming back to my issue. Unfortunately I have a lot of other stuff on my desk right now so I have to postpone finishing this issue. I will reach out to you again as soon as possible (especially if I was able to find a solution).

9 months ago
Show more results compactanswers