SarcasticSquirrel56

16 Questions, 144 Answers

Active since 10 January 2023

Last activity 7 months ago

Reputation

Badges 1

137 × Eureka!

Questions 16
Answers 144

0 Votes

14 Answers

941 Views

0 Votes 14 Answers 941 Views

Hi Folks, I Have Installed Clearml On Kubernets Using The Helm Chart, But I Had To Specify Three Different Domains For The Ui, Apiserver And Fileserver. Is There Any Way To Let Clearml Know That The Apiserver Is At

Hi folks, I have installed ClearML on kubernets using the helm chart, but I had to specify three different domains for the ui, apiserver and fileserver. Is t...

clearml

2 years ago

0 Votes

8 Answers

1K Views

0 Votes 8 Answers 1K Views

Hi Folks, A Question Regarding The Clearml-Agent With K8S Glue. In The Agents We Mount An Nfs Volume So That Some Artifacts And Data Would Be Available For Training. I Have Seen That The K8S Glue Runs As Root (I Guess To Be Able To Spawn New Pods?), But

Hi folks, a question regarding the clearml-agent with k8s glue. In the agents we mount an nfs volume so that some artifacts and data would be available for t...

clearml

2 years ago

0 Votes

15 Answers

1K Views

0 Votes 15 Answers 1K Views

Hi Everybody, I Am Having An Issues With A Self-Hosted Clearml Server... I Am Having A Problem Enqueuing Experiments Whose Code Is In A Git Repository, They Are In A Pending State And Proceed... However If I Copy The Same Code Out In A Folder With No Rep

Hi everybody, I am having an issues with a self-hosted clearml server... I am having a problem enqueuing experiments whose code is in a git repository, they ...

clearml

2 years ago

0 Votes

7 Answers

1K Views

0 Votes 7 Answers 1K Views

Hi Folks, I Am Having An Issue I Can'T Properly Understand: I Have Tried To Run The "Dataset" Example From The Official Clearml Repository (From My Laptop) For Some Reason It Got Stuck, So I Killed The Process, But In Clearml Ui It Still Results As "Runn

Hi folks, I am having an issue I can't properly understand: I have tried to run the "dataset" example from the official clearml repository (from my laptop) F...

clearml

2 years ago

0 Votes

4 Answers

1K Views

0 Votes 4 Answers 1K Views

I Do Have One Questions About Using The Helm Chart, Is There Any Way To Specify The Users In The Values.Yaml?

I do have one questions about using the helm chart, is there any way to specify the users in the values.yaml?

clearml

2 years ago

0 Votes

11 Answers

1K Views

0 Votes 11 Answers 1K Views

Good Morning Folks, I Am Setting Up Clearml On A (Self-Hosted) K8S Cluster Using The

Good morning folks, I am setting up ClearML on a (self-hosted) K8s cluster using the https://github.com/allegroai/clearml-helm-charts/blob/main/charts/clearm...

mlops

2 years ago

0 Votes

19 Answers

1K Views

0 Votes 19 Answers 1K Views

Hi Folks, One Question: I Have A Script That Looks Like:

Hi folks, one question: I have a script that looks like: import clearml as cml import numpy as np from sklearn.linear_model import LogisticRegression from sk...

clearml

2 years ago

0 Votes

6 Answers

998 Views

0 Votes 6 Answers 998 Views

Hi Folks, Good Morning

Hi folks, good morning 🙂 In our setup we have a set of queues that do not use any GPU resources. Yet, when I run an experiment in such queues, we see a Warn...

clearml

2 years ago

0 Votes

6 Answers

1K Views

0 Votes 6 Answers 1K Views

Hi Folks, I Have A Question On Something That It'S Not Clear To Me Reading The Documentation At

Hi folks, I have a question on something that it's not clear to me reading the documentation at https://clear.ml/docs/latest/docs/clearml_agent/ From what I ...

clearml

2 years ago

0 Votes

13 Answers

1K Views

0 Votes 13 Answers 1K Views

Hi Folks, I Have A Question Related To The Storage Of Artifacts, As It Is Not Entirely Clear To Me Where To Configure It. If I Read The Documentation

Hi folks, I have a question related to the storage of artifacts, as it is not entirely clear to me where to configure it. If I read the documentation https:/...

clearml

2 years ago

0 Votes

31 Answers

20K Views

0 Votes 31 Answers 20K Views

Hi Folks, I Just Deployed A Clearml Agent Using The Helm Chart. I Have A Few Doubts:

Hi folks, I just deployed a ClearML agent using the Helm chart. I have a few doubts: after the deployment, I see a new queue called k8s_scheduler, which I di...

mlops

2 years ago

0 Votes

6 Answers

1K Views

0 Votes 6 Answers 1K Views

Hi Folks, Any Of You Has Experience In Deploying Clearml To Kubernetes Using Argocd? I Managed To Make It Run Pointing It To The Clearml-Charts-Repo, It Recognizes The Helm Chart And It Works. But I Am Struggling A Bit To Write My Own Definition To Make I

Hi folks, any of you has experience in deploying ClearML to Kubernetes using ArgoCD? I managed to make it run pointing it to the clearml-charts-repo, it reco...

clearml

2 years ago

0 Votes

17 Answers

1K Views

0 Votes 17 Answers 1K Views

Hi Folks I Have A Problem I Can'T Understand. Plots Are Not Shown When Experiments Are Executed From The Ui. For Example, If I Run The Code On My Laptop, And I Go To The Experiment Page I Can See Correctly The Plots: But If I Then Clone The Task, And Ex

Hi folks I have a problem I can't understand. Plots are not shown when experiments are executed from the UI. For example, if I run the code on my laptop, and...

clearml

2 years ago

0 Votes

31 Answers

22K Views

0 Votes 31 Answers 22K Views

Hi Folks, Occasionally When I Clone A Job And Enqueue It, Instead Of Being Processed By The Expected Queue, A New Queue (With Some Id That Looks Like An Hash) Is Created Instead, And The Experiment Hangs In A "Pending" State. When This Happens, If I Abor

Hi folks, occasionally when I clone a job and enqueue it, instead of being processed by the expected queue, a new queue (with some id that looks like an hash...

clearml

2 years ago

0 Votes

31 Answers

23K Views

0 Votes 31 Answers 23K Views

Hi Folks, I Did A Deployment Of Clearml Using The K8S Helm Chart, And I Set The Agent Using K8S Glue. I Run A Task Locally, And I Went To The Ui Cloned The Experiment And Scheduled It In The Default Queue. After Doing This, I See That The Experiment Is Q

Hi folks, I did a deployment of ClearML using the K8s helm chart, and I set the agent using K8s Glue. I run a task locally, and I went to the UI cloned the e...

mlops

2 years ago

0 Votes

31 Answers

22K Views

0 Votes 31 Answers 22K Views

Hi Folks, I Am Trying To Run Clearml On A Local Kubernetes (Created With Rancher Desktop), But I Am Running Into Issues. I Admit I Am A Complete Noob To K8S, Helm, Clearml Etc. Etc. So I Am Not Sure If I Did Some Very Stupid Mistakes. I Have A K8S Cluste

Hi folks, I am trying to run ClearML on a local kubernetes (created with Rancher Desktop), but I am running into issues. I admit I am a complete noob to K8s,...

kubernetes

2 years ago

0 Hi Folks, Occasionally When I Clone A Job And Enqueue It, Instead Of Being Processed By The Expected Queue, A New Queue (With Some Id That Looks Like An Hash) Is Created Instead, And The Experiment Hangs In A "Pending" State. When This Happens, If I Abor

now, I go to experiment, clone an experiment that I previously executed on my laptop. In the newly created experiment, I modify some parameter, and enqueue the experiment in the CPU queue.

2 years ago

0 Hi Folks, Any Of You Has Experience In Deploying Clearml To Kubernetes Using Argocd? I Managed To Make It Run Pointing It To The Clearml-Charts-Repo, It Recognizes The Helm Chart And It Works. But I Am Struggling A Bit To Write My Own Definition To Make I

I see in bitnami's gh-pages branch a file https://github.com/bitnami-labs/sealed-secrets/blob/gh-pages/index.html to do the redirect that contains:

` <html>

<head> <meta http-equiv="refresh" content="0; url= ` ` "> </head> <p><a href=" ` ` ">Redirect to repo index.yaml</a></p> </html> ` A similar file is missing in the ` clearml-helm-chart ` ` gh-pages ` branch.

2 years ago

The workaround that works for me is:
clone the experiment that I run on my laptop in the newly cloned experiment, modify the hyperparameters and configurations to my need in user properties set "k8s-queue" to "cpu" (or the name of queue I want to use) enqueue the experiment to the same queue I just set...
When I do like that in the K8sGlue pod for the cpu queue I can see that it has been correctly picked up:
` No tasks in queue 54d3edb05a89462faaf51e1c878cf2c7
No tasks in Queues, sleeping fo...

2 years ago

0 Hi Folks, One Question: I Have A Script That Looks Like:

Thanks Martin! If I end up having sometime I'll dig into the code and check if I can bake something!

2 years ago

0 Good Morning Folks, I Am Setting Up Clearml On A (Self-Hosted) K8S Cluster Using The

What I still don't get, is how you would create different queues, targeting different nodes with different GPUs, and having them using the appropriate Cuda image.
Looking at the template, I don't understand how that's possible.

2 years ago

Ah sorry, I thought what where the names of the queue I created like (in case I used some weird character or stuff like that)

2 years ago

0 Hi Folks, One Question: I Have A Script That Looks Like:

Thanks Martin.. I'll add this and check whether it fixes the issue, but I don't get quite well this though.. The local code doesn't need to import pandas, because the get method returns a DataFrame object that has a .loc method.
I was expecting the remote experiment to behave similarly, why do I need to import pandas there?

2 years ago

If I now reset the experiment, and enqueue the experiment to the gpu queue (but in the experimet, the user-properties configuration for k8s-glue is still set to cpu) the experiment is left in a Pending state... and in the K8sGlue Agent for the gpu queue, I can see a similar error as the one in the cpu agent....

` No tasks in Queues, sleeping for 5.0 seconds
No tasks in queue 75174e0e7ac047f195ab4dce6e9f03f7
No tasks in Queues, sleeping for 5.0 seconds
FATAL ERROR:
Traceback (most recent call...

2 years ago

0 Good Morning Folks, I Am Setting Up Clearml On A (Self-Hosted) K8S Cluster Using The

Thanks Martin, so if I understand correctly, when I do the clearml-agent init command (I have to check the syntax), by providing the apiserver webeserver and fileserver url they'll be registered to the clearml cluster?

2 years ago

0 Good Morning Folks, I Am Setting Up Clearml On A (Self-Hosted) K8S Cluster Using The

Hi AgitatedDove14 I have spent some time going through the helm charts but I admit I still haven't clear how things should work.

I see that with the default values (mostly what I am using), the K8s Glue agent is deployed (which is what you suggested to use).

2 years ago

0 Hi Folks, One Question: I Have A Script That Looks Like:

Hi Jake, sorry I left the office yesterday. On my laptop I have clearml==1.6.4

2 years ago

0 Hi Everybody, I Am Having An Issues With A Self-Hosted Clearml Server... I Am Having A Problem Enqueuing Experiments Whose Code Is In A Git Repository, They Are In A Pending State And Proceed... However If I Copy The Same Code Out In A Folder With No Rep

Yes, I still see those errors, but queues are working :)

2 years ago

0 Hi All! Question Around Resource Management Using

Hi Martin, I admit I don't know about MIG I'll have to ask some of our engineers.

As for the memory, yes the reasoning is clear, the main thing we'll have to see is hot define the limits, because we have nodes with quite different resources available, and this might get tricky, but I'll try and let's see what happens 🙂

We actually plan to create different queues for different types of workloads, we are a bit seeing what the actual usage is to define what type of workloads make sense for us.

2 years ago

Super!!! many thanks CostlyFox64 !

2 years ago

0 Hi Folks, I Have A Question On Something That It'S Not Clear To Me Reading The Documentation At

but I don't understand the comment on GPUs as the documentation makes a lot of references on GPU configurations for agents

2 years ago

0 Hi Folks, One Question: I Have A Script That Looks Like:

but I can confirm that adding the requirement with Task.add_requirements() does the trick

2 years ago

Next week I can take some screenshots if you need them, ai just closed the laptop and will be off for a couple of days :))

2 years ago

0 Hi All, If Install Clearml Agent With Helm, How Do I Authenticate The Agent With Git? In Values.Yaml File, I Can Set Clearml_Agent_Git_User And Clearml_Agent_Git_Pass But When The Agent Run, Git Has Error "Remote: Support For Password Authentication Was R

this worked for me, at least

2 years ago

Hi SuccessfulKoala55 I can confirm that the "id-like" queue created by ClearML
actually correspond to the id of queue "k8s_scheduler" (so it looks like that instead of submitting the experiment to the scheduler to be enqueued to the right queue), a new queue whose name corresponds to the id of the k8s_scheduler is created instead.

Hope this helps 🙂

2 years ago

PunyWoodpecker71 just create a Personal Access Token and use it as the value for CLEARML_AGENT_GIT_PASS, https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token

2 years ago

0 Hi Folks, I Am Trying To Run Clearml On A Local Kubernetes (Created With Rancher Desktop), But I Am Running Into Issues. I Admit I Am A Complete Noob To K8S, Helm, Clearml Etc. Etc. So I Am Not Sure If I Did Some Very Stupid Mistakes. I Have A K8S Cluste

using the --set you adviced above right?

2 years ago

0 Hi Folks, I Have Installed Clearml On Kubernets Using The Helm Chart, But I Had To Specify Three Different Domains For The Ui, Apiserver And Fileserver. Is There Any Way To Let Clearml Know That The Apiserver Is At

is there a way I can check whether the apiserver are reachable?
(like: https://clearml-apiserver.ds.bumble.dev/health http://ds.bumble.dev/health )

2 years ago

Thanks SuccessfulKoala55 . Any idea why going to the address https://allegroai.github.io/clearml-helm-charts
returns a 404 error?

Other repositories that are used in Argo CD examples (e.g. https://bitnami-labs.github.io/sealed-secrets , which is also hosted on Github) instead of returning a 404, the index.yaml page is loaded instead.

I suspect this might be the reason why I can't make it work with ClearML.

2 years ago

Show more results