SarcasticSquirrel56

16 Questions, 144 Answers

Active since 10 January 2023

Last activity one month ago

Reputation

Badges 1

137 × Eureka!

Questions 16
Answers 144

0 Votes

6 Answers

616 Views

0 Votes 6 Answers 616 Views

Hi Folks, Good Morning

Hi folks, good morning 🙂 In our setup we have a set of queues that do not use any GPU resources. Yet, when I run an experiment in such queues, we see a Warn...

clearml

one year ago

0 Votes

13 Answers

706 Views

0 Votes 13 Answers 706 Views

Hi Folks, I Have A Question Related To The Storage Of Artifacts, As It Is Not Entirely Clear To Me Where To Configure It. If I Read The Documentation

Hi folks, I have a question related to the storage of artifacts, as it is not entirely clear to me where to configure it. If I read the documentation https:/...

clearml

one year ago

0 Votes

6 Answers

663 Views

0 Votes 6 Answers 663 Views

Hi Folks, I Have A Question On Something That It'S Not Clear To Me Reading The Documentation At

Hi folks, I have a question on something that it's not clear to me reading the documentation at https://clear.ml/docs/latest/docs/clearml_agent/ From what I ...

clearml

2 years ago

0 Votes

6 Answers

698 Views

0 Votes 6 Answers 698 Views

Hi Folks, Any Of You Has Experience In Deploying Clearml To Kubernetes Using Argocd? I Managed To Make It Run Pointing It To The Clearml-Charts-Repo, It Recognizes The Helm Chart And It Works. But I Am Struggling A Bit To Write My Own Definition To Make I

Hi folks, any of you has experience in deploying ClearML to Kubernetes using ArgoCD? I managed to make it run pointing it to the clearml-charts-repo, it reco...

clearml

2 years ago

0 Votes

19 Answers

617 Views

0 Votes 19 Answers 617 Views

Hi Folks, One Question: I Have A Script That Looks Like:

Hi folks, one question: I have a script that looks like: import clearml as cml import numpy as np from sklearn.linear_model import LogisticRegression from sk...

clearml

one year ago

0 Votes

17 Answers

686 Views

0 Votes 17 Answers 686 Views

Hi Folks I Have A Problem I Can'T Understand. Plots Are Not Shown When Experiments Are Executed From The Ui. For Example, If I Run The Code On My Laptop, And I Go To The Experiment Page I Can See Correctly The Plots: But If I Then Clone The Task, And Ex

Hi folks I have a problem I can't understand. Plots are not shown when experiments are executed from the UI. For example, if I run the code on my laptop, and...

clearml

one year ago

0 Votes

7 Answers

666 Views

0 Votes 7 Answers 666 Views

Hi Folks, I Am Having An Issue I Can'T Properly Understand: I Have Tried To Run The "Dataset" Example From The Official Clearml Repository (From My Laptop) For Some Reason It Got Stuck, So I Killed The Process, But In Clearml Ui It Still Results As "Runn

Hi folks, I am having an issue I can't properly understand: I have tried to run the "dataset" example from the official clearml repository (from my laptop) F...

clearml

one year ago

0 Votes

11 Answers

668 Views

0 Votes 11 Answers 668 Views

Good Morning Folks, I Am Setting Up Clearml On A (Self-Hosted) K8S Cluster Using The

Good morning folks, I am setting up ClearML on a (self-hosted) K8s cluster using the https://github.com/allegroai/clearml-helm-charts/blob/main/charts/clearm...

mlops

one year ago

0 Votes

8 Answers

660 Views

0 Votes 8 Answers 660 Views

Hi Folks, A Question Regarding The Clearml-Agent With K8S Glue. In The Agents We Mount An Nfs Volume So That Some Artifacts And Data Would Be Available For Training. I Have Seen That The K8S Glue Runs As Root (I Guess To Be Able To Spawn New Pods?), But

Hi folks, a question regarding the clearml-agent with k8s glue. In the agents we mount an nfs volume so that some artifacts and data would be available for t...

clearml

one year ago

0 Votes

31 Answers

4K Views

0 Votes 31 Answers 4K Views

Hi Folks, I Just Deployed A Clearml Agent Using The Helm Chart. I Have A Few Doubts:

Hi folks, I just deployed a ClearML agent using the Helm chart. I have a few doubts: after the deployment, I see a new queue called k8s_scheduler, which I di...

mlops

one year ago

0 Votes

31 Answers

4K Views

0 Votes 31 Answers 4K Views

Hi Folks, Occasionally When I Clone A Job And Enqueue It, Instead Of Being Processed By The Expected Queue, A New Queue (With Some Id That Looks Like An Hash) Is Created Instead, And The Experiment Hangs In A "Pending" State. When This Happens, If I Abor

Hi folks, occasionally when I clone a job and enqueue it, instead of being processed by the expected queue, a new queue (with some id that looks like an hash...

clearml

one year ago

0 Votes

14 Answers

535 Views

0 Votes 14 Answers 535 Views

Hi Folks, I Have Installed Clearml On Kubernets Using The Helm Chart, But I Had To Specify Three Different Domains For The Ui, Apiserver And Fileserver. Is There Any Way To Let Clearml Know That The Apiserver Is At

Hi folks, I have installed ClearML on kubernets using the helm chart, but I had to specify three different domains for the ui, apiserver and fileserver. Is t...

clearml

2 years ago

0 Votes

15 Answers

668 Views

0 Votes 15 Answers 668 Views

Hi Everybody, I Am Having An Issues With A Self-Hosted Clearml Server... I Am Having A Problem Enqueuing Experiments Whose Code Is In A Git Repository, They Are In A Pending State And Proceed... However If I Copy The Same Code Out In A Folder With No Rep

Hi everybody, I am having an issues with a self-hosted clearml server... I am having a problem enqueuing experiments whose code is in a git repository, they ...

clearml

one year ago

0 Votes

31 Answers

4K Views

0 Votes 31 Answers 4K Views

Hi Folks, I Am Trying To Run Clearml On A Local Kubernetes (Created With Rancher Desktop), But I Am Running Into Issues. I Admit I Am A Complete Noob To K8S, Helm, Clearml Etc. Etc. So I Am Not Sure If I Did Some Very Stupid Mistakes. I Have A K8S Cluste

Hi folks, I am trying to run ClearML on a local kubernetes (created with Rancher Desktop), but I am running into issues. I admit I am a complete noob to K8s,...

kubernetes

2 years ago

0 Votes

4 Answers

623 Views

0 Votes 4 Answers 623 Views

I Do Have One Questions About Using The Helm Chart, Is There Any Way To Specify The Users In The Values.Yaml?

I do have one questions about using the helm chart, is there any way to specify the users in the values.yaml?

clearml

2 years ago

0 Votes

31 Answers

4K Views

0 Votes 31 Answers 4K Views

Hi Folks, I Did A Deployment Of Clearml Using The K8S Helm Chart, And I Set The Agent Using K8S Glue. I Run A Task Locally, And I Went To The Ui Cloned The Experiment And Scheduled It In The Default Queue. After Doing This, I See That The Experiment Is Q

Hi folks, I did a deployment of ClearML using the K8s helm chart, and I set the agent using K8s Glue. I run a task locally, and I went to the UI cloned the e...

mlops

one year ago

0 Hi Folks, Occasionally When I Clone A Job And Enqueue It, Instead Of Being Processed By The Expected Queue, A New Queue (With Some Id That Looks Like An Hash) Is Created Instead, And The Experiment Hangs In A "Pending" State. When This Happens, If I Abor

Thanks, in DM I sent you the conf we use to deploy the agents.

one year ago

0 Hi Folks, I Am Trying To Run Clearml On A Local Kubernetes (Created With Rancher Desktop), But I Am Running Into Issues. I Admit I Am A Complete Noob To K8S, Helm, Clearml Etc. Etc. So I Am Not Sure If I Did Some Very Stupid Mistakes. I Have A K8S Cluste

thanks a lot! So as long as we have the storageclass in our kubernetes cluster configured correctly, the new helm chart should work out of the box?

2 years ago

0 Hi Folks, I Have A Question Related To The Storage Of Artifacts, As It Is Not Entirely Clear To Me Where To Configure It. If I Read The Documentation

do I need something else in the clearml.conf?

one year ago

0 Hi All! Question Around Resource Management Using

Thanks Martin!

one year ago

0 Hi Folks, I Have A Question Related To The Storage Of Artifacts, As It Is Not Entirely Clear To Me Where To Configure It. If I Read The Documentation

CostlyOstrich36 so I don't have to write the clearml.conf?

I would like to setup things so that a data scientist working on a project doesn't have to know about buckets and this sort of things... Ideally the server and the agents are configured with a default bucket...

one year ago

If now I abort the experiment (which is in a pending state and not running), and re-enqueue it again -- no parameters modifications this time...
and I re-enqueue it to the CPU queue, I see that it is sent to the right queue, and after a few seconds the job enters a running state and it completes correctly

one year ago

0 Hi Folks, I Did A Deployment Of Clearml Using The K8S Helm Chart, And I Set The Agent Using K8S Glue. I Run A Task Locally, And I Went To The Ui Cloned The Experiment And Scheduled It In The Default Queue. After Doing This, I See That The Experiment Is Q

After trying Gaspard changes to the helm chart values, I do now see that also a pod for the agentservice is deployed,
And some of the logs point to a misconfigurations on my side (the fact it can't access resources externally),
some others I don't understand:
Err:1 ` bionic InRelease
Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to archive.ubuntu.com...

one year ago

In the ClearML ui it stays in a Pending state

one year ago

I can see the outputs from argo, so I know if some resource has been created but I can't inspect the full logs,
the ones I have available are all records similar to
No tasks in queue 80247f703053470fa60718b4dff7a576

one year ago

Exactly that :) if I go in the queue tab, I see a new queue name (that I didn't create),
with a name like "4gh637aqetc"

one year ago

I have tried this several time and the behaviour is always the same. It looks like when I modify some hyperparameter, when I enqueue the experiment to one queue, things don't work if I didn't make sure to have previously set the value of k8s-queue to the name of the queue that I want to use. If I don't modify the configuration (e.g. I abort, or reset the job and enqueue it again, or clone and enqueue it without modifying the hyperparameters) then everything works as expected.

one year ago

0 Hi Everybody, I Am Having An Issues With A Self-Hosted Clearml Server... I Am Having A Problem Enqueuing Experiments Whose Code Is In A Git Repository, They Are In A Pending State And Proceed... However If I Copy The Same Code Out In A Folder With No Rep

Thanks Jake!

one year ago

2 years ago

Thanks, adding environment variables to the agentservice solved it, but for the agentgroup agent, I can't see any obvious way to inject environment variables. In the helm chart template I don't see any way to pass custom environment variables to the pod

one year ago

but I was a bit set off track seeing errors in the logs

one year ago

Hi Jake thanks for your answer!

So I just have a very simple file "project.py" with this content:

` from clearml import Task

task = Task.init(project_name='project-no-git', task_name='experiment-1')

import pandas as pd

print("OK") If I run python project.py ` from a folder that is not in a git repository, I can clone the task and enqueue it from the UI, and ti runs in the agent with no problems.
If I copy the same file, in a folder that is in a git repository, when I enqueue the ex...

one year ago

0 Hi Folks, Any Of You Has Experience In Deploying Clearml To Kubernetes Using Argocd? I Managed To Make It Run Pointing It To The Clearml-Charts-Repo, It Recognizes The Helm Chart And It Works. But I Am Struggling A Bit To Write My Own Definition To Make I

Thanks SuccessfulKoala55 . Any idea why going to the address https://allegroai.github.io/clearml-helm-charts
returns a 404 error?

Other repositories that are used in Argo CD examples (e.g. https://bitnami-labs.github.io/sealed-secrets , which is also hosted on Github) instead of returning a 404, the index.yaml page is loaded instead.

I suspect this might be the reason why I can't make it work with ClearML.

2 years ago

I see in bitnami's gh-pages branch a file https://github.com/bitnami-labs/sealed-secrets/blob/gh-pages/index.html to do the redirect that contains:

` <html>

<head> <meta http-equiv="refresh" content="0; url= ` ` "> </head> <p><a href=" ` ` ">Redirect to repo index.yaml</a></p> </html> ` A similar file is missing in the ` clearml-helm-chart ` ` gh-pages ` branch.

2 years ago

0 Hi All, If Install Clearml Agent With Helm, How Do I Authenticate The Agent With Git? In Values.Yaml File, I Can Set Clearml_Agent_Git_User And Clearml_Agent_Git_Pass But When The Agent Run, Git Has Error "Remote: Support For Password Authentication Was R

PunyWoodpecker71 just create a Personal Access Token and use it as the value for CLEARML_AGENT_GIT_PASS, https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token

one year ago

0 Hi Folks, One Question: I Have A Script That Looks Like:

actually there are some network issues right now, I'll share the output as soon as I manage to run it

one year ago

0 Hi Folks, A Question Regarding The Clearml-Agent With K8S Glue. In The Agents We Mount An Nfs Volume So That Some Artifacts And Data Would Be Available For Training. I Have Seen That The K8S Glue Runs As Root (I Guess To Be Able To Spawn New Pods?), But

I see... because the problem it would be with permissions when creating artifacts to store in the "/shared" folder

one year ago

0 Hi Folks, I Just Deployed A Clearml Agent Using The Helm Chart. I Have A Few Doubts:

thanks for the help!

one year ago

0 Hi Folks, I Have Installed Clearml On Kubernets Using The Helm Chart, But I Had To Specify Three Different Domains For The Ui, Apiserver And Fileserver. Is There Any Way To Let Clearml Know That The Apiserver Is At

Hi Jake unfortunately I realized we put a loadbalancer, so any address like addess.domain, would ping

2 years ago

0 Hi Folks I Have A Problem I Can'T Understand. Plots Are Not Shown When Experiments Are Executed From The Ui. For Example, If I Run The Code On My Laptop, And I Go To The Experiment Page I Can See Correctly The Plots: But If I Then Clone The Task, And Ex

This is the list of all the environment variables (starting with CLEARML) available in the Pod spawned by the K8s Glue Agent:
` CLEARML_MONGODB_PORT_27017_TCP_PORT
CLEARML_FILESERVER_PORT_8081_TCP_ADDR
CLEARML_ELASTIC_MASTER_PORT_9200_TCP
CLEARML_APISERVER_PORT_8008_TCP_PROTO
CLEARML_FILESERVER_PORT_8081_TCP_PORT
CLEARML_ELASTIC_MASTER_SERVICE_PORT_TRANSPORT
CLEARML_WEBSERVER_PORT_80_TCP
CLEARML_ELASTIC_MASTER_SERVICE_PORT
CLEARML_MONGODB_PORT_27017_TCP_ADDR
CLEARML_FILESERVER_PORT_8081_TCP_P...

one year ago

OK. In the pod spawned by the K8s Glue Agent, clearml.conf is the same as the K8S Glue Agent

one year ago

0 Hi Folks, I Have A Question Related To The Storage Of Artifacts, As It Is Not Entirely Clear To Me Where To Configure It. If I Read The Documentation

when I run it on my laptop...
what I am trying to achieve is not having to worry about this setting, and have all the artifacts and models uploaded to the file server automatically

one year ago

0 Hi Folks, I Am Having An Issue I Can'T Properly Understand: I Have Tried To Run The "Dataset" Example From The Official Clearml Repository (From My Laptop) For Some Reason It Got Stuck, So I Killed The Process, But In Clearml Ui It Still Results As "Runn

ah I see.. yes it makes sense 🙂

one year ago

0 I Do Have One Questions About Using The Helm Chart, Is There Any Way To Specify The Users In The Values.Yaml?

and one more question, in the values, I also see the values for the default tokens:

` credentials:
apiserver:
# -- Set for apiserver_key field
accessKey: "5442F3443MJMORWZA3ZH"
# -- Set for apiserver_secret field
secretKey: "BxapIRo9ZINi8x25CRxz8Wdmr2pQjzuWVB4PNASZqCtTyWgWVQ"
tests:
# -- Set for tests_user_key field
accessKey: "ENP39EQM4SLACGD5FXB7"
# -- Set for tests_user_secret field
secretKey: "lPcm0imbcBZ8mwgO7tpadutiS3gnJD05x9j7a...

2 years ago

0 Hi Folks, I Just Deployed A Clearml Agent Using The Helm Chart. I Have A Few Doubts:

Effectively kubectl commands don't work from within the agent pod, I'll try to figure out why

one year ago

0 Hi Folks, One Question: I Have A Script That Looks Like:

OK, so... when executed locally "train" prints:
` train:
SepalLength SepalWidth PetalLength PetalWidth Species
122 7.7 2.8 6.7 2.0 2.0
86 6.7 3.1 4.7 1.5 1.0
59 5.2 2.7 3.9 1.4 1.0
4 5.0 3.6 1.4 0.2 0.0
77 6.7 3.0 5.0 1.7 1.0
.. ... ... ... ... ......

one year ago

Show more results