CostlyOstrich36

0 Questions, 4213 Answers

Active since 10 January 2023

Last activity 2 years ago

Reputation

Answers 4213

0 There Is Something Weird Going On With Console Log After Latest Updates Of Clearml Server. It Doesn'T Show The Latest Updates, Instead It Often Jumps To The Seemingly Random Parts Of The Console Output

Hi DilapidatedDucks58 , what is your server version?

2 years ago

0 Hi All! I Am In The Process Of Setting Up Clearml-Serving On My Kubernetes Cluster Using The Provided Helm Charts. Currently I Am Stuck With Running The Control Task. When I Call

Hi @<1526371965655322624:profile|NuttyCamel41> , can you add the full log?

one year ago

0 Hi Team,

@<1533257278776414208:profile|SuperiorCockroach75> , excuse my ignorance, but doesn't it depend on the output model i.e. the training run that created it?

2 years ago

0 Cannot Import My Custom Python File When Creating A Pipeline Using The Decorator Do Anyone Have Any Idea As To How To Solve This Issue.

What do you mean? How are you running the pipeline - locally or remotely?

2 years ago

0 Hi. I Use Set_Base_Docker For Remote Execute. Its My Custom Image From Gitlab Registry. And My Question Is How I Can Use Script And Etc From My Image? On My Local Machine I’Ve Got In Docker Container Folder With Code From That Image If I Made It Like From

Hi @<1523701457835003904:profile|AbruptHedgehog21> , I'm not sure I understand - How do you use set_base_docker and what do you expect to happen?

2 years ago

0 Hello All, I Am New To Clearml, Need A Clarification. I Would Like To Enable Gpu-As-A-Service With Secure Multi-Tenancy And Real-Time Billing Per Tenant, Is This Feature Available In Open Source? Or Do We Need To Purchase The Enterprise Access For Multi-T

Hi @<1813745484821434368:profile|SuccessfulPigeon84> , these are Enterprise only features as far as I'm aware. I would suggest contacting ClearML's sales 🙂

8 months ago

0 I Have A Dataset Uploaded To Clearml. By Mistake I Didn'T Set The Destination Correctly And The Dataset Is Now Saved On A Computer Rather Than In S3. Is There An Easy Way To Rectify This?

Setting the upload destination correctly and doing the same steps again

one year ago

0 Hey Is There A Way For One To Extend Clearml Somehow? We Have Some Custom Evaluations We Want To Do And Our Ideal Scenario Would Be To Do Them Within Clearml Itself.

Hi @<1535069219354316800:profile|PerplexedRaccoon19> , not sure I understand what you mean, can you please elaborate on what you mean by doing the evaluations within ClearML?

one year ago

0 Hi, I'Ve Run Into A Problem And Would Appreciate Some Help. I Installed Clearml Locally. When I Run A New Task On A Remote Server And In The Python Training Code I Set It To Only Train On One Gpu. Everything Works Fine And I See All The Scalars Automatica

What versions of clearml-agent & clearml are you using? Is it a self hosted server?

11 months ago

0 Hi! For

How about this by the way?
https://clear.ml/docs/latest/docs/references/sdk/model_outputmodel#outputmodelset_default_upload_uri

3 years ago

0 Does Anyone Have A Good Idea On How To Set The Correct Python Version For A Task? Apparently I Can Set It On A Task Level Or Via The Web-Interface.

Hi @<1806135344731525120:profile|GrumpyDog7> , it shows the reason in the log:

Python executable with version '3.9' requested by the Task, not found in path, using '/usr/bin/python3' (v3.12.3) instead

You either need a container with the relevant python version available or have it installed using the bash script section.

Makes sense?

8 months ago

0 Hey All. Question Regarding Scheduling And Orchestration. Does Clearml Provide Any Tooling To Schedule Entire Training Pipelines And To Trigger Training Pipelines In Response To Events, E.G. Degraded Model Performance Alerting?

TenseOstrich47 , you could create a monitor task that reads model performance from your database and reports them as some scalar. According to that scalar you can create triggers 🙂

What do you think?

external trigger

What do you mean? Do you have a reference?

3 years ago

0 Is There A Way To Configure A Clearml-Agent So That It Shutdown The Server After It Has Been Idle For A Certain Time Period? We Are Using Gpu Resources From A Provider That Autoscaling Doesn'T Support (Such As Sagemaker Training Jobs).

Any specific reason not to use the autoscaler? I would imagine it would be even more cost effective

2 years ago

0 Helo, I Wonder If Someone Have An Idea How To Handle The "Error", Which I Mentioned In

Hi @<1856144871656525824:profile|SparklingFly7> , can you describe the issue you're experiencing? I saw there is a new response in github - None

4 months ago

0 Hi, I Am Trying To Use Agent, But I Have A Problem. Execution Of Task Stucks Like This

btw what os are you on?

3 years ago

0 There Is A Problem Starting From Clearml 1.7.0 With Python-Fire

try with pip install -U clearml==1.7.2rc1

3 years ago

Hi @<1632913939241111552:profile|HighRaccoon77> , the most 'basic' solution would be adding a piece of code at the end of your script to shut down the machine but obviously it would be unpleasant to run locally without Task.execute_remotely() - None

Are you specifically using Sagemaker? Do you have any api interface you could work with to manipulate shutdown of machines?

2 years ago

0 Hi All, How Can I Have A Global Variable Used In A Pipeline Step? I Have To Define Them In Each Pipeline Step, Otherwise They Are Not Included In The Pipeline Step

Hi @<1523701066867150848:profile|JitteryCoyote63> , you mean a global "env" variable that can be passed along the pipeline?

one year ago

0 Hi, I'M Using A Self Hosted Version Of Clearml. I Am Querying Experiments From Ui And I Am Getting Such Error:

Can you check the machine status? Is the storage running low?

2 years ago

0 Hi, I Have A Pipeline Designed Whenever I Run That All The Steps Will Be Present In All Experiment But I Want That To Be In The Project That I Have Specified Can Anyone Help Me With This

Yes. Run all the pipelines examples and see how the parameters are added via code to the controller.

For example:
None

one year ago

0 Hi, I Am Trying To Use Agent, But I Have A Problem. Execution Of Task Stucks Like This

I think something might be block ports on your local machine. Did you change ports mapping for the ClearML dockers?

3 years ago

0 Tasks Can Be Put In Draft State - If We Will Execute:

How do you run your pipeline currently?

2 years ago

0 Hi. I'D Like To Try The Gcp Autoscaler.

` Status: Downloaded newer image for nvidia/cuda:10.2-runtime-ubuntu18.04

1657737108941 dynamic_aws:cpu_services:n1-standard-1:4834718519308496943 DEBUG docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].
time="2022-07-13T18:31:45Z" level=error msg="error waiting for container: context canceled" `As can be seen here 🙂

3 years ago

0 Another Question About Pipelines: Is It Possible To Run A Component On The Same Machine Of The Main Pipeline Controller? I Have A Function That It'S Rather Fast To Execute, So I Don'T Want To Start A Separate Container Just For It, But I'D Like To Track I

Check the pre_execute_callback and post_execute_callback arguments of the component.

None

2 years ago

0 I'M On The Machine With Clearml Server Hosted. Is There Any Way To See Datasets Uploaded To Clearml Data Without Downloading Them Using Clearml Data?

Do you mean see the datasets in the UI?

3 years ago

0 Hello, I Have An Issue With Task.Close(). On A Mac It Works As Expected, But On My (Gpu Enabled) Linux Machine The Code Execution Freezes When Calling Task.Close(). Do You Have Any Idea What The Issue Could Be Or How I Cloud Debug It? The Log Does Not Sho

Hi @<1695969549783928832:profile|ObedientTurkey46> , is this happening when running on top of the agent or locally?

one year ago

0 Hi All. I'Ve Been Mistakenly Using

Hit F12 when on the browser 🙂

3 years ago

0 Hi! Any Idea Why Clearml Fails To Detect Iteration Reporting?

GrievingTurkey78 , I'm not sure. Let me check.
Do you have cpu/gpu tracking through both pytorch lightning AND ClearML reported in your task?

4 years ago

0 Hey Community!, I Am Trying To Run A Pipeline Using Pipeline Decorator In Two Steps , Now In My First Step I Am Trying To Import A Custom Python File Which Is Already Pushed To My Repository Online, I Am Executing The Pipeline Remotely But Still I Am Gett

Hi @<1533619725983027200:profile|BattyHedgehong22> , does the package appear in the installed packages section of the experiment?

2 years ago

0 In Pipelines. Is It Possible To Inject A Requierment.Txt Such That The Executing Node Will Install Before Running Tasks?

I'm sorry. I think I wrote something wrong. I'll elaborate:
The SDK detects all the packages that are used during the run - The Agent will install a venv with those packages.
I think there is also an option to specify a requirements file directly in the agent.

Is there a reason you want to install packages from a requirements file instead of just using the automatic detection + agent?

3 years ago

Show more results