CloudySwallow27

10 Questions, 35 Answers

Active since 10 January 2023

Last activity 2 months ago

Reputation

Badges 1

31 × Eureka!

Questions 10
Answers 35

0 Votes

15 Answers

2K Views

0 Votes 15 Answers 2K Views

We Have A Environment Variables Definitions.Py File Which Every User Configures On Their Local Machine. This File Includes Local Paths As Well As Aws/Api Credentials. This Is An Issue When Spinning Up Clearml Tasks Since It Is Not Included In The Git Repo

We have a environment variables definitions.py file which every user configures on their local machine. This file includes local paths as well as AWS/API cre...

clearml

3 years ago

0 Votes

4 Answers

2K Views

0 Votes 4 Answers 2K Views

Hi, I Am Successfully Starting Multiple Tasks Automatically, But They Dont Train To Completion They Start Training And Then At Some Point They Give Me This Error:

Hi, I am successfully starting multiple tasks automatically, but they dont train to completion they start training and then at some point they give me this e...

clearml

3 years ago

0 Votes

14 Answers

2K Views

0 Votes 14 Answers 2K Views

Hi, Trying To Spin Up A Clearml Agent And Gettting This Error:

Hi, trying to spin up a clearml agent and gettting this error: ERROR: Could not find a version that satisfies the requirement pywin32==303 (from -r /tmp/cach...

mlops

3 years ago

0 Votes

2 Answers

2K Views

0 Votes 2 Answers 2K Views

Hello, I Am Using The Autoscaler To Start Jobs. Previously, Everything Was Working. However, Now I Get This Error:

Hello, I am using the autoscaler to start jobs. Previously, everything was working. However, now I get this error: Using cached repository in "/home/ubuntu/....

mlops

3 years ago

0 Votes

7 Answers

2K Views

0 Votes 7 Answers 2K Views

Hi, I'M Using The Autoscaler And Getting The Error

Hi, I'm using the autoscaler and getting the error Process terminated by user even though I did not terminate anything. This error occurs randomly during tra...

mlops

3 years ago

0 Votes

5 Answers

2K Views

0 Votes 5 Answers 2K Views

Hi, When I Use The Autoscaler To Start Jobs, I Noticed Some Of Them Randomly Abort In The Middle Of The Jobs And Give The Following Error:

Hi, when I use the autoscaler to start jobs, I noticed some of them randomly abort in the middle of the jobs and give the following error: Process failed, ex...

mlops

3 years ago

0 Votes

8 Answers

513 Views

0 Votes 8 Answers 513 Views

Hey Everyone! I'Ve Been Using Clearml For Few Years Now And Love It. Recently Started Getting This Error When I Try To Enqueue A Task:

Hey everyone! I've been using ClearML for few years now and love it. Recently started getting this error when I try to enqueue a task: Action failed <404/0: ...

clearml

2 months ago

0 Votes

6 Answers

2K Views

0 Votes 6 Answers 2K Views

Hi, I See That Debug Samples Are Taking Up A Huge Amount Of Space. I Want To Limit The Amount Of Debug Images Which Are Stored. I See There Is An Option For That Here:

Hi, I see that debug samples are taking up a huge amount of space. I want to limit the amount of debug images which are stored. I see there is an option for ...

clearml

3 years ago

0 Votes

2 Answers

2K Views

0 Votes 2 Answers 2K Views

Hi, I Am Following The Programmatic Orchestration Example Here:

Hi, I am following the programmatic orchestration example here: https://clear.ml/docs/latest/docs/guides/automation/task_piping . My question is, when I setu...

clearml

3 years ago

0 Votes

5 Answers

1K Views

0 Votes 5 Answers 1K Views

Hey! I Am Running A A Web-App On A Clearml Agent (From A Gcp Queue) On Its Localhost (

Hey! I am running a a web-app on a clearml agent (from a GCP queue) on its localhost ( None ). How can I view the app over the internet?

mlops

one year ago

0 Hey Everyone! I'Ve Been Using Clearml For Few Years Now And Love It. Recently Started Getting This Error When I Try To Enqueue A Task:

I get that error whether I select "Remove all related artifacts and debug samples from ClearML file server" or not

2 months ago

0 Hello, How Can Increase The Number Of Debug Samples To Be Recorded In One Training?

Thanks AgitatedDove14 . Does this go in the local clearml.conf file w/ each user's credentials, or in the conf file for the server?

3 years ago

0 Hi, I'M Using The Autoscaler And Getting The Error

this worked, thank you!

3 years ago

0 Hey Everyone! I'Ve Been Using Clearml For Few Years Now And Love It. Recently Started Getting This Error When I Try To Enqueue A Task:

It seems to an a memory issue w/ the VM that hosts clearml filling up. I am trying to delete some experiments but now i get:

2 months ago

0 Hi, I'M Using The Autoscaler And Getting The Error

TimelyPenguin76 not sure what you mean by "as a service or via the apps", but we are self-hosting it. Does that answer the question?

Also, not sure what you mean by which "clearml version". How do we check this? The clearml python package is 1.1.4. Is that what you wanted?

3 years ago

0 Hey Everyone! I'Ve Been Using Clearml For Few Years Now And Love It. Recently Started Getting This Error When I Try To Enqueue A Task:

@<1855782498290634752:profile|AppetizingFly3>

2 months ago

0 Hi, I Am Successfully Starting Multiple Tasks Automatically, But They Dont Train To Completion They Start Training And Then At Some Point They Give Me This Error:

Thanks! I installed CUDA/CuDNN on the image and now the GPU is being utilized.

3 years ago

0 Hey! I Am Running A A Web-App On A Clearml Agent (From A Gcp Queue) On Its Localhost (

yes

one year ago

0 Hi, Trying To Spin Up A Clearml Agent And Gettting This Error:

yes, on my windows machine I am running:
cloned_task = Task.clone(source_task=base_task, name="Auto generated cloned task") Task.enqueue(cloned_task.id, queue_name='test_queue')I see the task successfully start in the clearml server. In the installed packages section it includes pywin32 == 303 even though that is not in my requirements.txt.

In the results --> console section, I see the agent is running and trying to install all packages, but then stops at pywin32. Some lines from t...

3 years ago

0 Hi, Trying To Spin Up A Clearml Agent And Gettting This Error:

yall thought of everything. this fixed it! Having another issue now, but will post seperately

3 years ago

0 Hi, Trying To Spin Up A Clearml Agent And Gettting This Error:

yes. i think the problem is that its trying to recreate the environment the task was spun up on - which was on a windows machine - on a linux ec2 instance

3 years ago

0 Hi, Trying To Spin Up A Clearml Agent And Gettting This Error:

no 64 bit. but do you mean the PC where I am spinning the task up or the machine where I am running the task

3 years ago

0 Hi, Trying To Spin Up A Clearml Agent And Gettting This Error:

Windows 10 python 3.8 virtualenv

3 years ago

0 Hi, Trying To Spin Up A Clearml Agent And Gettting This Error:

the latter is an ec2 instance

3 years ago

0 Hi, Trying To Spin Up A Clearml Agent And Gettting This Error:

is there a way to explicitly make it some install certain packages, or at least stick to the requirements.txt file rather than the actual environment

3 years ago

0 Hi

gotit. reg is indeed free 🙂

3 years ago

0 Hi

Awesome! any way to hear the talk w/o/ registering for the whole conference?

3 years ago

0 Hey! I Am Running A A Web-App On A Clearml Agent (From A Gcp Queue) On Its Localhost (

is there a way to get the instance's external IP address from clearml? i wouldve thought it would be in the info tab, but its not

one year ago

0 Hi, I See That Debug Samples Are Taking Up A Huge Amount Of Space. I Want To Limit The Amount Of Debug Images Which Are Stored. I See There Is An Option For That Here:

huh. i really like how easy it is w/ the automated TB. Is there a way to still use the auto_connect but limit the amount of debug imgs?

3 years ago

0 Hi, I See That Debug Samples Are Taking Up A Huge Amount Of Space. I Want To Limit The Amount Of Debug Images Which Are Stored. I See There Is An Option For That Here:

perfect. exactly what i was looking for!

3 years ago

0 Hi, I See That Debug Samples Are Taking Up A Huge Amount Of Space. I Want To Limit The Amount Of Debug Images Which Are Stored. I See There Is An Option For That Here:

SourLion48

3 years ago

0 Hi, When I Use The Autoscaler To Start Jobs, I Noticed Some Of Them Randomly Abort In The Middle Of The Jobs And Give The Following Error:

gotit. the instance logs showed
/var/log/syslog.1:May 5 03:25:27 ip-172-31-37-234 kernel: [53387.840425] python invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 /var/log/syslog.1:May 5 03:25:27 ip-172-31-37-234 kernel: [53387.840442] oom_kill_process+0xe6/0x120
I assume this is something I have to fix on my end (or increase instance memory). Does ClearML also happen to have solutions for this?

3 years ago

0 Hi, I Am Successfully Starting Multiple Tasks Automatically, But They Dont Train To Completion They Start Training And Then At Some Point They Give Me This Error:

relately, I just noticed that the GPU is not starting. This was in the logs:
2022-04-07 20:59:54.464854: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
Do we need to call a specific instance w/ CUDA preinstalled or does clearml take care of it?

3 years ago

0 Hi, I Am Successfully Starting Multiple Tasks Automatically, But They Dont Train To Completion They Start Training And Then At Some Point They Give Me This Error:

This is what the instance state looks like, as logged by clearml:

3 years ago

0 Hi, I'M Using The Autoscaler And Getting The Error

We were able to find an error from the autoscalaer agent:

Stuck spun instance dynamic_worker:clearml-agent-autoscale:p2.xlarge:i-015001a93e0910a09 of type clearml-agent-autoscale

2022-04-19 19:16:58,339 - clearml.auto_scaler - INFO - Spinning down stuck worker: 'dynamic_worker:clearml-agent-autoscale:p2.xlarge:i-015001a93e0910a09

3 years ago

0 Hello, I Am Using The Autoscaler To Start Jobs. Previously, Everything Was Working. However, Now I Get This Error:

Im thinking it may have something to do with
Using cached repository in "/home/ubuntu/.clearml/vcs-cache/ai_dev.git.42a0e941ddbf5c69216f37ceac2eca6b/ai_dev.git"We tried to reset the machines but the cache is still there. any idea how to clear it?

3 years ago

0 We Have A Environment Variables Definitions.Py File Which Every User Configures On Their Local Machine. This File Includes Local Paths As Well As Aws/Api Credentials. This Is An Issue When Spinning Up Clearml Tasks Since It Is Not Included In The Git Repo

the 2nd option looks good. would everyone's credentials be displayed on the server though?

3 years ago

ok, i suppose that will have to do for now. thank you!

3 years ago

0 Hi, When I Use The Autoscaler To Start Jobs, I Noticed Some Of Them Randomly Abort In The Middle Of The Jobs And Give The Following Error:

We are using self-hosted clearMl w/ the following versions:

Worker CLEARML-AGENT version 1.1.2
The autoscaler instance Clearml-AGENT version: 1.2.3
ClearML WebApp: 1.2.0-153 Server: 1.2.0-153 API: 2.16
python pip package 1.3.2

3 years ago

0 Hey Everyone! I'Ve Been Using Clearml For Few Years Now And Love It. Recently Started Getting This Error When I Try To Enqueue A Task:

@<1856144902816010240:profile|SuccessfulCow78> can you please help provide

2 months ago

Show more results