RoughTiger69

28 Questions, 101 Answers

Active since 10 January 2023

Last activity one year ago

Reputation

Badges 1

90 × Eureka!

Questions 28
Answers 101

0 Votes

0 Answers

2K Views

0 Votes 0 Answers 2K Views

Hi, I Am Catching Up With

Hi, I am catching up with http://clear.ml for stuff beyond exp. tracking, and have a few questions. Will ask them separately to allow threading:

clearml

4 years ago

0 Votes

3 Answers

2K Views

0 Votes 3 Answers 2K Views

Fyi I Am Getting A Lot Of Read Timeouts From The Community Server: Retrying (Retry(Total=235, Connect=240, Read=235, Redirect=240, Status=240)) After Connection Broken By ’Readtimeouterror(“Httpsconnectionpool(Host=‘

FYI I am getting a lot of read timeouts from the community server: Retrying (Retry(total=235, connect=240, read=235, redirect=240, status=240)) after connect...

clearml

3 years ago

0 Votes

5 Answers

2K Views

0 Votes 5 Answers 2K Views

Hi, I Am Running A Pipeline From Decorators. The Pipeline Runs Fine. Then I Try To Clone It By Clicking The (Successful) Run And Launching. The Pipeline Fails Immediately With The Error

hi, I am running a pipeline from decorators. the pipeline runs fine. Then I try to clone it by clicking the (successful) run and launching. The pipeline fail...

clearml

3 years ago

0 Votes

6 Answers

2K Views

0 Votes 6 Answers 2K Views

Autoscaler From Saas (Pro Version). I Attempted To Use The Autoscaler “Application” From Clearml Ui. Here Is What I Get In The Logs Of The Autoscaler Screen Itself (Consistent):

autoscaler from saas (pro version). I attempted to use the autoscaler “application” from clearml UI. here is what I get in the logs of the autoscaler screen ...

mlops

3 years ago

0 Votes

10 Answers

2K Views

0 Votes 10 Answers 2K Views

Hi Folks, Is There A Way To Force Clear-Ml Agent With --Docker To

hi folks, is there a way to force clear-ml agent with --docker to not create a virtualenv at all? And perhaps not even attempt to install requirements even? ...

mlops

3 years ago

0 Votes

14 Answers

2K Views

0 Votes 14 Answers 2K Views

Question About Pipeline And Long-Waiting Tasks: Say I Want To Generate A Dataset. The Workflow I Have Requires

question about pipeline and long-waiting tasks: Say I want to generate a dataset. The workflow I have requires query to a DB Creating a labeling assigment in...

clearml

3 years ago

0 Votes

0 Answers

2K Views

0 Votes 0 Answers 2K Views

Did More Digging, Seems That You Need To Start The Agent With Clearml_Agent_Skip_Pip_Venv_Install=1

Did more digging, seems that you need to start the agent with CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1

mlops

3 years ago

0 Votes

0 Answers

2K Views

0 Votes 0 Answers 2K Views

Autoscaler 101 Questions:

autoscaler 101 questions: What is the best practice for managing credentials so that they don’t get saved in clearml webapp? When the https://clear.ml/docs/l...

mlops

3 years ago

0 Votes

5 Answers

2K Views

0 Votes 5 Answers 2K Views

I Have A Logical Task That I Want To Split To Multiple Workers. The Task Involves Processing Media Files (Not Training). The Optimal Design For Me Would Be:

I have a logical task that I want to split to multiple workers. The task involves processing media files (not training). The optimal design for me would be: ...

clearml

3 years ago

0 Votes

6 Answers

2K Views

0 Votes 6 Answers 2K Views

Regarding The “Classic” Datasets (Not Hyper Datasets): Is There An Option To Do Something Equivalent To Dvc’S “

Regarding the “classic” datasets (not hyper datasets): Is there an option to do something equivalent to dvc’s “ https://dvc.org/doc/user-guide/managing-exter...

clearml

3 years ago

0 Votes

11 Answers

2K Views

0 Votes 11 Answers 2K Views

I Have A Local Folder A, And A Dataset B. A:

I have a local folder a, and a dataset B. a: a a/.DS_Store a/1.txt a/b a/b/.DS_Store a/b/1.txt a/b/c a/b/c/1.txtDataset B: b b/2.txt b/c b/c/2.txtI want to “...

clearml

3 years ago

0 Votes

3 Answers

2K Views

0 Votes 3 Answers 2K Views

Is There A Way To Tell The Agent To Run In Docker In A Way That Just “Trusts” The Installation Completely? For Example, I Have A Local Environment With Python 3.8.10 And I Am Trying To Run A Task In A Docker With Python 3.8.13 I’Ve Pointed The Notorious

Is there a way to tell the agent to run in docker in a way that just “trusts” the installation completely? for example, I have a local environment with pytho...

mlops

3 years ago

0 Votes

8 Answers

2K Views

0 Votes 8 Answers 2K Views

Hi, I Am Using Pipelinedecorator To Create Tasks. Is There A Way To Force It To Use The Entire Git Repo It Is Created From On The Pythonpath? Vs. Just The Decorated Function And Perhaps The Helper_Function=[Some_Function]?

Hi, I am using PipelineDecorator to create tasks. is there a way to force it to use the entire git repo it is created from on the pythonpath? vs. just the de...

clearml

3 years ago

0 Votes

18 Answers

2K Views

0 Votes 18 Answers 2K Views

2. Is There A Case-Study Or Ref. Architecture For Interacting With Ci/Cd I.E. Exposing Mature Pipelines To Be Triggered Upon Code Pushes (Taking Latest Git Hash) Or With Manual Ci Triggers?

Is there a case-study or ref. architecture for interacting with CI/CD i.e. exposing mature pipelines to be triggered upon code pushes (taking latest git hash...

clearml

4 years ago

0 Votes

5 Answers

2K Views

0 Votes 5 Answers 2K Views

Hi, I Am Trying To Use The Aws Autoscaler To Assign Instance Profiles To New Machines. This Is A Better Way Than Managing Credentials. I Added The Configuration To The Autoscaler Config Like So:

Hi, I am trying to use the aws autoscaler to assign instance profiles to new machines. This is a better way than managing credentials. I added the configurat...

aws mlops

3 years ago

0 Votes

8 Answers

3K Views

0 Votes 8 Answers 3K Views

Cannot Upload A Dataset With A Parent - Seems Very Odd! Clearml Versions I Tried: 1.6.1, 1.6.2 Scenario: * Create Parent Dataset (With Storage On S3) * Upload Data * Close Dataset * Create Child Dataset (Tried With Storage On Both S3 Or On Clearml Serv

Cannot upload a dataset with a parent - seems very odd! clearml versions I tried: 1.6.1, 1.6.2 scenario: * Create parent dataset (with storage on S3) * Uploa...

dataset

3 years ago

0 Votes

4 Answers

2K Views

0 Votes 4 Answers 2K Views

Hi, When Running A Training Script From Pycharm, It Seems That Clearml Logs Only Those Packages That Are Explicitly Imported By My .Py Files; It Seems To Not Take The Pacakges That Are In The Requirements.Txt My Training Uses Keras

hi, When running a training script from pycharm, it seems that clearml logs only those packages that are explicitly imported by my .py files; it seems to not...

mlops

4 years ago

0 Votes

7 Answers

2K Views

0 Votes 7 Answers 2K Views

I Have A Pipeline With Tasks A->B->C. I Want To Be Able To Trigger It Manually, And Skip A Regardless Of It’S Cache Status. I Want To Pass B Value That Represents A’S Output If Needed. What’S A Good Way To Achieve This (Can Be Ui-Based, Or Pipeline-Gymnas

I have a pipeline with tasks A->B->C. I want to be able to trigger it manually, and skip A regardless of it’s cache status. I want to pass B value that repre...

clearml

3 years ago

0 Votes

10 Answers

2K Views

0 Votes 10 Answers 2K Views

Hi, I Have A Task Which Uses Hydra For Configuration. I Want To Add This Taks To A Pipeline, And Pass The Full Hydra Config Objects To The Task. Is There A Way To Do It? I Get “Parameters Should Be In The Form Of “`Section-Name`/Parameter”, Example: “Args

hi, I have a task which uses hydra for configuration. I want to add this taks to a pipeline, and pass the full hydra config objects to the task. is there a w...

clearml

4 years ago

0 Votes

9 Answers

2K Views

0 Votes 9 Answers 2K Views

I Am Trying To Run A Task That Is Completely Detached From Git - Remotely. The Script Uploads Fine But In The Ui, The Git Repo Appears As “Origin”. When The Agent Tries To Pick This Up, It Fails On Trying To Clone “Origin”. What Can I Do To Let The Agent

I am trying to run a task that is completely detached from git - remotely. The script uploads fine but in the UI, the git repo appears as “origin”. When the ...

mlops

3 years ago

0 Votes

5 Answers

2K Views

0 Votes 5 Answers 2K Views

Avoiding

Avoiding http://Clear.ml glue code spaghetti - community best practices? Say I have training pipeline : Task 1 - data preprocessing -> create a dataset artif...

clearml

4 years ago

0 Votes

3 Answers

2K Views

0 Votes 3 Answers 2K Views

I Have A Situation Where I’D Like To “Promote” The Pipeline (And Dataset) By Creating It In A Completely Separate Instance Of Clearml Server Which Is Used For Production Retraining (Vs. The Dev. Clearml Server That Is Used For Experiments) A) Is This Some

I have a situation where I’d like to “promote” the pipeline (and dataset) by creating it in a completely separate instance of clearml server which is used fo...

clearml

4 years ago

0 Votes

10 Answers

2K Views

0 Votes 10 Answers 2K Views

Hello Folks! I Have A Pipeline With Three Tasks: A, B, And C I Want To Set It Up So That: A Gets Assigned A Machine (E.G. Based On The Queue) B Always Gets Assigned To The Same Machine As A (But May Run In A Different Docker Etc.) C Will Be Submitted To

Hello folks! I have a pipeline with three tasks: A, B, and C I want to set it up so that: A gets assigned a machine (e.g. based on the queue) B always gets a...

clearml

3 years ago

0 Votes

12 Answers

1K Views

0 Votes 12 Answers 1K Views

Is there a reference implmentation for a task in a pipeline that awaits user input?

clearml

4 years ago

0 Votes

9 Answers

2K Views

0 Votes 9 Answers 2K Views

I Have Code That Does Torch.Load(Path) And Deserializes A Model. I Am Performing This In Package A.B.C, And The Model’S Module Is Available In In A.B.C.Model Unfortunately, The Model Was Serialized With A Different Module Structure - It Was Originally Pla

I have code that does torch.load(path) and deserializes a model. I am performing this in package a.b.c, and the model’s module is available in in a.b.c.model...

clearml

3 years ago

0 Votes

3 Answers

2K Views

0 Votes 3 Answers 2K Views

What’S A Good Ami To Use For The Clearml Autoscaler On Aws? The Defaults Offered Confidently By The Various Auto Scaler Installers Don’T Seem To Exist…| E.G.

what’s a good ami to use for the clearml autoscaler on AWS? the defaults offered confidently by the various auto scaler installers don’t seem to exist…| e.g....

mlops

3 years ago

0 Votes

14 Answers

2K Views

0 Votes 14 Answers 2K Views

Two Simple Lineage Related Questions:

Two simple lineage related questions: Task B is a clone of Taks A. Does B store the information that it was cloned from A somewhere? Training task X loads Da...

clearml

4 years ago

0 Votes

4 Answers

2K Views

0 Votes 4 Answers 2K Views

Hi, I Created A Dataset With 20K Files, Total Of 20Gb, With Storage Pointing To S3. When I Upload (Or Close) The Dataset, During The Compression Phase, The Clear-Ml Cli Is Killed Due To Oom.

hi, I created a dataset with 20K files, total of 20GB, with storage pointing to S3. When I upload (or close) the dataset, during the compression phase, the c...

clearml

3 years ago

0 I Have A Local Folder A, And A Dataset B. A:

AgitatedDove14 ideas?

3 years ago

0 I Have Code That Does Torch.Load(Path) And Deserializes A Model. I Am Performing This In Package A.B.C, And The Model’S Module Is Available In In A.B.C.Model Unfortunately, The Model Was Serialized With A Different Module Structure - It Was Originally Pla

python 3.8
I’ve worked around the issue by doing:
sys.modules['model'] = local_model_package

3 years ago

0 Is It Possible To Use In Clearml

As far I know storage can be https://clear.ml/docs/latest/docs/integrations/storage/#direct-access .
typical EBS is limited to being mounted to 1 machine at a time.
so in this sense, it won’t be too easy to create a solution where multiple machines consume datasets from this storage type

PS https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volumes-multi.html is possible under some limitations

3 years ago

I think it has something to do with clearml since I can run this code as pure python without clearml, and when I activate clearml, I see that torch.load() hits the

import_bind

.

__patched_import3

when trying to deserialize the saved model

3 years ago

0 Regarding The “Classic” Datasets (Not Hyper Datasets): Is There An Option To Do Something Equivalent To Dvc’S “

AgitatedDove14 I haven’t done a full design for this 😉
Just referring to how DVC claims it can detect and invalidate changes in large remote files.
So I take it there is no such feature in http://clear.ml 🙂

3 years ago

0 I Am Trying To Run A Task That Is Completely Detached From Git - Remotely. The Script Uploads Fine But In The Ui, The Git Repo Appears As “Origin”. When The Agent Tries To Pick This Up, It Fails On Trying To Clone “Origin”. What Can I Do To Let The Agent

CostlyOstrich36 If I delete the origin and all other info and set it to tag_name=‘xxx’ then it is able to work

3 years ago

CostlyOstrich36 yes

3 years ago

0 Hi, I Am Trying To Setup Multi-Node Training With Pytorch Distributeddataparallel. Ddp Requres A Launch Script With A Set Of Parameters To Be Run On Each Node. One Of These Parameters Is Master Node Address. I Am Currently Using The Following Scheme:

AgitatedDove14 let me reach out to my pocket there 😉

3 years ago

0 Regarding The “Classic” Datasets (Not Hyper Datasets): Is There An Option To Do Something Equivalent To Dvc’S “

AgitatedDove14 nope… you can run md5 on the file as stored in the remote storage (nfs or s3)

3 years ago

AgitatedDove14 looks like service-writing-time for me!
PS can you point me to some official example/ doc for how to persist/restore state so that tasks are restartable?

3 years ago

0 Hi Everyone, We Train Our Ml Models Using The Aws Autoscaler On G4Dn Instances. We Currently Have A 24 Vcpu Limit For G Type Instances In Eu-West. I'M Trying To Get This Limit At Least Doubled Or Tripled. My Request Keeps Stagnating With The Service Team

AgitatedDove14 can you share if there is a plan to put the gcp autoscaler in the open source?

3 years ago

Show more results