ClearML FAQ | BoredHedgehog47

BoredHedgehog47

27 Questions, 213 Answers

Active since 10 January 2023

Last activity one year ago

Reputation

0

Badges 1

212 × Eureka!

Questions 27
Answers 213

0 Votes

10 Answers

2K Views

0 Votes 10 Answers 2K Views

Is There Any Examples Of Mounting An Aws Efs Mount To A Self Hosted K8 Agent Deploy?

Is there any examples of mounting an AWS EFS mount to a self hosted k8 agent deploy? https://github.com/allegroai/clearml-helm-charts/blob/main/charts/clearm...

3 years ago

0 Votes

0 Answers

2K Views

0 Votes 0 Answers 2K Views

Hey Everyone, I'M Trying To Add A Test User To The Api Server Config. Here Is A Snippet Form My Values.Yaml File. Do I Have This Formatted Correctly? I'M Not Seeing Api Config Map In The K8 Cluster

Hey everyone, I'm trying to add a test user to the api server config. Here is a snippet form my values.yaml file. Do I have this formatted correctly? I'm not...

3 years ago

0 Votes

14 Answers

2K Views

0 Votes 14 Answers 2K Views

Does Clearml Have The Ability To Run A Single Experiment Across Multiple Nodes/Gpus In A K8 Cluster?

Does ClearML have the ability to run a single experiment across multiple nodes/GPUs in a k8 cluster?

3 years ago

0 Votes

15 Answers

2K Views

0 Votes 15 Answers 2K Views

Hey All, I'M Testing The Usage Of

Hey all, I'm testing the usage of SETUP SHELL SCRIPT in the experiment window. I added a simple command but did not see it in the console. The task did execu...

3 years ago

0 Votes

30 Answers

3K Views

0 Votes 30 Answers 3K Views

When I Run An Experiment (Self Hosted), I Only See Scalars For Gpu And System Performance. How Do I See Additional Scalars? I Have

When I run an experiment (self hosted), I only see scalars for GPU and System performance. How do I see additional scalars? I have "tensorboard": { "enabled"...

3 years ago

0 Votes

17 Answers

2K Views

0 Votes 17 Answers 2K Views

Or Is It Just The Ubuntu Official Image

or is it just the ubuntu official image https://github.com/allegroai/clearml-helm-charts/blob/main/charts/clearml-agent/values.yaml#L59

3 years ago

0 Votes

7 Answers

2K Views

0 Votes 7 Answers 2K Views

Was There Any Changes To Clearml Python Sdk In The Past 24 Hours?

Was there any changes to clearML python SDK in the past 24 hours?

3 years ago

0 Votes

7 Answers

2K Views

0 Votes 7 Answers 2K Views

Is There Any Additional Configuration Needed For

Is there any additional configuration needed for PYTHONPATH to be setup properly in the clearml agent? I'm getting python import errors from the root directo...

3 years ago

0 Votes

3 Answers

2K Views

0 Votes 3 Answers 2K Views

In My Git Repo, I Have A

In my git repo, I have a setup.py , how would I run pip install -e . rather than using --packages or --requirements

3 years ago

0 Votes

8 Answers

2K Views

0 Votes 8 Answers 2K Views

When I Run

When I run clearml-data close on an 84mb file, I get the following response 413 Request Entity Too Large 413 Request Entity Too Large nginxYet the file is st...

3 years ago

0 Votes

1 Answers

2K Views

0 Votes 1 Answers 2K Views

Also Is This The Image That Is Used For Experiments?

Also is this the image that is used for experiments? https://github.com/allegroai/clearml-helm-charts/blob/main/charts/clearml-agent/values.yaml#L38-L39

3 years ago

0 Votes

30 Answers

2K Views

0 Votes 30 Answers 2K Views

When I Do

When I do Dataset.get why does the SDK use clearml.storage - ERROR - Could not download http://files.clearml.myhost.com vs using what is defined on the agent...

3 years ago

0 Votes

30 Answers

2K Views

0 Votes 30 Answers 2K Views

I'M Trying To Configure The Glue Agent To Use Aws Ecr Via Helm Charts. Below Is My Configuration. It Is Not Pulling The Image Though, It Is Failing With

I'm trying to configure the glue agent to use AWS ECR via helm charts. Below is my configuration. It is not pulling the image though, it is failing with K8S ...

3 years ago

0 Votes

30 Answers

2K Views

0 Votes 30 Answers 2K Views

In Order For A New Worker To Come Online In My K8 Cluster, Do I Need To Have An Ec2 Startup Script Init The Agent/Config, And Then Start The Daemon? Do I Have To Do This Manually Is This A Better Way?

In order for a new worker to come online in my k8 cluster, do I need to have an EC2 startup script init the agent/config, and then start the daemon? Do I hav...

3 years ago

0 Votes

2 Answers

2K Views

0 Votes 2 Answers 2K Views

If I Leave

If I leave WORKING DIRECTORY empty in the experiment configuration (in the UI), will that use the git project root by default?

3 years ago

0 Votes

30 Answers

2K Views

0 Votes 30 Answers 2K Views

Why Am I Getting A 403 From File Server When The K8 Glue Agent Is Initializing ?

Why am I getting a 403 from file server when the k8 glue agent is initializing ?

3 years ago

0 Votes

16 Answers

2K Views

0 Votes 16 Answers 2K Views

When I Try To Create Experiment In The Ui All I See Is This Dialogue

When I try to create experiment in the UI all I see is this dialogue

3 years ago

0 Votes

13 Answers

2K Views

0 Votes 13 Answers 2K Views

Apiserver: Service: Type: Clusterip Configuration: Additionalconfigs: Apiserver.Conf: | Auth { Fixed_Users { Enabled: True Pass_Hashed: False Users: [ {

apiserver: service: type: ClusterIP configuration: additionalConfigs: apiserver.conf: | auth { fixed_users { enabled: true pass_hashed: false users: [ { user...

3 years ago

0 Votes

4 Answers

2K Views

0 Votes 4 Answers 2K Views

Yesterday I Executed An Experiment In Our Hosted Clearml Cluster. After The Experiment Was Finished, We Got An Aws Guard Duty Notification About Suspicious Outbound Traffic From The Ec2 That Executed The Job. It Looks Like The Tag Being Used Is Hardcoded

Yesterday I executed an experiment in our hosted clearML cluster. After the experiment was finished, we got an AWS guard duty notification about suspicious o...

3 years ago

0 Votes

2 Answers

2K Views

0 Votes 2 Answers 2K Views

Where I Can Change This Host Name Using The Helm Charts? I Got This Error When My Task Is Fetching A Dataset.

Where I can change this host name using the helm charts? I got this error when my task is fetching a dataset. 2022-09-23 15:09:45,318 - clearml.storage - ERR...

3 years ago

0 Votes

1 Answers

2K Views

0 Votes 1 Answers 2K Views

In A Nutshell, What Do I Need For The Clearml Agent To Scale Ec2 Nodes In The K8 Cluster, In Terms Of Helm Configuration? I Assume Aws Credentials, Is There Anything Else?

In a nutshell, what do I need for the clearML agent to scale EC2 nodes in the k8 cluster, in terms of helm configuration? I assume AWS credentials, is there ...

3 years ago

0 Votes

6 Answers

2K Views

0 Votes 6 Answers 2K Views

How Do I Create An Experiment Where I Can Set The Github Repository/Branch Name/Script Path Like This Example Shows?

How do I create an experiment where I can set the github repository/branch name/script path like this example shows?

3 years ago

0 Votes

1 Answers

2K Views

0 Votes 1 Answers 2K Views

What Does This Log Message Mean

What does this log message mean ClearML Monitor: Could not detect iteration reporting, falling back to iterations as seconds-from-start ?

3 years ago

0 Votes

18 Answers

2K Views

0 Votes 18 Answers 2K Views

I'M New To Using Datasets, If My Git Project Root Is

I'm new to using datasets, if my git project root is myProject and I expect file.json to be at the root level, how do I accomplish this?

3 years ago

0 Votes

14 Answers

2K Views

0 Votes 14 Answers 2K Views

Hey All, Is There Any Reason The Python Sdk

Hey all, is there any reason the python sdk clearml would cause subprocess issues? I'm calling returncode = Popen(cmd).wait() and getting File "/usr/lib64/py...

3 years ago

0 Votes

2 Answers

2K Views

0 Votes 2 Answers 2K Views

If I Have An Aws Key/Secret For An Iam User, What Is The Best Way To Pass In These Credentials So The Task Docker Container Has Credentials Generated For Usage With Boto3?

If I have an AWS key/secret for an IAM user, what is the best way to pass in these credentials so the task docker container has credentials generated for usa...

3 years ago

0 Votes

31 Answers

135K Views

0 Votes 31 Answers 135K Views

When My Remote Task Is Installing The Python Dependencies

When my remote task is installing the python dependencies --packages requests for example, is there any caching "magic" that is done by the k8 agent? Or is i...

3 years ago

0 When I Do

The pods should be able to use internal DNS names

3 years ago

0 When I Do

Does that make sense

3 years ago

0 I'M Trying To Configure The Glue Agent To Use Aws Ecr Via Helm Charts. Below Is My Configuration. It Is Not Pulling The Image Though, It Is Failing With

I think the quotes don't effect the yaml

3 years ago

0 When I Run

okay I will try that

3 years ago

0 Does Clearml Have The Ability To Run A Single Experiment Across Multiple Nodes/Gpus In A K8 Cluster?

As they are singular not plural

3 years ago

0 Or Is It Just The Ubuntu Official Image

"title": "Unusual outbound communication seen from EC2 instance i-<> on server port 80.",

3 years ago

0 I'M Trying To Configure The Glue Agent To Use Aws Ecr Via Helm Charts. Below Is My Configuration. It Is Not Pulling The Image Though, It Is Failing With

https://github.com/allegroai/clearml-helm-charts/pull/107

3 years ago

0 When I Try To Create Experiment In The Ui All I See Is This Dialogue

I guess I'm confused on venv mode vs docker mode. It seems like I'm passing in my own docker image which is then used at run time?

3 years ago

0 When I Try To Create Experiment In The Ui All I See Is This Dialogue

Yep got it, I was under the impression I could set those values in the UI but I now see they are parsed from my local workstation

3 years ago

0 When I Try To Create Experiment In The Ui All I See Is This Dialogue

Ah I see now

3 years ago

0 When I Try To Create Experiment In The Ui All I See Is This Dialogue

Gotcha, and the agent default runtime mode is docker correct? So I could install all my system dependencies in my own docker image?

3 years ago

0 When I Try To Create Experiment In The Ui All I See Is This Dialogue

Also what is the purpose of the aws block in the clearml.conf? Where are those values used?

3 years ago

0 When I Try To Create Experiment In The Ui All I See Is This Dialogue

Wouldnt that be docker mode?

3 years ago

0 When I Try To Create Experiment In The Ui All I See Is This Dialogue

How does a task specify which docker image it needs?

3 years ago

0 Does Clearml Have The Ability To Run A Single Experiment Across Multiple Nodes/Gpus In A K8 Cluster?

Okay, so basically the DL framework manages the master/worker relationship. I just need to use pod replicas for my k8 agents.

3 years ago

0 When I Do

ahhh its possible my clearml.conf was using the public urls when I made it. Let me try this

3 years ago

0 Apiserver: Service: Type: Clusterip Configuration: Additionalconfigs: Apiserver.Conf: | Auth { Fixed_Users { Enabled: True Pass_Hashed: False Users: [ {

No

3 years ago

0 Apiserver: Service: Type: Clusterip Configuration: Additionalconfigs: Apiserver.Conf: | Auth { Fixed_Users { Enabled: True Pass_Hashed: False Users: [ {

https://github.com/allegroai/clearml-helm-charts/blob/main/charts/clearml/values.yaml#L111-L142

3 years ago

0 Or Is It Just The Ubuntu Official Image

Would using 22.04 Ubuntu still work in the task execution?

3 years ago

0 When I Run An Experiment (Self Hosted), I Only See Scalars For Gpu And System Performance. How Do I See Additional Scalars? I Have

Yes I will try that

3 years ago

0 I'M New To Using Datasets, If My Git Project Root Is

I wouldn't be able to pass in ~/.clearml/cache/storage_manager/datasets/ds_{ds_id}/my_file.json as an argument?

3 years ago

0 Hey, What Is The Recommended Approach To Speed Up The Spin Up Of A Task In A Gcp Autoscaled Instance ? It Takes 20Mins To Build The Venv Environment Needed By The Clearml-Agent To Run It, Would Providing A Vm Image With Preinstalled Pip Packages On It Hel

It takes about 30 seconds here for that step

3 years ago

0 Was There Any Changes To Clearml Python Sdk In The Past 24 Hours?

Was there a new release?

3 years ago

0 When I Run An Experiment (Self Hosted), I Only See Scalars For Gpu And System Performance. How Do I See Additional Scalars? I Have

I basically moved the Task.init() call below the imports

3 years ago

0 Hey All, I'M Testing The Usage Of

yes makes sense. So I wouldnt be able to setup the PYTHONPATH via the setup script?

3 years ago

0 In Order For A New Worker To Come Online In My K8 Cluster, Do I Need To Have An Ec2 Startup Script Init The Agent/Config, And Then Start The Daemon? Do I Have To Do This Manually Is This A Better Way?

So that it spins up nodes

3 years ago

0 When I Run An Experiment (Self Hosted), I Only See Scalars For Gpu And System Performance. How Do I See Additional Scalars? I Have

and removed the duplicate Task.init()

3 years ago

0 Apiserver: Service: Type: Clusterip Configuration: Additionalconfigs: Apiserver.Conf: | Auth { Fixed_Users { Enabled: True Pass_Hashed: False Users: [ {

Thanks for looking into this!

3 years ago

0 When I Run An Experiment (Self Hosted), I Only See Scalars For Gpu And System Performance. How Do I See Additional Scalars? I Have

How would I do os.fork? I'm not familiar with that

3 years ago

0 Hey All, Is There Any Reason The Python Sdk

yes but its limited

3 years ago

Show more results