Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Profile picture
JumpyClams73
Moderator
10 Questions, 57 Answers
  Active since 10 January 2023
  Last activity 2 years ago

Reputation

0

Badges 1

57 × Eureka!
0 Votes
29 Answers
2K Views
0 Votes 29 Answers 2K Views
Hi, I'm using ClearML's hosted free SaaS offering. I'm running model training in PyTorch on a server and pushing metrics to CML. I've noticed that anytime my...
3 years ago
0 Votes
8 Answers
2K Views
0 Votes 8 Answers 2K Views
3 years ago
0 Votes
30 Answers
2K Views
0 Votes 30 Answers 2K Views
3 years ago
0 Votes
3 Answers
2K Views
0 Votes 3 Answers 2K Views
Hi, I'm looking for documentation on GCP autoscalers. When I search on the docs site, it shows me the AWS autoscaler but not the GCP one. Can someone point m...
3 years ago
0 Votes
1 Answers
2K Views
0 Votes 1 Answers 2K Views
3 years ago
0 Votes
18 Answers
2K Views
0 Votes 18 Answers 2K Views
Hi, I'm trying to run the following API call # Imports ... client = APIClient() resp = client.events.get_scalar_metrics_and_variants("MY_TASK_ID")but it erro...
3 years ago
0 Votes
2 Answers
2K Views
0 Votes 2 Answers 2K Views
Hi, For the CML SaaS Pro tier - are the first 3 users still free and I'll only be charged for any additional users?
3 years ago
0 Votes
10 Answers
2K Views
0 Votes 10 Answers 2K Views
Hi, I've just started to evaluate ClearML for internal use at my org and am wondering if there's anyway to import data from old experiments into the dashboar...
3 years ago
0 Votes
2 Answers
2K Views
0 Votes 2 Answers 2K Views
Hi, I'm looking at https://clear.ml/docs/latest/docs/webapp/webapp_exp_tuning/#base-docker-image where it says To add, change, or delete a base Docker image:...
3 years ago
0 Votes
6 Answers
2K Views
0 Votes 6 Answers 2K Views
3 years ago
0 Hi, I'M Using Clearml'S Hosted Free Saas Offering. I'M Running Model Training In Pytorch On A Server And Pushing Metrics To Cml. I'Ve Noticed That Anytime My Training Job Fails Due To Gpu Oom Issues, Cml Marks The Job As

I haven't had much time to look into this but ran a quick debug and it seems like the exception on the __exit_hook variable is None even though the process failed. So seems like hydra maybe somehow preventing the hook callback from executing correctly. will dig in a bit more next week

3 years ago
0 Hi, I'M Using Clearml'S Hosted Free Saas Offering. I'M Running Model Training In Pytorch On A Server And Pushing Metrics To Cml. I'Ve Noticed That Anytime My Training Job Fails Due To Gpu Oom Issues, Cml Marks The Job As

No, we currently don't handle it gracefully. It just crashes. But we do use hydra which does sort of arrests that exception first. I'm wondering if it's Hydra causing this issue. I'll look into it later today

3 years ago
0 Hi, I'M Trying To Run The Following Api Call

the CML free SaaS offering. It'll probably hit https://app.clear.ml/api if I'm not wrong

3 years ago
0 Hi, I'M Using Clearml'S Hosted Free Saas Offering. I'M Running Model Training In Pytorch On A Server And Pushing Metrics To Cml. I'Ve Noticed That Anytime My Training Job Fails Due To Gpu Oom Issues, Cml Marks The Job As

AnxiousSeal95 I just checked and Hydra returns an exit code of 1 to mark the failure as does another toy program which just throws an exception. So my guess is CML is not using the exit code as a means to determine when the task failed. Are you able to share how CML determines when a task failed? If you could point me to the relevant code files, I'm happy to dive in and figure it out.

3 years ago
0 Hello, When I Clone And Enqueue A Task Using The Web-Console, Is There Anyway To Add A Pre-Execution Hook To That Cloned Task? More Specifically, My Code Uses A Bunch Of Resources Off The Local Disk Which Are Setup Independently Of The Code Itself. When I

I'm looking at the docs on docker mode and running the script. Is this script run after the venv and code dir are setup, or immediately after the container starts but before the environment for running the experiment is setup?

3 years ago
0 Hi, I'Ve Just Started To Evaluate Clearml For Internal Use At My Org And Am Wondering If There'S Anyway To Import Data From Old Experiments Into The Dashboard. Anyone Have Any Thoughts On This?

This is great! Thanks!
If I have access to the logs, python env and git commits, is there an API to log those to the experiments too?

3 years ago
0 Hi, I'M Trying To Run The Following Api Call

Ok. I think I misunderstood what you said. I thought you meant you've already opened a bug ticket. If that's not the case, do you want to me create one on github?

3 years ago
0 Hi, I'M Trying To Run The Following Api Call

I have no way to access your servers

3 years ago
0 Hi, I'Ve Just Started To Evaluate Clearml For Internal Use At My Org And Am Wondering If There'S Anyway To Import Data From Old Experiments Into The Dashboard. Anyone Have Any Thoughts On This?

We have run experiments in the past (before I put ClearML into my code) which has logged scalars, plots etc. to local tensorboard. Is there any way to import this data to ClearML cloud for tracking, visualization and comparison?

3 years ago
0 Hello, When I Clone And Enqueue A Task Using The Web-Console, Is There Anyway To Add A Pre-Execution Hook To That Cloned Task? More Specifically, My Code Uses A Bunch Of Resources Off The Local Disk Which Are Setup Independently Of The Code Itself. When I

Yes, but is it run after the requirements are installed and the code is mounted? The docs say
If we look at the console output in the web UI, the third entry should start with Executing: ['docker', 'run', '-t', '--gpus...', and towards the end of the entry, where the downloaded packages are mentioned, we can see the additional shell-script apt-get install -y bindfs.which seems like that would be the case but I'm not sure what the 1st or 2nd entries are and so want to confirm.

3 years ago
0 Hi, I'M Trying To Clone And Queue Experiments For Running Them On My Workers. I Am Able To Successfully Clone And Queue The Task, But Seems Like The Task Does Not Pass The Correct Parameters To My Python Script On The Worker. We Use Hydra For Configuring

yes, it seems like the command line args are recorded now but the connect call with my parameter dictionary now fails with exception:
` Error executing job with overrides: ['model_name=all-test', ...]
Traceback (most recent call last):
File "/home/binoydalal/miniconda3/envs/DS974/lib/python3.9/site-packages/clearml/binding/hydra_bind.py", line 146, in _patched_task_function
return task_function(a_config, *a_args, **a_kwargs)
....
File "/home/binoydalal/miniconda3/envs/DS974/li...

3 years ago
0 Hi, I'M Looking At

Aah I see it only says Image . Somehow I hit tunnel vision on Base Docker Image as stated in the docs and couldn't identify both to mean the same thing 😅 thanks

3 years ago
0 Hello, When I Clone And Enqueue A Task Using The Web-Console, Is There Anyway To Add A Pre-Execution Hook To That Cloned Task? More Specifically, My Code Uses A Bunch Of Resources Off The Local Disk Which Are Setup Independently Of The Code Itself. When I

The Agent pulls the Task, and then reproduces it, and now it will execute the extra_docker_shell_script that was put in the configuration file.Does this imply the former? Env is fully setup, then script is run, then experiment is started by calling the executable?

3 years ago
0 Hi, I'M Trying To Clone And Queue Experiments For Running Them On My Workers. I Am Able To Successfully Clone And Queue The Task, But Seems Like The Task Does Not Pass The Correct Parameters To My Python Script On The Worker. We Use Hydra For Configuring

I think the fire + hydra combination is not an issue anymore. We're going to separate the 2 out, and I tried it last night and argument modification and passing worked fine with hydra only.
In any case, thanks for you help Martin!

3 years ago
0 Hi, I'M Trying To Clone And Queue Experiments For Running Them On My Workers. I Am Able To Successfully Clone And Queue The Task, But Seems Like The Task Does Not Pass The Correct Parameters To My Python Script On The Worker. We Use Hydra For Configuring

I thought the agent created a new conda env and installed all packages, recorded during initial task run, from scratch (except for caching with venv). Is that not the case?

3 years ago
0 Hi, I'M Trying To Clone And Queue Experiments For Running Them On My Workers. I Am Able To Successfully Clone And Queue The Task, But Seems Like The Task Does Not Pass The Correct Parameters To My Python Script On The Worker. We Use Hydra For Configuring

Could it be the script itself is using vanilla sys.argv and not Argparser ? (edited)

Thanks for bringing this up. Our code uses fire to parse command line args and then sort of hands off to hydra, so yes it does use sys.argv initially. Is this a possible issue?

3 years ago
0 Hi, I'M Using Clearml'S Hosted Free Saas Offering. I'M Running Model Training In Pytorch On A Server And Pushing Metrics To Cml. I'Ve Noticed That Anytime My Training Job Fails Due To Gpu Oom Issues, Cml Marks The Job As

I didn't check with the toy task, I thought the error codes might be an issue here so was just looking for the difference. I'll check for that too.
But for my hydra task, it's always marked completed, never failed

3 years ago
Show more results compactanswers