Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Profile picture
GrievingTurkey78
Moderator
34 Questions, 125 Answers
  Active since 10 January 2023
  Last activity 2 years ago

Reputation

0

Badges 1

119 × Eureka!
0 Votes
3 Answers
1K Views
0 Votes 3 Answers 1K Views
Hi! I have some ClearML agents on GCP and sometimes the instance seems to reboot making the experiment fail and all the progress is lost. What is the best wa...
3 years ago
0 Votes
2 Answers
1K Views
0 Votes 2 Answers 1K Views
Hi ! While restarting the server I got ERROR: for agent-services removal of container 8f1d8539340d6d073eb5b51294f5f5d802048a3614d459b5c4fb1d38a05ce538 is alr...
4 years ago
0 Votes
3 Answers
2K Views
0 Votes 3 Answers 2K Views
Hi! I am trying to run some experiments on an agent I have configured to use the requirements.txt the problem is it only shows Cython on the list of installe...
4 years ago
0 Votes
30 Answers
2K Views
0 Votes 30 Answers 2K Views
Hi! Is there something happening with the ModelCheckpoint callback on tensorflow==2.4.0 ? Using 2.2.0 gave me an input model on the artifacts tab in the GUI 😒
4 years ago
0 Votes
9 Answers
2K Views
0 Votes 9 Answers 2K Views
Hi! Does ClearML have a way to turn on/off virtual machines depending if there are experiments on queue?
4 years ago
0 Votes
4 Answers
1K Views
0 Votes 4 Answers 1K Views
Hi! I am having some problems with a loss after a good amount of training, what would be the best way to log a value to have a better idea of what is happening?
3 years ago
0 Votes
6 Answers
2K Views
0 Votes 6 Answers 2K Views
Hi! I am saving some intermediate .pt files on the experiments and clearml automatically detects them as models, this makes the clearml.model - INFO message ...
3 years ago
0 Votes
4 Answers
2K Views
0 Votes 4 Answers 2K Views
Hi, is there a way to force the requirements.txt? I have a package I installed directly from github but the version is always wrong. Any other way to do this?
4 years ago
0 Votes
21 Answers
2K Views
0 Votes 21 Answers 2K Views
Hi! Any idea why clearml fails to detect iteration reporting? ClearML Monitor: Could not detect iteration reporting, falling back to iterations as seconds-fr...
3 years ago
0 Votes
5 Answers
2K Views
0 Votes 5 Answers 2K Views
Hi! I was taking a look at the https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_cli.html and wanted to know if anyone has used clearml wit...
4 years ago
0 Votes
5 Answers
1K Views
0 Votes 5 Answers 1K Views
Hi, with the upcoming version of Hydra it seems the binding breaks. Specifically in the run_job function the argument order changed from https://github.com/f...
4 years ago
0 Votes
15 Answers
2K Views
0 Votes 15 Answers 2K Views
Hi
Hi πŸ‘‹ I am trying to set up a trains server on GCP. I followed all the steps listed here https://allegro.ai/docs/deploying_trains/trains_server_gcp/ . I also...
4 years ago
0 Votes
2 Answers
1K Views
0 Votes 2 Answers 1K Views
Hi! Regarding the artifact.get_local_copy() method, since there is no way to specify the path where the artifact will be downloaded, I wanted to confirm that...
4 years ago
0 Votes
10 Answers
1K Views
0 Votes 10 Answers 1K Views
Hi! I am getting the following error on an agent: /usr/local/bin/python3.8: No module named virtualenv clearml_agent: ERROR: Command '['python3.8', '-m', 'vi...
3 years ago
0 Votes
8 Answers
1K Views
0 Votes 8 Answers 1K Views
Hello πŸ‘‹ I am using a self hosted clearml setup using the requirments file of the project. When I run the task it is failing and I get: Collecting torch==2.0...
2 years ago
0 Votes
2 Answers
1K Views
0 Votes 2 Answers 1K Views
Hi
Hi AgitatedDove14 ! Regarding the Hydra integration, which pattern should be used? Call the task inside the decorated function? Will this store the parameter...
4 years ago
0 Votes
2 Answers
1K Views
0 Votes 2 Answers 1K Views
Hi! I have the previous trains server configured with multiple experiments; I created it using the gcloud images provided. If I want to update the server to ...
4 years ago
0 Votes
2 Answers
1K Views
0 Votes 2 Answers 1K Views
Hi all! Currently I am trying to create a tool that can perform certain operations on dataset ids, this is a skeleton of what I have in mind (based on the ex...
4 years ago
0 Votes
10 Answers
1K Views
0 Votes 10 Answers 1K Views
I am also experiencing a weird behaviour when running a script using the module flag. For example I run: python -m module.script arg1 arg 2And after the scri...
4 years ago
0 Votes
2 Answers
1K Views
0 Votes 2 Answers 1K Views
I am trying to upgrade from clearml server 0.16 to the newest version but I am getting some errors when spinning up the new containers: WiredTiger error (-31...
4 years ago
0 Votes
13 Answers
2K Views
0 Votes 13 Answers 2K Views
4 years ago
0 Votes
4 Answers
1K Views
0 Votes 4 Answers 1K Views
Hi! If I have a folder with multiple ckpt files would the manual way to upload them be the following: output_model = OutputModel(task) output_model.update_we...
3 years ago
0 Votes
2 Answers
1K Views
0 Votes 2 Answers 1K Views
Hi! What would be the way for manually uploading a model? I have intermediate .pt files which I don't want to upload. Is there a way to turn off clearml capt...
3 years ago
0 Votes
7 Answers
2K Views
0 Votes 7 Answers 2K Views
Hi! I am trying to download data from GS using StorageManager.get_local_copy() . It works fine when I point it to a file i.e gs://bucket/dataset/image.png bu...
4 years ago
0 Votes
6 Answers
1K Views
0 Votes 6 Answers 1K Views
Hi! I have some agents on GCP. Lately I have been getting some experiments that simply stop running (no signs that the experiment crashed). Here is a plot th...
3 years ago
0 Votes
2 Answers
1K Views
0 Votes 2 Answers 1K Views
Hi! I changed from trains to clearml and ran some experiments using keras but it seems the metrics are not being tracked automagically, has anyone ran into t...
4 years ago
0 Votes
7 Answers
2K Views
0 Votes 7 Answers 2K Views
Hi! I am currently using Hydra+ClearML and wanted to know if there are still some updates coming. At the moment, if I change the defaults hydra uses from the...
4 years ago
0 Votes
1 Answers
1K Views
0 Votes 1 Answers 1K Views
Quick question on the clearml-data package, Can I add files to a dataset from google storage instead of having to download them?
4 years ago
0 Votes
7 Answers
2K Views
0 Votes 7 Answers 2K Views
Hi! If I have a pipeline on gitlab that uses ClearML for some tests is there some way to setup the credentials so that it doesn’t fail?
4 years ago
0 Votes
3 Answers
1K Views
0 Votes 3 Answers 1K Views
Hi! I recently updated my server and my clearml version, now when I set a task to be executed remotely its default state is aborted hence I have to reset and...
4 years ago
Show more results questions
0 Hi! I Am Trying To Run Some Experiments On An Agent I Have Configured To Use The Requirements.Txt The Problem Is It Only Shows Cython On The List Of Installed Packages. It Crashes Due To Missing Packages.

` [package_manager.force_repo_requirements_txt=true] Skipping requirements, using repository "requirements.txt"
Using base prefix '/opt/conda'
New python executable in /home/ramon/.clearml/venvs-builds/3.7/bin/python3.7
Also creating executable in /home/ramon/.clearml/venvs-builds/3.7/bin/python
Installing setuptools, pip, wheel...
2021-06-10 09:57:56
done.
Collecting pip<20.2
Using cached pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: p...

4 years ago
0 Hi! I Am Trying To Download Data From Gs Using

Yes! How can I help? AgitatedDove14

4 years ago
0 Hi! Is There A Way To Run A Task Without Reporting To The Server? For Example If I Want To Debug A Script By Running It Locally Without It Appearing On The Server

AgitatedDove14 task.set_archived(True) + the cleanup service should do it πŸ‘Œ If we run in debug mode the experiment goes directly to the archive and gets cleaned and we don’t pollute the main experiment page.

4 years ago
0 Hi! Is There A Way To Run A Task Without Reporting To The Server? For Example If I Want To Debug A Script By Running It Locally Without It Appearing On The Server

I feel it’s easier not to report than cleaning after but please correct me if I am overthinking it. I’ll check if I could wrap the code in something that calls the Task.delete if debugging

4 years ago
0 Hi! Is There A Way To Run A Task Without Reporting To The Server? For Example If I Want To Debug A Script By Running It Locally Without It Appearing On The Server

Yes! I think thats what I will do πŸ‘Œ Let me know if there is a way to contribute a mode to keep logging off. We just don’t want to pollute the server when debugging.

4 years ago
0 Hi! I Have Some Agents On Gcp. Lately I Have Been Getting Some Experiments That Simply Stop Running (No Signs That The Experiment Crashed). Here Is A Plot That Shows The Resource Monitoring. Any Ideas On What Could Be Causing This?

Hey CostlyOstrich36 ! I am using clearml==1.1.2 and clearml-agent==1.1.0 . Stopped is not the right word, more like frozen, it just froze at an epoch. The console on the agent shows epoch 33 first batch and the one at the server epoch 32 last batch. The experiment was running for ~6 hours.

3 years ago
0 Hey Everyone- I Have An Issue Started Today With Trains-Agent Which I’M Getting This Error On Startup:

` File "/home/ramon/.trains/venvs-builds/3.7/lib/python3.7/site-packages/trains/backend_api/session/token_manager.py", line 72, in _get_token_exp
return jwt.decode(token, verify=False).get('exp', sys.maxsize)
File "/home/ramon/.trains/venvs-builds/3.7/lib/python3.7/site-packages/jwt/api_jwt.py", line 113, in decode
decoded = self.decode_complete(jwt, key, algorithms, options, **kwargs)
File "/home/ramon/.trains/venvs-builds/3.7/lib/python3.7/site-packages/jwt/api_jwt.py", line 80, in decode_c...

4 years ago
0 Hi! Does Clearml Have A Way To Turn On/Off Virtual Machines Depending If There Are Experiments On Queue?

Hey AgitatedDove14 do you have an implementation for gcloud? this is awesome

4 years ago
0 Hello

Yes, I configured it that way πŸ‘Œ Thanks! I'll use the flag!

2 years ago
0 Hi! Regarding The

Thanks for the info AgitatedDove14 !

4 years ago
0 Hi! I Am Getting The Following Error On An Agent:

Not yet AgitatedDove14 , does the agent use by default the python version the command is run with? I installed conda and tried using package_manager.type=conda but then get an error:
clearml_agent: ERROR: 'NoneType' object has no attribute 'lower'

3 years ago
0 Hi! I Am Getting The Following Error On An Agent:

With pip I get the first error I showed, I tried conda and it starts running but at some point crashes with:
clearml_agent: ERROR: 'NoneType' object has no attribute 'lower'

3 years ago
0 Hi! I Am Getting The Following Error On An Agent:

Give me a couple of minutes πŸ™Œ

3 years ago
3 years ago
0 Hi! If I Have A Folder With Multiple

My bad :man-facepalming: It was just specifying weights_path=dirpath since the first argument is weights_filename

3 years ago
0 Hello

Managed to get:

clearml_agent: ERROR: Command '['/home/ramon/.clearml/venvs-builds/3.9/bin/python', '-m', 'pip', '--disable-pip-version-check', 'install', '-r', '/var/tmp/requirements_tb0x2i3j.txt', '--extra-index-url', '
 died with <Signals.SIGKILL: 9>.

while building the task with the id on the agent

2 years ago
0 Hi! I Was Taking A Look At The

Yes AgitatedDove14 , I am not sure what they use by default. Here is a simple working example:
` from typing import Optional

import torch
from clearml import Task
from pytorch_lightning import LightningDataModule, LightningModule
from pytorch_lightning.utilities.cli import LightningCLI
from torch.utils.data import DataLoader, Dataset, Subset

class RandomDataset(Dataset):
def init(self, size, length):
self.len = length
self.data = torch.randn(length, size)

def ...
4 years ago
0 Hello

What additional context do you need?

2 years ago
0 Hi! Is There A Way To Run A Task Without Reporting To The Server? For Example If I Want To Debug A Script By Running It Locally Without It Appearing On The Server

AgitatedDove14 Downloading a dataset would not be possible using this right? I want to be able to access the data just avoid reporting the experiment results

4 years ago
0 Hi

Thanks!

4 years ago
0 Hi, I Was Getting A Really Weird Error Due To Mismatch On The Versions Between The Installed Libraries In My Environment And The Ones Ran In The Node (I Manually Changed The Installed Packages And Everything Worked). How Can I Force Trains To Use Exactly

No, I have all the packages with a version. I just want to know if there is a way to override the requirements versions detected by Pigar when using detect_with_pip_freeze: false . I have locally cloudpickle==1.4.1 but when running the code and sending the task to the node the environment uses cloudpickle==1.6.0 . I have to manually change the version on the UI. Is there a way to force this single package to have a version? Maybe on the requirments.txt or something similar

4 years ago
0 Hi, I Was Getting A Really Weird Error Due To Mismatch On The Versions Between The Installed Libraries In My Environment And The Ones Ran In The Node (I Manually Changed The Installed Packages And Everything Worked). How Can I Force Trains To Use Exactly

AgitatedDove14 I am not sure why the packages get different versions, maybe since the package is not directly imported in my code it is possible to get a different version to what I have locally (?). Should all the libraries versions match exactly between local and the code that runs in the agent? The Task.add_requirements(package_name, package_version=None) workaround works perfectly! I just add the previous version that doesn’t break the code. Yes, definitely a force flag could help ...

4 years ago
Show more results compactanswers