Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Profile picture
RoughTiger69
Moderator
28 Questions, 101 Answers
  Active since 10 January 2023
  Last activity one year ago

Reputation

0

Badges 1

90 × Eureka!
0 Votes
4 Answers
2K Views
0 Votes 4 Answers 2K Views
hi, I created a dataset with 20K files, total of 20GB, with storage pointing to S3. When I upload (or close) the dataset, during the compression phase, the c...
2 years ago
0 Votes
5 Answers
2K Views
0 Votes 5 Answers 2K Views
Avoiding http://Clear.ml glue code spaghetti - community best practices? Say I have training pipeline : Task 1 - data preprocessing -> create a dataset artif...
3 years ago
0 Votes
0 Answers
1K Views
0 Votes 0 Answers 1K Views
autoscaler 101 questions: What is the best practice for managing credentials so that they don’t get saved in clearml webapp? When the https://clear.ml/docs/l...
2 years ago
0 Votes
5 Answers
2K Views
0 Votes 5 Answers 2K Views
Hi, I am trying to use the aws autoscaler to assign instance profiles to new machines. This is a better way than managing credentials. I added the configurat...
2 years ago
0 Votes
8 Answers
2K Views
0 Votes 8 Answers 2K Views
2 years ago
0 Votes
5 Answers
2K Views
0 Votes 5 Answers 2K Views
hi, I am running a pipeline from decorators. the pipeline runs fine. Then I try to clone it by clicking the (successful) run and launching. The pipeline fail...
2 years ago
0 Votes
12 Answers
985 Views
0 Votes 12 Answers 985 Views
Is there a reference implmentation for a task in a pipeline that awaits user input?
3 years ago
0 Votes
8 Answers
2K Views
0 Votes 8 Answers 2K Views
Hi, I am using PipelineDecorator to create tasks. is there a way to force it to use the entire git repo it is created from on the pythonpath? vs. just the de...
2 years ago
0 Votes
3 Answers
2K Views
0 Votes 3 Answers 2K Views
FYI I am getting a lot of read timeouts from the community server: Retrying (Retry(total=235, connect=240, read=235, redirect=240, status=240)) after connect...
3 years ago
0 Votes
10 Answers
2K Views
0 Votes 10 Answers 2K Views
hi folks, is there a way to force clear-ml agent with --docker to not create a virtualenv at all? And perhaps not even attempt to install requirements even? ...
3 years ago
0 Votes
0 Answers
1K Views
0 Votes 0 Answers 1K Views
Did more digging, seems that you need to start the agent with CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1
3 years ago
0 Votes
14 Answers
2K Views
0 Votes 14 Answers 2K Views
Two simple lineage related questions: Task B is a clone of Taks A. Does B store the information that it was cloned from A somewhere? Training task X loads Da...
3 years ago
0 Votes
18 Answers
2K Views
0 Votes 18 Answers 2K Views
Is there a case-study or ref. architecture for interacting with CI/CD i.e. exposing mature pipelines to be triggered upon code pushes (taking latest git hash...
3 years ago
0 Votes
0 Answers
1K Views
0 Votes 0 Answers 1K Views
Hi, I am catching up with http://clear.ml for stuff beyond exp. tracking, and have a few questions. Will ask them separately to allow threading:
3 years ago
0 Votes
11 Answers
2K Views
0 Votes 11 Answers 2K Views
I have a local folder a, and a dataset B. a: a a/.DS_Store a/1.txt a/b a/b/.DS_Store a/b/1.txt a/b/c a/b/c/1.txtDataset B: b b/2.txt b/c b/c/2.txtI want to “...
2 years ago
0 Votes
3 Answers
2K Views
0 Votes 3 Answers 2K Views
what’s a good ami to use for the clearml autoscaler on AWS? the defaults offered confidently by the various auto scaler installers don’t seem to exist…| e.g....
2 years ago
0 Votes
9 Answers
2K Views
0 Votes 9 Answers 2K Views
2 years ago
0 Votes
7 Answers
2K Views
0 Votes 7 Answers 2K Views
2 years ago
0 Votes
5 Answers
2K Views
0 Votes 5 Answers 2K Views
I have a logical task that I want to split to multiple workers. The task involves processing media files (not training). The optimal design for me would be: ...
2 years ago
0 Votes
3 Answers
2K Views
0 Votes 3 Answers 2K Views
2 years ago
0 Votes
10 Answers
2K Views
0 Votes 10 Answers 2K Views
3 years ago
0 Votes
4 Answers
2K Views
0 Votes 4 Answers 2K Views
hi, When running a training script from pycharm, it seems that clearml logs only those packages that are explicitly imported by my .py files; it seems to not...
3 years ago
0 Votes
10 Answers
2K Views
0 Votes 10 Answers 2K Views
2 years ago
0 Votes
6 Answers
2K Views
0 Votes 6 Answers 2K Views
Regarding the “classic” datasets (not hyper datasets): Is there an option to do something equivalent to dvc’s “ https://dvc.org/doc/user-guide/managing-exter...
3 years ago
0 Votes
14 Answers
2K Views
0 Votes 14 Answers 2K Views
question about pipeline and long-waiting tasks: Say I want to generate a dataset. The workflow I have requires query to a DB Creating a labeling assigment in...
3 years ago
0 Votes
3 Answers
2K Views
0 Votes 3 Answers 2K Views
3 years ago
0 Votes
9 Answers
2K Views
0 Votes 9 Answers 2K Views
2 years ago
0 Votes
6 Answers
2K Views
0 Votes 6 Answers 2K Views
autoscaler from saas (pro version). I attempted to use the autoscaler “application” from clearml UI. here is what I get in the logs of the autoscaler screen ...
2 years ago
0 Hi, I Am Trying To Use The Aws Autoscaler To Assign Instance Profiles To New Machines. This Is A Better Way Than Managing Credentials. I Added The Configuration To The Autoscaler Config Like So:

But you already have all the entries defined here:

yes but it’s missing a field that is actually found and parsed from my local autoscaler.yaml….

2 years ago
0 Hi, I Am Trying To Use The Aws Autoscaler To Assign Instance Profiles To New Machines. This Is A Better Way Than Managing Credentials. I Added The Configuration To The Autoscaler Config Like So:

Trust me, I had to add this field to this default dict just so that clearml doesn’t delete it for me
it does appear on the task in the UI, just somehow not repopulated in the remote run if it’s not a part of the default empty dict…

2 years ago
0 <no title>

yeah, its a tradeoff that is dependent on parameters that lie outside the realm of human comprehension.
Let’s call if voodoo.
Yes, the manual selection can be done via tagging a model.
The main thing is that I want the selection to be part of the overall flow.
I want the task of human tagging a model to be “just another step in the pipeline”

3 years ago
0 I Have A Pipeline With Tasks A->B->C. I Want To Be Able To Trigger It Manually, And Skip A Regardless Of It’S Cache Status. I Want To Pass B Value That Represents A’S Output If Needed. What’S A Good Way To Achieve This (Can Be Ui-Based, Or Pipeline-Gymnas

It’s more like this:
I have a pipeline, ran on all data.
Now I change/add a sub-dag to the pipeline
I want to run only that sub-dag on all historical data in ad-hoc manner
And then next runs will run the full dag (e.g. only on new data)

2 years ago
0 Two Simple Lineage Related Questions:

Sure, but was wondering if it has more of a “first class citizen” status for tracking… e.g. something you can visualize in the UI or query via API

3 years ago
2 years ago
0 Hi Folks, Is There A Way To Force Clear-Ml Agent With --Docker To

SweetBadger76 thanks for your reply.
One quirk I found was that even with this flag on, the agent decides to install whatever is in the requirements.txt.

3 years ago
0 Regarding The “Classic” Datasets (Not Hyper Datasets): Is There An Option To Do Something Equivalent To Dvc’S “

AgitatedDove14 nope… you can run md5 on the file as stored in the remote storage (nfs or s3)

3 years ago
0 Hi, I Am Trying To Setup Multi-Node Training With Pytorch Distributeddataparallel. Ddp Requres A Launch Script With A Set Of Parameters To Be Run On Each Node. One Of These Parameters Is Master Node Address. I Am Currently Using The Following Scheme:

AgitatedDove14 looks like service-writing-time for me!
PS can you point me to some official example/ doc for how to persist/restore state so that tasks are restartable?

3 years ago
0 Hi Folks, Is There A Way To Force Clear-Ml Agent With --Docker To

SweetBadger76 I think it’s not related to the flag or whether or not I am running in a virtual env.
I just noticed that even when I clear the list of installed packages in the UI, upon startup, clearml agent still picks up the requirements.txt (after checking out the code) and tries to install it.
I wonder if there’s a way to tell it to skip this step too?

3 years ago
0 Hi Folks, Is There A Way To Force Clear-Ml Agent With --Docker To

AgitatedDove14 yes, i am passing this flag to the agent with CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1 clearml-agent….
running inside docker
and it still tries to install the requirements.txt

Using 1.3.1

3 years ago
0 Two Simple Lineage Related Questions:

so I think it will just be confusing

3 years ago
0 Two Simple Lineage Related Questions:

I think that in principal, if you “intercept” the calls to Model.get() or Dataset.get() from within a task, you can collect the ID’s and do various stuff with them. You can store and visualize it for lineage, or expose it as another hyper parameter I suppose.

You’ll just need the user to name them as part of loading them in the code (in case they are loading multiple datasets/models).

3 years ago
0 Two Simple Lineage Related Questions:

CostlyOstrich36 Lineage information for datasets - oversimplifying but bare with me:
Task should have a section called “input datasets”)
each time I do a Dataset.get() inside a current_task, add the dataset ID to this section

Same can work with InputModel()

This way you can have a full lineage graph (also queryable/visualizable)

3 years ago
0 Is It Possible To Use In Clearml

As far I know storage can be https://clear.ml/docs/latest/docs/integrations/storage/#direct-access .
typical EBS is limited to being mounted to 1 machine at a time.
so in this sense, it won’t be too easy to create a solution where multiple machines consume datasets from this storage type

PS https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volumes-multi.html is possible under some limitations

2 years ago
0 Ui Suggestion:

which configuration are you passing? are you using any framework for configuration?

3 years ago
0 Two Simple Lineage Related Questions:

I mean, if it’s not tracked, I think it would be a good feature!

3 years ago
0 Hi, I Am Using Pipelinedecorator To Create Tasks. Is There A Way To Force It To Use The Entire Git Repo It Is Created From On The Pythonpath? Vs. Just The Decorated Function And Perhaps The Helper_Function=[Some_Function]?

AgitatedDove14 the emphasis is that the imports I am doing are not from external/pipe packages, they are just neighbouring modules to the function I am importing. Imports that rely on pip installed packages work well

2 years ago
0 Question About Pipeline And Long-Waiting Tasks: Say I Want To Generate A Dataset. The Workflow I Have Requires

AgitatedDove14 thanks, good idea.

My main issue with this approach is that it breaks the workflow into “a-sync” set of tasks:

One task sends a list of images for labeling and terminates an external webhook calls http://clear.ml and creates a dataset from the labels returned from the labeling task a trigger wakes up the label post processing/splitting logic.
It will be hard to understand where things are standing from looking at the UI.

I was wondering if the “waiting” operator can actua...

3 years ago
0 What’S A Good Ami To Use For The Clearml Autoscaler On Aws? The Defaults Offered Confidently By The Various Auto Scaler Installers Don’T Seem To Exist…| E.G.

AgitatedDove14 thanks, it was late and I wasn’t sure if I needed to use one of clearml “certified” AMI’s or just a vanilla one.

2 years ago
0 <no title>

that’s the thing. I want to it to appear like one long pipeline, vs. trigger a new set of steps after the approval. So “wait” is a better metaphore for me

3 years ago
Show more results compactanswers