Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Profile picture
RoughTiger69
Moderator
27 Questions, 94 Answers
  Active since 10 January 2023
  Last activity 10 months ago

Reputation

0

Badges 1

83 × Eureka!
0 Votes
8 Answers
441 Views
0 Votes 8 Answers 441 Views
one year ago
0 Votes
4 Answers
452 Views
0 Votes 4 Answers 452 Views
hi, I created a dataset with 20K files, total of 20GB, with storage pointing to S3. When I upload (or close) the dataset, during the compression phase, the c...
one year ago
0 Votes
3 Answers
420 Views
0 Votes 3 Answers 420 Views
FYI I am getting a lot of read timeouts from the community server: Retrying (Retry(total=235, connect=240, read=235, redirect=240, status=240)) after connect...
one year ago
0 Votes
5 Answers
373 Views
0 Votes 5 Answers 373 Views
Avoiding http://Clear.ml glue code spaghetti - community best practices? Say I have training pipeline : Task 1 - data preprocessing -> create a dataset artif...
2 years ago
0 Votes
10 Answers
433 Views
0 Votes 10 Answers 433 Views
2 years ago
0 Votes
5 Answers
432 Views
0 Votes 5 Answers 432 Views
autoscaler from saas (pro version). I attempted to use the autoscaler “application” from clearml UI. here is what I get in the logs of the autoscaler screen ...
one year ago
0 Votes
8 Answers
410 Views
0 Votes 8 Answers 410 Views
Hi, I am using PipelineDecorator to create tasks. is there a way to force it to use the entire git repo it is created from on the pythonpath? vs. just the de...
one year ago
0 Votes
0 Answers
388 Views
0 Votes 0 Answers 388 Views
Did more digging, seems that you need to start the agent with CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1
one year ago
0 Votes
9 Answers
442 Views
0 Votes 9 Answers 442 Views
one year ago
0 Votes
9 Answers
433 Views
0 Votes 9 Answers 433 Views
one year ago
0 Votes
6 Answers
393 Views
0 Votes 6 Answers 393 Views
Regarding the “classic” datasets (not hyper datasets): Is there an option to do something equivalent to dvc’s “ https://dvc.org/doc/user-guide/managing-exter...
one year ago
0 Votes
0 Answers
340 Views
0 Votes 0 Answers 340 Views
autoscaler 101 questions: What is the best practice for managing credentials so that they don’t get saved in clearml webapp? When the https://clear.ml/docs/l...
one year ago
0 Votes
10 Answers
400 Views
0 Votes 10 Answers 400 Views
hi folks, is there a way to force clear-ml agent with --docker to not create a virtualenv at all? And perhaps not even attempt to install requirements even? ...
one year ago
0 Votes
14 Answers
427 Views
0 Votes 14 Answers 427 Views
question about pipeline and long-waiting tasks: Say I want to generate a dataset. The workflow I have requires query to a DB Creating a labeling assigment in...
one year ago
0 Votes
3 Answers
405 Views
0 Votes 3 Answers 405 Views
one year ago
0 Votes
14 Answers
364 Views
0 Votes 14 Answers 364 Views
Two simple lineage related questions: Task B is a clone of Taks A. Does B store the information that it was cloned from A somewhere? Training task X loads Da...
2 years ago
0 Votes
5 Answers
469 Views
0 Votes 5 Answers 469 Views
hi, I am running a pipeline from decorators. the pipeline runs fine. Then I try to clone it by clicking the (successful) run and launching. The pipeline fail...
one year ago
0 Votes
5 Answers
391 Views
0 Votes 5 Answers 391 Views
I have a logical task that I want to split to multiple workers. The task involves processing media files (not training). The optimal design for me would be: ...
one year ago
0 Votes
5 Answers
485 Views
0 Votes 5 Answers 485 Views
Hi, I am trying to use the aws autoscaler to assign instance profiles to new machines. This is a better way than managing credentials. I added the configurat...
one year ago
0 Votes
10 Answers
423 Views
0 Votes 10 Answers 423 Views
one year ago
0 Votes
18 Answers
392 Views
0 Votes 18 Answers 392 Views
Is there a case-study or ref. architecture for interacting with CI/CD i.e. exposing mature pipelines to be triggered upon code pushes (taking latest git hash...
2 years ago
0 Votes
11 Answers
413 Views
0 Votes 11 Answers 413 Views
I have a local folder a, and a dataset B. a: a a/.DS_Store a/1.txt a/b a/b/.DS_Store a/b/1.txt a/b/c a/b/c/1.txtDataset B: b b/2.txt b/c b/c/2.txtI want to “...
one year ago
0 Votes
4 Answers
379 Views
0 Votes 4 Answers 379 Views
hi, When running a training script from pycharm, it seems that clearml logs only those packages that are explicitly imported by my .py files; it seems to not...
2 years ago
0 Votes
3 Answers
418 Views
0 Votes 3 Answers 418 Views
what’s a good ami to use for the clearml autoscaler on AWS? the defaults offered confidently by the various auto scaler installers don’t seem to exist…| e.g....
one year ago
0 Votes
7 Answers
389 Views
0 Votes 7 Answers 389 Views
one year ago
0 Votes
0 Answers
360 Views
0 Votes 0 Answers 360 Views
Hi, I am catching up with http://clear.ml for stuff beyond exp. tracking, and have a few questions. Will ask them separately to allow threading:
2 years ago
0 Votes
3 Answers
388 Views
0 Votes 3 Answers 388 Views
2 years ago
0 I Have A Local Folder A, And A Dataset B. A:

AgitatedDove14 mv command requires empty folders… so moving b in to a won’t work if some subfolders are already there

one year ago
0 I Have A Local Folder A, And A Dataset B. A:

if the state is :
a:
a a/.DS_Store a/1.txt a/b a/b/.DS_Store a/b/1.txt a/b/c a/b/c/1.txtDataset B:
b b/2.txt b/c b/c/2.txtThen the command
mv b a/returns error since a/ is not empty.
That’s exactly the issue…

As a result, I need to do somethig which copies the files (e.g. cp -r or StorageManager.upload_folder(‘b’, ‘a’)
but this is expensive

one year ago
one year ago
0 Two Simple Lineage Related Questions:

Sure, but was wondering if it has more of a “first class citizen” status for tracking… e.g. something you can visualize in the UI or query via API

2 years ago
0 Two Simple Lineage Related Questions:

I mean, if it’s not tracked, I think it would be a good feature!

2 years ago
0 Two Simple Lineage Related Questions:

Re. “which task did I clone from” - to my understanding “parent’ field is used for “runtime parent” - i.e. what task started me.
This is not the same as “which task was I cloned from”

2 years ago
2 years ago
0 Two Simple Lineage Related Questions:

I think that in principal, if you “intercept” the calls to Model.get() or Dataset.get() from within a task, you can collect the ID’s and do various stuff with them. You can store and visualize it for lineage, or expose it as another hyper parameter I suppose.

You’ll just need the user to name them as part of loading them in the code (in case they are loading multiple datasets/models).

2 years ago
0 I Have A Local Folder A, And A Dataset B. A:

What I’d like is to do Dataset.get(“b”, to=‘a’) and have the download land the files directly there

one year ago
0 Ui Suggestion:

which configuration are you passing? are you using any framework for configuration?

2 years ago
0 I Have A Local Folder A, And A Dataset B. A:

Yes, but this is not the use-case.
The use-case is that I have a local folder and I want to merge a dataset into it without re-fetching the local folder…

one year ago
0 2. Is There A Case-Study Or Ref. Architecture For Interacting With Ci/Cd I.E. Exposing Mature Pipelines To Be Triggered Upon Code Pushes (Taking Latest Git Hash) Or With Manual Ci Triggers?

IrritableGiraffe81 AgitatedDove14 there are multiple levels of what the CI/CD should automate/validate.
This one is the minimal option.
Another option is:
CI deploys (executes) the pipeline fresh, from the committed code http://2.CI waits and extracts the results (various artifacts, metrics etc.) CI compares them to the latest (published) pipeline or to absolute numbers CI decides if to publish it or not (or at least tag it as RC.Steps 2-4 can be themselves encapsulated in a clearml task ...

one year ago
0 What’S A Good Ami To Use For The Clearml Autoscaler On Aws? The Defaults Offered Confidently By The Various Auto Scaler Installers Don’T Seem To Exist…| E.G.

AgitatedDove14 thanks, it was late and I wasn’t sure if I needed to use one of clearml “certified” AMI’s or just a vanilla one.

one year ago
0 2. Is There A Case-Study Or Ref. Architecture For Interacting With Ci/Cd I.E. Exposing Mature Pipelines To Be Triggered Upon Code Pushes (Taking Latest Git Hash) Or With Manual Ci Triggers?

However I see I should really have made my question clearer.
My workflow is as follows:
Engineer A develops a pipeline with a number of steps. She experiments with this pipeline until she is happy with the flow and her code

2 years ago
2 years ago
0 2. Is There A Case-Study Or Ref. Architecture For Interacting With Ci/Cd I.E. Exposing Mature Pipelines To Be Triggered Upon Code Pushes (Taking Latest Git Hash) Or With Manual Ci Triggers?

I want to have a CI/CD pipeline that, upon Engineer A commit, ensures that the pipeline is re-deployed such that with Engineer B uses it as template, it’s definitely the latest version of the code and process

2 years ago
0 2. Is There A Case-Study Or Ref. Architecture For Interacting With Ci/Cd I.E. Exposing Mature Pipelines To Be Triggered Upon Code Pushes (Taking Latest Git Hash) Or With Manual Ci Triggers?

The training pipeline that is considered “best of breed” is committed to Git and deployed by CI/CD; tagged in ClearML clearly.
Users of this pipeline know it’s the “official” training flow that they can now play with using configuration.

Goal is to ensure that “official” pipelines are source controlled.

makes sense?

2 years ago
2 years ago
0 Two Simple Lineage Related Questions:

CostlyOstrich36 Lineage information for datasets - oversimplifying but bare with me:
Task should have a section called “input datasets”)
each time I do a Dataset.get() inside a current_task, add the dataset ID to this section

Same can work with InputModel()

This way you can have a full lineage graph (also queryable/visualizable)

2 years ago
Show more results compactanswers