Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Profile picture
RoughTiger69
Moderator
28 Questions, 101 Answers
  Active since 10 January 2023
  Last activity 10 months ago

Reputation

0

Badges 1

90 × Eureka!
0 Votes
8 Answers
1K Views
0 Votes 8 Answers 1K Views
Hi, I am using PipelineDecorator to create tasks. is there a way to force it to use the entire git repo it is created from on the pythonpath? vs. just the de...
2 years ago
0 Votes
3 Answers
1K Views
0 Votes 3 Answers 1K Views
3 years ago
0 Votes
10 Answers
1K Views
0 Votes 10 Answers 1K Views
3 years ago
0 Votes
6 Answers
1K Views
0 Votes 6 Answers 1K Views
Regarding the “classic” datasets (not hyper datasets): Is there an option to do something equivalent to dvc’s “ https://dvc.org/doc/user-guide/managing-exter...
3 years ago
0 Votes
5 Answers
1K Views
0 Votes 5 Answers 1K Views
I have a logical task that I want to split to multiple workers. The task involves processing media files (not training). The optimal design for me would be: ...
2 years ago
0 Votes
0 Answers
1K Views
0 Votes 0 Answers 1K Views
autoscaler 101 questions: What is the best practice for managing credentials so that they don’t get saved in clearml webapp? When the https://clear.ml/docs/l...
2 years ago
0 Votes
7 Answers
1K Views
0 Votes 7 Answers 1K Views
2 years ago
0 Votes
10 Answers
1K Views
0 Votes 10 Answers 1K Views
hi folks, is there a way to force clear-ml agent with --docker to not create a virtualenv at all? And perhaps not even attempt to install requirements even? ...
2 years ago
0 Votes
3 Answers
1K Views
0 Votes 3 Answers 1K Views
FYI I am getting a lot of read timeouts from the community server: Retrying (Retry(total=235, connect=240, read=235, redirect=240, status=240)) after connect...
2 years ago
0 Votes
3 Answers
1K Views
0 Votes 3 Answers 1K Views
what’s a good ami to use for the clearml autoscaler on AWS? the defaults offered confidently by the various auto scaler installers don’t seem to exist…| e.g....
2 years ago
0 Votes
0 Answers
1K Views
0 Votes 0 Answers 1K Views
Hi, I am catching up with http://clear.ml for stuff beyond exp. tracking, and have a few questions. Will ask them separately to allow threading:
3 years ago
0 Votes
0 Answers
1K Views
0 Votes 0 Answers 1K Views
Did more digging, seems that you need to start the agent with CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1
2 years ago
0 Votes
4 Answers
1K Views
0 Votes 4 Answers 1K Views
hi, I created a dataset with 20K files, total of 20GB, with storage pointing to S3. When I upload (or close) the dataset, during the compression phase, the c...
2 years ago
0 Votes
18 Answers
1K Views
0 Votes 18 Answers 1K Views
Is there a case-study or ref. architecture for interacting with CI/CD i.e. exposing mature pipelines to be triggered upon code pushes (taking latest git hash...
3 years ago
0 Votes
5 Answers
2K Views
0 Votes 5 Answers 2K Views
Hi, I am trying to use the aws autoscaler to assign instance profiles to new machines. This is a better way than managing credentials. I added the configurat...
2 years ago
0 Votes
6 Answers
1K Views
0 Votes 6 Answers 1K Views
autoscaler from saas (pro version). I attempted to use the autoscaler “application” from clearml UI. here is what I get in the logs of the autoscaler screen ...
2 years ago
0 Votes
8 Answers
1K Views
0 Votes 8 Answers 1K Views
2 years ago
0 Votes
12 Answers
629 Views
0 Votes 12 Answers 629 Views
Is there a reference implmentation for a task in a pipeline that awaits user input?
3 years ago
0 Votes
9 Answers
1K Views
0 Votes 9 Answers 1K Views
2 years ago
0 Votes
4 Answers
1K Views
0 Votes 4 Answers 1K Views
hi, When running a training script from pycharm, it seems that clearml logs only those packages that are explicitly imported by my .py files; it seems to not...
3 years ago
0 Votes
10 Answers
1K Views
0 Votes 10 Answers 1K Views
2 years ago
0 Votes
14 Answers
1K Views
0 Votes 14 Answers 1K Views
question about pipeline and long-waiting tasks: Say I want to generate a dataset. The workflow I have requires query to a DB Creating a labeling assigment in...
2 years ago
0 Votes
5 Answers
1K Views
0 Votes 5 Answers 1K Views
hi, I am running a pipeline from decorators. the pipeline runs fine. Then I try to clone it by clicking the (successful) run and launching. The pipeline fail...
2 years ago
0 Votes
5 Answers
1K Views
0 Votes 5 Answers 1K Views
Avoiding http://Clear.ml glue code spaghetti - community best practices? Say I have training pipeline : Task 1 - data preprocessing -> create a dataset artif...
3 years ago
0 Votes
11 Answers
1K Views
0 Votes 11 Answers 1K Views
I have a local folder a, and a dataset B. a: a a/.DS_Store a/1.txt a/b a/b/.DS_Store a/b/1.txt a/b/c a/b/c/1.txtDataset B: b b/2.txt b/c b/c/2.txtI want to “...
2 years ago
0 Votes
9 Answers
1K Views
0 Votes 9 Answers 1K Views
2 years ago
0 Votes
14 Answers
1K Views
0 Votes 14 Answers 1K Views
Two simple lineage related questions: Task B is a clone of Taks A. Does B store the information that it was cloned from A somewhere? Training task X loads Da...
3 years ago
0 Votes
3 Answers
1K Views
0 Votes 3 Answers 1K Views
2 years ago
0 <no title>

sounds workable, will start with that

3 years ago
0 I Have A Local Folder A, And A Dataset B. A:

What I’d like is to do Dataset.get(“b”, to=‘a’) and have the download land the files directly there

2 years ago
0 2. Is There A Case-Study Or Ref. Architecture For Interacting With Ci/Cd I.E. Exposing Mature Pipelines To Be Triggered Upon Code Pushes (Taking Latest Git Hash) Or With Manual Ci Triggers?

So “The” pipeline Engineer A creates, once updated with the latest code, and perhaps ran once as test by CI CD, should be “tainted” as “The production” version of that pipeline, so that Engineer B’s code always uses the latest released pipeline code

3 years ago
0 2. Is There A Case-Study Or Ref. Architecture For Interacting With Ci/Cd I.E. Exposing Mature Pipelines To Be Triggered Upon Code Pushes (Taking Latest Git Hash) Or With Manual Ci Triggers?

I want to have a CI/CD pipeline that, upon Engineer A commit, ensures that the pipeline is re-deployed such that with Engineer B uses it as template, it’s definitely the latest version of the code and process

3 years ago
2 years ago
0 I Have A Local Folder A, And A Dataset B. A:

Yes, but this is not the use-case.
The use-case is that I have a local folder and I want to merge a dataset into it without re-fetching the local folder…

2 years ago
0 Question About Pipeline And Long-Waiting Tasks: Say I Want To Generate A Dataset. The Workflow I Have Requires

AgitatedDove14

What was important for me was that the user can define the entire workflow and that I can see its status as one ‘pipeline’ in the UI (vs. disparate tasks).

perform query process records into a labeling assignment Call labeling system API wait for and external hook when labels are ready clean the labels upload them to a dataset
Do you know what specific API do I need to signal “resume” after “abort”?
not “reset” I presume?

2 years ago
0 Hi, I Am Using Pipelinedecorator To Create Tasks. Is There A Way To Force It To Use The Entire Git Repo It Is Created From On The Pythonpath? Vs. Just The Decorated Function And Perhaps The Helper_Function=[Some_Function]?

AgitatedDove14 the emphasis is that the imports I am doing are not from external/pipe packages, they are just neighbouring modules to the function I am importing. Imports that rely on pip installed packages work well

2 years ago
0 <no title>

yes and no.
This is a pseudo flow:
Data download -> pre-processing -> model training (e.g. HPT) - > model evaluation (per variant) -> model comparison dashboard -> human selects the best model using a heuristic and the status of the weather -> model packaging -> inference tests etc.

I could divide it into two pipelines:
Data download --> dashboard
Packaging --> …

Where packaging takes a parameter which is the human selected ID of the model.
However, this way, I lose the context of the ent...

3 years ago
0 Hi, I Am Trying To Use The Aws Autoscaler To Assign Instance Profiles To New Machines. This Is A Better Way Than Managing Credentials. I Added The Configuration To The Autoscaler Config Like So:

Trust me, I had to add this field to this default dict just so that clearml doesn’t delete it for me
it does appear on the task in the UI, just somehow not repopulated in the remote run if it’s not a part of the default empty dict…

2 years ago
0 Hi, I Am Using Pipelinedecorator To Create Tasks. Is There A Way To Force It To Use The Entire Git Repo It Is Created From On The Pythonpath? Vs. Just The Decorated Function And Perhaps The Helper_Function=[Some_Function]?

sure CostlyOstrich36
I have something like the following:

@PipelineDecorator.component(....) def my_task(...) from my_module1 import my_func1 from my_modeul2 import ....my_module1 and 2 are modules that are a part of the same project source. they don’t come as a separate package.

Now when I run this in clearml, these imports don’t work.

These functions may require transitive imports of course, so the following doesn’t work:
` PipelineDecorator.component(helper_function=[my_fu...

2 years ago
0 <no title>

could work! is there a way to visualize the pipeline such that this step is “stuck” in executing?

3 years ago
0 <no title>

thanks so much for taking the time by the way!

3 years ago
0 Hello Folks! I Have A Pipeline With Three Tasks: A, B, And C I Want To Set It Up So That: A Gets Assigned A Machine (E.G. Based On The Queue) B Always Gets Assigned To The Same Machine As A (But May Run In A Different Docker Etc.) C Will Be Submitted To

I don’t think so.
In most cases I woudl have multiple agents pulling from the same queue. I can’t have a queue per pipeline execution.
So if I submit A and B to the same queue, it still doesn’t gurantee that they will be pulled by the same agent….

2 years ago
0 I Have Code That Does Torch.Load(Path) And Deserializes A Model. I Am Performing This In Package A.B.C, And The Model’S Module Is Available In In A.B.C.Model Unfortunately, The Model Was Serialized With A Different Module Structure - It Was Originally Pla

I think it has something to do with clearml since I can run this code as pure python without clearml, and when I activate clearml, I see that torch.load() hits the

import_bind

.

__patched_import3

when trying to deserialize the saved model

2 years ago
0 I Have A Pipeline With Tasks A->B->C. I Want To Be Able To Trigger It Manually, And Skip A Regardless Of It’S Cache Status. I Want To Pass B Value That Represents A’S Output If Needed. What’S A Good Way To Achieve This (Can Be Ui-Based, Or Pipeline-Gymnas

It’s more like this:
I have a pipeline, ran on all data.
Now I change/add a sub-dag to the pipeline
I want to run only that sub-dag on all historical data in ad-hoc manner
And then next runs will run the full dag (e.g. only on new data)

2 years ago
0 Regarding The “Classic” Datasets (Not Hyper Datasets): Is There An Option To Do Something Equivalent To Dvc’S “

AgitatedDove14 I haven’t done a full design for this 😉
Just referring to how DVC claims it can detect and invalidate changes in large remote files.
So I take it there is no such feature in http://clear.ml 🙂

3 years ago
0 I Have A Pipeline With Tasks A->B->C. I Want To Be Able To Trigger It Manually, And Skip A Regardless Of It’S Cache Status. I Want To Pass B Value That Represents A’S Output If Needed. What’S A Good Way To Achieve This (Can Be Ui-Based, Or Pipeline-Gymnas

AgitatedDove14
Sort of.
I would go with something which is more like:
` execution_plan = {'step_b':'b_result', step_c: None, ...}
@PipelineDecorator.pipeline(...)

def pipeline(execution_plan):
step_results = {}
for step in pipeline.get_dag():
if step.name in execution_plan.keys():
step_results[step.name] = execution_plan[step.name] or step(**step_results)

`The ‘execution plan’ specifies list of steps to run (keys) and for each, whether we should use a u...
2 years ago
Show more results compactanswers