Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Profile picture
PanickyMoth78
Moderator
34 Questions, 167 Answers
  Active since 10 January 2023
  Last activity 11 months ago

Reputation

0

Badges 1

166 × Eureka!
0 Votes
6 Answers
1K Views
0 Votes 6 Answers 1K Views
Is there some built-in way in clearml to trigger further action on task fail (or pipeline fail)?
2 years ago
0 Votes
2 Answers
2K Views
0 Votes 2 Answers 2K Views
I have a training task that auto-magically saves a model for me to GCS task = Task.init( project_name=project_name, task_name=f"Image classification training...
2 years ago
0 Votes
3 Answers
1K Views
0 Votes 3 Answers 1K Views
Hi. Shoulf this command succeed in the presence of project lavi-testing and absence of dataset tmp_datset within it? from clearml import Dataset tmp_dataset ...
2 years ago
0 Votes
2 Answers
2K Views
0 Votes 2 Answers 2K Views
I am using the AWS autoscaler and I wish to set my files server to be gs. I tried to do so by having this in the ADDITIONAL CLEARML CONFIGURATION window: api...
2 years ago
0 Votes
8 Answers
2K Views
0 Votes 8 Answers 2K Views
2 years ago
0 Votes
4 Answers
946 Views
0 Votes 4 Answers 946 Views
Hi. I'm using clearml agent 1.16.1 My code is running a multi-process pool with "spawn" (see here for why) from multiprocessing import get_context ... with g...
11 months ago
0 Votes
7 Answers
1K Views
0 Votes 7 Answers 1K Views
Hi. I have a problem accessing repo code in pipeline components running in an AWS autoscaler (first attempts at doing this) My local clearml.conf file has ag...
2 years ago
0 Votes
1 Answers
2K Views
0 Votes 1 Answers 2K Views
2 years ago
0 Votes
2 Answers
1K Views
0 Votes 2 Answers 1K Views
Hi. I'm using @PipelineDecorator.component to define a task from a function (to run in a pipeline) I'd like to get the task object within this function so th...
2 years ago
0 Votes
14 Answers
2K Views
0 Votes 14 Answers 2K Views
Hi there. I'm trying to switch pipeline code from a local run using PipelineDecorator.run_locally()to a slightly-less-local run using PipelineDecorator.set_d...
2 years ago
0 Votes
7 Answers
1K Views
0 Votes 7 Answers 1K Views
I have 5 unarchived pipeline runs that were defined with this decorator: @PipelineDecorator.pipeline( name="fastai_image_classification_pipeline", project="l...
2 years ago
0 Votes
22 Answers
2K Views
0 Votes 22 Answers 2K Views
I started two pipelines (using AWS autoscaler in app.clear.ml ). The pipelines ran concurrently, using the same pipeline code. Both failed in the same compon...
2 years ago
0 Votes
8 Answers
2K Views
0 Votes 8 Answers 2K Views
2 years ago
0 Votes
16 Answers
2K Views
0 Votes 16 Answers 2K Views
Hi. Question about Dataset upload errors: When uploading a clearml.Dataset created with output_uri=" gs://lavi_test/datasets after adding 20 files of size 50...
gcp
2 years ago
0 Votes
22 Answers
2K Views
0 Votes 22 Answers 2K Views
Hi. I'm encountering a problem with model.name At least, for models that where auto-magically uploaded. I see it in my own code but you can see it if you run...
2 years ago
0 Votes
20 Answers
2K Views
0 Votes 20 Answers 2K Views
task struck at task.flush(wait_for_uploads=True) : I've been running a model training task - a variation on this clearml dataset example: https://github.com/...
2 years ago
0 Votes
9 Answers
2K Views
0 Votes 9 Answers 2K Views
Hi. I have a question about pipelines and their generated dependency graphs. I took the code of the clearml pipeline from decorator example: https://github.c...
2 years ago
0 Votes
3 Answers
1K Views
0 Votes 3 Answers 1K Views
Hi. First time user here 👋 I have experienced a problem following the getting started documentation. I opened an account on https://app.clear.ml/ I then fol...
2 years ago
0 Votes
1 Answers
1K Views
0 Votes 1 Answers 1K Views
suppose I use a pipeline decorator to define a pipeline: @PipelineDecorator.pipeline(name='my-pipeline', project='my-project', version='0.2') def my_pipeline...
2 years ago
0 Votes
9 Answers
1K Views
0 Votes 9 Answers 1K Views
Hi. Help 🥺 I have a clearml.Datase which I can't get
2 years ago
0 Votes
25 Answers
2K Views
0 Votes 25 Answers 2K Views
Autoscaler parallelization issue: I have an AWS Autoscaler set up with a resource that has a max of 3 instances assigned to the default queue I've given it a...
2 years ago
0 Votes
27 Answers
2K Views
0 Votes 27 Answers 2K Views
Hi. I'm running this little pipeline: from clearml.automation.controller import PipelineDecorator from clearml import TaskTypes @PipelineDecorator.component(...
2 years ago
0 Votes
2 Answers
2K Views
0 Votes 2 Answers 2K Views
Hi. Suppose I want to report on what my task has done by having it generate a markdown (.md) file with links to some "local" figure files. looking at the rep...
2 years ago
0 Votes
13 Answers
1K Views
0 Votes 13 Answers 1K Views
Another question on the topic of how a remote execution of a pipeline kills the calling process (previously discussed https://clearml.slack.com/archives/CTK2...
2 years ago
0 Votes
3 Answers
1K Views
0 Votes 3 Answers 1K Views
2 years ago
0 Votes
14 Answers
1K Views
0 Votes 14 Answers 1K Views
Bug? dataset name is ignored if use_current_task=True
2 years ago
0 Votes
14 Answers
2K Views
0 Votes 14 Answers 2K Views
Hi. I have a job that processes images and creates ~5 GB of processed image files (lots of small ones). At the end - it creates a clearml.Dataset and perform...
2 years ago
0 Votes
11 Answers
2K Views
0 Votes 11 Answers 2K Views
Hi. I have a few questions about the snippet attached re-running this code produces the same printouts... I chose 47 out of 100 in the pipeline ... I chose 8...
2 years ago
0 Votes
8 Answers
2K Views
0 Votes 8 Answers 2K Views
Hi (again... sorry for asking so many questions) Question about using google cloud storage in a clearml agent running in AWS ec2 instance. my clearml.conf ha...
2 years ago
0 Votes
7 Answers
1K Views
0 Votes 7 Answers 1K Views
Hi. I am experimenting with clearml.Dataset and encountering an error. LockException: [Errno 11] Resource temporarily unavailable In my experiment, I make a ...
2 years ago
Show more results questions
2 years ago
0 Hi. I'M Encountering A Problem With

Right. Thanks.
With several models saved by the training process (whose code is not task-aware) I suspect that doing the update call after training completed will only update the last of the uploaded models.
I'm currently looking at a workaround where:
I disable auto saving by https://clear.ml/docs/latest/docs/clearml_sdk/task_sdk/#automatic-logging Manually upload the models Manually register the models with https://github.com/allegroai/clearml/blob/cf7361e134554f4effd939ca67e8ecb2345b...

2 years ago
0 Hi. I'M Encountering A Problem With

To be specific there is "model name" which is not unique , and there is model-key which is unique to the Task

not sure why the two fields don't simply match. I guess that there may be situations where file name (without the full path) may be used several times.

2 years ago
0 Autoscaler Parallelization Issue: I Have An Aws Autoscaler Set Up With A Resource That Has A Max Of 3 Instances Assigned To The

erm,
this parallelization has led to the pipeline task issuing a bunch of:
model_path/run_2022_07_20T22_11_15.209_0.zip , err: [Errno 28] No space left on deviceand quitting on me.
my train_image_classifier_component is programmed to save model files to a local path which is returned (and, thanks to clearml, the path's contents are zipped uploded to the files service).

I take it that these files are also brought into pipeline tasks's local disk?
Why is that? If that is indeed what...

2 years ago
0 Hi. I Have A Few Questions About The Snippet Attached

That is a good point, I'll make sure we mention it somewhere in the docs. Any thoughts on where?

maybe in (all of) these places:
https://clear.ml/docs/latest/docs/faq
https://clear.ml/docs/latest/docs/fundamentals/task
https://clear.ml/docs/latest/docs/clearml_sdk/task_sdk

2 years ago
0 Hi. I Have A Job That Processes Images And Creates ~5 Gb Of Processed Image Files (Lots Of Small Ones). At The End - It Creates A

That job was using clearml 1.8.3 so I take it that setting max_workers to 1 would not make a difference?
Looking at the docs:
https://clear.ml/docs/latest/docs/references/sdk/dataset/#upload
they say that max_workers = number of cores but looking at the log it does seem like it's doing one chunk every 5 minutes (long time for 500mb upload for a node running in gcp...)

2 years ago
0 Bug?

Oh sure, use

they will be visible on the Dataset page on the version in question

That sounds simple enough.
Though I imagine I'd need to explicitly report every figure. Correct?

2 years ago
0 Bug?

I don't mind assigning to the task the same name that I'd assign to the dataset. I just think that the create function should expect dataset_name to be None in the case of use_current_task=True (or allow the dataset name to differ from the task name)

2 years ago
0 Bug?

here is what I do:
` try:
dataset = Dataset.get(
dataset_project=bucket_name,
dataset_name=dataset_name,
dataset_version=dataset_version,
)
print(
f"dataset found {dataset.project}/{dataset.name} v{dataset.version}\n(id: {dataset.id})"
)
return dataset
except ValueError:
pass

task = Task.current_task()
if task is None:
    task = Task.init(
        project_name=bucket_name,...
2 years ago
0 Hi. I'M Encountering A Problem With

Ooh nice.
I wasn't aware task.models["output"] also acts like a dict.
I can get the one I care about in my code with something like task.models["output"]["best_model"]
however can you see the inconsistency between the key and the name there:

2 years ago
0 Hi. I'M Encountering A Problem With

I imagine that one workaround is to
Disable automatic model uploads Perform manual model upload (with the correct name).Can you point me to how to do these?

2 years ago
0 Hi. I'M Encountering A Problem With

another weird thing:
Before my training task is done:
print(task.models['output'].keys())outputs
odict_keys(['Output Model #0', 'Output Model #1', 'Output Model #2'])
after task.close()
I can do:
task = Task.get_task(task_id) for i in range(100): print(task.models["output"].keys())which prints
odict_keys(['Output Model #0', 'Output Model #1', 'Output Model #2'])in the first iteration
and prints the file names in the latter iterations:
` od...

2 years ago
0 Hi. I Am Experimenting With

I'm on clearml==1.6.3rc1

2 years ago
0 I Have 5 Unarchived Pipeline Runs That Were Defined With This Decorator:

In fact, all my projects seems empty of tasks.

2 years ago
0 Hi. I Am Experimenting With

TimelyPenguin76 , this turned out to be the reason I was having locking issues https://clearml.slack.com/archives/CTK20V944/p1658761943458649 :
SweetBadger76 , CostlyOstrich36 : I've attempted essentially the same thing before https://clearml.slack.com/archives/CTK20V944/p1657124102133519 and I thought it had worked in the past so I'm not sure why it is failing me now.

2 years ago
11 months ago
0 Hi. I'M Using Clearml Agent 1.16.1 My Code Is Running A Multi-Process Pool With "Spawn" (See

Oh, cool. So would this then report the activities of the spawned processes to the same task as that of the spawning process?

11 months ago
0 Is There Some Built-In Way In Clearml To Trigger Further Action On Task Fail (Or Pipeline Fail)?

There may be cases where failure occurs before my code starts to run (and, perhaps, after it completes)

2 years ago
0 Hi I'M Looking Into How Clearml Supports Datasets And Dataset Versioning And I'M A Bit Confused. Is Dataset Versioning Not Supported At All In The Non-Enterprise Or Is Versioning Available By A Different Mechanism? I See That

This idea seems to work.
I tested this for a scenario where data is periodically added to a dataset and, to "version" the steps, I create a new dataset with the old as parent:
To do so, I split a set of image files into separate folders (pets_000, pets_001, ... pets_015), each with 500 image files
I then run the code here to make the datasets.

2 years ago
0 Hi I'M Looking Into How Clearml Supports Datasets And Dataset Versioning And I'M A Bit Confused. Is Dataset Versioning Not Supported At All In The Non-Enterprise Or Is Versioning Available By A Different Mechanism? I See That

console output shows uploads of 500 files on every new dataset. The lineage is as expected, each additional upload is the same size as the previous ones (~50mb) and Dataset.get on the last dataset's ID retreives all the files from the separate parts to one local folder.
Checking the remote storage location (gs://) shows artifact zip files, each with 500 files

2 years ago
0 Hi. I Have A Few Questions About The Snippet Attached

Re
re-running this code produces the same printoutsI guess repeatable behaviour is a great default to have for, well, repeatability 🙂

I'm able to "randomize" my results by adding a seed pipeline argument and calling random.seed(seed)
within the pipeline and component. Results then change with change of seed.

I think most veteran ML practitioners are bitten at some point by randomising when they shouldn't and not randomising when they should. It would be nice to have some docu...

2 years ago
0 Hi. I Have A Few Questions About The Snippet Attached

Thanks,

Just to be clear, you are saying the "random" results are consistent over runs ?

yes !
By re-runs I mean re-running this script (not cloning the pipeline)

2 years ago
0 Hi. First Time User Here

In case anyone else is interested. We found two alternative solutions:
Repeating the first steps but from within a Docker container ( docker run -it --rm python:3.9 bash ) worked for me.alternatively
The example tasks (or at least those I've tried) that appear in the clear ml examples within a new workspace have clearml==0.17.5 (an old clearml version) listed in "INSTALLED PACKAGES". Updating the clearml package within the task to 1.5.0 let me run the clear-ml agent daemon lo...

2 years ago
0 Hi. I Have A Job That Processes Images And Creates ~5 Gb Of Processed Image Files (Lots Of Small Ones). At The End - It Creates A

Q: is there an equivalent env var for sdk.google.storage.pool_connections/pool_maxsize ? My jobs are running remotely and not within a clearml agent at the moment so they get clearml config through env vars.

2 years ago
0 Hi. I Have A Job That Processes Images And Creates ~5 Gb Of Processed Image Files (Lots Of Small Ones). At The End - It Creates A

I ran another version of the above code where
output_uri="./random_dataset_local_target"
(i.e. db target on local disk instead of gcp).
I still see large memory usage.
I also find it worrisome that while generating the random dataset and writing it to disk took under 3 minutes, generating the hash took 9 minutes and saving the files to a dataset target in an adjacent folder took 30 minutes (10 times longer than writing the original files)! Simply copying the files to an adjacent folde...

2 years ago
0 Another Question On The Topic Of How A Remote Execution Of A Pipeline Kills The Calling Process (Previously Discussed

You can have

parents

as one of the

@PipelineDecorator.component

args. The step will be executed only after all the

parents

are executed and completed

Is there an example of using parents some place? Im not sure what to pass and also, how to pass a component from one pipeline that was just kicked off to execute remotely (which I'd like to block on) to a component of the next pipeline's run

2 years ago
0 Another Question On The Topic Of How A Remote Execution Of A Pipeline Kills The Calling Process (Previously Discussed

I've also not figured out how to modify the examples above to wait for one pipline to end before the next begins

2 years ago
Show more results compactanswers