Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Profile picture
PanickyMoth78
Moderator
33 Questions, 163 Answers
  Active since 10 January 2023
  Last activity 8 months ago

Reputation

0

Badges 1

162 × Eureka!
0 Votes
7 Answers
216 Views
0 Votes 7 Answers 216 Views
Hi. I have a problem accessing repo code in pipeline components running in an AWS autoscaler (first attempts at doing this) My local clearml.conf file has ag...
one year ago
0 Votes
3 Answers
214 Views
0 Votes 3 Answers 214 Views
Hi. First time user here 👋 I have experienced a problem following the getting started documentation. I opened an account on https://app.clear.ml/ I then fol...
one year ago
0 Votes
9 Answers
205 Views
0 Votes 9 Answers 205 Views
Hi. Help 🥺 I have a clearml.Datase which I can't get
one year ago
0 Votes
2 Answers
211 Views
0 Votes 2 Answers 211 Views
Hi. I'm using @PipelineDecorator.component to define a task from a function (to run in a pipeline) I'd like to get the task object within this function so th...
one year ago
0 Votes
25 Answers
228 Views
0 Votes 25 Answers 228 Views
Autoscaler parallelization issue: I have an AWS Autoscaler set up with a resource that has a max of 3 instances assigned to the default queue I've given it a...
one year ago
0 Votes
1 Answers
200 Views
0 Votes 1 Answers 200 Views
suppose I use a pipeline decorator to define a pipeline: @PipelineDecorator.pipeline(name='my-pipeline', project='my-project', version='0.2') def my_pipeline...
one year ago
0 Votes
6 Answers
225 Views
0 Votes 6 Answers 225 Views
Is there some built-in way in clearml to trigger further action on task fail (or pipeline fail)?
one year ago
0 Votes
8 Answers
220 Views
0 Votes 8 Answers 220 Views
Hi (again... sorry for asking so many questions) Question about using google cloud storage in a clearml agent running in AWS ec2 instance. my clearml.conf ha...
one year ago
0 Votes
3 Answers
218 Views
0 Votes 3 Answers 218 Views
Hi. Shoulf this command succeed in the presence of project lavi-testing and absence of dataset tmp_datset within it? from clearml import Dataset tmp_dataset ...
one year ago
0 Votes
14 Answers
262 Views
0 Votes 14 Answers 262 Views
Hi. I have a job that processes images and creates ~5 GB of processed image files (lots of small ones). At the end - it creates a clearml.Dataset and perform...
9 months ago
0 Votes
14 Answers
221 Views
0 Votes 14 Answers 221 Views
Hi there. I'm trying to switch pipeline code from a local run using PipelineDecorator.run_locally()to a slightly-less-local run using PipelineDecorator.set_d...
one year ago
0 Votes
9 Answers
232 Views
0 Votes 9 Answers 232 Views
Hi. I have a question about pipelines and their generated dependency graphs. I took the code of the clearml pipeline from decorator example: https://github.c...
one year ago
0 Votes
7 Answers
211 Views
0 Votes 7 Answers 211 Views
Hi. I am experimenting with clearml.Dataset and encountering an error. LockException: [Errno 11] Resource temporarily unavailable In my experiment, I make a ...
one year ago
0 Votes
2 Answers
254 Views
0 Votes 2 Answers 254 Views
Hi. Suppose I want to report on what my task has done by having it generate a markdown (.md) file with links to some "local" figure files. looking at the rep...
10 months ago
0 Votes
20 Answers
238 Views
0 Votes 20 Answers 238 Views
task struck at task.flush(wait_for_uploads=True) : I've been running a model training task - a variation on this clearml dataset example: https://github.com/...
10 months ago
0 Votes
11 Answers
219 Views
0 Votes 11 Answers 219 Views
Hi. I have a few questions about the snippet attached re-running this code produces the same printouts... I chose 47 out of 100 in the pipeline ... I chose 8...
one year ago
0 Votes
2 Answers
267 Views
0 Votes 2 Answers 267 Views
I have a training task that auto-magically saves a model for me to GCS task = Task.init( project_name=project_name, task_name=f"Image classification training...
10 months ago
0 Votes
25 Answers
226 Views
0 Votes 25 Answers 226 Views
Hi. I'm running this little pipeline: from clearml.automation.controller import PipelineDecorator from clearml import TaskTypes @PipelineDecorator.component(...
one year ago
0 Votes
8 Answers
250 Views
0 Votes 8 Answers 250 Views
one year ago
0 Votes
14 Answers
203 Views
0 Votes 14 Answers 203 Views
Bug? dataset name is ignored if use_current_task=True
10 months ago
0 Votes
16 Answers
280 Views
0 Votes 16 Answers 280 Views
Hi. Question about Dataset upload errors: When uploading a clearml.Dataset created with output_uri=" gs://lavi_test/datasets after adding 20 files of size 50...
gcp
10 months ago
0 Votes
13 Answers
224 Views
0 Votes 13 Answers 224 Views
Another question on the topic of how a remote execution of a pipeline kills the calling process (previously discussed https://clearml.slack.com/archives/CTK2...
one year ago
0 Votes
3 Answers
215 Views
0 Votes 3 Answers 215 Views
one year ago
0 Votes
1 Answers
242 Views
0 Votes 1 Answers 242 Views
11 months ago
0 Votes
22 Answers
237 Views
0 Votes 22 Answers 237 Views
Hi. I'm encountering a problem with model.name At least, for models that where auto-magically uploaded. I see it in my own code but you can see it if you run...
9 months ago
0 Votes
30 Answers
211 Views
0 Votes 30 Answers 211 Views
Hi. I'd like to try the GCP autoscaler. What permissions does the service account that I provide to clearml need? (and what GCP API should I enable in the GC...
one year ago
0 Votes
22 Answers
299 Views
0 Votes 22 Answers 299 Views
I started two pipelines (using AWS autoscaler in app.clear.ml ). The pipelines ran concurrently, using the same pipeline code. Both failed in the same compon...
one year ago
0 Votes
7 Answers
232 Views
0 Votes 7 Answers 232 Views
Hi I'm looking into how clearml supports datasets and dataset versioning and I'm a bit confused. Is dataset versioning not supported at all in the non-enterp...
one year ago
0 Votes
3 Answers
239 Views
0 Votes 3 Answers 239 Views
one year ago
0 Votes
2 Answers
222 Views
0 Votes 2 Answers 222 Views
Hi. I've noticed that my clearml.conf has both: agent.git_user="" agent.git_pass=""and agent { ... git_user: "" git_pass: "" ... }What's the difference? Shou...
one year ago
Show more results questions
0 Hi There. I'M Trying To Switch Pipeline Code From A Local Run Using

I'm on clearml 1.6.2
The jupyter notebook service and two clear-ml agents ( version1.3.0, one in queue "default" and one in queue "services" and with --cpu-only flag) ) are all running inside a docker container

one year ago
0 Hi There. I'M Trying To Switch Pipeline Code From A Local Run Using

Hmm interesting, so like a callback?!

like https://github.com/allegroai/clearml/blob/bca9a6de3095f411ae5b766d00967535a13e8401/examples/pipeline/pipeline_from_tasks.py#L54-L55 pipe-step level callbacks? I guess that mechanism could serve. Where do these callbacks run? In the instantiating process? If so, that would work (since the callback function can be any code I wish, right?)

I might want to dispatch other jobs from within the same process.

This is actually something t...

one year ago
0 Hi. I'M Encountering A Problem With

Right. Thanks.
With several models saved by the training process (whose code is not task-aware) I suspect that doing the update call after training completed will only update the last of the uploaded models.
I'm currently looking at a workaround where:
I disable auto saving by https://clear.ml/docs/latest/docs/clearml_sdk/task_sdk/#automatic-logging Manually upload the models Manually register the models with https://github.com/allegroai/clearml/blob/cf7361e134554f4effd939ca67e8ecb2345b...

9 months ago
0 Autoscaler Parallelization Issue: I Have An Aws Autoscaler Set Up With A Resource That Has A Max Of 3 Instances Assigned To The

Where was it running?

this message appears in the pipeline task's log. It is preceded by lines that reflect the storage manager downloading a corresponding zip file

I take it that these files are also brought into pipeline tasks's local disk?

Unless you changed the object, then no, they should not be downloaded (the "link" is passed)

The object is run_model_path
I don't seem to be changing it. I just pass it along from the training component to the evaluation compo...

one year ago
0 Autoscaler Parallelization Issue: I Have An Aws Autoscaler Set Up With A Resource That Has A Max Of 3 Instances Assigned To The

Two values:
`
@PipelineDecorator.component(
return_values=["run_model_path", "run_tb_path"],
cache=False,
task_type=TaskTypes.training,
packages=[
"clearml",
"tensorboard_logger",
"timm",
"fastai",
"torch==1.11.0",
"torchvision==0.12.0",
"protobuf==3.19.*",
"tensorboard",
"google-cloud-storage>=1.13.2",
],
repo="git@github.com:shpigi/clearml_evaluation.git",
repo_branch="main",
)
def train_ima...

one year ago
0 Autoscaler Parallelization Issue: I Have An Aws Autoscaler Set Up With A Resource That Has A Max Of 3 Instances Assigned To The

erm,
this parallelization has led to the pipeline task issuing a bunch of:
model_path/run_2022_07_20T22_11_15.209_0.zip , err: [Errno 28] No space left on deviceand quitting on me.
my train_image_classifier_component is programmed to save model files to a local path which is returned (and, thanks to clearml, the path's contents are zipped uploded to the files service).

I take it that these files are also brought into pipeline tasks's local disk?
Why is that? If that is indeed what...

one year ago
0 Autoscaler Parallelization Issue: I Have An Aws Autoscaler Set Up With A Resource That Has A Max Of 3 Instances Assigned To The

Note that the same models files were previously also generated by a non-paralelized version of the same pipeline without the out-of-space error but a storage manager was downloading zip files in that version as well (maybe these files were downloaded and removed as the object reference counts went to 0?)

one year ago
0 Hi There. I'M Trying To Switch Pipeline Code From A Local Run Using

Thanks ! 🎉
I'll give it a try.
I think that clearml should be able to do parameter sweeps using pipelines in a manner that makes use of parallelisation.
If that's not happening with the new RC, I wonder how I would do a parameter sweep within the pipelines framework.

For example - how would this task-based example be done with pipelines?
https://github.com/allegroai/clearml/blob/master/examples/automation/manual_random_param_search_example.py

I'm thinking of a case where you want t...

one year ago
0 Hi. I'M Encountering A Problem With

I imagine that one workaround is to
Disable automatic model uploads Perform manual model upload (with the correct name).Can you point me to how to do these?

9 months ago
0 Hi. I'M Encountering A Problem With

BTW:

If I try to find the right model in the

task.models["output"]

(this time there is just one but in my code there may be several) it appears with the

(see other attached screenshot).

What would make sense here ? (I have to be honest I'm not sure).

If the model was saved with a file name (is that the trigger for auto-upload?), I think it makes sense for the model name to match the file name (not the task name), especially when there may be ...

9 months ago
0 Hi. I Have A Question About Pipelines And Their Generated Dependency Graphs. I Took The Code Of The Clearml Pipeline From Decorator Example:

I imagine that these phantom dependencies will prevent parallelization. Is there a workaround?

one year ago
0 Hi There. I'M Trying To Switch Pipeline Code From A Local Run Using

What I think would be preferable is that the pipeline be deployed and that the python process that deployed it were allowed to continue on to whatever I had planned for it to do next (i.e. not exit)

one year ago
0 Hi There. I'M Trying To Switch Pipeline Code From A Local Run Using

first, thanks for having these discussions. I appreciate this kind of support is an effort 🙏
Yes. i perfectly understand that once a pipeline job (or a task) is sent off in this manner, it executes separately (and, most likely in a different machine) from the process that instantiated it.
I still feel strongly that such a command should not be thought of as a fire and exit operation. I can think of several scenarios where continued execution of the instantiating process is desired:
I ...

one year ago
0 Hi There. I'M Trying To Switch Pipeline Code From A Local Run Using

actually, re-running pipeline_from_decorator.py a second time (and a third time) from the command line seem to have executed without the that ValueError so maybe that issue was some fluke.
Nevertheless, those runs exit prior to line
print('process completed')
and I would definitely prefer the command executing_pipeline to not kill the process that called it.
For example, maybe, having started the pipeline I'd like my code to also report having started the pipeline to som...

one year ago
0 Hi. I'D Like To Try The Gcp Autoscaler.

did you mean that I was running in CPU mode? I'll tried both but I'll try cpu mode with that base docker image

one year ago
0 Hi. First Time User Here

thanks, I'll DM

one year ago
0 Hi. I'D Like To Try The Gcp Autoscaler.

Is there any chance the experiment itself has a docker image specified?

It does not as far as I know. The decorators do not have docker fields specified

one year ago
0 Hi. I'D Like To Try The Gcp Autoscaler.

I'll give it a try.
And if I wanted to support GPU in the default queue, are you saying that I'd need a different machine from the n1-standard-1 ?

one year ago
0 Hi. I'M Running This Little Pipeline:

I'm connecting to the hosted clear.ml
packages in use are:
# Python 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0] clearml == 1.6.2 fastai == 2.7.5
in case it matters, I'm running this code in a jupyter notebook within a docker container (to keep things vell isolated). The /data path is volume mapped to my local filesystem (and, in fact, already contains the dataset files, so the fastai call to untar_data should see the data there and return immediately)
That same make_data fu...

one year ago
0 Hi. I'M Running This Little Pipeline:

The pipeline eventually completed after ~20 minutes and the log shows it has downloaded a 755mb file.
I can also download the zip file from the artifacts tab for the component now.
Why is the data being up/down loaded? Can I prevent that?
I get that clearml likes to take good care of my data but I must be doing something wrong here as it doesn't make sense for a dataset to be uploaded to files.clear.ml .

one year ago
0 Hi. I'M Running This Little Pipeline:

Note that if I change the component to return a regular meaningless string - "mock_path" , the pipeline completes rather quickly and the dataset is not uploaded.

one year ago
0 Hi. I'M Running This Little Pipeline:

Thanks TimelyPenguin76 .
From your reply I understand that I have control over what the destination is but that all files generated in a task get transferred regardless of the return_values decorator argument. Is that correct? Can I disable auto-save of artifacts?
Ideally, I'd like to have better control over what gets auto-saved. E.g. I'm happy for tensorboard events to be captured and shown in clearml and for matplotlib figures to be uploaded (perhaps to gcs) but I'd like to avoid ...

one year ago
0 Hi (Again... Sorry For Asking So Many Questions) Question About Using Google Cloud Storage In A Clearml Agent Running In Aws Ec2 Instance. My

My local environment has clearml version 1.6.3rc0
and agents in aws were started with the AWS Autoscaler which has no explicit place for google credentials.

I see a place for Additional ClearML Configuration in the AWS autoscaler UI which I suspect may help but I don't see how I can pass a secrets file along with my agent.

one year ago
0 Hi (Again... Sorry For Asking So Many Questions) Question About Using Google Cloud Storage In A Clearml Agent Running In Aws Ec2 Instance. My

For anyone following, you can "inject" a credentials json file for a google cloud service account so at to get access to your google cloud storage from agents on aws ec2 instances that are managed by the AWS autoscaler by providing the following in the ADDITIONAL CLEARML CONFIGURATION when starting the autoscaler:
` sdk.google.storage.credentials_json: "/root/gs.cred"
sdk.google.storage.project: "<my-gcp-project-id>"
files {
gsc {
contents: """<copy-paste the contents of yo...

one year ago
0 Hi. I'D Like To Try The Gcp Autoscaler.

Trying to switch to a resources using gpu-enabled VMs failed with that same error above.
Looking at spawned VMs, they were spawned by the autoscaler without gpu even though I checked that my settings ( n1-standard-1 and nvidia-tesla-t4 and https://console.cloud.google.com/compute/imagesDetail/projects/ml-images/global/images/c0-deeplearning-common-cu113-v20220701-debian-10?project=ml-tooling-test-external image for the VM) can be used to make vm instances and my gcp autoscaler...

one year ago
0 Hi There. I'M Trying To Switch Pipeline Code From A Local Run Using

Thanks for the fix and the mock HPO example code !
Pipeline behaviour with the fix is looking good.
I see the point about changes to data inside the controller possibly causing dependencies for step 3 (or, at least, making it harder for the interpreter to know).

one year ago
0 Hi. I'D Like To Try The Gcp Autoscaler.

Hi TimelyPenguin76
Thanks for working on this. The clearml gcp autoscaler is a major feature for us to have. I can't really evaluate clearml without some means of instantiating multiple agents on GCP machines and I'd really prefer not to have to set up a k8 cluster with agents and manage scaling it myself.

I tried the settings above with two resources, one for default queue and one for the services queue (making sure I use that image you suggested above for both).
The autoscaler started up...

one year ago
Show more results compactanswers