Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi. I Have A

Hi.
I have a problem accessing repo code in pipeline components running in an AWS autoscaler (first attempts at doing this)

My local clearml.conf file has agent's git_user and git_pass defined as in my https://clearml.slack.com/archives/CTK20V944/p1658150427357609
(but the file is not included in my repo)

The agent running the pipeline task has access to the repo code:
subprocess.run(["tree", "."]) shows a image_classifier_training folder with my repo code and
from image_classifier_training import pipeline_functions executes within the pipline agent without having specified image_classifier_training as a package anywhere.

so far - so good.

The first component that the pipeline calls also has a
from image_classifier_training import pipeline_functions
however, that results in
ModuleNotFoundError: No module named 'image_classifier_training
Performing subprocess.run(["tree", "."]) in the component call shows just the component function name as a file (I make components with decorators) and nothing else in that local path.
. └── make_new_dataset_component.py
whereas the pipeline log has lines:
cloning: git@github.com:shpigi/clearml_evaluation.git Using user/pass credentials - replacing ssh url 'git@github.com:shpigi/clearml_evaluation.git' with https url ' '
Those lines do not appear in the component's log

  
  
Posted one year ago
Votes Newest

Answers 7


in order for the autoscaler to access your git , in the wizard you have to provide the git user/token

git_pass has the token
Perhaps I should have mentined that I start the AWS autoscaler with the https://app.clear.ml/applications/aws-autoscaler/ .

Hmm, how does the decorator of the component looks like ? meaning did you specify a repo/branch/commit there

Neither my pipeline decorator not my component specify any repos:

# pipeline @PipelineDecorator.pipeline( name="fastai_image_classification_pipeline", project="lavi-testing", version="0.2", multi_instance_support=True, )
# component @PipelineDecorator.component( return_values=["the_dataset"], cache=False, task_type=TaskTypes.data_processing, packages=[ "clearml", "tensorboard_logger", "timm", "fastai", "torch==1.11.0", "torchvision==0.12.0", "protobuf==3.19.*", "tensorboard", "google-cloud-storage>=1.13.2", ], )
Yet the agent running the pipeline task is pulling the repo just fine..

  
  
Posted one year ago

Hi PanickyMoth78

My local

clearml.conf

file has agent's

git_user

and

git_pass

defined as in my

in order for the autoscaler to access your git , in the wizard you have to provide the git user/token

The component agent's log has:

Executing task id [90de043e354b4b28a84d5cc0788fe63c]: repository = branch = version_num =Hmm, how does the decorator of the component looks like ? meaning did you specify a repo/branch/commit there ?

  
  
Posted one year ago

Yes. I thought this happened automagically with the current git repo when I send a pipeline for execution from my local python environment. Shouldn't it?
It seems to have happened with the agent running the pipeline task.

I'll try adding repo and repo_branch to the pipeline.component decorator

  
  
Posted one year ago

also, whereas the pipeline agent's log has:
Executing task id [7a0ad1fb243a4ff3b9e6c477442ded4a]: repository = git@github.com:shpigi/clearml_evaluation.git branch = main version_num = e045904094cf2f4fa61ce92f7b91682f5de64ab8
The component agent's log has:
Executing task id [90de043e354b4b28a84d5cc0788fe63c]: repository = branch = version_num =

  
  
Posted one year ago

Are you saying this component should pull a specific git repo?
PipelineDecorator.component( ..., )seems like there is no reference to a specific repo (arguments repo and repo_branch etc are missing) is that correct?

  
  
Posted one year ago

AgitatedDove14
Adding adding repo and repo_branch to the pipeline.component decorator worked (and I can move on to my next issue 🙂 ).
I'm still unclear on why cloning the repo in use happens automatically for the pipeline task and not for component tasks.

  
  
Posted one year ago

I'm still unclear on why cloning the repo in use happens automatically for the pipeline task and not for component tasks.

I think in the pipeline it was the original default, but it turns out for a lot of users this was not their defualt use case ...
Anyhow you can also pass repo="." which will load + detect the repo in the execution environemtn and automatically fill it in

  
  
Posted one year ago