Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi. I Have A

Hi.
I have a problem accessing repo code in pipeline components running in an AWS autoscaler (first attempts at doing this)

My local clearml.conf file has agent's git_user and git_pass defined as in my https://clearml.slack.com/archives/CTK20V944/p1658150427357609
(but the file is not included in my repo)

The agent running the pipeline task has access to the repo code:
subprocess.run(["tree", "."]) shows a image_classifier_training folder with my repo code and
from image_classifier_training import pipeline_functions executes within the pipline agent without having specified image_classifier_training as a package anywhere.

so far - so good.

The first component that the pipeline calls also has a
from image_classifier_training import pipeline_functions
however, that results in
ModuleNotFoundError: No module named 'image_classifier_training
Performing subprocess.run(["tree", "."]) in the component call shows just the component function name as a file (I make components with decorators) and nothing else in that local path.
. └── make_new_dataset_component.py
whereas the pipeline log has lines:
cloning: git@github.com:shpigi/clearml_evaluation.git Using user/pass credentials - replacing ssh url 'git@github.com:shpigi/clearml_evaluation.git' with https url ' '
Those lines do not appear in the component's log

  
  
Posted 2 years ago
Votes Newest

Answers 7


also, whereas the pipeline agent's log has:
Executing task id [7a0ad1fb243a4ff3b9e6c477442ded4a]: repository = git@github.com:shpigi/clearml_evaluation.git branch = main version_num = e045904094cf2f4fa61ce92f7b91682f5de64ab8
The component agent's log has:
Executing task id [90de043e354b4b28a84d5cc0788fe63c]: repository = branch = version_num =

  
  
Posted 2 years ago

Yes. I thought this happened automagically with the current git repo when I send a pipeline for execution from my local python environment. Shouldn't it?
It seems to have happened with the agent running the pipeline task.

I'll try adding repo and repo_branch to the pipeline.component decorator

  
  
Posted 2 years ago

I'm still unclear on why cloning the repo in use happens automatically for the pipeline task and not for component tasks.

I think in the pipeline it was the original default, but it turns out for a lot of users this was not their defualt use case ...
Anyhow you can also pass repo="." which will load + detect the repo in the execution environemtn and automatically fill it in

  
  
Posted 2 years ago

Are you saying this component should pull a specific git repo?
PipelineDecorator.component( ..., )seems like there is no reference to a specific repo (arguments repo and repo_branch etc are missing) is that correct?

  
  
Posted 2 years ago

in order for the autoscaler to access your git , in the wizard you have to provide the git user/token

git_pass has the token
Perhaps I should have mentined that I start the AWS autoscaler with the https://app.clear.ml/applications/aws-autoscaler/ .

Hmm, how does the decorator of the component looks like ? meaning did you specify a repo/branch/commit there

Neither my pipeline decorator not my component specify any repos:

# pipeline @PipelineDecorator.pipeline( name="fastai_image_classification_pipeline", project="lavi-testing", version="0.2", multi_instance_support=True, )
# component @PipelineDecorator.component( return_values=["the_dataset"], cache=False, task_type=TaskTypes.data_processing, packages=[ "clearml", "tensorboard_logger", "timm", "fastai", "torch==1.11.0", "torchvision==0.12.0", "protobuf==3.19.*", "tensorboard", "google-cloud-storage>=1.13.2", ], )
Yet the agent running the pipeline task is pulling the repo just fine..

  
  
Posted 2 years ago

Hi PanickyMoth78

My local

clearml.conf

file has agent's

git_user

and

git_pass

defined as in my

in order for the autoscaler to access your git , in the wizard you have to provide the git user/token

The component agent's log has:

Executing task id [90de043e354b4b28a84d5cc0788fe63c]: repository = branch = version_num =Hmm, how does the decorator of the component looks like ? meaning did you specify a repo/branch/commit there ?

  
  
Posted 2 years ago

AgitatedDove14
Adding adding repo and repo_branch to the pipeline.component decorator worked (and I can move on to my next issue 🙂 ).
I'm still unclear on why cloning the repo in use happens automatically for the pipeline task and not for component tasks.

  
  
Posted 2 years ago