Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
In Pipelines. Is It Possible To Inject A Requierment.Txt Such That The Executing Node Will Install Before Running Tasks?

In Pipelines.

Is it possible to inject a requierment.txt such that the executing node will install before running tasks?

  
  
Posted 2 years ago
Votes Newest

Answers 28


from clearml import PipelineController

  
  
Posted 2 years ago

based on what reqeuirment.txt manifest?

  
  
Posted 2 years ago

I think I am missing something

  
  
Posted 2 years ago

the question how does ClearML know to create env and what files does it copy to the task

Either automatically detecting the packages in requirements.txt OR using the packages listed in the task itself

  
  
Posted 2 years ago

or how can I add some libraries I'd like it to pip install in the new environment

  
  
Posted 2 years ago

locally and remotely

  
  
Posted 2 years ago

if this was possible we wouldn't need pip

  
  
Posted 2 years ago

how this is possible?

  
  
Posted 2 years ago

we have a diff name for this file in the repo

  
  
Posted 2 years ago

Yep Correct

  
  
Posted 2 years ago

nice! Thank you

  
  
Posted 2 years ago

and just making sure - by pipeline we're talking about the ClearML pipelines, correct?
https://clear.ml/docs/latest/docs/references/sdk/automation_controller_pipelinecontroller

  
  
Posted 2 years ago

Have you run experiments with the SDK? i.e added Task.init()

  
  
Posted 2 years ago

so it looks for requirement.txt?

  
  
Posted 2 years ago

The one sitting in the repository

  
  
Posted 2 years ago

I have a simple pipline that works

  
  
Posted 2 years ago

now the question how to make the production stuff to work

  
  
Posted 2 years ago

PIP Mode

By default, ClearML Agent works in PIP Mode, in which it uses

as the package manager. When ClearML runs, it will create a virtual environment (or reuse an existing one, see

). Task dependencies (Python packages) will be installed in the virtual environment.

how does it know what are the dependencies of a task?

  
  
Posted 2 years ago

if a==True: import torch else: import tensorflow

  
  
Posted 2 years ago

it's an undecidable problem

  
  
Posted 2 years ago

the only way any computer could figure this out is by running it

  
  
Posted 2 years ago

Task.add_requirements

  
  
Posted 2 years ago

Hi AverageRabbit65 , can you elaborate on what you're trying to do?
ClearML-Agent will automatically create a venv and install everything

  
  
Posted 2 years ago

And the pipeline runs with agents or locally?

  
  
Posted 2 years ago

I'm sorry. I think I wrote something wrong. I'll elaborate:
The SDK detects all the packages that are used during the run - The Agent will install a venv with those packages.
I think there is also an option to specify a requirements file directly in the agent.

Is there a reason you want to install packages from a requirements file instead of just using the automatic detection + agent?

  
  
Posted 2 years ago

the question how does ClearML know to create env and what files does it copy to the task
/homes/yossi/.clearml/venvs-builds/3.7/

  
  
Posted 2 years ago

consider this case:

  
  
Posted 2 years ago
945 Views
28 Answers
2 years ago
one year ago
Tags