would love to hear your thoughts

solved with Alon's help 1x1

based on what reqeuirment.txt manifest?

I created a worker that listens to the queue, but the worker doesn't pull the tasks (they are pending)

PIP Mode

By default, ClearML Agent works in PIP Mode, in which it uses

as the package manager. When ClearML runs, it will create a virtual environment (or reuse an existing one, see

). Task dependencies (Python packages) will be installed in the virtual environment.

how does it know what are the dependencies of a task?

or how can I add some libraries I'd like it to pip install in the new environment

I think I am missing something

Yep Correct

the only way any computer could figure this out is by running it

consider this case:

the question how does ClearML know to create env and what files does it copy to the task
/homes/yossi/.clearml/venvs-builds/3.7/

if a==True: import torch else: import tensorflow

nice! Thank you

TimelyPenguin76

thanks again... I have tried what you have suggested. It does inject the dir, but not cloning the whole content.

os.system("tree") from within demo.py pipeline step function:

└── test-demo-clearml.git ├── demo.py ├── __init__.py ├── local_dir └── step_one_task.pyvs.
from the actual tree (under ~/clearml_using_local_lib )
` tree
.
├── demo.py
├── init.py
└── local_dir
├── hello.py
├── init.py
└── pycache
└── __i...

Missing the last piece of the puzzle I believe.

Indeed locally it does work if I run pipe.start_locally

ClearML Task: created new task id=bda6736172df497a83290b2927df28a2 ClearML results page: 2022-06-30 12:35:16,119 - clearml.Task - INFO - No repository found, storing script code instead ClearML pipeline page: `
2022-06-30 12:35:25,434 - clearml.automation.controller - INFO - Node "step_two" missing parent reference, adding: {'step_one'}
2022-06-30 12:35:25,436 - clearml.automation.controller - INFO - Node "step_three" missing parent reference, adding: {'step_two'}
2022-06-30 12:35...

thank you, guys! I think now it works! Amazing step-by-step support! This is sublime!

for future ref., this is a summary of what I have done:

create a project on clearml webUI create a queue on clearml webUI run an agent /homes/yosefhaie/.conda/envs/devops/bin/clearml-agent daemon --create-queue --queue <queue-name> use this test script:` from clearml import PipelineController

We will use the following function an independent pipeline component step

notice all package imports ins...

ran clearml-agent daemon --queue rudolf --detached

from clearml import PipelineController

Task.add_requirements

if this was possible we wouldn't need pip

it's an undecidable problem

Still missing some info about how to make the worker actually pull the task from the queue.

we have a diff name for this file in the repo

so it looks for requirement.txt?

how this is possible?

locally and remotely

I have a simple pipline that works

you are amazing guys! thank yall