what you mean by different script ?

very hard to diagnose with this tiny bit of log ...

some clearml cache folder

there is a whole discussion about it here: None

I found that if pip is upgraded to latest version 25.0.1 then the package install fine.
The question become: why does the agent downgrade pip ?

Ignoring pip: markers 'python_version < "3.10"' don't match your environment
Collecting pip<22.3
  Downloading pip-22.2.2-py3-none-any.whl.metadata (4.2 kB)
Downloading pip-22.2.2-py3-none-any.whl (2.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 3.9 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
  ...

have you try a different browser ?

I didn;t know that from the client side, you can specify the storage elsewhere than the clearML server. Good to know !
But I still want to know, if possible, to use a blob storage by default, configured on the ClearML server, and each client don't need to do that ...

To me the whole point of having pipeline is to have a system that "know" previous state and make "smart" decision on what should run and what not. If it's just about if then else, then code already handle all that.
And what I struggle a bit is to find doc on how it determine the existing state and how it make decision what to run. thus the initial question

For #2: it's a pull rather than a push system: you need to have a script that do pulling at regular interval and need to keep track what new and what not?

and just came across this: None
That sounds like what you may be looking for

So "Using env ..." take minutes without any output ?

hard to see with your croppout here an there ...

Can you paste here what inside "Installed package" to double check ?

So I tried:

CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/data/hieu/opt/python-venv/fastai/bin/python3.10
clearml-agent  daemon  --queue no_venv

Then enqueue a cloned task to no_venv
It is still trying to create a venv (and fail):

[...]
tag = 
docker_cmd = 
entry_point = debug.py
working_dir = apple_ic
created virtual environment CPython3.10.10.final.0-64 in 140ms
  creator CPython3Posix(dest=/data/hieu/deleteme/clearml-agent/venvs-builds/3.10, clear=False, no_vcs_ignore=False, gl...

if you have 2 agent serving the same queue and then send 2 task to that queue, each agent should take one task
But if you queue sequentially one task then wait until that task to finish and queue the next: then it will be random which agent will take the task. Can be the same on from the previous task
Are you saying that you have 1 agent running task, 1 agent sitting idle while there is a task waiting in the queue and no one is processing it ??

You don't need agent on your local machine.
You want an agent running on the GPU machine.
Local code will create an experiment in ClearML Server, then run up to the line remotely_execute() then stop
Once local code stop, the Clearml Server will take over and enqueue the experiment to the prescribe queue
The agent on the GPU see there is a experiment on its queue and then pull it and execute it. This time, clearml lib magic will make the code on the GPU machine, launched by the agent, run...

I use CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/path/to/my/vemv/bin/python3.12 and it work for me

@<1523701868901961728:profile|ReassuredTiger98> I found that you an set the file_server in your local clearml.conf to your own cloud storage. In our case, we use something like this in our clearml.conf:

api {
   file_server: "azure://<account>..../container"
}

All non artifact model are then store in our azure storage. In our self-hosted clearml setup, we don't even have a file server running alltogether

never tried 😛

you will need to provide more context than that if you don't want the answer: Have you try to turn it off and back on again ?

once you install manually your package inside the docker container, check that your file module_b/templates/my_template.yml is where it should be

this bug: None

Should I put that in the clearml.conf file?

Do you want to use https or ssh to do git clone ? Setting up both in the same time is confusing

so the issue is that for some reason, the pip install by the agent don't behave the same way as your local pip install ?
Have you tried to manually install your module_b with pip install inside the machine that is running clearml-agent ? Seeing your example, looks like you are even running inside docker ?

not sure how that work with Docker and machine that is not set up with ssh public key ... We will go to that path sometime in the future so I am quite interested too, on how people do it without ssh public key

I mean, what happen if I import and use function from another py file ? And that function code changes ?
Or you are expecting code should be frozen and only parameters changes between runs ?

So I tried:

import livsdk.livbatch

import clearml

clearml.Task.add_requirements("livsdk","

")
task = clearml.Task.init(project_name="hieu-test", task_name='base_config')

print("Done")

Which give me this list of Packages Installed:

# Python 3.10.10 (main, Mar 05 2023, 19:07:49) [GCC]

# Local modules found - skipping:
# livsdk == ../[REDACTED]/livsdk/__init__.py

Augmentor == 0.2.10
Pillow == 9.2.0
PyYAML == 6.0
albumentations == 1.2.1
azure_storage_blob == 12.1...

What should I put in there? What is the syntax for git package?