depend on how the agent is launched ...

@<1523701087100473344:profile|SuccessfulKoala55> Should I raise a github issue ?

CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/path/to/my/vemv/bin/python3.12 clearml-agent bla

I am wondering, should it be True be default ???

(I never played with pipeline feature so I am not really sure that it works as I imagined ...)

hard to see with your croppout here an there ...

To me the whole point of having pipeline is to have a system that "know" previous state and make "smart" decision on what should run and what not. If it's just about if then else, then code already handle all that.
And what I struggle a bit is to find doc on how it determine the existing state and how it make decision what to run. thus the initial question

So I tried:

CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/data/hieu/opt/python-venv/fastai/bin/python3.10
clearml-agent  daemon  --queue no_venv

Then enqueue a cloned task to no_venv
It is still trying to create a venv (and fail):

[...]
tag = 
docker_cmd = 
entry_point = debug.py
working_dir = apple_ic
created virtual environment CPython3.10.10.final.0-64 in 140ms
  creator CPython3Posix(dest=/data/hieu/deleteme/clearml-agent/venvs-builds/3.10, clear=False, no_vcs_ignore=False, gl...

may be specific to fastai as I cannot reproduce it with another training using yolov5

I don;t think ClearML is designed to handle secrets other than git and storage ...

None
CLEARML_AGENT_SKIP_PIP_VENV_INSTALL
CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL

in my case, I set eveything inside the container, including the agent and not using docker mode altogether.
When my container start, it start the agent inside it in "normal" mode

are you using the agent docker mode ?

I don;t think there is a "kill task" code. By principle, in Linux, as a parent process, ClearML agent launch the training process. When a parent process is terminated, the linux kernel will, in most of the case, kill all child processes, including your training process.
There may be some way to resume a task from ClearML agent when it restart, but I don;t think that is the default behavior

one specify the venv python, the other tell it to not do anything

@<1523701070390366208:profile|CostlyOstrich36> Thanks !! That's look much cleaner than task.export_task()['runtime']['gpu_type'] :D

not sure if related but clearml 1.14 tend to not "show" the gpu_type

May be create a Feature request on github ?

like for dataset_dir I would expect a single path, not an array of 2 paths duplicated

Anything else to migrate ?

it is actually in the repo root folder.

Mongo db data migration?

If you care about the local destination then you may want to use this None

inside the script that launch the agent, I set all the env need (aka disable installation with the var above)

oh, looks like I need to empty the Installed Package before enqueue the cloned task

Nice ! That is handy !!
thanks !

oh ... maybe the bottleneck is augmentation in CPU !
But is it normal that the agent don't detect the GPU count and type properly ?

you can upload the df as artifact.
Or the statistics as a DataFrame and upload as artifact ?

had you made sure that the agent inside GCP VM have access to your repository ? Can you ssh into that VM and try to do a git clone ?

So if i spin up a new clearml server in the cloud and use the same file server mount point, i will see all task and expriment that i had on the in prem server in the cloud server?