In the web UI, in the queue/worker tab, you should see a service queue and a worker available in that queue. Otherwise the service agent is not running. Refer to John c above

I don't have it so I don't know how things are setup and how to pass on credentials in this case

So I now just need to find a way to not populate Installed Package in the first place

Oh, so you mean CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/path/to/my/venv/bin/python3 ??

Found a trick to have empty Installed package:
clearml.Task.force_requirements_env_freeze(force=True,requirements_file="/dev/null")
Not sure if this is the right way or not ...

@<1523701070390366208:profile|CostlyOstrich36> Is there a way to tell clearml to not try to detect the Installed package ?

oh, looks like I need to empty the Installed Package before enqueue the cloned task

But then how did the agent know where is the venv that it needs to use?

Should I put that in the clearml.conf file?

So I tried:

CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/data/hieu/opt/python-venv/fastai/bin/python3.10
clearml-agent  daemon  --queue no_venv

Then enqueue a cloned task to no_venv
It is still trying to create a venv (and fail):

[...]
tag = 
docker_cmd = 
entry_point = debug.py
working_dir = apple_ic
created virtual environment CPython3.10.10.final.0-64 in 140ms
  creator CPython3Posix(dest=/data/hieu/deleteme/clearml-agent/venvs-builds/3.10, clear=False, no_vcs_ignore=False, gl...

Set that env var in the terminal before running the agent ?

Looks like your issue is not that ClearML is not tracking your changes but more about your Configuration is overwrriten.
This often happen to me. The way I debug this is put a lot of print statement along the code to track when the Configuration is overwriten and narrow down why. print statement will show up in the Console tab.

you mean this dump section ?

Nevermind: None

By default, the File Server is not secured even if Web Login Authentication has been configured. Using an object storage solution that has built-in security is recommended.

My bad

thanks for all the pointer ! I will try to have a good play around

I don;t think there is a "kill task" code. By principle, in Linux, as a parent process, ClearML agent launch the training process. When a parent process is terminated, the linux kernel will, in most of the case, kill all child processes, including your training process.
There may be some way to resume a task from ClearML agent when it restart, but I don;t think that is the default behavior

Done: None

so it's not suppose to say "illegal output destination ..." ?

If the agent is the one running the experiment, very likely that your task will be killed.
And when the agent come back, immediately or later, probably nothing will happen. It won't resume ...

but when I spin up a new server in the cloud, that server will have it's own mongodb and that will be empty no ?

Are you talking about this: None
It seems to not doing anything aboout the database data ...

I am more curious about how to migrate all the information stored in the local clearml server to the clearml server in the cloud

nevermind, all the database files are in data folder

What about migrating existing expriment in the on prem server?

Anything else to migrate ?

I understand to from the agent, point of view, I just need to update the conf file to use new credential and new server address.

I will try it. But it's a bit random when this happen so ... We will see

you can upload the df as artifact.
Or the statistics as a DataFrame and upload as artifact ?

if you want plot, you can simply generate plot with matplotlib and clearml can upload them in the Plot or Debug Sample section

None