Based on this : it feels like S3 is supported

Are you running within a zero-trust environment like ZScaler ?
Feels like your issue is not ClearML itself, but issue with https/SSL and certificate from your zero-trust system

Should I put that in the clearml.conf file?

Last time I tried docker compose, elastic take a lot of RAM !!
You need to limit its RAM usage with mem_limit :

[...]
  elasticsearch:
    networks:
      - backend
    container_name: clearml-elastic
    mem_limit: 2g
    environment:
      bootstrap.memory_lock: "true"
      cluster.name: clearml
[...]

my code looks like this :

    parser = argparse.ArgumentParser()
    parser.add_argument('-c', '--config-file', type=str, default='train_config.yaml',
                        help='train config file')
    parser.add_argument('-t', '--train-times', type=int, default=1,
                        help='train the same model several times')
    parser.add_argument('--dataset_dir', help='path to folder containing the preped dataset.', required=True)
    parser.add_argument('--backup', action='s...

@<1523701087100473344:profile|SuccessfulKoala55> Thanks. Manage to get it working now with

export REQUESTS_CA_BUNDLE=/etc/ssl/certs/zscaler.crt

(Ubuntu system)

what about having 2 agents, one on each GPU, on the same machine, serving the same queue ? So that when you enqueue, which ever agent (thus GPU) available will take the new task

if you want to replace MLflow by ClearML: do it !! It's like "Should I use sandal or running shoes for my next marathon ..."
Let your user try ClearML, and I am pretty sure all of them will want to swap over !!!

@<1523701087100473344:profile|SuccessfulKoala55> Actually it failed now: failed to talked to our storage in Azure:

ClearML Task: created new task id=c47dd71dea2f421db05647a21d78ed26



2024-01-25 21:45:23,926 - clearml.storage - ERROR - Failed uploading: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)
2024-01-25 21:46:48,877 - clearml.storage - WARNING - Storage helper problem for .clearml.0149daec-7a03-4853-a0cd-a7e2b295...

no. I set apo.file_server to the None in Both the remote agent clearml.conf and my local clearml.conf
In which case, both case where the code is ran from local or remote, will store metrics to cloud storage

so what was the solution/hack then ?

CLEARML_AGENT_SKIP_PIP_VENV_INSTALL need to be a path

You are using CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL the wrong way

all good. Just wanted to know in case I missed it

but afaik this only works locally and not if you run your task on a clearml-agent!

Isn;t the agent using the same clearml.conf ?
We have our agent running task and uploading everything to Cloud. As I said, we don;t even have file server running

I think ES use a greedy strategy where it allocate first then use it from there ...

I also have the same issue. Default argument are fine but all supplied argument in command line become duplicated !

If you are using multi storage place, I don't see any other choice than putting multi credential in the conf file ... Free or Paid Clearml Server ...

some clearml cache folder

(wrong tab sorry :P)

Feels like Docker, Kubernetes is more fit for that purpose ...

while the other may need to be 1 instead of true

Solved @<1533620191232004096:profile|NuttyLobster9> . In my case:
I need to from clearml import Task very early in the code (like first line), before importing argparse
And not calling task.connect(parser)

you should know where your latest model is located then just call task.upload_artifact on that file ?

you should be able to explicitly upload a file of your choice as artefact using something like this: None

we are not using docker compose. We are deploying in Azure with each database as a standalone service

Set that env var in the terminal before running the agent ?

Oh, so you mean CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/path/to/my/venv/bin/python3 ??

None

Onprem: User management is not "live" as you need to reboot and password are hardcoded ... No permission distinction, as everyone is admin ...