Based on this : it feels like S3 is supported

Are you running within a zero-trust environment like ZScaler ?
Feels like your issue is not ClearML itself, but issue with https/SSL and certificate from your zero-trust system

Should I put that in the clearml.conf file?

About the caching: how does it work ? ClearML maintain it own cache and monitor if any of you code changes? Even code that get change inside an import ?

Last time I tried docker compose, elastic take a lot of RAM !!
You need to limit its RAM usage with mem_limit :

[...]
  elasticsearch:
    networks:
      - backend
    container_name: clearml-elastic
    mem_limit: 2g
    environment:
      bootstrap.memory_lock: "true"
      cluster.name: clearml
[...]

None

my code looks like this :

    parser = argparse.ArgumentParser()
    parser.add_argument('-c', '--config-file', type=str, default='train_config.yaml',
                        help='train config file')
    parser.add_argument('-t', '--train-times', type=int, default=1,
                        help='train the same model several times')
    parser.add_argument('--dataset_dir', help='path to folder containing the preped dataset.', required=True)
    parser.add_argument('--backup', action='s...

@<1523701087100473344:profile|SuccessfulKoala55> Thanks. Manage to get it working now with

export REQUESTS_CA_BUNDLE=/etc/ssl/certs/zscaler.crt

(Ubuntu system)

what about having 2 agents, one on each GPU, on the same machine, serving the same queue ? So that when you enqueue, which ever agent (thus GPU) available will take the new task

From SDK ?

if you want to replace MLflow by ClearML: do it !! It's like "Should I use sandal or running shoes for my next marathon ..."
Let your user try ClearML, and I am pretty sure all of them will want to swap over !!!

the config that I mention above are the clearml.conf for each agent

@<1523701087100473344:profile|SuccessfulKoala55> Actually it failed now: failed to talked to our storage in Azure:

ClearML Task: created new task id=c47dd71dea2f421db05647a21d78ed26



2024-01-25 21:45:23,926 - clearml.storage - ERROR - Failed uploading: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)
2024-01-25 21:46:48,877 - clearml.storage - WARNING - Storage helper problem for .clearml.0149daec-7a03-4853-a0cd-a7e2b295...

no. I set apo.file_server to the None in Both the remote agent clearml.conf and my local clearml.conf
In which case, both case where the code is ran from local or remote, will store metrics to cloud storage

so what was the solution/hack then ?

CLEARML_AGENT_SKIP_PIP_VENV_INSTALL need to be a path

so in your case, in the clearml-agent conf, it contains multiple credential, each for different cloud storage that you potential use ?

please share your .service content too as there are a lot of way to "spawn" in systemd

Done: None

You are using CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL the wrong way

all good. Just wanted to know in case I missed it

but afaik this only works locally and not if you run your task on a clearml-agent!

Isn;t the agent using the same clearml.conf ?
We have our agent running task and uploading everything to Cloud. As I said, we don;t even have file server running

I think ES use a greedy strategy where it allocate first then use it from there ...

I also have the same issue. Default argument are fine but all supplied argument in command line become duplicated !

If you are using multi storage place, I don't see any other choice than putting multi credential in the conf file ... Free or Paid Clearml Server ...

some clearml cache folder

(wrong tab sorry :P)

Feels like Docker, Kubernetes is more fit for that purpose ...