the config that I mention above are the clearml.conf for each agent

@<1576381444509405184:profile|ManiacalLizard2> Maybe you are using the enterprise version with the vault? I suppose the enterprise version is running differently, but I dont have experience with it.
For the open-source version, each clearml-agent is using it's own clearml.conf

right, in which case you want to dynamically change with your code, not with the config file. This is where the Logger.set_default_output_upload come in

None

@<1576381444509405184:profile|ManiacalLizard2> Yea, that makes sense. However, my problem is that I do not want to set it on the remote clearml-agent, since every use may have a different storage. E.g. one user pushes to Azure, while another one pushes to S3

nope, we are self-hosted in Azure

so in your case, in the clearml-agent conf, it contains multiple credential, each for different cloud storage that you potential use ?

@<1576381444509405184:profile|ManiacalLizard2> Just so I understand correctly:
You are saying that in your local, user-specific, clearml.conf you set the api.files_server , but in your remote, clearml-agent, clearml.conf you left it empty?

oh ..... did not know about that ...

Makes sense, but this means that we are not able to tell clearml-agent where to save on a per-task basis?

The debug samples? or the artifacts/models?

Also it is not possible to use multiple files server? E.g. log tasks on different S3 buckets without changing clearml.conf

Yes, change the Task's output destination in the UI (or programmatically)

@<1576381444509405184:profile|ManiacalLizard2> Yes, exactly. I just didn't know how, but now it is all working 🙂
And yes, I have multiple credentials in the clearml.conf of the agents. It's not a good solution, but since I am currently limited to the free version of ClearML, it is the best I could do.

@<1576381444509405184:profile|ManiacalLizard2> I ll check again 🙂 thanks

@<1576381444509405184:profile|ManiacalLizard2> Thank you, but afaik this only works locally and not if you run your task on a clearml-agent!

Hi @<1523701868901961728:profile|ReassuredTiger98>
The sdk.development.default_output_uri is used for Artifacts and Models. debug samples (or anything else the Logger class creates) will use the api.file_server
On the Task itself, you have the "output destination" (in the Execution tab) which would override the "output_uri" on a Task level
Does that make sense ?

The debug samples? or the artifacts/models?

Both.

Yes, change the Task's output destination in the UI (or programmatically)

This has no effect. I am not able to change the files_sever, e.g. I can not change from None to None
If my files_server is None , it will always look there no matter what I set as output destination.

@<1523701868901961728:profile|ReassuredTiger98>
Manually set both:
None
None
To where you want your files to be uploaded

Debug samples can only be controlled via api.file_server (or programatically)
Model/Artifacts see above

This has no effect. I am not able to change the files_sever, e.g. I can not change from

You are Not changing the files_server just where your Taskj uploads Models/Artifacts, these are two diff things (and again Only applies to Artifacts/Models)

@<1523701205467926528:profile|AgitatedDove14> Thank you very much for your guidance. Setting these manually works for me!

If you are using multi storage place, I don't see any other choice than putting multi credential in the conf file ... Free or Paid Clearml Server ...

but afaik this only works locally and not if you run your task on a clearml-agent!

Isn;t the agent using the same clearml.conf ?
We have our agent running task and uploading everything to Cloud. As I said, we don;t even have file server running

I think in the paid version there is this configuration vault, so that the user can pass their own credentials securely to the agent.

no. I set apo.file_server to the None in Both the remote agent clearml.conf and my local clearml.conf
In which case, both case where the code is ran from local or remote, will store metrics to cloud storage

@<1523701868901961728:profile|ReassuredTiger98> I found that you an set the file_server in your local clearml.conf to your own cloud storage. In our case, we use something like this in our clearml.conf:

api {
   file_server: "azure://<account>..../container"
}

All non artifact model are then store in our azure storage. In our self-hosted clearml setup, we don't even have a file server running alltogether

Thanks a lot, now I think I understand.

Debug samples can only be controlled via api.file_server (or programatically)

Could you guide me how to approach this programmatically? Can I implement my own storage adapter for debug samples with ClearML interfaces or am I on my own?

Makes sense, but this means that we are not able to tell clearml-agent where to save on a per-task basis? I see the output_destination set correctly in clearml web interface, but as you say, clearml-agent always uses its api.fileserver ?

Or maybe a different question: What is not

Artifacts and Models. debug samples (or anything else the Logger class creates)

?

Also it is not possible to use multiple files server? E.g. log tasks on different S3 buckets without changing clearml.conf