Answered

I Want To Run

I want to run clearml sdk to create task (without running it) and upload artifact in a container.
I have code to pull clearml API key from Azure Keyvault and auth with clearml using

    clearml.Task.set_credentials(
        api_host=creds['api_url'],
        web_host=creds['web_url'],
        key=creds['access_key'],
        secret=creds['secret_key']
    )

But how do I set the auth for our blob storage account ? I would like to not use clearml.conf

  				
Posted 
	14 days ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

Votes Newest

Answers 6

Yes just use AZURE_STORAGE_ACCOUNT, it is used when no specific config exist.
Perfect reference BTW :)

  				
Posted 
	11 days ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Awesome. I will try that !
And also found this based on your suggestion that clearml use azure sdk underneath: None
Just not sure under which conditions from_config is actually called ...

  				
Posted 
	12 days ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

So this can be translated to

CLEARML__SDK__AZURE__STORAGE__CONTAINERS__0__ACOUNT_NAME=abcd

  				
Posted 
	13 days ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Yes the easiest is os.environ call before the import
Regarding azure blob

General azure env vars should work because it configure the underlying azure sdk, but I would double check

Generally speaking
Generic Override Format
ClearML allows you to override any config entry using this format:

bash
CLEARML__<section>__<key>=<value>
Double underscores __ separate the hierarchy levels.

All keys and values are treated as strings.

This works for nested entries in clearml.conf.

  				
Posted 
	13 days ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

@<1523701205467926528:profile|AgitatedDove14>
What is the env var name for Azure Blob storage ? That the one we use for our Artifiact.

Also, is there function call rather than env var ?
It would be simplier in our case to call a function to set credential for clearml rather than fetch secret and set env var prior to running the python code.

If there is only the option of using env var, I am thinking fetchcing secrets and set env var from python, eg: os.environ["MY_VARIABLE"] = "hello" prior to import clearml That should work too right ?

  				
Posted 
	13 days ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

Hi @<1576381444509405184:profile|ManiacalLizard2>
You can also use env vars, it might be easier, I'm assuming this is kind of CI/CD process
'''
export CLEARML_API_ACCESS_KEY="your-public-key"
export CLEARML_API_SECRET_KEY="your-private-secret"
export CLEARML_API_HOST=" https://api.clear.ml "
export CLEARML_WEB_HOST=" https://app.clear.ml "
export CLEARML_FILES_HOST=" https://files.clear.ml "

'''
https://clear.ml/docs/latest/docs/configs/env_vars/
Re aws s3 credentials
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_DEFAULT_REGION="us-east-1"

Optional for S3-compatible storage:

export CLEARML_AWS_S3_ENDPOINT=" https://minio.mycompany.com "

  				
Posted 
	13 days ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Write your answer

311 Views

6 Answers

14 days ago

10 days ago