Cool 🙂 - as for your question, clearml will use the env vars as the global settings, but if you set the configuration for a specific bucket, it should use that for that bucket
JitteryCoyote63 , well I'm happy to hear that everything worked out in the end 🙂
Sorry both of you, my problem was actually lying somewhere else (both buckets are in the same region) - thanks for you time!
But clearml does read from env vars as well right? It’s not just delegating resolution to the aws cli, so it should be possible to specify the region to use for the logger, right?
SuccessfulKoala55 I want to avoid writing creds in plain text in the config file
Yes, I stayed with an older version for a compatibility reason I cannot remember now 😄 - just tested with 1.1.2 and it’s the same
I tried specifying the bucket directly in my clearml.conf, same problem. I guess clearml just reads from the env vars first
JitteryCoyote63 using multiple regions while using env vars is something that's not even supported in the AWS CLI... 🙂
If you want to mix regions, you'll have to use the clearml.conf configuration (as CostlyOstrich36 suggested)
Do you explicitly specify the region?
Also, please try specifying the buckets directly in your ~/clearml.conf
The relevant section is sdk.aws.s3.credentials
It would look something like this:
{ host: "<BUCKET_A_HOST>" key: "<KEY>" secret: "<SECRET>" region: "<REGION>" }
JitteryCoyote63 , can you try upgrading to the latest versions? Is there any reason why you stayed on older versions?
I am using clearml_agent v1.0.0 and clearml 0.17.5 btw
Yes that’s correct - the weird thing is that the error shows the right detected region
Traceback (most recent call last): File "devops/train.py", line 73, in <module> train(parse_args) File "devops/train.py", line 37, in train train_task.get_logger().set_default_upload_destination(args.artifacts + '/clearml_debug_images/') File "/home/machine/miniconda3/envs/py36/lib/python3.6/site-packages/clearml/logger.py", line 1038, in set_default_upload_destination uri = storage.verify_upload(folder_uri=uri) File "/home/machine/miniconda3/envs/py36/lib/python3.6/site-packages/clearml/storage/helper.py", line 494, in verify_upload log_on_error=log_on_error, File "/home/machine/miniconda3/envs/py36/lib/python3.6/site-packages/clearml/storage/helper.py", line 1424, in _test_bucket_config raise StorageError(msg) clearml.storage.helper.StorageError: Failed testing access to bucket clearml-artefacts: incorrect region specified for bucket clearml-artefacts (detected region eu-central-1)
JitteryCoyote63 Hello 🙂
So That I understand correctly you have a task where you pull data from us-east-1 and then want to upload the data on eu-central-1 but it's failing due to the region change, correct? Can you please give an example of the failure in console?