Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Would Like To Configure Clearml-Server To Connect To An S3 Bucket In Order To Store Artefacts - I'Ve Taken A Look On This Page

Hi, I would like to configure clearml-server to connect to an S3 bucket in order to store artefacts - i've taken a look on this page None and it mentions 'The default configuration files are in the clearml-server repository.'. The link on this page takes me to a HTTP 404 - what is the correct link please?

  
  
Posted 3 years ago
Votes Newest

Answers 18


is there any documentation for connecting to an S3 bucket?

  
  
Posted 3 years ago

@<1523701205467926528:profile|AgitatedDove14> - any thoughts on this. Would like to use profile / iam roles as well.

  
  
Posted 3 years ago

Same credentials configuration for the ClearML-Agent.
Notice that when a task is created, in the UI, under EXECUTION tab, you can find (and change if you like) the output destination.

  
  
Posted 3 years ago

None

  
  
Posted 3 years ago

@<1687643893996195840:profile|RoundCat60> I'm assuming we are still talking about the S3 credentials, sadly no 😞
Are you familiar with boto and IAM roles ?

  
  
Posted 3 years ago

yep still referring to the S3 credentials, somewhat familiar with boto and IAM roles

  
  
Posted 3 years ago

As we can’t create keys in our AWS due to infosec requirements

  
  
Posted 3 years ago

As we can’t create keys in our AWS due to infosec requirements

Hmmm

  
  
Posted 3 years ago

Is it possible to use an IAM role rather than user credentials in the clearml.conf file?

  
  
Posted 3 years ago

Was asking about using iam roles without keys

  
  
Posted 3 years ago

@<1687643893996195840:profile|RoundCat60> Looks like the docs have not caught up yet with recent structure change in the repo which renamed the 'server' folder to 'apiserver'.
So... the correct link would be None

  
  
Posted 3 years ago

I suggest a bump in the GitHub issue

  
  
Posted 3 years ago

thank you

  
  
Posted 3 years ago

None
So this is the only place we need to change to support it, do you feel like messing around with it and adding IAM roles ?

  
  
Posted 3 years ago

Hi @<1687643893996195840:profile|RoundCat60>

I think the best way will be to configure a default output_uri to be used by all tasks:  None , under default_output_uri just write your bucket path ( None ).
When using S3/google storage/azure, you will also need to add your credentials in this section - None (s3 in this link),
and install clearml with storage requirements:
pip install clearml[s3] for S3
pip install clearml[azure] for azure
pip install clearml[gs] for google storage

  
  
Posted 3 years ago

or have I got this wrong, and it's the clearml-agent that needs to read/write to S3?

  
  
Posted 3 years ago

Let me check

  
  
Posted 3 years ago

Still not supported 😞

  
  
Posted 3 years ago
132 Views
18 Answers
3 years ago
2 months ago
Tags