Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Is It Possible To Avoid The Clearml-Agent For Local Installations, And Have The File Server Automatically Use An S3 Bucket? I'Ve Found

Is it possible to avoid the clearml-agent for local installations, and have the file server automatically use an S3 bucket?
I've found https://clearml.slack.com/archives/CTK20V944/p1632219057464900 but I'm not sure if they had to use clearml-agent or edit some of the .conf files under /opt/clearml/config

  
  
Posted 3 years ago
Votes Newest

Answers 30


The odd thing is that it was already defined, and then when I clicked an S3 link, it asked me to fill it in again, adding a duplicate credentials row

  
  
Posted 3 years ago

Thanks! That's what I thought, but then I get
2021-12-21 22:08:35,376 - clearml.storage - ERROR - Failed uploading: Parameter validation failed: Invalid bucket name "": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$"

  
  
Posted 3 years ago

and I've made a script to edit it to our needs as part of the installation process 

 Thanks Martin!

My pleasure, btw: there is no actual need to configure all the clearml.conf values. It will actually take the defaults from the clearml package itself. This means you only need something like:
` api {

server config here

}
sdk.aws.s3{

minio config here

} `

  
  
Posted 3 years ago

AgitatedDove14 another option I thought would be nice is to actually self-sign the internal MinIO bucket, but then I get
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1076)Are you aware of any other way then (other than the secure: false flag?

  
  
Posted 3 years ago

That would be nice :)

  
  
Posted 3 years ago

One last MinIO-related question (sorry for the long thread!)
While I do have the access and secret defined in clearml.conf, and even in the WebUI, I still get similar warnings as David does here - https://clearml.slack.com/archives/CTK20V944/p1640135359125200

  
  
Posted 3 years ago

Sorry to keep this up - what about support for minio using the environment variable? Do I set the CLEARML_FILES_HOST to the end point instead of an s3 bucket?

  
  
Posted 3 years ago

Ah! Makes sense. Thanks!

  
  
Posted 3 years ago

I will TIAS, but maybe worthwhile to also mention if it has to be the absolute path or if relative path is fine too!

Good point! (absolute but you can use ~, and I "think" also $ENV )

  
  
Posted 3 years ago

That's fine for the current use-case I believe.
Once the team is happy with the logging functionality, we'll move on to remote execution and things will update.

  
  
Posted 3 years ago

If I add the bucket to that (so CLEARML_FILES_HOST= s3://minio_ip:9000/minio/bucket ), I then get the following error instead --

2021-12-21 22:14:55,518 - clearml.storage - ERROR - Failed uploading: SSL validation failed for ... [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1076)

  
  
Posted 3 years ago

Or did you mean I can couple a short "mini config" with the package and redirect clearml to use this local one (instead of the one at ~/clearml.conf)?

Actually yes, you can set a "fixed" config point to it with ENV variable, then setup per user just the access/secret .
wdyt?
(I was also pointing to the fact you do not have to use clearml-init you can create a simple partial config template and let user just fill in the missing "key"/"secret")

  
  
Posted 3 years ago

Are you aware of any other way then (other than the 

secure: false

 flag?

Actually self -signing and providing certificate file is already supported with boto (and thus clearml)
AWS_CA_BUNDLE
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html

  
  
Posted 3 years ago

~ is a bit weird since it's not part of the package (might as well let the user go through clearml-init ), but using ${PWD} works! 👍 👍

(Though I still had to add the CLEARML_API_HOST and CLEARML_WEB_HOST ofc, or define them in the clearml.conf)

  
  
Posted 3 years ago

In the Profile section, yes, they are well defined (bucket, secret, key, and endpoint)

  
  
Posted 3 years ago

maybe we should add some ENV setting it? (I'm not sure we should disable SSL for all S3 connections... so somehow specify the mino it should use http with)

  
  
Posted 3 years ago

Always great to find a bug! I'll make relevant SDK updates then.

  
  
Posted 3 years ago

UnevenDolphin73 it seems this is a UI browser limit, this means we will need to move it into the server ...
See here: https://clearml.slack.com/archives/CTK20V944/p1640247879153700?thread_ts=1640135359.125200&cid=CTK20V944

  
  
Posted 3 years ago

Once the team is happy with the logging functionality, we'll move on to remote execution and things will update.

🎉

While I do have the access and secret defined in clearml.conf, and even in the WebUI, I still get similar

and you have your credentials in the browser when deleting a Task ?

  
  
Posted 3 years ago

That's what I found as well, but it did not like it after all (boto is fine with it, but underlying urllib and requests were not?)

It's fine -- I see the added benefit in making sure the users set up their clearml.conf and I've made a script to edit it to our needs as part of the installation process 🙂 Thanks Martin!

  
  
Posted 3 years ago

If I add the bucket to that ....

Oh no .... you should also set SSL off for the connection, but I think this is only in the clearml.conf:
https://github.com/allegroai/clearml/blob/fd2d6c6f5d46cad3e406e88eeb4d805455b5b3d8/docs/clearml.conf#L101

  
  
Posted 3 years ago

${PWD} works!

This will be resolved every call to Task.init (so I would recommend against it), how about "$HOME/" ?

  
  
Posted 3 years ago

Is it CLEARML_CONFIG_FILE ? (I had to dig this from the GH code 😅 )

  
  
Posted 3 years ago

I will TIAS, but maybe worthwhile to also mention if it has to be the absolute path or if relative path is fine too!

  
  
Posted 3 years ago

hmm that is odd, let me check

  
  
Posted 3 years ago

Is it 

CLEARML_CONFIG_FILE

? (I had to dig this from the GH code 

 )

Yes it is !
https://clear.ml/docs/latest/docs/faq#clearml-configuration
(I will make sure we add it to https://clear.ml/docs/latest/docs/configs/env_vars#server-connection as well 🙂 )

  
  
Posted 3 years ago

Do I set the 

CLEARML_FILES_HOST

 to the end point instead of an s3 bucket?

Yes you are right this is not straight forward:
CLEARML_FILES_HOST=" s3://minio_ip:9001 "
Notice you must specify "port" , this is how it knows this is not AWS. I would avoid using an IP and register the minio as a host on your local DNS / firewall. This way if you change the IP the links will not get broken 🙂

  
  
Posted 3 years ago

👍

  
  
Posted 3 years ago

Yes, exactly! I've added instructions for the users on creating their account and running clearml-init , and then they run the snippet that updates the api and sdk sections.

Or did you mean I can couple a short "mini config" with the package and redirect clearml to use this local one (instead of the one at ~/clearml.conf)?

  
  
Posted 3 years ago

The key/secret is also shared internally so that sounds like a nice mitigation actually!
Which environment variable am I looking for? I couldn't spot anything specifically in that environment variables page

  
  
Posted 3 years ago