Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello, We Recently Moved Our Clearml Self-Hosted Server (On Gce) To Be Behind A Load Balancer With Certs Etc. At The Address

Hello, we recently moved our clearml self-hosted server (on GCE) to be behind a load balancer with certs etc. at the address {app,files,api}.dev.companyname.tld , it is in the same project (and vpc) as our instances where we run the trainings so they connect to clearml with the private GCE hostname (which is clearml-v1-2-0.europe-something.c.myproject.internal ), should that be ok ?
Because on the plots and debug samples pages, the front tries to request the files from the internal address instead of http://files.dev .companyname.tld and so nothing can load, is there a fix ? The docs says somewhere that when the front runs on a app.address it automatically fetchs from {api,files}.address but it's not the case for us.
Thanks

  
  
Posted 2 years ago
Votes Newest

Answers 21


Hello, sorry the second is for models and not images

  
  
Posted 2 years ago

for plots also

  
  
Posted 2 years ago

Sorry, you're right 🙂

  
  
Posted 2 years ago

Is there a way to store relative urls in clearml ? We can't connect to our server with a public address, it only works with the internal dns from GCE

Relative URLs are currently not supported, bit would be a good feature, albeit only useful for the fileserver

  
  
Posted 2 years ago

oookay so we found that for kubernetes, if we allow only tls v1.3 on the ingress controller, clearml-inits breaks with 2022-03-04 10:32:02,814 - clearml.session - WARNING - SSLError Retrying HTTPSConnectionPool(host=' http://api.clear-ml.dev.monk.ai ', port=443): Max retries exceeded with url: /auth.login (Caused by SSLError(SSLError(1, '[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:1129)'))) or sometimes just could not verify credentials

  
  
Posted 2 years ago

Yeah, but only for stuff stored on the fileserver (not on S3 or other object storage)

  
  
Posted 2 years ago

we still don't what was happening with the VM + docker compose + load balancers

  
  
Posted 2 years ago

I suppose the images are in db.task but I can't find them

  
  
Posted 2 years ago

Also it would be awesome if the front-end integrated a small reverse-proxy to have everything on 1 address, I don't know if this is somewhere on the roadmap ? Or what are advantages of having 3 separate addresses ?

  
  
Posted 2 years ago

Well, there's actually a reverse proxy in place, so you can use app.zzz, app.zzz/api and app.zzz/files

  
  
Posted 2 years ago

Hi VirtuousFish83 , the debug images and other URLs are registered when the event is origonally sent, which is the reason why you see the old address being used. To fix that, you will need to run a migration on the server

  
  
Posted 2 years ago

I welcome the day clearml saves relative urls by default ^^ it is supported by browsers (i.e. fetching /someurl is relative to the current hostname) so maybe only the clearml client would need to be updated right ? to push images with a relative url instead of the clearml server url.

  
  
Posted 2 years ago

We tried with a docker-compose on a GCE VM + load balancers, and then in kube, we get the same error: clearml-init returns Error: could not verify credentials: key=241... secret=NhC...

  
  
Posted 2 years ago

Oh ok I thought it would be relative to the server, how do i run this migration ?

  
  
Posted 2 years ago

Regarding the migration, there's been some discussion on it here, it'll take me some time to find it 🙂

  
  
Posted 2 years ago

so if anybody needs this someday (migrating your hostname which is saved inside your experiments (debug images and plots with images)) you need this https://github.com/allegroai/clearml-server/issues/83
but it's slow , you can restrict the query to the items that are actually updated, with:
` # on index events-training_debug_image-yourid

OLDHOST/ should be something like or

NEWHOST/ same

"script": {
"source": "ctx._source.url = ctx._source.url.replace('OLDHOST/', 'NEWHOST/')",
"lang": "painless"
},
"query": {
"query_string": {
"query": "http\:OLD.HOST/*" # you need to escape : with \, and add * at the end
}
} then for the plots do # on index events-plot-yourid
"script": {
"source": "ctx._source.plot_str = ctx._source.plot_str.replace('OLDHOST/', 'NEWHOST/')",
"lang": "painless"
},
"query": {
"query_string": {
"query": "http\:OLD.HOST/*"
}
} and the last one, also for the plots # on index events-plot-yourid
"script": {
"source": """ctx._source.source_urls = ctx._source.source_urls.stream()
.map(url -> url.replace('OLDHOST/', 'NEWHOST/'))
.collect(Collectors.toList())""",
"lang": "painless"
},
"query": {
"query_string": {
"query": "http\:OLD.HOST/*"
}
} `

  
  
Posted 2 years ago

Yeah, that's true, only some internal stuff would need to be sorted out (for example, if embedded inside a plot, this could be a problem)

  
  
Posted 2 years ago

Is there a way to store relative urls in clearml ? We can't connect to our server with a public address, it only works with the internal dns from GCE

  
  
Posted 2 years ago

VirtuousFish83 this is an example on how to do that for artifact and model URLS in MongoDB: https://clearml.slack.com/archives/CTK20V944/p1645797943141509?thread_ts=1645702655.726019&cid=CTK20V944

  
  
Posted 2 years ago
1K Views
21 Answers
2 years ago
one year ago
Tags