oookay so we found that for kubernetes, if we allow only tls v1.3 on the ingress controller, clearml-inits breaks with 2022-03-04 10:32:02,814 - clearml.session - WARNING - SSLError Retrying HTTPSConnectionPool(host=' http://api.clear-ml.dev.monk.ai ', port=443): Max retries exceeded with url: /auth.login (Caused by SSLError(SSLError(1, '[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:1129)'))) or sometimes just could not verify credentials
so if anybody needs this someday (migrating your hostname which is saved inside your experiments (debug images and plots with images)) you need this https://github.com/allegroai/clearml-server/issues/83
but it's slow , you can restrict the query to the items that are actually updated, with:
` # on index events-training_debug_image-yourid
OLDHOST/ should be something like
NEWHOST/ same
"script": {
"source": "ctx._source.url = ctx._source.url.replace('OLDHOST/', 'NEWHOST/')",
"lang": "painless"
},
"query": {
"query_string": {
"query": "http\:OLD.HOST/*" # you need to escape : with \, and add * at the end
}
} then for the plots do # on index events-plot-yourid
"script": {
"source": "ctx._source.plot_str = ctx._source.plot_str.replace('OLDHOST/', 'NEWHOST/')",
"lang": "painless"
},
"query": {
"query_string": {
"query": "http\:OLD.HOST/*"
}
} and the last one, also for the plots # on index events-plot-yourid
"script": {
"source": """ctx._source.source_urls = ctx._source.source_urls.stream()
.map(url -> url.replace('OLDHOST/', 'NEWHOST/'))
.collect(Collectors.toList())""",
"lang": "painless"
},
"query": {
"query_string": {
"query": "http\:OLD.HOST/*"
}
} `
VirtuousFish83 this is an example on how to do that for artifact and model URLS in MongoDB: https://clearml.slack.com/archives/CTK20V944/p1645797943141509?thread_ts=1645702655.726019&cid=CTK20V944
And this is for debug image URLs in Elastic: https://clearml.slack.com/archives/CTK20V944/p1618473431491800?thread_ts=1618471688.490400&cid=CTK20V944
Regarding the migration, there's been some discussion on it here, it'll take me some time to find it 🙂
I welcome the day clearml saves relative urls by default ^^ it is supported by browsers (i.e. fetching /someurl is relative to the current hostname) so maybe only the clearml client would need to be updated right ? to push images with a relative url instead of the clearml server url.
Well, there's actually a reverse proxy in place, so you can use app.zzz, app.zzz/api and app.zzz/files
Hello, sorry the second is for models and not images
we still don't what was happening with the VM + docker compose + load balancers
Also it would be awesome if the front-end integrated a small reverse-proxy to have everything on 1 address, I don't know if this is somewhere on the roadmap ? Or what are advantages of having 3 separate addresses ?
Is there a way to store relative urls in clearml ? We can't connect to our server with a public address, it only works with the internal dns from GCE
Yeah, that's true, only some internal stuff would need to be sorted out (for example, if embedded inside a plot, this could be a problem)
Yeah, but only for stuff stored on the fileserver (not on S3 or other object storage)
Oh ok I thought it would be relative to the server, how do i run this migration ?
Hi VirtuousFish83 , the debug images and other URLs are registered when the event is origonally sent, which is the reason why you see the old address being used. To fix that, you will need to run a migration on the server
I suppose the images are in db.task but I can't find them
We tried with a docker-compose on a GCE VM + load balancers, and then in kube, we get the same error: clearml-init returns Error: could not verify credentials: key=241... secret=NhC...
Is there a way to store relative urls in clearml ? We can't connect to our server with a public address, it only works with the internal dns from GCE
Relative URLs are currently not supported, bit would be a good feature, albeit only useful for the fileserver