Thanks for looking at this.
- Pasted the body below
{
"aws": {
"key": "emEijEB2wZtj1rgaUN3y",
"secret": "oTxr3w3nlygv85oULOBWkaJi6Zj41OFBLB1e1m0L",
"region": "",
"token": "",
"use_credentials_chain": false,
"buckets": [{
"bucket": "clearml",
"host": "
",
"key": "emEijEB2wZtj1rgaUN3y",
"secret": "oTxr3w3nlygv85oULOBWkaJi6Zj41OFBLB1e1m0L",
"token": "",
"secure": false,
"region": "",
"verify": null,
"use_credentials_chain": false
}]
},
"google": {
"project": null,
"credentials_json": {},
"buckets": []
},
"azure": {
"containers": []
}
}
I noticed I got this response
{
"meta": {
"id": "a2abdf8724014e01bf8ecd3a98887082",
"trx": "a2abdf8724014e01bf8ecd3a98887082",
"endpoint": {
"name": "storage.set_settings",
"requested_version": "2.31",
"actual_version": "1.0"
},
"result_code": 400,
"result_subcode": 12,
"result_msg": "Validation error (error for field 'google': error for field 'credentials_json': ('value is wrong, expected type \"str\"', {})..)",
"error_stack": null,
"error_data": {}
},
"data": {}
}
I’m not setting google credentials so idk why this would be part of the error
I think I found it.
The google credentials were “set” out of the box but had an empty Object {}. I removed the object and saved and the save was succesful.
I was then able to add my minio credentials and save them as well.
Should I make a bug report for this?
- I managed to recreate it.
#In my jupyter notebook
%pip install clearml --upgrade pip
from clearml import Task, Logger
%env CLEARML_WEB_HOST=
%env CLEARML_API_HOST=
%env CLEARML_FILES_HOST=
%env CLEARML_API_ACCESS_KEY=GZBUVZYG1YHU6QSPMM3PI19EZB51ZY"
%env CLEARML_API_SECRET_KEY=MlirsAtDI_V4_OdzGFk5aB8-Z8ivLV3bZfcLzMzZD6oYSt503Lbmg0HHJDFJW9PGMuE
task = Task.init(project_name='Test', task_name='test')
I get a timeout error
etrying (Retry(total=239, connect=239, read=240, redirect=240, status=240)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7a91cd5add30>, 'Connection to api-clearml.domain.duckdns.org timed out. (connect timeout=3.0)')': /auth.login
I’m wondering if this is because the url is a local dns set in my unify router and jupyter runs inside the cluster and is trying to resolve it locally?
Also. Can I skip setting the bucket name in the UI as it’s part of the URL?
I also couldn’t see any example credentials in the documentation but the bucket is not public so should I assume that credentials will be used if set?
It’s running behind an ingress so the port is there.
Oh. I see what you mean now.
“To force usage of a non-AWS endpoint, port declaration is always needed (e.g. host: "my-minio-host:9000" ), even for standard ports like 433 for HTTPS (e.g. host: "my-minio-host:433" ).”
Spun up a jupyterlab locally and was able to connect with clearml. This is obviously a problem with networking on my k8s cluster.
Looks like a network issue.
As a side note, I would suggest removing & revoking all credentials you've pasted here 🙂
minio creds, server creds, someone with ill intentions could have accessed your cluster by now
It’s a homelab k8s cluster that I tear down for laughs and giggles 🙂
No problem. This is all very ephemeral and will die imminently.
# This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
host: "my-minio-host:9000"
You need to add the port for minio
Just to make sure the domain is available I did
jovyan@hub-54bbb78ff4-bphnj:/srv/jupyterhub$ dig api-clearml.domain.duckdns.org
; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> api-clearml.domain.duckdns.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1149
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 3ac65d14c9554b0f (echoed)
;; QUESTION SECTION:
;api-clearml.domain.duckdns.org. IN A
;; ANSWER SECTION:
api-clearml.domain.duckdns.org. 5 IN A 192.168.70.20
;; Query time: 3 msec
;; SERVER: 10.96.0.10#53(10.96.0.10) (UDP)
;; WHEN: Mon Feb 17 20:29:43 UTC 2025
;; MSG SIZE rcvd: 117
Fair enough 🙂
BTW I noticed that your minio access is misconfigured, check here for the proper syntax - None
Is there a way to test if the connection is working?
Hi PerfectSeaurchin36 , on your points:
- what was the body of the API call, you got 400 so it looks like the body was incorrect
- Not sure what the issue is
Not exactly sure I understand what the issue is, can you please elaborate?
I also tried
Task.set_credentials(
api_host="
",
web_host="
",
files_host="
",
key='E5BYXIM0JXX5N9MRZSHEE2ACKXPTY2',
secret='govErSIYdtu-67EBGVPhriMOOB0QCT_OZ_B2073rGjYO14uYP802dMuOk1_oVV4STxY'
)
Same result
Highly unlikely as those are all local dns records. 🙂