It seems that the SDK can't reach the API server. Are you seeing anything in the API server logs? Is it possible you're being blocked by an internal firewall?
hi @<1734020208089108480:profile|WickedHare16> what was the issue here with the api server? im getting the same warning log:
[2024-08-18 14:03:26,881] [9] [WARNING] [clearml.service_repo] Returned 400 for in 0ms, msg=Invalid request path /
not sure why
I have also added the certificated to keychain
WebApp: 1.16.0-494 • Server: 1.16.0-494 • API: 2.30
Hi @<1523701070390366208:profile|CostlyOstrich36>
Getting issue in setting python local config using clearml-init
Verifying credentials ...
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)'))': /auth.login
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)'))': /auth.login
Error: could not verify credentials: key=<key> secret=<secret>
@<1523701070390366208:profile|CostlyOstrich36> Found the issues and fixed it
api server contain these logs
[2024-08-13 18:06:32,523] [9] [WARNING] [clearml.service_repo] Returned 400 for in 0ms, msg=Invalid request path /
[2024-08-13 18:06:35,178] [9] [INFO] [clearml.service_repo] Returned 200 for debug.ping in 0ms
[2024-08-13 18:06:35,179] [9] [INFO] [clearml.service_repo] Returned 200 for debug.ping in 0ms
[2024-08-13 18:06:45,179] [9] [INFO] [clearml.service_repo] Returned 200 for debug.ping in 0ms
[2024-08-13 18:06:45,181] [9] [INFO] [clearml.service_repo] Returned 200 for debug.ping in 0ms
[2024-08-13 18:06:46,714] [9] [WARNING] [clearml.service_repo] Returned 400 for in 0ms, msg=Invalid request path /
[2024-08-13 18:06:47,089] [9] [WARNING] [clearml.service_repo] Returned 400 for in 0ms, msg=Invalid request path /
[2024-08-13 18:06:47,532] [9] [WARNING] [clearml.service_repo] Returned 400 for in 0ms, msg=Invalid request path /
[2024-08-13 18:06:55,178] [9] [INFO] [clearml.service_repo] Returned 200 for debug.ping in 0ms
[2024-08-13 18:06:55,180] [9] [INFO] [clearml.service_repo] Returned 200 for debug.ping in 0ms
[2024-08-13 18:07:01,715] [9] [WARNING] [clearml.service_repo] Returned 400 for in 0ms, msg=Invalid request path /
[2024-08-13 18:07:02,101] [9] [WARNING] [clearml.service_repo] Returned 400 for in 0ms, msg=Invalid request path /
[2024-08-13 18:07:02,539] [9] [WARNING] [clearml.service_repo] Returned 400 for in 0ms, msg=Invalid request path /
[2024-08-13 18:07:05,178] [9] [INFO] [clearml.service_repo] Returned 200 for debug.ping in 0ms
[2024-08-13 18:07:05,180] [9] [INFO] [clearml.service_repo] Returned 200 for debug.ping in 0ms
In the open source you don't have users & groups, user management is done via fixed users - None
What errors are you seeing in the apiserver pod?
@<1734020208089108480:profile|WickedHare16> , I suggest going through the documentation and seeing how each part works with the other
None
I have setup the infra in k8s and it working now. Now i am trying to create the new user, but the User & group tab in settings page is not visible to any pre-existing users
@<1523701070390366208:profile|CostlyOstrich36> can you also explain the dependency of these components with each other
@<1523701070390366208:profile|CostlyOstrich36> thanks for doc
Hi @<1734020208089108480:profile|WickedHare16> , what issues are you facing?
@<1734020208089108480:profile|WickedHare16> , many different things, RBAC, users & groups, K8s dedicated support with advanced features, HyperDatasets, SSO/LDAP integration, dedicated support, dynamic GPU allocation, advanced GPU fractioning on top of K8s and much more.
You can see a more detailed list here - None
I would suggest contacting sales@clear.ml for more information 🙂
And what other features are not available in open source?
mongodb is not getting failed. with log
[38;5;6mmongodb [38;5;5m12:59:37.86 [0m
[38;5;6mmongodb [38;5;5m12:59:37.86 [0m[1mWelcome to the Bitnami mongodb container[0m
[38;5;6mmongodb [38;5;5m12:59:37.86 [0mSubscribe to project updates by watching [1m
[0m
[38;5;6mmongodb [38;5;5m12:59:37.86 [0mSubmit issues and feature requests at [1m
[0m
[38;5;6mmongodb [38;5;5m12:59:37.86 [0m
[38;5;6mmongodb [38;5;5m12:59:37.86 [0m[38;5;2mINFO [0m ==> ** Starting MongoDB setup **
[38;5;6mmongodb [38;5;5m12:59:37.87 [0m[38;5;2mINFO [0m ==> Validating settings in MONGODB_* env vars...
[38;5;6mmongodb [38;5;5m12:59:37.89 [0m[38;5;3mWARN [0m ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
[38;5;6mmongodb [38;5;5m12:59:37.90 [0m[38;5;2mINFO [0m ==> Initializing MongoDB...
[38;5;6mmongodb [38;5;5m12:59:37.93 [0m[38;5;2mINFO [0m ==> Deploying MongoDB with persisted data...
[38;5;6mmongodb [38;5;5m12:59:37.94 [0m[38;5;2mINFO [0m ==> ** MongoDB setup finished! **
[38;5;6mmongodb [38;5;5m12:59:37.94 [0m[38;5;2mINFO [0m ==> ** Starting MongoDB **
{"t":{"$date":"2024-08-12T12:59:37.964+00:00"},"s":"I", "c":"CONTROL", "id":20698, "ctx":"-","msg":"***** SERVER RESTARTED *****"}
{"t":{"$date":"2024-08-12T12:59:37.965+00:00"},"s":"I", "c":"NETWORK", "id":4915701, "ctx":"main","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":13},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":13},"outgoing":{"minWireVersion":0,"maxWireVersion":13},"isInternalClient":true}}}
{"t":{"$date":"2024-08-12T12:59:37.965+00:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2024-08-12T12:59:37.966+00:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2024-08-12T12:59:37.966+00:00"},"s":"I", "c":"NETWORK", "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2024-08-12T12:59:37.967+00:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2024-08-12T12:59:37.967+00:00"},"s":"I", "c":"REPL", "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","ns":"config.tenantMigrationDonors"}}
{"t":{"$date":"2024-08-12T12:59:37.967+00:00"},"s":"I", "c":"REPL", "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","ns":"config.tenantMigrationRecipients"}}
{"t":{"$date":"2024-08-12T12:59:37.967+00:00"},"s":"I", "c":"CONTROL", "id":5945603, "ctx":"main","msg":"Multi threading initialized"}
{"t":{"$date":"2024-08-12T12:59:37.967+00:00"},"s":"I", "c":"CONTROL", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/bitnami/mongodb/data/db","architecture":"64-bit","host":"clearml-mongodb-5f776cb685-52ctl"}}
{"t":{"$date":"2024-08-12T12:59:37.967+00:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"5.0.10","gitVersion":"bbf5bc7e16d1713c9349a09adf4901ca37472e66","openSSLVersion":"OpenSSL 1.1.1n 15 Mar 2022","modules":[],"allocator":"tcmalloc","environment":{"distmod":"debian11","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2024-08-12T12:59:37.967+00:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"PRETTY_NAME=\"Debian GNU/Linux 11 (bullseye)\"","version":"Kernel 5.10.219-208.866.amzn2.x86_64"}}}
{"t":{"$date":"2024-08-12T12:59:37.967+00:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/opt/bitnami/mongodb/conf/mongodb.conf","net":{"bindIp":"*","ipv6":false,"port":27017,"unixDomainSocket":{"enabled":true,"pathPrefix":"/opt/bitnami/mongodb/tmp"}},"processManagement":{"fork":false,"pidFilePath":"/opt/bitnami/mongodb/tmp/mongodb.pid"},"security":{"authorization":"disabled"},"setParameter":{"enableLocalhostAuthBypass":"true"},"storage":{"dbPath":"/bitnami/mongodb/data/db","directoryPerDB":false,"journal":{"enabled":true}},"systemLog":{"destination":"file","logAppend":true,"logRotate":"reopen","path":"/opt/bitnami/mongodb/logs/mongodb.log","quiet":false,"verbosity":0}}}}
{"t":{"$date":"2024-08-12T12:59:37.975+00:00"},"s":"I", "c":"STORAGE", "id":22315, "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=7339M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
{"t":{"$date":"2024-08-12T12:59:38.554+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":17,"message":"[1723467578:554085][1:0x7f0ba1978c80], connection: __posix_open_file, 808: /bitnami/mongodb/data/db/WiredTiger.wt: handle-open: open: File exists"}}
{"t":{"$date":"2024-08-12T12:59:38.563+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.297"}}
{"t":{"$date":"2024-08-12T12:59:38.570+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":1,"message":"[1723467578:570973][1:0x7f0ba1978c80], connection: __posix_open_file, 808: /bitnami/mongodb/data/db/WiredTiger.wt: handle-open: open: Operation not permitted"}}
{"t":{"$date":"2024-08-12T12:59:38.606+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":17,"message":"[1723467578:606551][1:0x7f0ba1978c80], connection: __posix_open_file, 808: /bitnami/mongodb/data/db/WiredTiger.wt: handle-open: open: File exists"}}
{"t":{"$date":"2024-08-12T12:59:38.615+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.298"}}
{"t":{"$date":"2024-08-12T12:59:38.620+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":1,"message":"[1723467578:620755][1:0x7f0ba1978c80], connection: __posix_open_file, 808: /bitnami/mongodb/data/db/WiredTiger.wt: handle-open: open: Operation not permitted"}}
{"t":{"$date":"2024-08-12T12:59:38.657+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":17,"message":"[1723467578:657849][1:0x7f0ba1978c80], connection: __posix_open_file, 808: /bitnami/mongodb/data/db/WiredTiger.wt: handle-open: open: File exists"}}
{"t":{"$date":"2024-08-12T12:59:38.667+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.299"}}
{"t":{"$date":"2024-08-12T12:59:38.673+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":1,"message":"[1723467578:673261][1:0x7f0ba1978c80], connection: __posix_open_file, 808: /bitnami/mongodb/data/db/WiredTiger.wt: handle-open: open: Operation not permitted"}}
{"t":{"$date":"2024-08-12T12:59:38.678+00:00"},"s":"W", "c":"STORAGE", "id":22347, "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
{"t":{"$date":"2024-08-12T12:59:38.678+00:00"},"s":"F", "c":"STORAGE", "id":28595, "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"1: Operation not permitted"}}
{"t":{"$date":"2024-08-12T12:59:38.678+00:00"},"s":"F", "c":"-", "id":23091, "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":687}}
{"t":{"$date":"2024-08-12T12:59:38.678+00:00"},"s":"F", "c":"-", "id":23092, "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
