Hello! I have an issue on a last-week self-hosted ClearML installation (Server: 1.16.0-494): after starting containers with docker compose everything goes we...
5 months ago
TroubledCamel37
1
Question,
3
Answers
Active since 11 July 2024
Last activity
5 months ago
Reputation
0
Badges 1
Eureka!
0
I Executed A Hyperparameter Optimization Task, But Just One Single Experiment Was Performed And Nothing. The Task Wasn'T Terminated Automatically, But Still Not Starting The Next Experiments.
What'S Wrong Here? How Can I Fix This?
Was "task.close()" called for the early-stopped task?
What is the experiment status in Web UI?
5 months ago
0
Hello! I Have An Issue On A Last-Week Self-Hosted Clearml Installation (Server: 1.16.0-494): After Starting Containers With Docker Compose Everything Goes Well Except For
Hi @<1523701070390366208:profile|CostlyOstrich36> ! Thanks for responding!
In the apiserver.log there are periodic (~1 min) messages like this:[9] [WARNING] [clearml.service_repo] Returned 401 for auth.login in 2ms, msg=Unauthorized (invalid credentials) (failed to locate provided credentials)
In the fileserver.log there are no fresh messages - probably because the container could not start normally.
5 months ago
0
I Executed A Hyperparameter Optimization Task, But Just One Single Experiment Was Performed And Nothing. The Task Wasn'T Terminated Automatically, But Still Not Starting The Next Experiments.
What'S Wrong Here? How Can I Fix This?
@<1664079296102141952:profile|DangerousStarfish38> Yep, you are right, according to the docs, the optimizer.stop() should be used, not task.close(). Sorry for confusing.
I guess the issue is in connectivity/auth problems between ClearML components - there are many timeout messages in the log. I have similar messages for fileserver container, not yet resolved.
5 months ago