Reputation
Badges 1
69 × Eureka!SucculentCrab55 I've had this problem when I tried to launch UI too quickly, try to wait a bit and then check UI again
CostlyOstrich36
The error appears regardless of --foreground tag. This is not full stacktrace, I will provide it with the next message.
clearml 1.9.0
clearml-agent 1.5.1
Ubuntu1 8.04.6 LTS
CostlyOstrich36 Yep, it seems it was the case. I did not provide credentials for API in docker compose. I did that but now agent-services just keeps restarting. I looked into containers logs and it seems to be a proxy error. Why this container is trying to connect somewhere?
@<1523701435869433856:profile|SmugDolphin23> Thanks a lot, that actually worked! It was very difficult to figure out you have to plug those exact values given you have https endpoint:
- Using s3 protocol instead of https together with bucket name in output URI
- Not providing a bucket name in credentials section where it is by default
- Providing default secure port for both host and output URI
- Disabling credentials chainI think a common use case for many people that they get S3 storage wi...
I looked through agent-services logs and found new error I haven't seen before:clearml_agent: ERROR: Connection Error: it seems *api_server* is misconfigured. Is this the ClearML API server http://<my_ip>:8008 ?
Traceback (most recent call last):
File "/home/<home>/.local/bin/clearml-agent", line 8, in <module>
sys.exit(main())
File "/home/<home>/.local/lib/python3.8/site-packages/clearml_agent/__main__.py", line 83, in main
return run_command(parser, args, command_name)
File "/home/<home>/.local/lib/python3.8/site-packages/clearml_agent/__main__.py", line 46, in run_command
return func(**args_dict)
` File "/home/<home>/.local/lib/python3....
Try to run docker ps
and check if all of your clearml containers up and running (should be 8 total)
@<1523701087100473344:profile|SuccessfulKoala55>
When I run clearml-agent init
I don't have a file prior to this. I tried running agent daemon with clearml.conf
created by clearml-init
but that doesn't work since it has no agent section, right? I know I can add it myself but I think clearml-agent init
should function too
@<1523701087100473344:profile|SuccessfulKoala55> I run it from local machine, that's right. When I run the task it says it can't clone repository. In the web UI on my task there's a REPOSITORY string. It's a correct ssh URL to my repo but it's missing git@
after ssh://
If I add the git part to it by editing the task and queuing again it works. In my config file I have option force_git_ssh_user: git
enabled.
@<1523701087100473344:profile|SuccessfulKoala55> I reloaded agent couple of times, cleared cache and for some reason it works now! Anyways, thanks for your help!
Thank you, got it. I tried it because I couldn't figure out how to make auto-detection work. When I run a task from my local project folder (which is also a git repo) via Task.init
it says that no repository was found. Also there is Task.create
method which lets you pass git URL but I suspect the Task.init
is more preferrable method
@<1523701087100473344:profile|SuccessfulKoala55>
from random import random
from clearml import Task, TaskTypes
import pandas as pd
task: Task = Task.init(
project_name="My Project",
task_name='Sample task',
task_type=TaskTypes.inference
)
task.connect(args)
task.execute_remotely(queue_name="default")
value = random()
task.get_logger().report_single_value(name="sample_value", value=value)
df = pd.DataFrame.from_dict({'col_1': [3, 2, 1, 0], 'col_2': ['a', 'b', 'c', 'd']})...
@<1523701087100473344:profile|SuccessfulKoala55>
I managed to create clearml.conf file with clearml-agent init
after fixing proxy problem. And now trying to run daemon with this conf file. I suspect something is missing from it since request validator fails with missing attribute
Sorry for bothering but I am really lost, I think I exhausted all my options. I really have no clue what is going on.
Also, previous problem was in incorrect proxy configuration on agent machine
SmugDolphin23 Sorry to bother again, output_uri should be a URI to S3 endpoint or clear ml fileserver? If it's not provided artifacts are stored locally, right?
@<1523701070390366208:profile|CostlyOstrich36>
What agent-services is doing on start up? Seems like something is preventing it from properly working. I already added a command to entrypoint to configure pip.conf since we have to use a trusted mirror to download python packages. Also I managed to connect local agent to ClearML server by using 127.0.0.1 host in credentials. Still no luck with remote agent
It works like I mentioned before: the terminal jumps on a new line and sits there, no output after that, nothing is happening in the console. But if you go to UI you see that "Last used" is updating
SuccessfulKoala55 So my question is how to setup auto-detection properly so worker knows what git repo to pull from
Right, seems the lib was severely outdated
` % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100k 100 100k 0 0 10236 0 0:00:10 0:00:10 --:--:-- 21354
Warning: Transient problem: HTTP error Will retry in 10 seconds. 10 retries
Warning: left.
100 100k 100 100k 0 0 10237 0 0:00:10 0:00:10 --:--:-- 21345
Warning: Transient problem: HTTP error Will retry in 10 seconds. 9 retries
Warning: left...
CostlyOstrich36 Any thoughts?
clearml-agent daemon --foreground
@<1523701087100473344:profile|SuccessfulKoala55> Fixed it by setting env var with path to certificates. I was sure that wouldn't help since I can curl and python get request to my endpoint from shell just fine. Now it says I am missing security headers, seems it's something on my side. Will try to fix this
clearml 1.9.0
clearml-agent 1.5.1
NAME="Ubuntu"
VERSION="18.04.6 LTS (Bionic Beaver)"
The terminal hangs on the command
SmugDolphin23 That fixed the issue, thank you very much!
Console output of clearml-agent daemon --foreground
?
CostlyOstrich36 Am I right I should also provide this URLS in agent-services section in docker-compose file?
CLEARML_HOST_IP: ${CLEARML_HOST_IP:-}
CLEARML_WEB_HOST: ${CLEARML_WEB_HOST:-}
CLEARML_API_HOST: http://apiserver:8008