Reputation
Badges 1
104 × Eureka!After I run my experiment I have a console error that says I am missing security headers. This is a custom XML response. The same behaviour could be achieved when just trying to curl the endpoint or plug it in the browser. When I run e.g. boto3 client where I explicitly specify endpoint, ak, sk and bucket I could do whatever I want. So it seems to me ClearML is trying to get to this endpoint in some incorrect way
Sorry, forgot to mention. I used the command with --foreground tag. It is the same. Terminal just sits at a new line, no logs, no worker in UI
Traceback (most recent call last):File "/home/<home>/.local/bin/clearml-agent", line 8, in <module>sys.exit(main())File "/home/<home>/.local/lib/python3.8/site-packages/clearml_agent/__main__.py", line 83, in mainreturn run_command(parser, args, command_name)File "/home/<home>/.local/lib/python3.8/site-packages/clearml_agent/__main__.py", line 46, in run_commandreturn func(**args_dict)
` File "/home/<home>/.local/lib/python3....
@<1722061389024989184:profile|ResponsiveKoala38> There were errors everywhere, in almost every container. I decided to manually move working clearml image from old installation and run from it. It worked. It's not the latest ClearML but it'll do for now. Thanks for your help! 💪
@<1523701087100473344:profile|SuccessfulKoala55> I provided following env vars:
CLEARML_HOST_IP: "<my_ip>"
CLEARML_WEB_HOST: " http://<my_ip>:8080 "
CLEARML_API_HOST: " http://<my_ip>:8008 "
CLEARML_FILES_HOST: " http://<my_ip>:8081 "
CLEARML_API_ACCESS_KEY: <my_access_key>
CLEARML_API_SECRET_KEY: <my_secret_key>
also I changed IP in entrypoint from apiserver:8008 to <my_ip>:8008
Yes, I run both commands from the same place — dedicated user on my worker m...
@<1523701304709353472:profile|OddShrimp85> I fixed my SSL error by putting REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt in .bashrc file
My question could be this: what's get plugged into endpoint_url in boto3 client inside ClearML?
` from random import random
from clearml import Task, TaskTypes
args = {}
task: Task = Task.init(
project_name="My Proj",
task_name='Sample task',
task_type=TaskTypes.inference,
auto_connect_frameworks=False
)
task.connect(args)
task.execute_remotely(queue_name="default")
value = random()
task.get_logger().report_single_value(name="sample_value", value=value)
with open("some_artifact.txt", "w") as f:
f.write(f"Some random value: {value}\n")
task.upload_artifact(name="test...
` s3 {
# S3 credentials, used for read/write access by various SDK elements
# default, used for any bucket not specified below
key: "mykey"
secret: "mysecret"
region: " ` ` "
credentials: [
{
bucket: "mybucket"
key: "mykey"
secret: "mysecret"
region: " ` ` "
}, `
@<1523701435869433856:profile|SmugDolphin23> Thanks a lot, that actually worked! It was very difficult to figure out you have to plug those exact values given you have https endpoint:
- Using s3 protocol instead of https together with bucket name in output URI
- Not providing a bucket name in credentials section where it is by default
- Providing default secure port for both host and output URI
- Disabling credentials chainI think a common use case for many people that they get S3 storage wi...
@<1523701435869433856:profile|SmugDolphin23> @<1523701087100473344:profile|SuccessfulKoala55>
2023-02-03 20:38:14,515 - clearml.metrics - WARNING - Failed uploading to <my-endpoint> (HTTPSConnectionPool(host=' e ndpoint', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))))
2023-02-03 20:38:14,517 - clearml.metrics - ERROR - Not uploa...
@<1523701304709353472:profile|OddShrimp85> I haven't done it, for me it worked as-is
SmugDolphin23 Sorry to bother again, output_uri should be a URI to S3 endpoint or clear ml fileserver? If it's not provided artifacts are stored locally, right?
The code is run from another machine where clearml.conf configured to connect to ClearML server, no other configurations are provided
@<1523701435869433856:profile|SmugDolphin23> I didn't use a region at first and that was not working. Now I use a region and it still doesn't work.
From the boto3 inside a Python I could create a session where I specify ak and sk, and create a client from the session where I pass service_name and endpoint_url. It works just fine
SmugDolphin23 I added a region, run experiment again. Didn't work
It's the same request you provided just without "case_sensitive" option and with my endpoints @<1722061389024989184:profile|ResponsiveKoala38>
