it's pretty much the same

SmugDolphin23

Oh, it's configured o agent machine, got you

SuccessfulKoala55 So my question is how to setup auto-detection properly so worker knows what git repo to pull from

SucculentCrab55 I've had this problem when I tried to launch UI too quickly, try to wait a bit and then check UI again

Try to run docker ps and check if all of your clearml containers up and running (should be 8 total)

@<1523701087100473344:profile|SuccessfulKoala55>

from random import random
from clearml import Task, TaskTypes
import pandas as pd

task: Task = Task.init(
    project_name="My Project",
    task_name='Sample task',
    task_type=TaskTypes.inference
)

task.connect(args)
task.execute_remotely(queue_name="default")

value = random()
task.get_logger().report_single_value(name="sample_value", value=value)

df = pd.DataFrame.from_dict({'col_1': [3, 2, 1, 0], 'col_2': ['a', 'b', 'c', 'd']})...

@<1523701087100473344:profile|SuccessfulKoala55> I run it from local machine, that's right. When I run the task it says it can't clone repository. In the web UI on my task there's a REPOSITORY string. It's a correct ssh URL to my repo but it's missing git@ after ssh:// If I add the git part to it by editing the task and queuing again it works. In my config file I have option force_git_ssh_user: git enabled.

@<1523701087100473344:profile|SuccessfulKoala55> I reloaded agent couple of times, cleared cache and for some reason it works now! Anyways, thanks for your help!

@<1523701435869433856:profile|SmugDolphin23> I didn't use a region at first and that was not working. Now I use a region and it still doesn't work.
From the boto3 inside a Python I could create a session where I specify ak and sk, and create a client from the session where I pass service_name and endpoint_url. It works just fine

Right, seems the lib was severely outdated

jsonschema 2.6.0

SmugDolphin23 That fixed the issue, thank you very much!

Traceback (most recent call last):
File "/home/<home>/.local/bin/clearml-agent", line 8, in <module>
sys.exit(main())
File "/home/<home>/.local/lib/python3.8/site-packages/clearml_agent/__main__.py", line 83, in main
return run_command(parser, args, command_name)
File "/home/<home>/.local/lib/python3.8/site-packages/clearml_agent/__main__.py", line 46, in run_command
return func(**args_dict)
` File "/home/<home>/.local/lib/python3....

CostlyOstrich36
The error appears regardless of --foreground tag. This is not full stacktrace, I will provide it with the next message.
clearml 1.9.0
clearml-agent 1.5.1
Ubuntu1 8.04.6 LTS

@<1523701087100473344:profile|SuccessfulKoala55> Hey, Jake, getting back to you. I couldn't be able to resolve my issue. I can access my bucket by any means just fine, e.g. by S3 CLI client. All the tools I use require 4 params: AK, SK, endpoint, bucket. I wonder why ClearML doesn't have explicit endpoint parameter and you have to use output_uri for it and why is there a region when other tools don't require it.

@<1523701435869433856:profile|SmugDolphin23> Hello, again! I tried to fill the values by your example. Still no luck. I noticed console log on my task says that I have certificate error. I disabled it in api section in clearml.conf like this: verify_certificate = false and I still have SSL error. Any clues why would that be?

SmugDolphin23 Sorry to bother again, output_uri should be a URI to S3 endpoint or clear ml fileserver? If it's not provided artifacts are stored locally, right?

Thank you, got it. I tried it because I couldn't figure out how to make auto-detection work. When I run a task from my local project folder (which is also a git repo) via Task.init it says that no repository was found. Also there is Task.create method which lets you pass git URL but I suspect the Task.init is more preferrable method

@<1523701087100473344:profile|SuccessfulKoala55> Could you provide a sample of how to properly fill all the necessary config values to make S3 work, please?
My endpoint starts with https:// and I don't know what my region is, endpoint URL doesn't contain it.
Right now I fill it like this:

aws.s3.key = <access-key>
aws.s3.secret = <secret-key>
aws.s3.region = <blank>
aws.s3.credentials.0.bucket = <just_bucket_name>
aws.s3.credentials.0.key = <access-key>
aws.s3.credentials.0.secret ...

clearml 1.9.0
clearml-agent 1.5.1
NAME="Ubuntu"
VERSION="18.04.6 LTS (Bionic Beaver)"

Sorry, forgot to mention. I used the command with --foreground tag. It is the same. Terminal just sits at a new line, no logs, no worker in UI

Docstring from inside the boto3 lib says:

:param endpoint_url: The complete URL to use for the constructed
    client. Normally, botocore will automatically construct the
    appropriate URL to use when communicating with a service.  You
    can specify a complete URL (including the "http/https" scheme)
    to override this behavior.  If this value is provided,
    then ``use_ssl`` is ignored.

I want ClearML to use my endpoint

@<1523701087100473344:profile|SuccessfulKoala55> It's the URL I use when creating boto3 session from Python like this fro example

s3 = self.session.client(
    service_name='s3',
    endpoint_url=endpoint,
    verify=False
)

@<1523701087100473344:profile|SuccessfulKoala55> No port needed when accessing this URL from things like boto3 or s3-client CLI

My question could be this: what's get plugged into endpoint_url in boto3 client inside ClearML?

session = boto3.Session(
    aws_access_key_id=self.access_key,
    aws_secret_access_key=self.secret_key)

@<1523701087100473344:profile|SuccessfulKoala55> Right

Sorry, guys, maybe I am not expressing myself clear or it's something I am missing, I am not a native speaker so I'll try to reformulate. What we have is enterprise solution built on S3 technology, I don't have an access to servers on where it's run, I don't have a port. All I have been provided with are: secret key, access key, endpoint that looks like a regular web URL and a bucket name. Using these creds I can access this cloud storage just fine by any means except ClearML

After I run my experiment I have a console error that says I am missing security headers. This is a custom XML response. The same behaviour could be achieved when just trying to curl the endpoint or plug it in the browser. When I run e.g. boto3 client where I explicitly specify endpoint, ak, sk and bucket I could do whatever I want. So it seems to me ClearML is trying to get to this endpoint in some incorrect way