Versions in compose are:
image: allegroai/clearml:1
image: elasticsearch:7.6.2
image: mongo:4.4.9

I am not quite sure that backups were made on those versions. Is there a way to see service versions from backup?

Console output of clearml-agent daemon --foreground ?

CostlyOstrich36 Any thoughts?

clearml-agent daemon --foreground

I looked through agent-services logs and found new error I haven't seen before:
clearml_agent: ERROR: Connection Error: it seems *api_server* is misconfigured. Is this the ClearML API server http://<my_ip>:8008 ?

SucculentCrab55 I've had this problem when I tried to launch UI too quickly, try to wait a bit and then check UI again

Thanks for the reply! We have a custom S3 server, it has an URL — endpoint like https://<some-domain>.<sub-domain>. I've read in docs that when you provide credentials.host — port must be specified. @<1523701070390366208:profile|CostlyOstrich36>

@<1523701087100473344:profile|SuccessfulKoala55>
When I run clearml-agent init I don't have a file prior to this. I tried running agent daemon with clearml.conf created by clearml-init but that doesn't work since it has no agent section, right? I know I can add it myself but I think clearml-agent init should function too

CostlyOstrich36 Yep, it seems it was the case. I did not provide credentials for API in docker compose. I did that but now agent-services just keeps restarting. I looked into containers logs and it seems to be a proxy error. Why this container is trying to connect somewhere?

It works like I mentioned before: the terminal jumps on a new line and sits there, no output after that, nothing is happening in the console. But if you go to UI you see that "Last used" is updating

CostlyOstrich36 Am I right I should also provide this URLS in agent-services section in docker-compose file?
CLEARML_HOST_IP: ${CLEARML_HOST_IP:-}
CLEARML_WEB_HOST: ${CLEARML_WEB_HOST:-}
CLEARML_API_HOST: http://apiserver:8008

The strange thing also is that I see that the credentials are being used in web UI: last used timestamp is updated constantly to present time. So apparently daemon is trying to do something but can't launch properly all the way

~/.local/bin/clearml-agent daemon --foreground

@<1523701087100473344:profile|SuccessfulKoala55>
So, I did it with debug and got this stacktrace error:
type_checker=validator.TYPE_CHECKER.redefine_many({
AttributeError: type object 'Draft4Validator' has no attribute 'TYPE_CHECKER'

@<1523701070390366208:profile|CostlyOstrich36>

@<1523701070390366208:profile|CostlyOstrich36> You mean using port in credentials.host ?

@<1523701087100473344:profile|SuccessfulKoala55> Right

After I run my experiment I have a console error that says I am missing security headers. This is a custom XML response. The same behaviour could be achieved when just trying to curl the endpoint or plug it in the browser. When I run e.g. boto3 client where I explicitly specify endpoint, ak, sk and bucket I could do whatever I want. So it seems to me ClearML is trying to get to this endpoint in some incorrect way

My question could be this: what's get plugged into endpoint_url in boto3 client inside ClearML?

Also, previous problem was in incorrect proxy configuration on agent machine

session = boto3.Session(
    aws_access_key_id=self.access_key,
    aws_secret_access_key=self.secret_key)

He tried to help me in another thread but I still couldn't make things work

@<1722061389024989184:profile|ResponsiveKoala38> It fixed the issue!

@<1722061389024989184:profile|ResponsiveKoala38> Sure, I'll get back to you as it finishes

Should I remove "case-sensitive" option from a query?

I'll get back to you in a minute

@<1523701435869433856:profile|SmugDolphin23> Hello, again! I tried to fill the values by your example. Still no luck. I noticed console log on my task says that I have certificate error. I disabled it in api section in clearml.conf like this: verify_certificate = false and I still have SSL error. Any clues why would that be?

curl -XPOST -H 'Content-Type: application/json' 'localhost:9200/events-training_debug_image-*/_update_by_query?conflicts=proceed' -d'{
    "script": {
        "source": "ctx._source.url = ctx._source.url.replace('http://<MY_OLD_ADDRESS>', '

.<NEW_ADDRESS>')",
        "lang": "painless"
    },
    "query": {"prefix": {"url": {"value": "http://<MY_OLD_ADDRESS>"}}}
}'

I figured it was something with syntax but I thought single ticks inside double quotes should work without escaping 😀

• 482e96243041 allegroai/clearml:latest "python3 -m jobs.asy…" 18 months ago Up 7 weeks 8008/tcp, 8080-8081/tcp async_delete
• 26c677f2b70f allegroai/clearml:1 "/opt/clearml/wrappe…" 18 months ago Up 16 months 8008/tcp, 8080-8081/tcp, 0.0.0.0:8080->80/tcp, :::8080->80/tcp clearml-webserver
• `7e2cf4462f44 allegroai/clearml:1 "/opt/clearml/wrappe…" 18 months ago Up 7 months 0.0.0.0:8008->8008/tcp, :::8008->8008/tcp, 8080-8081/tcp clearml-apiserv...