Reputation
Badges 1
27 × Eureka!SuccessfulKoala55 created file run after clearml-init
on cli
Is this bug resolved in latest version?
SuccessfulKoala55
The dataset file URL is set on upload and stored on the server
This might be a reason. I think server IP in machine A is set to "localhost:port"
Then, after I change IP "localhost" to "<server IP>" in server A and re upload Dataset, Is it accessible remotely?
` >>> d = Dataset.get(dataset_name="Anonymous task (user@beryl 2022-03-23 04:05:19)", dataset_project="test_project").get_local
_copy()
Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0c23e4b490>: Failed to establish a new connection: [Errno 111] Connection refused')': /test_project/Anonymous%20task%20%28user%2540beryl%202022-03-23%2004%253A05%253A19%29.c05641c2e1c74389b471fb...
SuccessfulKoala55 Thanks a lot. problem solved. Have a good day!
CostlyOstrich36
I'm taking a look if it's possible
Thank you for response. Dataset.squash
works fine. But squash function squash after download all datasets, so I think it's not proper to me cuz dataset size is huge. I'll try upload at once. BTW, is this a bug? or I did something wrong?
AbruptCow41 Yes, it's possible to do so, but I wanted to upload parallelly if I can and I'm wonder it's a kind of bug.
I just added manually to config file and it works fine
also, all dataset don't have any dependency(parent) on other dataset
It says some files not deleted on several experiments. But the files also not deleted on project which error not occurred.
CostlyOstrich36 I'm using WebApp: 1.3.1-169 • Server: 1.3.1-169 • API: 2.17
Thanks.
http://<IP>:8081/ranix_pp_result/2022%252F04%252F04.29adfa8dbfa24489a0d9e9[…]tifacts/data/dataset.29adfa8dbfa24489a0d9e959d947b971.zip
TimelyPenguin76 I think http://clear.ml webserver is down. I can't access webserver nor clearml API
What is desired output when I typed ls -al
at clearml folder?
I just use clearml-init
command and paste api key from localhost webserver like below. Is it normal that there's no files_server key in api dict?api { web_server:
api_server:
credentials { "access_key" = "my key" "secret_key" = "my key" } }
Error: Insufficient permissions for //:8081/files_server:
http://localhost:8081 This is my whole output.
BTW, Is there a possibility this problem have related to access permission of /opt/clearml folder?
In this docs step #9 , https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_linux_mac
It says "Grant access to the Dockers, depending upon the operating system."
Name: clearml Version: 1.1.6 Summary: ClearML - Auto-Magical Experiment Manager, Version Control, and MLOps for AI Home-page:
Author: Allegroai Author-email: clearml@allegro.ai License: Apache License 2.0 Location: /home/junpyo/.conda/envs/minikube/lib/python3.8/site-packages Requires: psutil, numpy, jsonschema, requests, urllib3, PyYAML, python-dateutil, future, attrs, pathlib2, Pillow, furl, pyjwt, six, pyparsing Required-by:
The weird thing is after I run clearml-data create ...
, I can check the project and dataset are created in draft mode withCLI spit out error above
I'm using ubuntu 18.04, and docker command was docker-compose -f /opt/clearml/docker-compose.yml up -d
as described at docs.
Also I tired to put fileserver information to configuration file, but nothing changed
How can I check SDK version?
I changed my directory to clearml-data installed and just run the same command with sudo ./clearml-data
and it works
SuccessfulKoala55 OMG I solved problem. the cause was the permission of the clearml-data
Thank you for supporting Jake
TimelyPenguin76 It's working now. Thx for your support.
I'm not sure about how airflow workers run. What I trying to do is upload "different files" to "one clearrml-dataset" in parallel. My dag looks like below, each task from "transform_group " execute clearml-related dataset tasks. Sorry for my bad explanation