DepravedBee82

5 Questions, 36 Answers

Active since 19 July 2024

Last activity 4 months ago

Reputation

Badges 1

36 × Eureka!

Questions 5
Answers 36

0 Votes

3 Answers

613 Views

0 Votes 3 Answers 613 Views

Hi All, Is There A Way To Completely Disable All Artifact Logging?

Hi all, is there a way to completely disable all artifact logging?

clearml

9 months ago

0 Votes

4 Answers

554 Views

0 Votes 4 Answers 554 Views

Hi All, What Is The Best Way Of Getting Clearml To Pull Code From Github Repos? At The Moment We Can Pull Using A Users Ssh Credentials, But Afaik It'S Not Possible To Create "Service Accounts" For Github. The Preferred Method For

Hi all, what is the best way of getting ClearML to pull code from GitHub repos? At the moment we can pull using a users SSH credentials, but AFAIK it's not p...

clearml

4 months ago

0 Votes

6 Answers

883 Views

0 Votes 6 Answers 883 Views

Hi All, We Have Clearml-Server Running On A Kube Pod, And Then A Gpu Server Running The Clearml-Agent Which We Use To Queue Jobs. For Some Reason, Our Kube Pod Restarted (We'Re Looking Into Why), But In The Process Of This Happening All Jobs On The Worke

Hi all, we have clearml-server running on a kube pod, and then a GPU server running the clearml-agent which we use to queue jobs. For some reason, our kube p...

clearml

9 months ago

0 Votes

46 Answers

108K Views

0 Votes 46 Answers 108K Views

Hi All, I'Ve Successfully Run A Task Locally, And Now I'M Trying To Clone It And Send It To A Queue. It Looks Like The Environment Is Built Successfully, But It Hangs Here:

Hi all, I've successfully run a Task locally, and now I'm trying to clone it and send it to a Queue. It looks like the environment is built successfully, but...

clearml

one year ago

0 Votes

3 Answers

720 Views

0 Votes 3 Answers 720 Views

Hi All - I Have A Large Dataset And Have Preprocessed It And Saved Each Item In .Pt Files, Which Are Loaded Using

Hi all - I have a large dataset and have preprocessed it and saved each item in .pt files, which are loaded using torch.load in my Dataset . The issue is tha...

clearml

9 months ago

0 Hi All, I'Ve Successfully Run A Task Locally, And Now I'M Trying To Clone It And Send It To A Queue. It Looks Like The Environment Is Built Successfully, But It Hangs Here:

Thank you! Although it's still really weird how it was failing silently - would it be worth changing the logging level for that error somewhere?

one year ago

0 Hi All, I'Ve Successfully Run A Task Locally, And Now I'M Trying To Clone It And Send It To A Queue. It Looks Like The Environment Is Built Successfully, But It Hangs Here:

If there was an SSL issue it should log to console right?

ClearML is hosted on an on-prem kube cluster and to get it to log locally I needed to append my company cert to the file located at certifi.where() . Do you think the same needs to be done for the Python installation for the worker?

one year ago

0 Hi All, I'Ve Successfully Run A Task Locally, And Now I'M Trying To Clone It And Send It To A Queue. It Looks Like The Environment Is Built Successfully, But It Hangs Here:

Ah yes you were right, it does still print on remote. Here you go:

environ({'LANG': 'en_GB.UTF-8', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'HOME': '/root', 'LOGNAME': 'root', 'USER': 'root', 'SHELL': '/bin/bash', 'INVOCATION_ID': '2cf51dc43b78470cb14c29f5f653ee18', 'JOURNAL_STREAM': '8:224108', 'SYSTEMD_EXEC_PID': '134947', 'PYTHONUNBUFFERED': '1', 'CUDA_DEVICE_ORDER': 'PCI_BUS_ID', 'CLEARML_WORKER_ID': 'mrl-plswh100:0', 'TRAINS_WORKER_ID': 'mrl-plswh100:0', 'CLEARM...

one year ago

0 Hi All, I'Ve Successfully Run A Task Locally, And Now I'M Trying To Clone It And Send It To A Queue. It Looks Like The Environment Is Built Successfully, But It Hangs Here:

I managed to set up my (Windows) laptop as a worker and reproduce the issue. Would that suggest an issue with ClearML server?

Our server is deployed on a kube cluster. I'm not too clear on how Helm charts etc. work, but if there are any obvious things we should check, let me know and I can ask our DevOps engineer

one year ago

0 Hi All, I'Ve Successfully Run A Task Locally, And Now I'M Trying To Clone It And Send It To A Queue. It Looks Like The Environment Is Built Successfully, But It Hangs Here:

I've added that flag, removed all PL loggers & callbacks and all references to Hydra, but no luck 😞

one year ago

0 Hi All, I'Ve Successfully Run A Task Locally, And Now I'M Trying To Clone It And Send It To A Queue. It Looks Like The Environment Is Built Successfully, But It Hangs Here:

Yes the agent is running in venv mode afaik. As for why it’s running as root - I’ll ask our engineer …

one year ago

Show more results