this is the response. We are using the clearml hosted server

Looks like it's picking up the projects but then viewing on the UI they disappear

cool thanks - I'll try it out

great thank you

ahhh okay thank you 🙂

gdn4.xlarge (the best price for 16GB of GPU ram). Not so surprising they would want a switch

we normally do something like that - not sure what why it's freezing for you without more info

Can you try to go into 'Settings' -> 'Configuration' and verify that you have 'Show Hidden Projects' enabled

thank you guys 😄 😄

This was the response from AWS:

"Thank you for for sharing the requested details with us. As we discussed, I'd like to share that our internal service team is currently unable to support any G type vCPU increase request for limit increase. 

The issue is we are currently facing capacity scarcity to accommodate P and G instances. Our engineers are working towards fixing this issue. However, until then, we are unable to expand the capacity and process limit increase."

Nope AWS aren't approving the increased vCPU request. I've explained the use case several times and they've not approved

<queues.GetQueueMetricsResponse: { "queues": [] }>

Usually that is shown here... for some reason it's not showing

okay so this could be a python script that generates the clearml.conf in the working dir in the container?

the agent it for replicating what you run locally elsewhere i.e. remote GPU machine

the Model object

next one I'm looking for is average queue time

From SuccessfulKoala55 suggestion

This was the error I was getting from uploads using the old SDK
has been rejected for invalid domain. heap-2443312637.js:2:108655 Referrer Policy: Ignoring the less restricted referrer policy "no-referrer-when-downgrade" for the cross-site request:

I'll like to call Run Time via the task object.... I think I need to calculate manually

i.e.

task = clearml.Task.get_task(id) time = task.data.last_update - task.data.started

Yep figured this out yesterday. I had been tagging G type instances with an alarm as a fail safe if the AWS autoscaler failed. The alarm only stopped the instance and didn't terminate it (which deletes the drive). Thanks anyway CostlyOstrich36 and TimelyPenguin76 🙂

Still debugging.... That fixed the issue with the
nvcr.io/nvidia/tritonserver:22.02-py3 container which now returns
` =============================
== Triton Inference Server ==

NVIDIA Release 22.02 (build 32400308)

Copyright (c) 2018-2021, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Co...

Hi yes all sorted ! 🙂

`
import os
import glob
from clearml import Dataset

DATASET_NAME = "Bug"
DATASET_PROJECT = "ProjectFolder"
TARGET_FOLDER = "clearml_bug"
S3_BUCKET = os.getenv('S3_BUCKET')

if not os.path.exists(TARGET_FOLDER):
os.makedirs(TARGET_FOLDER)

with open(f'{TARGET_FOLDER}/data.txt', 'w') as f:
f.writelines('Hello, ClearML')

target_files = glob.glob(TARGET_FOLDER + "/**/*", recursive=True)

# upload dataset

dataset = Dataset.create(dataset_name=DATASET_NAME, dataset_project=DATASET_PR...

` python upload_data_to_clearml_copy.py
Generating SHA2 hash for 1 files
100%|████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 733.91it/s]
Hash generation completed
0%| | 0/1 [00:00<?, ?it/s]
Compressing local files, chunk 1 [remaining 1 files]
100%|████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 538.77it/s]
File compression completed: t...

It was the CORS like error we were getting

Yes already tried that but it seems there's some form of mismatch with a C/C++ lib.

It might only be a req for the docker/docker-compose-triton-gpu.yml file but I'd need to check

I'll add a more detailed response once it's working

Yes correct . In fact we cannot see any old datasets