2023-10-03 20:46:07,100 - clearml.Auto-Scaler - INFO - Spinning new instance resource='clearml-autoscaler-vm', prefix='dynamic_gcp', queue='default'
2023-10-03 20:46:07,107 - googleapiclient.discovery_cache - INFO - file_cache is only supported with oauth2client<4.0.0
2023-10-03 20:46:07,122 - clearml.Auto-Scaler - INFO - Creating regular instance for resource clearml-autoscaler-vm
2023-10-03 20:46:07,264 - clearml.Auto-Scaler - INFO - --- Cloud instances (0):
2023-10-03 20:46:07,482 - clearml.Auto-Scaler - INFO - stopping
2023-10-03 20:46:07,482 - clearml.Auto-Scaler - INFO - state change: State.RUNNING -> State.STOPPED
2023-10-03 20:46:07,482 - clearml.Auto-Scaler - INFO - Autoscaler exits
2023-10-03 20:46:07,556 - clearml.Auto-Scaler - ERROR - Failed to start new instance (resource 'clearml-autoscaler-vm'), Error: <HttpError 404 when requesting https://compute.googleapis.com/compute/v1/projects/../zones/ returned "The resource 'projects/../zones/us-central1-a/ was not found". Details: "[{'message': "The resource 'projects/../zones/ was not found", 'domain': 'global', 'reason': 'notFound'}]">
Traceback (most recent call last):
File "/root/.clearml/venvs-builds/3/task_repository/clearml-apps.git/apps/auto_scaler/auto_scaler.py", line 742, in launch_one
instance_id = self.driver.spin_up_worker(resource_conf, worker_prefix, queue, task_id=task_id)
File "/root/.clearml/venvs-builds/3/task_repository/clearml-apps.git/apps/auto_scaler/cloud_driver.py", line 281, in spin_up_worker
instance_id, region = self._spin_up_worker(resource_conf, worker_prefix, queue_name, task_id)
File "/root/.clearml/venvs-builds/3/task_repository/clearml-apps.git/apps/auto_scaler/gcp_driver.py", line 194, in _spin_up_worker
exc, response = f(*args)
File "/root/.clearml/venvs-builds/3/task_repository/clearml-apps.git/apps/auto_scaler/networking.py", line 18, in wrapper
return func(obj_instance, *args, **kwargs)
File "/root/.clearml/venvs-builds/3/task_repository/clearml-apps.git/apps/auto_scaler/gcp_driver.py", line 137, in attempt_launch
spin_up_client.instances().insert(project=self.gcp_project_id, zone=zone, body=launch_spec).execute()
File "/root/venv/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 131, in positional_wrapper
return wrapped(*args, **kwargs)
File "/root/venv/lib/python3.8/site-packages/googleapiclient/http.py", line 937, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 404 when requesting https://compute.googleapis.com/compute/v1/projects/zones/../ returned "The resource 'projects/../zones/' was not found". Details: "[{'message': "The resource 'projects/../zones/us-central1-a/acceleratorTypes/NVIDIA T4' was not found", 'domain': 'global', 'reason': 'notFound'}]">

1696365971074 apps-agent-i-08bf8b26b6175ea1f-1:service:8d816e475307473885aaa87b52a5c526 DEBUG Process aborted by user