I was able to set up a GCP VM manually earlier, like without the autoscaler

Well the VM is running in the default docker nvidia/cuda:10.2-cudnn7-runtime-ubuntu18.04, but it's not spinning up the agent when the VM is intialized

so my model is not able to find the dataset

I did provide the credentials, and also I am running up the autoscaler for the first time, so no it hasn't worked before

When the package installation is done in the task

All I need to do is

pip install -r requirements.txt
pip install .

I am able to get the requirements installed for each task

2023-10-03 20:46:07,100 - clearml.Auto-Scaler - INFO - Spinning new instance resource='clearml-autoscaler-vm', prefix='dynamic_gcp', queue='default'
2023-10-03 20:46:07,107 - googleapiclient.discovery_cache - INFO - file_cache is only supported with oauth2client<4.0.0
2023-10-03 20:46:07,122 - clearml.Auto-Scaler - INFO - Creating regular instance for resource clearml-autoscaler-vm
2023-10-03 20:46:07,264 - clearml.Auto-Scaler - INFO - --- Cloud instances (0):
2023-10-03 20:46:07,482 - clearm...

So, one of my tasks requires GCP credentials json file, is there a way that I can pass in the json file and set the environment variable for that?

It has my project ID and zone

Hey, so I am able to spin up the GCP instance using the autoscaler, I wanted to confirm one thing does the autoscaler spins up the agent automatically in the VM or do I need to add the script for that to the bash script

so inside /Users/adityachaudhry/.clearml/venvs-builds.1/3.10/task_repository/ I have my git repo, I have one component that make a dataset directory inside this git repo, but when the other component starts executing this dataset directory is not there

So, I am able to resolve the above issues

I don't think it has issues with this

is there a way that there is only one environment for the whole pipeline?

I want to know how to execute pip install . to import all the custom packages

yes same env for all the components

how's that?

thanks for the help though!!

I am providing pillow>=8.3.1 in my req.txt and I think clearml has Pillow==10.0.0 already

I think I got it resolved

So I am running a pipeline on a GCP VM, my VM has 1 NVIDIA GPU, and my requirements.txt has torch==1.13.1+cu117
torchvision==0.14.1+cu117

When I am running the Yolo training step I am getting the above error.

Ok, I think I was able to resolve that issue, but now when it's installing the packages I am getting Double requirement given Error for pillow

Ok, I'll try that out

Because I think I need to have the following two lines in the .bashrc and the Google_Application_credentials
git config --global user.email 'email'
git config --global user.name "user_name"

Just a follow up on this issue, @<1523701087100473344:profile|SuccessfulKoala55> @<1523701205467926528:profile|AgitatedDove14> I would very much appreciate it if you could help me with this.

I am not able to see cu117 there

Ok I was able to resolve the above issue, but now I am getting the following error while executing a task

import cv2
File "/root/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/cv2/init.py", line 181, in <module>
bootstrap()
File "/root/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/cv2/init.py", line 153, in bootstrap
native_module = importlib.import_module("cv2")
File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module
return _boots...