I think I am missing one part — which command do I use on my local machine, to indicate the job needs to be run remotely? I’m imagining something like
clearml-remote run python3 my_train.py

AgitatedDove14 thanks yes I assume I would follow these instructions:
https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_gcp

(and a way to specify which remote server)

got it, nice, thanks

thanks, so I got clearml-task working, sent to a queue and the agent on gcp picked it up. I had a question — for a job that runs on the order of minutes, it’s not worth re-creating the whole python virtual env from scratch on the remote (that itself takes 5mins). So is the --folder ` option meant for running it in an existing folder in an existing virtual env?

Yes after installing , it listed the installed packages in the console , with version of each

I used
task.execute_remotely(queue_name=..., clone=True)and indeed it instantly activates the venv on the remote. I assume clone=True is fine

I'm not familiar with “installed package s” list in the task

That seems pretty powerful, will give that a try, thanks

So net-net does this mean it’s behaving as expected, or is there something I need to do enable “full venv cache”? It spends nearly 2 mins starting from
created virtual environment CPython3.8.10.final.0-64 in 97ms creator CPython3Posix(dest=/home/pchalasani/.clearml/venvs-builds/3.8, clear=False, global=False)and then printing several lines lines like this
` Successfully installed pip-20.1.1
Collecting Cython
Using cached Cython-0.29.30-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86...

Thanks for the quick response . Will look into this later , I think I understand

CLI doesn’t care about the state of my git repo right?

should I nuke the .clearml/cache

it finally finished no worries

Actually that did not help. Still same behavior

But “cloning” via UI runs an exact copy of the code/config, not a variant, unless I edit those via UI (which is not ideal). So it looks like the following workflow that is trivial to do locally is not possible via remote agents:
run exp tweak code/configs in IDE, or tweak configs via CLI have it re-rerun in exact same venv (with no install overhead etc)
So maybe the remote agents are more meant for enqueuing a whole collection of settings (via code) and checking back in a few hours (in which ...

I see, so there’s no way to launch a variant of my last run (with say some config/code tweaks) via CLI, and have it re-use the cached venv?

I think I’m starting to “get” this 🙂

created a new dataset 5GB, no update since 20 mins, is that normal?

didn’t help

saas

thanks

Great, and this would show up in the description column in the dashboard ?

Dataset.get works fine from python script, it pulls in the data into cache. Just the cli seems broken

this is great… so it looks like best to do it in a new dir

Thanks, I guess I need to have a bucket under Cloud Storage?

I would also be interested in a GCP autoscaler, I did not know it was possible/available yet.

I guess I follow these steps on a GCP instance?
https://clear.ml/docs/latest/docs/clearml_agent

So if I want to train with a remote agent on a remote machine, I have to:
spin up clearml-agent on the remote create a dataset using clearml-data, populate with data… from my local machine use clearml-data to upload data to google gs:// bucket modify my code so it accesses data from the dataset as here https://clear.ml/docs/latest/docs/clearml_data/clearml_data_sdk#accessing-datasetsAm I understanding right?

👍