Badges 141 × Eureka!
should I nuke the
CLI doesn’t care about the state of my git repo right?
Dataset.get works fine from python script, it pulls in the data into cache. Just the cli seems broken
it finally finished no worries
got it, nice, thanks
no containers for me 😁
So if I do this in my local repo, will it mess up my git state, or should I do it in a fresh directory?
this is great… so it looks like best to do it in a new dir
So if I want to train with a remote agent on a remote machine, I have to:
spin up clearml-agent on the remote create a dataset using clearml-data, populate with data… from my local machine use clearml-data to upload data to google gs:// bucket modify my code so it accesses data from the dataset as here https://clear.ml/docs/latest/docs/clearml_data/clearml_data_sdk#accessing-datasetsAm I understanding right?
I have been using 1.6.2
I mean it is in Pip mode and the agent installs deps from git repo that it pulls
Thanks, I guess I need to have a bucket under Cloud Storage?
thanks, so I got clearml-task working, sent to a queue and the agent on gcp picked it up. I had a question — for a job that runs on the order of minutes, it’s not worth re-creating the whole python virtual env from scratch on the remote (that itself takes 5mins). So is the
--folder ` option meant for running it in an existing folder in an existing virtual env?
AgitatedDove14 thanks yes I assume I would follow these instructions:
I think I am missing one part — which command do I use on my local machine, to indicate the job needs to be run remotely? I’m imagining something like
clearml-remote run python3 my_train.py
I would also be interested in a GCP autoscaler, I did not know it was possible/available yet.
I'm not familiar with “installed package s” list in the task
Thanks for the quick response . Will look into this later , I think I understand
So net-net does this mean it’s behaving as expected, or is there something I need to do enable “full venv cache”? It spends nearly 2 mins starting from
created virtual environment CPython3.8.10.final.0-64 in 97ms creator CPython3Posix(dest=/home/pchalasani/.clearml/venvs-builds/3.8, clear=False, global=False)and then printing several lines lines like this
` Successfully installed pip-20.1.1
Using cached Cython-0.29.30-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86...
I see, so there’s no way to launch a variant of my last run (with say some config/code tweaks) via CLI, and have it re-use the cached venv?
Actually that did not help. Still same behavior
I have a strong attachment to a workflow based on CLI, nice zsh auto-suggestions, Hydra and the like. Hence why I moved away from dvc 🙂
base-task-id it uses the cached venv, thanks for this suggestion! Seems like this is equivalent to cloning via UI.
And I will look into the non-cli workflow you’re suggesting.
… but I have a feeling they will not give me the “instant venv activation” behavior I’m looking for.
task.execute_remotely(queue_name=..., clone=True)and indeed it instantly activates the venv on the remote. I assume clone=True is fine
I use a CLI arg remote=True so depending on that it will run locally or remotely.
This works great, thanks AgitatedDove14 👍
I think I’m starting to “get” this 🙂
A quick note for others who may visit this… it looks like you have to do:
Task.force_requirements_env_freeze(force=True, requirements_file="requirements.txt")to ensure any changes in requirements.txt are reflected in the remote venv