Sounds doable, I will give it a try.
The task.execute_remotely
thing is quite interesting, I didn't know about that!
Hi SmallDeer34
Is the Dataset in clearml-data ? If it is then Dataset.get().get_local_copy() will get you a cached local copy of the entire dataset.
If it is not, then you can use StorageManager.get_local_copy(url_here) to download the dataset.
- Any Argparser is automatically logged (and later can be overridden from the UI). Specifically HfArgumentParser will be automatically logged https://github.com/huggingface/transformers/blob/e43e11260ff3c0a1b3cb0f4f39782d71a51c0191/examples/pytorch/language-modeling/run_mlm.py#L200
Basically I would do the following:
Clone the huggingface repo to your dev machine.
Edit locally the run_mlm.py:
Add Task.init call add the Dataset download / StorageManager download. add `task.execute_remotely(queue_name='my_gpu_queue')This will make sure that all the local changes are automatically restored on the remote machine, it will auto populate the default arguments, and it will stop the local execution and relaunch the Task on the remote GPU.
wdyt?
Gave it a try, it seems our GPU Queue doesn't have the S3 creds set up correctly. Making a separate thread about that