For reproducibility, it kind of makes sense though. The existence of the file is contingent on the worker cloning the source code. I'm sure things can be done to maintain state differently but I personally adapted to the git-based workflow for managing files pretty quickly.
though yes I will admit I had the same thought first: why must I run it each time?
Beware: squash merges will ruin the ability to reproduce the experiment at that time since the git commit will be lost (presuming the branch was deleted post-merge)
huh, ok … the need to run stuff locally first is a little odd, ngl… just have to get a sense of what the workflow here is
but it looks like execute_remotely is indeed what I want, and I should just standardize on using that,
either in-situ, or in an outer runner script that just calls that first, then calls down to my actual implementation
you can put task.execute_remotely() to create it in draft mode. I've taken to configuring defaults to run things very quickly just in case i forget though (e.g. placeholder string for dataset, bail out early if not changed… or just do one epoch on a small subset of samples, etc).
🤔 could one create empty draft experiment placeholders, finalize all their code, commit it, and then and manually connect the impl as it exists in a file in a commit in a repo to that draft, once I’ve decided that all development is done?
probably, but the syntax would be in that of a git diff, so it’d be a touch clunky if you asked me
Are you trying to avoid local development?
but I’ll try out what you said - parameterize it all, and run a short version locally, then scale up in the UI
local development is fine, but local-running-first I am just unsure about
Yup if you scroll through the logs in the console, near the top (post config dump), you’ll see a git clone and checkout to the specific hash.
PS You can actually change this parameter in an experiment’s configuration if it is in draft mode.
🤔 does a job/experiment always reproduce from git + commit hash + specific file?
(again, haven’t read through all the docs yet, a lot to take in)