Oh neat! I want to take a look at this. Only a few more weeks at the client but it’d be nice to reduce the complexity of the software stack if I can before handoff.

Can you please elaborate on the latter point? My jupyterhub’s fully containerized and allows users to select their own containers (from a list i built) at launch, and launch multiple containers at the same time, not sure I follow how toes are stepped on.

Can vouch, this works well. Had my server hard reboot (maybe bc of clearml? maybe bc of hardware, maybe both… haven’t figured it out), and busy remote workers still managed to update the backend once it came back up.

Re: backups… what would happen if zipped while running but no work was being performed? Still an issue potentially?

and what happens if docker compose down is run while there’s work in the services queue? Will it be restored? What are the implications if a backup is perform...

I tried mounting a config file (in the structure of the one on github but with just the relevant s3 section) into the agent-services container at /root/clearml.conf and after restarting the container, it seems to have had an impact. thank you!

When I inspect the console of the task I'm trying to run, I see there's a call to cp /tmp/clearml.conf ~/default_clearml.conf in the docker command and that the volume /tmp/clearml.conf is picked up from the host at some custom-named file ...

tasks that create pipelines feels like a hack and i found they dont show up in the UI (have to use the link in the console).

I've found that sometimes i need to right click "Run" a couple of times before the parameters are filled in properly.

probably, but the syntax would be in that of a git diff, so it’d be a touch clunky if you asked me
Are you trying to avoid local development?

Yup if you scroll through the logs in the console, near the top (post config dump), you’ll see a git clone and checkout to the specific hash.

PS You can actually change this parameter in an experiment’s configuration if it is in draft mode.

thank you!
I'll add a volume mount to the services-agent container, and from what I understand that will become the template it uses?

is this the structure of the file?
None

or is it the "dot" syntax (like what shows up in the console when the task executes / your snippet)?

For reproducibility, it kind of makes sense though. The existence of the file is contingent on the worker cloning the source code. I'm sure things can be done to maintain state differently but I personally adapted to the git-based workflow for managing files pretty quickly.

though yes I will admit I had the same thought first: why must I run it each time?

Beware: squash merges will ruin the ability to reproduce the experiment at that time since the git commit will be lost (presuming th...

namely, I'm very interested in testing this unmerged feature, will be trying to leverage it as soon as possible
None

ah, thank you for the clarity. A quarterly release schedule makes sense, it's about what I've observed.
Let me know if I can be of any assistance in early testing!

the dataset, task, and pipeline were under the same project name. i'm seeing what happens if the dataset project name was different ( f"{project_name}_data" ). which project would get deleted... the dataset one or the project of the task that kicked it off?

and the answer is...
the project is preserved, the dataset's project hidden.

so ... empty dataset names due to a small typo in parameter override + the choice for the dataset to have the same project name as the task that created it (...