@<1798162804348293120:profile|FlutteringSeahorse49> wants to start HPO though, so the desire is to deploy agents to listen to queues on the slurm cluster (perhaps the controller runs on his laptop).

would that still make sense?

credentials for the server to do things with s3 will be in /opt/clearml/apiserver.conf.

I think you’d have to run the cleanup service. That’s what seems to be what is controlling deletion based on archived status and some other temporal filters

the clearml github, search for a file named cleanup service dot py (or something to that effect)

@<1541954607595393024:profile|BattyCrocodile47> put together None

maybe an important note: I mounted the same cache directory for the agents.

Might be under examples

dug deeper. if i'm to make a guess...
/root/clearml.conf -> used on startup of agent-services as a template of sorts to create .clearml_agent.<id>.cfg on demand -> this task-specific file is used to mount to /tmp/clearml_default.conf in a new container (docker in docker bc of the socket mounted to the agent-services) -> used to execute the task

one note is that it happened after I tried deploying a set of workers to a new queue, which she tried to use to run the tasks in parallel instead of our default queue which is only serviced by one worker (a container i built)

then back to CLI, updated the pipeline to point the tasks to the new queue. run it, shows up in the UI (same container as default worker, just replicated w a new docker-compose and CMD to point to the new queue).

i think we may have found the frankenbug?

the argument to the dataset name was not being overridden correctly (mistyped), so the default value of an empty string (instead of a placeholder like "CHANGE_ME") in the parent task caused the dataset to basically get created with an empty name, and somehow that hid the whole project, despite hundreds of existing tasks in it.

and no way to un-hide it as far as I can tell?

the dataset, task, and pipeline were under the same project name. i'm seeing what happens if the dataset project name was different ( f"{project_name}_data" ). which project would get deleted... the dataset one or the project of the task that kicked it off?

and the answer is...
the project is preserved, the dataset's project hidden.

so ... empty dataset names due to a small typo in parameter override + the choice for the dataset to have the same project name as the task that created it (...

Oh yes. I see. Yeah, no ML here actually (doing the testing infra of endpoints), but certainly when there is its an issue.

How does clearml session avoid it? I guess only if autoscaling is used (one worker one machine)?

i will attempt to start that now.

you could also take the route of NOT specifying num_layers, and instead write your own code to create a set of viable layer designs to choose from and pass that as a parameter, so optuna selects from a countable set instead of suggesting integer values .

the downside of this is the lack of gradient information in the optimization process

the project wasn't hidden before. I'm aware of the pipeline tasks being hidden, that makes sense for organization. but the actual project itself as an entirety has a ghost icon.

she created a new project and started working in there, it was visible in the UI... and just now it disappeared again. it's kind of like running the pipeline makes it disappear.

ah . that's a shame its under Enterprise only . no wonder I missed it .

im helping train my friend @<1798162804348293120:profile|FlutteringSeahorse49> on clearml to assist with his astrophysics research, and his university has a slurm cluster . So we're trying to figure out if we can launch an agent process on the cluster to pull work from the clearml queue (fwiw: on their cluster containers is not supported ) .

and now its gone.

I opened github.com/allegroai/clearml/pull/1083 as an attempt to help catch this.

Can vouch, this works well. Had my server hard reboot (maybe bc of clearml? maybe bc of hardware, maybe both… haven’t figured it out), and busy remote workers still managed to update the backend once it came back up.

Re: backups… what would happen if zipped while running but no work was being performed? Still an issue potentially?

and what happens if docker compose down is run while there’s work in the services queue? Will it be restored? What are the implications if a backup is perform...

If you can hit the endpoint with curl, you for sure can hook it up to many frontend frameworks.

Personal recs: gradio, streamlit

Abstract the interaction into a function call, and wrap it all in some UI elements using python.

so when the task completed successfully (changed the queue to default and let it finish instead of aborting), the project disappeared.

Oh neat! I want to take a look at this. Only a few more weeks at the client but it’d be nice to reduce the complexity of the software stack if I can before handoff.

Can you please elaborate on the latter point? My jupyterhub’s fully containerized and allows users to select their own containers (from a list i built) at launch, and launch multiple containers at the same time, not sure I follow how toes are stepped on.

youre basically asking to sample from a distribution where not all parameters are mutually independent .

the short answer is no- this is not directly supported . optuna needs each hyperparam to be independent, so its up to you to handle the dependencies between parameters yourself unfortunately .

your solution of defining them independently and then using num_layers to potentially ignore other parameters is a valid one .

I ran into something similar during deployment. Hopefully this helps with your debugging: if the agent was launched separately from the rest of the stack, it may not have proper docker-DNS resolution to None . (e.g. if in the same docker-compose, perhaps you didnt add the backend network field, or if it was launched separately through docker run without an explicit external network defined)

if the agent's on the same machine, try docker network connect to add...

waiting now to see if they disappear.
any problems you may have spotted with the versions used?

project hasn't disappeared just yet. but it's happened twice now

didnt disappear this time.

Yup if you scroll through the logs in the console, near the top (post config dump), you’ll see a git clone and checkout to the specific hash.

PS You can actually change this parameter in an experiment’s configuration if it is in draft mode.

probably, but the syntax would be in that of a git diff, so it’d be a touch clunky if you asked me
Are you trying to avoid local development?

hm, you should be able to hit None if docker networking is working properly. it shouldn't need to go through the internet to get back to your machine.