then back to CLI, updated the pipeline to point the tasks to the new queue. run it, shows up in the UI (same container as default worker, just replicated w a new docker-compose and CMD to point to the new queue).

yeah let's step through this, i'm having her execute these steps as we speak.

create a task with the new project name. its created as a draft. can see it in the UI under the new project.

pipeline script is updated with new project name for. execute script to create pipeline. now see in UI under this new project name. nothing hidden.

the pipeline is running. when the queue is default (only serviced by one container with agent in it ( clearml-agent==1.5.2 ). abort it. everything is still ...

Yup if you scroll through the logs in the console, near the top (post config dump), you’ll see a git clone and checkout to the specific hash.

PS You can actually change this parameter in an experiment’s configuration if it is in draft mode.

For reproducibility, it kind of makes sense though. The existence of the file is contingent on the worker cloning the source code. I'm sure things can be done to maintain state differently but I personally adapted to the git-based workflow for managing files pretty quickly.

though yes I will admit I had the same thought first: why must I run it each time?

Beware: squash merges will ruin the ability to reproduce the experiment at that time since the git commit will be lost (presuming th...

you can put task.execute_remotely() to create it in draft mode. I've taken to configuring defaults to run things very quickly just in case i forget though (e.g. placeholder string for dataset, bail out early if not changed… or just do one epoch on a small subset of samples, etc).

If you can hit the endpoint with curl, you for sure can hook it up to many frontend frameworks.

Personal recs: gradio, streamlit

Abstract the interaction into a function call, and wrap it all in some UI elements using python.

youre basically asking to sample from a distribution where not all parameters are mutually independent .

the short answer is no- this is not directly supported . optuna needs each hyperparam to be independent, so its up to you to handle the dependencies between parameters yourself unfortunately .

your solution of defining them independently and then using num_layers to potentially ignore other parameters is a valid one .

and now its gone.

I believe pipe.connect_configuration is what you're looking for?

ah . that's a shame its under Enterprise only . no wonder I missed it .

im helping train my friend @<1798162804348293120:profile|FlutteringSeahorse49> on clearml to assist with his astrophysics research, and his university has a slurm cluster . So we're trying to figure out if we can launch an agent process on the cluster to pull work from the clearml queue (fwiw: on their cluster containers is not supported ) .

hm, you should be able to hit None if docker networking is working properly. it shouldn't need to go through the internet to get back to your machine.

@<1541954607595393024:profile|BattyCrocodile47> put together None

one note is that it happened after I tried deploying a set of workers to a new queue, which she tried to use to run the tasks in parallel instead of our default queue which is only serviced by one worker (a container i built)

the clearml github, search for a file named cleanup service dot py (or something to that effect)

ah, thank you for the clarity. A quarterly release schedule makes sense, it's about what I've observed.
Let me know if I can be of any assistance in early testing!

the project wasn't hidden before. I'm aware of the pipeline tasks being hidden, that makes sense for organization. but the actual project itself as an entirety has a ghost icon.

she created a new project and started working in there, it was visible in the UI... and just now it disappeared again. it's kind of like running the pipeline makes it disappear.

i will attempt to start that now.

but isnt that just the same as running agent in daemon mode? thats what i was hoping James could do

if you commit but do not push, the metadata tells clearml that it needs to pull a non-existant commit. any changes you made on top may be saved as a diff, but they'd fail to apply.

for clearml to work on un-pushed commits, it'd have to wait for a push to register a new diff target, which can become a problem (what if you have multiple remotes? which one will it wait for?) so rather, it assumes it can access the most recent commit from your remote repo, and records this as the "base" upon whi...

credentials for the server to do things with s3 will be in /opt/clearml/apiserver.conf.

Might be under examples

namely, I'm very interested in testing this unmerged feature, will be trying to leverage it as soon as possible
None

so when the task completed successfully (changed the queue to default and let it finish instead of aborting), the project disappeared.

probably, but the syntax would be in that of a git diff, so it’d be a touch clunky if you asked me
Are you trying to avoid local development?

I opened github.com/allegroai/clearml/pull/1083 as an attempt to help catch this.

@<1798162804348293120:profile|FlutteringSeahorse49> wants to start HPO though, so the desire is to deploy agents to listen to queues on the slurm cluster (perhaps the controller runs on his laptop).

would that still make sense?

my approach was to spin up an EC2 and run the deployment there from within the EBS volume mount.

I symlinked /opt/clearml to /mnt/xvda/clearml to minimize docker-compose changes. been working out fine so far.

with aws-cdk, the deployment steps can be automated (format the volume, clone a repo with the config, etc). I can link you to a resource that may help with that if you're interested.

didnt disappear this time.

I think you’d have to run the cleanup service. That’s what seems to be what is controlling deletion based on archived status and some other temporal filters

Weird . I recently implemented a function that talked to this exact endpoint and found it had to exclude the version and api paths . Is there some sort of redirect that happens?