Yup if you scroll through the logs in the console, near the top (post config dump), you’ll see a git clone and checkout to the specific hash.

PS You can actually change this parameter in an experiment’s configuration if it is in draft mode.

Might be under examples

I ran into something similar during deployment. Hopefully this helps with your debugging: if the agent was launched separately from the rest of the stack, it may not have proper docker-DNS resolution to None . (e.g. if in the same docker-compose, perhaps you didnt add the backend network field, or if it was launched separately through docker run without an explicit external network defined)

if the agent's on the same machine, try docker network connect to add...

I believe pipe.connect_configuration is what you're looking for?

I'm guessing this is done through code-server?

I'm currently rolling a JupyterHub instance (multiuser, with codeserver inside) on the same machine as clearml-server. That’s where tasks are executed etc. so, all browser dev env.

It sounds like there’s an option to basically bypass this latter step and just use clearml’s credentialing to accomplish much the same thing? Am I understanding clearml-session correctly?

i think we may have found the frankenbug?

the argument to the dataset name was not being overridden correctly (mistyped), so the default value of an empty string (instead of a placeholder like "CHANGE_ME") in the parent task caused the dataset to basically get created with an empty name, and somehow that hid the whole project, despite hundreds of existing tasks in it.

and no way to un-hide it as far as I can tell?

you can put task.execute_remotely() to create it in draft mode. I've taken to configuring defaults to run things very quickly just in case i forget though (e.g. placeholder string for dataset, bail out early if not changed… or just do one epoch on a small subset of samples, etc).

then back to CLI, updated the pipeline to point the tasks to the new queue. run it, shows up in the UI (same container as default worker, just replicated w a new docker-compose and CMD to point to the new queue).

waiting now to see if they disappear.
any problems you may have spotted with the versions used?

project hasn't disappeared just yet. but it's happened twice now

For reproducibility, it kind of makes sense though. The existence of the file is contingent on the worker cloning the source code. I'm sure things can be done to maintain state differently but I personally adapted to the git-based workflow for managing files pretty quickly.

though yes I will admit I had the same thought first: why must I run it each time?

Beware: squash merges will ruin the ability to reproduce the experiment at that time since the git commit will be lost (presuming th...

If you can hit the endpoint with curl, you for sure can hook it up to many frontend frameworks.

Personal recs: gradio, streamlit

Abstract the interaction into a function call, and wrap it all in some UI elements using python.

the project wasn't hidden before. I'm aware of the pipeline tasks being hidden, that makes sense for organization. but the actual project itself as an entirety has a ghost icon.

she created a new project and started working in there, it was visible in the UI... and just now it disappeared again. it's kind of like running the pipeline makes it disappear.

ah, thank you for the clarity. A quarterly release schedule makes sense, it's about what I've observed.
Let me know if I can be of any assistance in early testing!

@<1541954607595393024:profile|BattyCrocodile47> put together None

my approach was to spin up an EC2 and run the deployment there from within the EBS volume mount.

I symlinked /opt/clearml to /mnt/xvda/clearml to minimize docker-compose changes. been working out fine so far.

with aws-cdk, the deployment steps can be automated (format the volume, clone a repo with the config, etc). I can link you to a resource that may help with that if you're interested.

I think you’d have to run the cleanup service. That’s what seems to be what is controlling deletion based on archived status and some other temporal filters

Oh yes. I see. Yeah, no ML here actually (doing the testing infra of endpoints), but certainly when there is its an issue.

How does clearml session avoid it? I guess only if autoscaling is used (one worker one machine)?

I tried mounting a config file (in the structure of the one on github but with just the relevant s3 section) into the agent-services container at /root/clearml.conf and after restarting the container, it seems to have had an impact. thank you!

When I inspect the console of the task I'm trying to run, I see there's a call to cp /tmp/clearml.conf ~/default_clearml.conf in the docker command and that the volume /tmp/clearml.conf is picked up from the host at some custom-named file ...

dug deeper. if i'm to make a guess...
/root/clearml.conf -> used on startup of agent-services as a template of sorts to create .clearml_agent.<id>.cfg on demand -> this task-specific file is used to mount to /tmp/clearml_default.conf in a new container (docker in docker bc of the socket mounted to the agent-services) -> used to execute the task

and now its gone.

i will attempt to start that now.

👀 following.
I have much the same issue, and it's mission-critical that I resolve it soon.

I opened github.com/allegroai/clearml/pull/1083 as an attempt to help catch this.

probably, but the syntax would be in that of a git diff, so it’d be a touch clunky if you asked me
Are you trying to avoid local development?

didnt disappear this time.

namely, I'm very interested in testing this unmerged feature, will be trying to leverage it as soon as possible
None

oh i see. you're talking about the agent-services, not a separate agent in a container.
yup, I've got the same thing going there.
fwiw...
for me, HOST_IP is 0.0.0.0 and the other "HOSTS" env vars don't contain "http" in them.
and my server is publicly reachable, not sure if that matter either.

thank you!
I'll add a volume mount to the services-agent container, and from what I understand that will become the template it uses?

is this the structure of the file?
None

or is it the "dot" syntax (like what shows up in the console when the task executes / your snippet)?

yeah let's step through this, i'm having her execute these steps as we speak.

create a task with the new project name. its created as a draft. can see it in the UI under the new project.

pipeline script is updated with new project name for. execute script to create pipeline. now see in UI under this new project name. nothing hidden.

the pipeline is running. when the queue is default (only serviced by one container with agent in it ( clearml-agent==1.5.2 ). abort it. everything is still ...

Weird . I recently implemented a function that talked to this exact endpoint and found it had to exclude the version and api paths . Is there some sort of redirect that happens?