This thread should be immortalized. Super stoked to try this out!

Ah, okay thanks!

I took a look

• I think the Outerbounds extension (the one in my screenshot) is currently closed source. That makes sense to me. A bit sad because it is highly similar.
• Another example could be the AWS ToolKit extension. But sadly, it's hardly a "minimal example". I was thinking it's relevant because it uses your local ~/.aws/ folder, which is similar to what we'd want to do.

I do agree with your earlier observation that the target of that mount seems wrong. I would think that the volume mount should be -v /root/.ssh:/root/.ssh but instead it's -v /root.ssh:/.ssh

I may be able to prepare a PR that only allows specifying the subnet ID. Can you help me brainstorm scenarios you’d want to see tested? Also, do these need to be automated tests?

Disclaimer: I'm not familiar enouch with the ClearML codebase to vouch for the quality of this PR, although it is short which is typically good . The feature we're interested in is the ability to specify the subnet_id .

I think it will work. There's a lot of really useful code in the black extension. I'm recruiting people now to join in on Friday. I'm actually very confident about it after messing around.

Oh awesome @<1523701132025663488:profile|SlimyElephant79> ! If you want to take a look, I made a big list of things to add. I'm working on a docker-compose.yaml file so we can have a good local development environment.

There's a lot of room to improve this from cleaning up the code to adding features on the list.

None

Playing around this weekend to learn the ins and outs of extensions

Totally worked!

Hey @<1523701482157772800:profile|AnxiousSeal95> ! I think ClearML's orchestrator is a great fit for ad-hoc experimentation, but not for (event-triggered) batch inference jobs that need to be relied on in production.

I'd only feel comfortable supporting pipelines that serve end users on a tool that is known for that, e.g. Metaflow, Dagster, or Airflow--mainly because those tools emphasize good monitoring and integration with the wider data ecosystem.

Dang! @<1590514584836378624:profile|AmiableSeaturtle81> awesome answer thank you! You seem like an awesome person to know. Definitely connect if you'd like to talk ops stuff sometime. None

I see. Is it possible for two agents to be utilizing the same GPU? (like if the machine has a terrific GPU, but only one of them?)

The question I'm exploring remains: is it possible to acquire that initial set of ClearML API keys programmatically so that the manual steps of 1-4 above can be avoided for an initial deployment?

You have no idea what is committed to disk vs what is still contained in memory.

If you ran docker-compose down and allowed ES to gracefully shut down, would ES finish writing everything to disk, therefore guaranteeing that the backups wouldn't get corrupted?

So, we've already got a model registry: MLFlow

And we've got a serving framework we really like: BentoML and FastAPI

The debate is between ClearML and Metaflow for

• training models during the research phase
• re-training models on a schedule or event-based trigger in production
• running batch inference jobs on a schedule or event-based trigger in production

That could work! Is that an option? Something that lets me spin up the ClearML and get a services worker to connect to it without manual steps.

Yes, it's pretty lame that a clearml-agent can only process one task at a time if it's not listening to a services queue 🤔

Aren't they two different auth systems? One for humans and one for machines?

Oh wow. If this works, that will be insanely cool. Like, I guess what I'm going for is that if I specify "username: test" and "password: test" in that file, that I can specify "api.access_key: test" and "api.secret_key: test" in the clearml.conf used for CI. I'll give it a try tonight!

Oh, right... the Docker image running on the instance takes care of the library versions. You guys are great!

Oh interesting. Is the hope that doing that would somehow result in being able to use those credentials to make authenticated API calls?

But the extension will need credentials to connect to it.

For now, I've written a headless selenium script to generate credentials for the fresh ClearML instance in CI.

Actually, dumb question: how do I set the setup script for a task?

I ultimately resorted to creating a selenium script combined with docker-compose. Not a beautiful solution but I can confirm that it works 😕 None

I don't know that you'd have to pre-build credentials into docker. If you could specify a set of credentials as environment variables to the docker run ... command or something, that would work just fine.

The goal is to be able to run docker-compose up in CI, which starts a clearml-server. And then make several API calls to the started ClearML server to prove that the VS Code extension code is working.

Examples:

• Assert that the extension can auth with ClearML
• Assert that the ext...

I did a quick local experiment and observed that credentials created from the UI indeed become invalid if you delete the ClearML volumes.

• starting docker-compose locally
• creating a set of credentials from the UI
• hardcodign those credentials into the docker-compose file
• restarting
• the agent-services container started up and successfully became a registered worker
• I killed the docker-compose and deleted the volume folders
• restarted the docker-compose (with the same hard-coded...

Oh this is thought provoking. Yeah, the idea of using ClearML for R&D is super appealing (to me speaking as an MLOps engineer 😆 ). And having the power of Metaflow's scheduler (on Step Functions with Event Bridge since we'd do the AWS-native deployment) also makes sense to me.

I'll keep asking questions about how we could do event-based jobs with alerting built in on ClearML in a different thread later on.

I pasted your points (anonymously) onto the Metaflow slack to le...

So the problem came back even with this new URL. I discovered clearing your cookies fixes it.