Disclaimer: I'm not familiar enouch with the ClearML codebase to vouch for the quality of this PR, although it is short which is typically good . The feature we're interested in is the ability to specify the subnet_id .

Will do!

Actually, dumb question: how do I set the setup script for a task?

cc: @<1565509803839590400:profile|MoodyBear54>

Oh my word, is this it? None

Can you set these to any strings and have them function as API keys?

I do agree with your earlier observation that the target of that mount seems wrong. I would think that the volume mount should be -v /root/.ssh:/root/.ssh but instead it's -v /root.ssh:/.ssh

None

That's with the key at /root/.ssh/id_rsa

That could work! Is that an option? Something that lets me spin up the ClearML and get a services worker to connect to it without manual steps.

I did a post on Linkedin with several slides on how I plan to build it here

That is great! This is all the motivation I needed to decide to do a POC at some point.

Here's a screenshot if a session where I first try to clone as ssm-user , but it fails, then I change to root and it succeeds

Oh hooray! So docker-compose manages the restarting of crashed containers? I didn't know that, and that is great 😄

Oh, right... the Docker image running on the instance takes care of the library versions. You guys are great!

Oh, that is cool. I captured all this. Maybe I'll make a user-data.sh script and docker-compose.yml file that brings all these things together. Probably won't have time for a few weeks.

I did a quick local experiment and observed that credentials created from the UI indeed become invalid if you delete the ClearML volumes.

• starting docker-compose locally
• creating a set of credentials from the UI
• hardcodign those credentials into the docker-compose file
• restarting
• the agent-services container started up and successfully became a registered worker
• I killed the docker-compose and deleted the volume folders
• restarted the docker-compose (with the same hard-coded...

Playing around this weekend to learn the ins and outs of extensions

@<1523701070390366208:profile|CostlyOstrich36> Oh that’s smart. Is that to make sure no transactions happen during the backup? Would there be a risk of ongoing or pending tasks somehow getting corrupted if you shut the server down?

you mean as experiment management / model registry / data? I think this is the bread&butter of clearml

💯 . I was wondering if anyone had had experience using ClearML together with one of these others.

I think most of them are alternatives to metaflow

Totally.

Like, if you google "dagster and clearml" or "prefect and clearml" or "airflow and clearml" -- I don't find any blogs written by people talking about how they use both of them together.

That's strange to me, becau...

Hi! Yes, it just logs the .py file.

This is the event: None

So the problem came back even with this new URL. I discovered clearing your cookies fixes it.

I don't see it as an argument in Task.init or Task.execute_remotely

I SOLVED IT, NO NEED TO READ FURTHER 😄

I'm a chump and didn't read the docs: None

Oh, I think I got overexcited and didn't look at this closely. So this ACCESS/SECRET key pair is on the agent-services container.

I can see that agent-services is simply a container running `clearml-agent daemon --queue ser...

On it

I took a look

• I think the Outerbounds extension (the one in my screenshot) is currently closed source. That makes sense to me. A bit sad because it is highly similar.
• Another example could be the AWS ToolKit extension. But sadly, it's hardly a "minimal example". I was thinking it's relevant because it uses your local ~/.aws/ folder, which is similar to what we'd want to do.

Thanks Vasil! Can you elaborate on what you mean by using boto3? Do you mean writing a script using boto that pulls the credentials down and writes to the user's clearml.conf

Also, I've been seeing references to "credentials vault" in the docs. I can see this is the problem that it solves.

You have no idea what is committed to disk vs what is still contained in memory.

If you ran docker-compose down and allowed ES to gracefully shut down, would ES finish writing everything to disk, therefore guaranteeing that the backups wouldn't get corrupted?

I have the same behavior whether or not I put task.execute_remotely(...) before or after the call to run_shell_script()