Oh there's parallelization as well. You could have step 1 gather the data, and then fan out to N parallel steps that all do different things with the data, for example hyper parameter tuning

Yay! Man, I want to do ClearML with "hard mode" (non-enterprise, self-hosted) first, before trying to sell BENlabs (my work) on it. I could see us paying for enterprise to get the Hyper Datasets and Vault features if our scientists/developers fall in love with it--they probably will if we can get them to adopt it since right now we have a homemade system that isn't nearly as nice as ClearML.

@<1523701087100473344:profile|SuccessfulKoala55> how exactly do you configure ClearML to use the cr...

I did a post on Linkedin with several slides on how I plan to build it here

This is the event: None

That's with the key at /root/.ssh/id_rsa

Hmm... these people are recommending restarting docker completely. I may have tried that already, but I'll do it again when I get some time to be sure.

And for the session

clearml-session --queue sessions --docker python:3.9

The key seems to be placed in the expected location

So, we've been able to run sudo su and then git clone with our private repos a few times now

The agent commands are nothing special.

clearml-agent daemon --queue sessions --cpu-only --create-queue true --docker

Wow, it really does not want to show the output of those print statements in stdout. Here's the output of the task from the console after cloning it. Confirmed that the setup script and all code changes are present:

Or the log of the init script?

Oh my word, is this it? None

Can you set these to any strings and have them function as API keys?

But from your other answer, I think I'm understanding that you can have multiple agents on a single instance listening to the same queue.

So we could maybe initialize 4 instances of the agent on a single EC2 instance which would allow us to handle a higher volume of small batches concurrently without tying up the entire instance.

I can't think of any changes we might have made on our side to cause that 🤔

Caching can be a reason. Say you do some heavy data loading / processing in step 1. Now you're developing step 2.

It'd be nice not to have to re-run Step 1 every time you want to test a change to step 2.

You could find a way to simply write your output of step1 to disk and do everything in one step, or you could let ClearML handle that caching for you--with the added benefit that others collaborating remotely can also use the outputs of steps you've cached with ClearML

I don't know about this, but could you turn your whole project into a pip-installable package using a setup.py and/or pyproject.toml ?

I've never tried this, but maybe then you could do pip install -e . locally before executing the task. Then execute. And then maybe the pip freeze that ClearML does would contain the symlink to your directory.

(so that from my_package import ... statements would work)

That's fabulous. This is definitely how my team prefers to structure projects. I hadn't gotten around to trying that out in our POC of ClearML yet, but I'm certain this is how our group will solve this problem

I think it will work. There's a lot of really useful code in the black extension. I'm recruiting people now to join in on Friday. I'm actually very confident about it after messing around.

I'm trying to add a docker-compose.yaml to the repo to

• make it more convenient for contributors to develop locally
• spin up a local ClearML instance in CI to run automated tests
  Here's the docker-compose file (mostly the standard file, except I altered the volume mounts, and I added minIO)
  Here's [the clearml.conf file](https://github.com/mlops-club/vscode-clearml-sessi...

That is great! This is all the motivation I needed to decide to do a POC at some point.

Oh my goodness. Thank you! I'd seen that before, but for some reason it didn't register I could run that with VS Code...

But this config should almost never need to change!

Host clearml-session
    HostName localhost
    User root
    Port 8022

Thank you! For now, it's kind of nice that it just picks up your credentials from your conf file. No extra setup required beyond the onboarding ClearML has you do 😄

And look! It's working, assuming you start the clearml session up yourself:

The issue went away. I'm still not sure why, but what finally made it work was creating a set of credentials manually in the UI and then setting those in my ~/clearml.conf file.

Do you happen to have a link to a docker-compose.yaml file that has a hardcoded set of credentials?

I want to seed the clearml instance with a set of credentials and ~/clearml.conf to run automated tests.

Hi friends, I'm just seeing these new messages. I read these links and I agree with @<1557175205510516736:profile|ShallowSwan53> . It's nice that the webapp has these pages, but what is the workflow to actually use this registry?

Also, @<1557175205510516736:profile|ShallowSwan53> , do you have a specific workflow in mind that you're hoping to get from ClearML?

At BEN, we're experimenting with

• BentoML for model serving. It's a Python REST framework a lot like FastAPI, but with some nice...

Here's a screenshot if a session where I first try to clone as ssm-user , but it fails, then I change to root and it succeeds

cc: @<1565509803839590400:profile|MoodyBear54>

This thread should be immortalized. Super stoked to try this out!

While I'm wishing for things: it'd be awesome if it had a queue already set up. But if there's not a way to do that in the docker compose file, I could potentially write a script that uses the creds to create one using API calls

Here's the repo: I've recorded a few update videos documenting how we learned about authoring VS Code extensions and how we got it to it's current state. Linked to those in order in the README.

ChatGPT has made working with TypeScript and the VSCode extension framework really nice! None