I need to mock it - because I'm writing some unittests

I tried that earlier - that checks out , it matches the s3 path I provide in the conf

this doesn't interrupt jobs, but it slows it down, and it takes a lot of time to quit (adds ~2 hours for the process to end)

Thanks, I can have docker + poetry execution modes then?

How does it work with k8s? how can I request the two pods to sit on the same gpu?

AMAZING!

None

How?

where is it persisted? if I have multiple sessions I want to persist, is that possible?

Sorry false alarm

is it in the OSS version too?

so the 192.xxxx network is the physical network, and not on the tailscale network

I believe so

feels like a typo somewhere

any luck, is this expected?

I set it up like this: clearml-agent daemon --detached --gpus 0,1,2 --queue single-gpu-24 --docker

but when I create the session : clearml-session --docker xyz --git-credentials and I run nvidia-smi I only see one gpu

want to work on it together?

@<1523701070390366208:profile|CostlyOstrich36> , as written above, I've done that. It still tries to send to 8081

I've also overriden CLEARML_FILES_HOST= None , and configured it in clearml.conf file. Don't know where its picking 8081 😕

@<1523701087100473344:profile|SuccessfulKoala55> Could you elaborate? I believe both the ips are visible to the container.

This is making things slightly complicated because now I have to introduce a jumphost for people who aren’t on the same physical network and are on the same tail scale network

thanks!

With respect to unstructured data, do hyperdatasets work well with audio data (and associated metadata) ?

This is the issue

Setting up connection to remote session
Starting SSH tunnel to root@192.168.1.185, port 10022
SSH tunneling failed, retrying in 3 seconds

None

No, it was fixed by restarting clearml then and some services. But currently, we gave up and we use debug=True so we dont use the services queue

We have some scenario where a group of clearml experiments might represent a logical experiment. We then want to use all the trained models in a pipeline to generate some output.

With that output, we probably want to some third party like mechanical turk, do some custom evaluations - and some times more than once. We then want to connect (and present) these evaluations alongwith ClearML experiments.

we have various services internally to do this --> however, we have to manually link it up w...

As mentioned above, I've tried both (env and clearml.conf). Here are my configs (I've blacked out urls and creds)

conf file

api { 
    web_server: 

    api_server: 

    files_server: 

    credentials {
        "access_key" = "xyz"
        "secret_key"  = "xyz"
    }
}

Relevant log (it uploads to S3, I can see the artefact fine on clearml's experiment tracker, but it still causes the job to hang)

2023-12-11 16:06:44,008 - clearml.sto...

In the end I forked the clearml-session library and removed mechanisms to access the interactive terminal. I added ipc=host.

There's one identifiable issue with clearml-session+tailscale though - while it does launch the daemon properly, it registers the wrong ip address to the task (sometimes the external ip address even when --external is not passed). At the end of the day, if we know which machine it was launched on, we're able to replace that ip address with a tailscale equivalent and st...

I'm thinking of using s3fs on the entire /opt/clearml/data folder. What do you think?

Its a simple training loop that trains models for 2-3 epochs for a total of 200-300 iterations, saves a few checkpoints and saves a final model at the end of it