is the agent execution dependent on some CMD in my docker file?

No, it was fixed by restarting clearml then and some services. But currently, we gave up and we use debug=True so we dont use the services queue

I set it up like this: clearml-agent daemon --detached --gpus 0,1,2 --queue single-gpu-24 --docker

but when I create the session : clearml-session --docker xyz --git-credentials and I run nvidia-smi I only see one gpu

how? just tar and untar?

found out the command swaps singular and plural. It's --gpus 0 and --gpu 0,1,2

I want the script to be agnostic to whether it is run using clearml or not, with a particular queue or not

I believe so

where is it persisted? if I have multiple sessions I want to persist, is that possible?

@<1523701087100473344:profile|SuccessfulKoala55> Could you elaborate? I believe both the ips are visible to the container.

This is making things slightly complicated because now I have to introduce a jumphost for people who aren’t on the same physical network and are on the same tail scale network

so the 192.xxxx network is the physical network, and not on the tailscale network

which path would that be?

As mentioned above, I've tried both (env and clearml.conf). Here are my configs (I've blacked out urls and creds)

conf file

api { 
    web_server: 

    api_server: 

    files_server: 

    credentials {
        "access_key" = "xyz"
        "secret_key"  = "xyz"
    }
}

Relevant log (it uploads to S3, I can see the artefact fine on clearml's experiment tracker, but it still causes the job to hang)

2023-12-11 16:06:44,008 - clearml.sto...

In the end I forked the clearml-session library and removed mechanisms to access the interactive terminal. I added ipc=host.

There's one identifiable issue with clearml-session+tailscale though - while it does launch the daemon properly, it registers the wrong ip address to the task (sometimes the external ip address even when --external is not passed). At the end of the day, if we know which machine it was launched on, we're able to replace that ip address with a tailscale equivalent and st...

Amazing. Going to play with it this weekend !

by special run args?

That makes sense, but that would mean that each client/user has to manage the upload themselves, right?

(I'm trying to use clearml to create an abstraction over the compute / cloud)

Also @<1523701070390366208:profile|CostlyOstrich36> - are these actions available for on prem OSS clearml-server deployments too?

nice! I was wondering whether we can trigger it by the UI, like "on publishing" an experiment

I'm thinking of using s3fs on the entire /opt/clearml/data folder. What do you think?

With respect to unstructured data, do hyperdatasets work well with audio data (and associated metadata) ?

We have some scenario where a group of clearml experiments might represent a logical experiment. We then want to use all the trained models in a pipeline to generate some output.

With that output, we probably want to some third party like mechanical turk, do some custom evaluations - and some times more than once. We then want to connect (and present) these evaluations alongwith ClearML experiments.

we have various services internally to do this --> however, we have to manually link it up w...

Thanks, I can have docker + poetry execution modes then?

Hey @<1577106212921544704:profile|WickedSquirrel54> , I would definitely be interested in this. A gist would be cool too

Would I also be able to change the task name from within the subprocess?

@<1537605940121964544:profile|EnthusiasticShrimp49> , now that I have run the task on remote, can I copy the artefacts/files it creates back to my local fs?

Lets say the artefacts are something like
artefacts = [checkpoint.pth, dvc.lock, some_other_dynamically_generated_file]

Sorry false alarm

it worked. The env variables definitely do not work! Had to use clearml.conf along with use_credential_chain=True

any luck, is this expected?

Thanks! so it seems like the key is the Task.connect and bubble up params to original task, correct?

I do change the task and the project name, the task name change works fine but the project name change silently fails