How?

by special run args?

@<1537605940121964544:profile|EnthusiasticShrimp49> , now that I have run the task on remote, can I copy the artefacts/files it creates back to my local fs?

Lets say the artefacts are something like
artefacts = [checkpoint.pth, dvc.lock, some_other_dynamically_generated_file]

nice! I was wondering whether we can trigger it by the UI, like "on publishing" an experiment

I want the script to be agnostic to whether it is run using clearml or not, with a particular queue or not

@<1523701087100473344:profile|SuccessfulKoala55> Could you elaborate? I believe both the ips are visible to the container.

This is making things slightly complicated because now I have to introduce a jumphost for people who aren’t on the same physical network and are on the same tail scale network

can I combine docker and poetry mode?

This is the issue

Setting up connection to remote session
Starting SSH tunnel to root@192.168.1.185, port 10022
SSH tunneling failed, retrying in 3 seconds

I've also overriden CLEARML_FILES_HOST= None , and configured it in clearml.conf file. Don't know where its picking 8081 😕

this doesn't interrupt jobs, but it slows it down, and it takes a lot of time to quit (adds ~2 hours for the process to end)

I tried that earlier - that checks out , it matches the s3 path I provide in the conf

Thanks! so it seems like the key is the Task.connect and bubble up params to original task, correct?

I do change the task and the project name, the task name change works fine but the project name change silently fails

Would I also be able to change the task name from within the subprocess?

I'm using clearml 1.9.3 client side

how? just tar and untar?

We have some scenario where a group of clearml experiments might represent a logical experiment. We then want to use all the trained models in a pipeline to generate some output.

With that output, we probably want to some third party like mechanical turk, do some custom evaluations - and some times more than once. We then want to connect (and present) these evaluations alongwith ClearML experiments.

we have various services internally to do this --> however, we have to manually link it up w...

I'm thinking of using s3fs on the entire /opt/clearml/data folder. What do you think?

where is it persisted? if I have multiple sessions I want to persist, is that possible?

That makes sense, but that would mean that each client/user has to manage the upload themselves, right?

(I'm trying to use clearml to create an abstraction over the compute / cloud)

With respect to unstructured data, do hyperdatasets work well with audio data (and associated metadata) ?

Also @<1523701070390366208:profile|CostlyOstrich36> - are these actions available for on prem OSS clearml-server deployments too?

it worked. The env variables definitely do not work! Had to use clearml.conf along with use_credential_chain=True

None

want to work on it together?

found out the command swaps singular and plural. It's --gpus 0 and --gpu 0,1,2

How does it work with k8s? how can I request the two pods to sit on the same gpu?

@<1523701070390366208:profile|CostlyOstrich36> , as written above, I've done that. It still tries to send to 8081

AMAZING!

In the end I forked the clearml-session library and removed mechanisms to access the interactive terminal. I added ipc=host.

There's one identifiable issue with clearml-session+tailscale though - while it does launch the daemon properly, it registers the wrong ip address to the task (sometimes the external ip address even when --external is not passed). At the end of the day, if we know which machine it was launched on, we're able to replace that ip address with a tailscale equivalent and st...