Might be a bit redundant but maybe running proxmox and then all the clearml stuff inside of vms. mainly so we can also try out different things and just split off a gpu for something else

Nice one. How is he funding this?

Best way imo is to set-up ClearML Server on a cloud machine (or go for paid, hosted, Enterprise version) and use the machines with RTX6000 as ClearML agents that will run pipelines (once you set them up - isn't a simple job for serious things, dockerized workflows etc.). Make sure the pay is good, prepare a strategy and plan for this and you can at least get some insanely valuable experience in this interesting-boss-job.

ah he only got 4x actually, 8k CHF/ea swiss supplier alltron. They havent arrived yet.

I do plan to use this for my studies most likely.

Pay is good for what I do.

its going to be one single machine actually.

You can use single PC and have multi agent running in the same time, each assigned one or multi GPU.
You likely to hit CPU bottleneck, depending on how much augmentation you are applying when training ....

PS: Do you know what was the price for each RTX and where he got them from?