But I can't do this from the web ui, can I?

I will only cancel daemons which are idle.

AgitatedDove14 I need to finish working now, will be back in the evening and have a look.

No, they were not SuccessfulKoala55

ok, there is probably a problem on my side, because when I ran the sample code from the repo it works, sorry to bother you

the images do not show up in debug_samples. How can I check what is wrong?

I implemented the first version and it seems to work.

since I am using the demo server, I should make sure that in the configuration file the images will be uploaded to the appropriate server, right? Can you please point me to the proper line in the config file?

that was quick, thanks!

that would be great!

it does, thanks!

For the images themselves, you can get heir urls

how can I do it?

so far everything works, the only problem I can think of is a race condition, but I will probably ignore it, which happens in the following scenario:
a) a worker finishes its current run, turns into an idle state,
b) my script scrapes the status of the worker, which is idle,
c) a new task is enqueued and picked by the worker,
d) the worker is killed after it managed to pull a task from the queue, so the task will be cancelled as well.

Yes, replace image viewer with a custom widget, but perhaps we can implement this externally (I am not an expert on UI to be honest)

yes

I hope you can do this without containers.

AgitatedDove14 thanks for the additional information:
yes, the report_image problem was resolved after I reordered dimensions in the tensor. is there an advantage in using tensorboard over your reporting? html reporting looks powerfull, can one inject some javascript inside?

AgitatedDove14 thanks, that will be helpful!

Unfortunately there is no docker, there is only singularity. This cluster is used by many users and docker is not secure enough.

AgitatedDove14 I do not want to push you in any way, but if you could give me an estimate of the slurm glue code, that would be helpful. I should have a local installation of the trains server to experiment with next week.

yes, happy to help! In fact I am also interested in the k8s glue, since in one of our use cases we are using jobs and not pods (to allow for spot instances in the cloud), but I need to dig digger into the architecture to understand what we need exactly from k8s glue.

sure, we can deal with the drivers

AgitatedDove14 Is there a way to say to a worker that it should not take new tasks? If there is such a feature then one could avoid the race condition.

AgitatedDove14 going back to the slurm subject, I have local trains installed on the cluster with slurm so I am ready to test. At the same time I was thinking whether a simple solution would do the job:
a) [scale up agents] monitor the trains queue, if there is something that was not started for some amount of time, and the number of agents is not yet at the maximum, then add an agent,
b) [scale down agents] if all the tasks are running and there are idle agents, kill an idle agent.
Or do yo...

AgitatedDove14 I looked at the K8s glue code, having something similar but for SLURM would be great!

got it, thanks!

👍

AgitatedDove14 if I use report_image can I get a URL to it somehow?

ok, got it, thanks!

Not sure yet, I will get back to you on this later, in 1-2 weeks, thanks.