my use case is more like 1st one where run the training at a certain given schedule

Hi PompousParrot44
Unfortunately this is still not available in the UI. As part of the Controllers, we thought of having a "Cron" controller that Clones base xperiments at a given time and schedulers them for execution. We are looking for specific use cases, to make sure this will actually answer the requirements of users.
It looks as if that might be what you are after, is this correct? What exactly is the use case here? Is it a stable daily cron job (for example retrain the an experiment at the end of the day with the latest code from the git master) ? Or is it a way to add another layer over the Job Scheduler (for example: start training in 2 hours because I still need the resource) ?

PompousParrot44 That should be very easy to do, basically a service mode code that clones a base task and puts it into a queue:
This should more or less do what you need :)
` from trains import Task

task = Task.init('devops', 'daily train', task_type='controller')

stop the local execution of this code, and put it into the service queue, so we have a remote machine running it.

task = execute_remotely('services')

while True:
a_task = Task.clone(base_task_id='aaabb111')
Task.enqueue(a_task.id, queue_name='default')
# wait until the next day
sleep(606024) `