I don't think I expressed myself well 😅

My problem is I don't know how to run a jupyterhub Task. Basically what I want is a clearml-session but with a docker container running JupyterHub instead of JupyterLab.

Do I write a Python script? If yes, how can I approach writing it? If not, what are the altenatives?

Is the trigger controller running on the services queue ?

Yes, yes it is

Mostly the configurabilty of clearml-session and how it was designed. Jupyterhub spawns a process at :8000 which we had to port foreward by hand, but spawning new docker containers using jupyterhub.Dockerspawner and connecting them to the correct network (the hub should talk to them without --network host ) seem too difficult or even impossible.

Oh, and there was no JupyterHub stdout in the console output on clearml server, it shows the jupyterlab's output by default

I guess I'll let you know the next time this happens haha

No errors in logs, but that's because I restarted the deployment :(

SuccessfulKoala55 sorry for the bump, what's the status of the fix?

CostlyOstrich36 jupyterhub is a multi-user server, which allows many users to login and spawn their own jupyterlab instances (with custom dependencies, data etc) for runing notebooks

AgitatedDove14 no errors, because I don't know how to start 😅 I am just exploring if anyone did this before I get my hands dirty

MelancholyElk85 thank you, however I am not sure where do I put that label?

the entire index is 35G

It could work but slack demands a minimum of 512x512

FrothyDog40 thanks, the . works 😄

we didn't change a thing from the defaults that's in your github 😄 so it's 500M?

AgitatedDove14 Well, we have gotten relatively close to the goal, i suppose you wouldn't have to do a lot of work to support it natively

Haha we manage our own deployment without k8s, so no dice there

But, it turns out we are using nginx as a reverse proxy so putting a client_max_body_size inside a nginx.conf solved it for us. Thanks :))

Yeah, sorry I typoed 😅 "newer than 18.04" was I supposed to say

What I meant was that we rebuilt them with 22.04

Thanks, I'll try it out first thing Monday and I'll let you know 😃

Yes, that is correct

SOLVED: It was an expired service account key in a clearml config

That's only a part of a solution.
You'd also have to allow specifying jupyterhub_config.py , mounting it inside a container at a right place, mounting the docker socket in a secure manner to allow spawning user containers, connecting them to the correct network ( --host won't work), persisting the user database and user data...

We have deployed clearml-agents as systemd services. This allows you to tell systemd to restart the agent whenever it crashes, and it automatically starts them up when the server boots!

Errors pop in occasionally in the Web UI. All we see is a dialog with the text "Error"

Hello, a similar thing happened today. In the developer's console there was this line

https://server/api/v2.19/tasks.reset_many 504 (Gateway time-out)

CostlyOstrich36 this sounds great. How do I accomplish that?

Thank you, I understand now :D

This means that an agent only ever spins up one particular image? I'd like to define different container images for different tasks, possibly even build them in the process of starting a task. Is such a thing possible?

You are not missing nothing, it is what we would like to have, to allow multiple people have their own notebook servers. We have multiple people doing different experiments, and JupyterHub would be their "playground" environment

For now, docker compose down && docker compose up -d helps

I succeeded with your instructions, so thank you!

However, we concluded that we don't want to run it through ClearML after all, so we ran it standalone.

But, I'll update you if we ever run it with ClearML so you could also provide it

By language, I meant the syntax. What is Args and what is batch in Args/batch and what other values exist 😀

By commit hash, I mean the hash od the commit a task was run from. I wish to refer to that commit hash in another task (started with a triggerscheduler) in code

trigger.add_task_trigger(name='export', schedule_task_id=SCHEDULE_ID, task_overrides={...})I would like to override the commit hash of the SCHEDULE_ID with task_overrides