GPU scheduling | potocolom docs

GPU time is the scarce and expensive resource. Generation jobs and live drawing sessions both run on the same workers. This page describes what the API and the workers do today, then the parts that are planned.

What runs today

The job queue

Generation jobs wait in a heap inside the API process. The heap is InProcessQueues in backend/app/jobs.py. On start, the API rebuilds the heap from PostgreSQL.

A worker can hold up to two queued jobs at once, set by JOB_DISPATCH_DEPTH in backend/app/jobs.py. The second job lets the worker encode and upload one result while it denoises the next.
A job that does not name a model_id is routed by the API. The API picks the cheapest registered model whose tier (draft, standard or premium), capabilities and parameter schema match the request. Routing uses backend/app/jobs.py and backend/app/manifests.py. A worker's hello lists only the models it has installed, and a model installed while a connection is open is not routed to that connection until it reconnects.
A job is retried once, and only after the worker holding it dies.

Realtime slots

A worker measures how fast it runs one frame for each model. It reports that time and the number of realtime slots it can run in its hello message. Admission uses a 500 ms per-frame p95 budget. Compatible frames from several sessions can share one GPU cycle.

Realtime needs the full-residency rung of the memory ladder, described next. Only that rung advertises the realtime capability.

The memory ladder

At model load, the worker measures free VRAM and takes the highest rung that fits:

Full residency: the whole pipeline stays on the GPU. This is the only rung that serves realtime.
Model offload: whole components move to the GPU for their own stage. Jobs run, more slowly. Realtime does not.
Group offload: layer groups stream through the GPU, and groups can spill to disk when system RAM is short. Jobs run, slowly. Realtime does not.

The rung is set per model. A worker can keep a small model fully resident while a larger model uses offload. The MEMORY_MODE worker setting pins a rung. It accepts auto (the default), full, model_offload or group_offload. A pinned rung never moves down. With auto, an out-of-memory error during a job moves the model down one rung, reloads it, and retries that job once.

The admission queue for sessions

A session request with no free slot waits in the admission queue. The session is in state queued.

The queue order is the time each session first joined. A session that is sent back keeps its place.
A freed slot, a newly registered worker, or the 30 second session sweep admits sessions. Admission is first fit: a session whose model has no room does not hold up a session whose model has room.
The browser sees its position in the queue. It does not see an estimated wait, because the API keeps no session-duration data to estimate from.
There is no queue timeout. Closing the socket removes the session from the queue.

An account can hold two realtime sessions. A third request is refused with close code 4003. Queued sessions count against that limit. See Realtime loop for the close codes and the session states.

Idle release

A live session with no canvas input for about 60 seconds gives up its slot. Usage metering stops when the slot is released. The canvas stays in the browser. The next stroke places the session on a worker again. When no slot is free, the session waits in the queue.

Failed attempts

A worker that cannot make the model resident answers session_refused. That fails one attempt, not the session. The API tries another worker, or queues the session when no worker has room.

Planned scheduling

The parts below are designed but not built. The shipped status notes say so directly: preemption and checkpoints between steps are not implemented, there is no scheduler that places models across machines, and the Redis-backed queue remains designed.