Rozalia

Rozalia lends the language model hardware sitting on your desk to the
people you choose, behind one OpenAI-compatible endpoint.

The machines with the GPUs live at home, behind NAT, where nothing on the
internet can dial them. So they dial out instead: rozalia-proxy opens a
WebSocket to the server and holds it there, and the server pushes
requests down it, to whichever machine advertises the model asked for and
has the fewest requests in flight.

This snap ships the proxy. Enrol the machine with a one-time code from
whoever runs the server, say what to offer and where, and turn it on:

 sudo rozalia.proxyctl enroll -server https://ai.example.org -code rze_...

 sudo snap set rozalia \

     proxy-server=https://ai.example.org \

     proxy-backends=http://127.0.0.1:11434/v1 \

     proxy=true

proxy-backends takes a comma-separated list, for a machine running more
than one inference server. proxy-concurrency says how many requests to
run at once, and defaults to one, because a single GPU answering two
completions at once finishes neither sooner.

The cloud half is optional and ships as a separate component, so a
machine that only lends its GPU never downloads it:

 sudo snap install rozalia+server

 sudo snap set rozalia base-url=https://ai.example.org listen-addr=127.0.0.1:8080

 sudo snap set rozalia server=true

 sudo rozalia.serverctl account create -email you@example.org -admin

The server does not start on install: a fresh database has no accounts,
and the address it should listen on is not something a package can
guess.