Janus is a single Go binary that runs .gguf models on your machine (GPU or CPU) and exposes an OpenAI-compatible API. No Python, no Docker, no Ollama required.
Use it your way: call it from the command line (curl, PowerShell, scripts), wire it into Cursor / Cline / any OpenAI client — same local models, whatever workflow fits you.

What you get

Local inference — llama.cpp via Vulkan (AMD / Intel / NVIDIA) or CPU fallback
OpenAI-compatible API — /v1/chat/completions, /v1/models
Hot-swap models — change .gguf without restarting
Thinking model support — <think> reasoning split into reasoning_content
Chat template auto-detection — uses the template from GGUF metadata
Zero dependencies — one .exe on Windows, no Python, no Docker


Requirements



Platform
What you need




Windows (primary)
Windows 10/11, Go 1.22+, Vulkan-capable GPU recommended


Linux
Go 1.22+, Vulkan or CPU


macOS
Go 1.22+, CPU backend (Vulkan varies by hardware)



Disk: plan for the model size (often 2–8 GB per model) plus ~50 MB for Janus + llama.dll.

Quick start (Windows)
1. Clone and build
git clone https://github.com/Vibra-Ingenn/Janus.git
cd Janus
.\build.ps1
build.ps1 downloads pre-built llama.cpp Vulkan DLLs and compiles dist\janus.exe.
2. Download a model
Put a .gguf file in the models\ folder. Use the included downloader:
go build -o dist\modelget.exe .\cmd\modelget
.\dist\modelget.exe -repo meta-llama/Llama-3.2-3B-Instruct -file Llama-3.2-3B-Instruct-Q8_0.gguf -out .\models\
Or download any GGUF from Hugging Face.
3. Configure

Edit .env:
INFERENCE_BACKEND=vulkan
JANUS_MODEL_PATH=./models/Llama-3.2-3B-Instruct-Q8_0.gguf
JANUS_MAX_TOKENS=4096



Variable
Default
Meaning




INFERENCE_BACKEND
vulkan
vulkan, cpu, or openrouter


JANUS_MODEL_PATH
(required)
Path to your .gguf file


JANUS_GPU_LAYERS
-1
-1 = all layers on GPU, 0 = CPU only


JANUS_VRAM_CEILING_MB
9216
VRAM budget hint (MiB)


JANUS_MAX_TOKENS
4096
Max tokens per reply


JANUS_LISTEN_ADDR
127.0.0.1:8990
Bind address



4. Run

Opens http://127.0.0.1:8990 in your browser.
5. Verify
curl http://127.0.0.1:8990/health

Quick start (Linux / macOS)
git clone https://github.com/Vibra-Ingenn/Janus.git
cd Janus
go mod tidy
go build -o dist/janus ./cmd/janus
cp .env.example .env
# edit .env — set JANUS_MODEL_PATH and INFERENCE_BACKEND=cpu if no Vulkan
./dist/janus
On Linux you need libllama.so next to the binary or on LD_LIBRARY_PATH.

OpenAI-compatible API
curl http://127.0.0.1:8990/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
Base URL: http://127.0.0.1:8990/v1
Endpoints



Method
Path
Description




GET
/health
Liveness check (?deep=true for details)


GET
/v1/models
Model list


POST
/v1/chat/completions
Chat (streaming supported)


POST
/models/load
Hot-swap model


GET
/models/list
Available .gguf files


GET
/engine/status
VRAM and backend info



Streaming
curl http://127.0.0.1:8990/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"local","stream":true,"messages":[{"role":"user","content":"Tell me a joke"}]}'

Connect to Cursor / Cline / other clients
Base URL:  http://127.0.0.1:8990/v1
API Key:   (leave blank)


Pitfalls (learned the hard way)



Problem
What's going on
Fix




"It built but my changes aren't there"
On Windows, Go can't overwrite a running .exe.
Stop all janus.exe in Task Manager, then rebuild.


"Address already in use"
A leftover process holds port 8990.
Task Manager → end all janus.exe.


Server starts, chat fails
JANUS_MODEL_PATH wrong or no .gguf in models/.
Set the path in .env, put the file in models/.


"local engine failed to start"
Missing llama.dll or GPU driver issue.
Run .\build.ps1. Update GPU drivers, or set INFERENCE_BACKEND=cpu.


Wrong URL
Default is http://127.0.0.1:8990, not 8080.
Bookmark 8990.


First reply takes forever
Model loading into VRAM — normal.
Wait 10–60s; smaller quants (Q4) load faster.




Project layout
cmd/janus/          Main server (OpenAI-compatible API)
cmd/modelget/       Hugging Face model downloader
internal/engine/    llama.cpp Vulkan/CPU backend
internal/bridge/    DLL loader and FFI bindings
internal/singleton/ Single-instance guard
models/             Put .gguf files here (not committed)
dist/               janus.exe + llama.dll after build


Development
Get-Process -Name "janus" -ErrorAction SilentlyContinue | Stop-Process -Force
go test ./...
go build -o dist\janus.exe .\cmd\janus
.\dist\janus.exe
Or just .\run.ps1. For a full build including llama DLLs, use .\build.ps1.
Contributions welcome — see CONTRIBUTING.md.

License
MIT — see LICENSE.