Run a 284B-parameter reasoning model on your own machine.
Connect it to a coding agent that edits code, runs commands, and builds.
No cloud. No API bills. No privacy leaks.
Most AI coding workflows today ship your code, prompts, and conversation history to a remote API. A local harness flips the model: the LLM runs on your hardware, your agent talks to it over a local HTTP server, and nothing leaves your machine.
Your prompts, your code, your reasoning traces — all processed locally. No data leaves your Mac. No API logs. No training on your conversations.
One hardware purchase. No per-token pricing. No monthly subscription. Run unlimited agent sessions for the cost of electricity.
DeepSeek V4 Flash's MoE architecture activates only ~35B parameters per token. On a Mac Studio or M3 Max you get 26–36 t/s — faster than many hosted APIs.
284B parameters with chain-of-thought thinking. The model reasons before answering, and the thinking length scales with problem complexity — not a fixed overhead.
ds4-server speaks OpenAI and Anthropic wire protocols natively. Pi, Claude Code, and opencode connect with zero adapter code. Tool calling, streaming, reasoning effort — all built in.
Long agent sessions survive server restarts. The KV cache lives on SSD, keyed by token hash. Restart the server and your conversation resumes from the exact checkpoint — no re-prefill.
ds4 is optimized for Apple Silicon. The model requires significant RAM to load and run efficiently.
The inference engine uses Apple's Metal Shading Language for GPU acceleration. This is a Mac-native build — no Docker, no CUDA, no Linux.
Note: 64 GB machines can run q3 quantization (~63 GB) at reduced reasoning quality.
From zero to a running agent in about 10 minutes.
Clone the ds4 repository and download the 2-bit quantized GGUF. At ~81 GB it's the largest single file you'll download, but it's the model — the entire 284B parameter DeepSeek V4 Flash, compressed to run on 128 GB machines.
# Clone ds4 and download the q2 quantized model
git clone https://github.com/antirez/ds4.c
cd ds4.c
./download_model.sh q2 # ~81 GB, for 128 GB RAM machinescurl -C - for resume support.
Compile the Metal-native inference engine. No Python, no CUDA, no containers — just a single C codebase that maps the model via mmap and executes on Apple's GPU through Metal Shading Language.
# builds ds4 (CLI) and ds4-server (HTTP)
makeds4 for interactive chat and ds4-server for agent integration.
Launch the OpenAI-compatible HTTP server with disk KV persistence. This is the bridge between the model and your agent.
./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192--ctx 100000 gives you 100k token context — enough for substantial agent sessions without exhausting your 128 GB (the compressed KV cache alone can use ~26 GB at 1M tokens). --kv-disk-dir enables the disk-backed KV cache: checkpoints are written to SSD as <sha1>.kv files, keyed by token hash. This means agent sessions survive server restarts — the next time the same prompt prefix arrives, the server loads the cached checkpoint instead of re-prefilling from token zero.
Add ds4 as a provider in Pi's model registry. This tells Pi where to find your local model and how to talk to it.
{
"providers": {
"ds4": {
"name": "ds4 local",
"baseUrl": "http://127.0.0.1:8000/v1",
"api": "openai-completions",
"apiKey": "dsv4-local",
"compat": {
"supportsStore": false,
"supportsDeveloperRole": false,
"supportsReasoningEffort": true,
"supportsUsageInStreaming": true,
"maxTokensField": "max_tokens",
"supportsStrictMode": false,
"thinkingFormat": "deepseek",
"requiresReasoningContentOnAssistantMessages": true
},
"models": [
{
"id": "deepseek-v4-flash",
"name": "DeepSeek V4 Flash (ds4 local)",
"reasoning": true,
"thinkingLevelMap": {
"off": null,
"minimal": "low",
"low": "low",
"medium": "medium",
"high": "high",
"xhigh": "xhigh"
},
"input": ["text"],
"contextWindow": 100000,
"maxTokens": 384000,
"cost": {
"input": 0,
"output": 0,
"cacheRead": 0,
"cacheWrite": 0
}
}
]
}
}
}compat block is critical — it tells Pi that this provider speaks OpenAI-compatible chat completions, supports reasoning effort (thinking modes), and uses DeepSeek's native thinking format. The thinkingLevelMap maps Pi's abstract levels (off/minimal/low/medium/high/xhigh) to ds4-server's reasoning_effort values. Setting cost to zero means Pi won't warn about token budgets — because local inference is free.
Set ds4 as Pi's default provider and model so every session uses your local harness automatically.
{
"defaultProvider": "ds4",
"defaultModel": "deepseek-v4-flash"
}pi command — pi "refactor this module", pi "explain this bug", pi "write tests for this" — routes through your local ds4-server. No cloud dependency. No API key management. No rate limits.
Run a quick smoke test to confirm Pi can reach your local model and generate a response.
# Smoke test — Pi will print the model name and respond
pi "Hello from the local harness. What model are you running?"ds4 local and a response, the harness is working. The first request will include a cold prefill (the server renders and processes your prompt), but subsequent requests with overlapping prefixes reuse the cached KV state — no re-prefill.
Understanding the moving parts helps you tune and debug your setup. Here's what happens under the hood.
Data flows from Pi (your agent) through ds4-server's HTTP API to the Metal GPU. KV checkpoints are persisted to SSD.
ds4-server keeps exactly one mutable Metal graph + KV checkpoint in memory. Concurrent requests queue behind a single worker. This isn't a limitation — it's the design: agent tools call the model sequentially anyway, and the single session means the KV cache is always hot for the active conversation.
When a new session replaces the live one, the old checkpoint is written to SSD as a .kv file. The key is a SHA1 of the exact token IDs. On the next request with the same prefix, the server loads the cached checkpoint and resumes from the exact token — logits, attention state, and all.
Three modes: non-thinking (direct answer), thinking (chain-of-thought before answering), and Think Max (extended reasoning). Pi maps its reasoning_effort levels to these modes. The model's thinking length scales with problem complexity.
ds4-server renders OpenAI tool schemas into DeepSeek's DSML format, and maps generated DSML tool calls back to OpenAI tool calls. The exact-DSML replay map stores verbatim sampled text for each tool call ID, so client JSON history can be re-rendered byte-for-byte.
Agent clients resend the full conversation every request. ds4-server compares the rendered token stream with cached prefixes — both in-memory and on-disk. If a prefix matches, it skips the prefill and starts generation from the cached checkpoint. This saves massive amounts of compute on long agent sessions.
Every engine change is validated against official DeepSeek API logprobs at multiple context sizes. The test vectors capture greedy continuations with top_logprobs from the hosted API. Local --dump-logprobs output must match token-by-token.
A frank comparison of running DeepSeek V4 Flash locally vs. using hosted APIs.
| Dimension | Local (ds4 + pi) | Cloud API |
|---|---|---|
| Privacy | ✔ Your code never leaves your machine | ✖ Prompts & code uploaded to remote servers |
| Cost | ✔ One-time hardware, $0/token | ✖ Recurring per-token billing |
| Speed | ✔ 26–36 t/s on M3 Max/Ultra | ✔ Fast, but adds latency |
| Context Window | ✔ Up to 1M tokens (configurable) | ✔ Usually 128k–1M |
| Model Quality | ✔ Same 284B frontier model | ✔ Same model (may be newer) |
| Availability | ✔ Always on, no outages | ✖ Rate limits, outages, deprecations |
| Thinking | ✔ Native chain-of-thought, scales with complexity | ✔ Supported (but costs extra) |
| Tool Calling | ✔ Full OpenAI/Anthropic tool format | ✔ Native tool support |
| Setup Effort | ✖ One-time download + config (~10 min) | ✔ Just an API key |
| Hardware Required | ✖ Mac with 128 GB+ RAM | ✔ Nothing — runs on their servers |
The tradeoff is simple: you trade a one-time hardware investment and a 10-minute setup for total privacy, zero recurring cost, and always-available inference. For professional developers who spend all day in an agent, the ROI is immediate.
— The local-first philosophyHere's what a real harness session looks like — from starting the server to having Pi refactor code using your local model.
# ── 1. Start the server (terminal 1) ──
./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192
# ── 2. Configure Pi (done once) ──
# Edit ~/.pi/agent/models.json and settings.json as shown above
# ── 3. Use Pi with your local model (terminal 2) ──
pi "Read the current project structure and suggest a refactor."
# Pi will:
# 1. Read files using its built-in tools
# 2. Send the conversation to ds4-server at http://127.0.0.1:8000
# 3. ds4-server renders the prompt, runs inference on Metal
# 4. Streams the response back with tool call suggestions
# 5. Pi executes the tools (edit files, run commands)
# 6. Sends the results back to the model for the next turn
# All locally, all private, all free.
# ── 4. Check the KV cache (debugging) ──
ls /tmp/ds4-kv/ # See .kv checkpoint files
hexdump -C /tmp/ds4-kv/*.kv | head 20 # Inspect cached prompt textThis is the key insight: the agent (Pi) and the model (ds4) are separate processes communicating over HTTP. Pi doesn't know or care that the model is running locally — it just sees an OpenAI-compatible API. ds4-server doesn't know about Pi's tool system — it just generates tokens and maps tool calls. The harness is the glue between them.
— Loose coupling, local executionTips from running ds4 + Pi in production for real agent work.
With 128 GB RAM and q2 quant (~81 GB), you have ~47 GB left for KV cache. At 1M tokens the compressed cache uses ~26 GB. Use --ctx 100000 to --ctx 300000 for a comfortable margin. Set maxTokens: 384000 in Pi config to avoid artificial caps.
Always use --kv-disk-dir. Agent clients resend the full conversation every request — without disk cache, every new session re-prefills from scratch. The cache directory is disposable: stop the server, rm -rf /tmp/ds4-kv, restart.
The first request to a new session does a full prefill. But ds4-server saves a "cold" checkpoint after the first prompt, trimmed to a prefill boundary. Subsequent requests with the same prefix skip the prefill entirely. For agent work, the first tool call is slow; everything after is fast.
Start the server with --trace /tmp/ds4-trace.txt. The trace log shows rendered prompts, cache decisions, generated text, and tool-parser events. It's the single best debugging tool for understanding what's happening under the hood.
Run make test after any engine changes. The test suite compares local logprobs against official DeepSeek API vectors at multiple context sizes. If the vectors pass, your engine is producing the same token distribution as the hosted model.
Thinking mode produces better results for complex tasks but uses more tokens. For simple edits, use /nothink or set reasoning_effort: "off". For architecture or debugging, use reasoning_effort: "high". The model's thinking length adapts to the problem.
Common issues and solutions.
./download_model.sh q3. You can also try reducing --ctx to free up KV cache memory: ./ds4-server --ctx 32768 ...
--trace /tmp/ds4-trace.txt to see the prefill phase. After the cold start, ds4-server saves a checkpoint and subsequent requests reuse it. For interactive work, keep the server running rather than restarting between sessions.
~/.pi/agent/models.json should have ds4 as a provider, and ~/.pi/agent/settings.json should set defaultProvider to "ds4". Run pi --debug "hello" to see which API endpoint Pi is hitting. Check that ds4-server is running (curl http://127.0.0.1:8000/v1/models).
--trace /tmp/ds4-trace.txt and look for cache HIT or cache MISS entries. Ensure Pi is sending the same prompt format each time.
sudo mdutil -a -i off to disable Spotlight indexing during inference. For sustained workloads, consider raising --kv-disk-space-mb to keep more cache on SSD.
~/.pi/agent/models.json has "thinkingFormat": "deepseek" and "requiresReasoningContentOnAssistantMessages": true in the compat block. Also run the server with --trace to see the tool call mapping in action.
./download_model.sh q3 (or q4, q5 for higher quality if you have more RAM). Stop the server, swap the file, restart. Your Pi config stays the same — just restart ds4-server with the new model file.
./ds4 opens an interactive REPL with the model. Useful for quick experiments and debugging. Run ./ds4 --help to see all flags. For programmatic access, ./ds4-server exposes an HTTP API — curl -X POST http://127.0.0.1:8000/v1/chat/completions -d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"Hello"}],"max_tokens":256}'