Two Rust Clients for Gemma 4: Calling the Endpoint vs. Calling the MCP Server 🦀

Two Rust Clients for Gemma 4: Calling the Endpoint vs. Calling the MCP Server

What is this project trying to Do?

This article provides a step-by-step guide to two small Rust CLIs that ask a self-hosted Gemma 4 E2B the same question. The first calls the model's OpenAI-compatible HTTP endpoint directly. The second is an MCP client: it launches the rig's own MCP server and asks through its tools. Both CLIs are demos, and their output is read by an audience. So neither hides anything behind a --verbose flag: every run prints the target, the health check, the request, the answer, the model's reasoning, token counts, latency, and whatever the server says about itself.

Why Two Clients?

Because they answer two different questions. gemma-rust shows what the model said. One HTTP call, the raw OpenAI-style response, every field printed. gemma-rust-mcp shows what an agent sees. An MCP client like Claude Code never touches the endpoint. It calls tools, and gets back whatever those tools choose to report. Writing a second client in Rust - one that is not Claude Code and not the Python SDK the servers were built with - is the fastest way to find out what those servers actually return.

Neither one starts, stops or deploys anything. The rigs do that.

How Does This All Fit Together?

gemma-rust ─────── HTTP (reqwest) ───────────────────────┐
├──▶ llama-server GTX 1650 Ti, local │
vLLM NVIDIA L4, Cloud Run
gemma-rust-mcp ─── MCP over stdio (rmcp) ──▶ server.py ──┘
(Python, the rig's own)

The MCP path makes the same HTTP call in the end. It just makes it from inside a Python process that the Rust client launched, and hands back markdown instead of JSON.

Where do I start?

The strategy for building the two clients is an incremental step-by-step approach.

Step 1 - Install Rust

Use rustup:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source ~/.cargo/env
rustc --version

Step 2 - Build llama.cpp

With CUDA:

git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp
cd ~/llama.cpp
git checkout 95ef7fc
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=75 -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j --target llama-server

Step 3 - Download the Gemma 4 Checkpoint

The rig serves Google's QAT q4_0 Gemma 4 E2B:

hf auth login
hf download google/gemma-4-E2B-it-qat-q4_0-gguf --local-dir ~/models/gemma-4-E2B-it-qat-q4_0

Step 4 - Start the Model Server

Run it in the foreground; Ctrl-C is the whole teardown:

~/llama.cpp/build/bin/llama-server \
  -m ~/models/gemma-4-E2B-it-qat-q4_0/gemma-4-E2B_q4_0-it.gguf \
  --host 127.0.0.1 --port 8080 -ngl 99 -c 8192

Step 5 - Build the HTTP Client

cd ~
git clone https://github.com/xbill9/gemma-rust
cd gemma-rust
make prod

Step 6 - Ask the Local Model

./target/release/gemma-rust "In one sentence, what is a TPU?"

What the HTTP Client Is Doing

Three decisions make one code path work on two servers that disagree about almost everything. The model id comes from the server. llama.cpp accepts any model value; vLLM returns 404 unless it is exactly the served id. The only default that works on both is the first id from /v1/models:

let model = served.first().and_then(|m| m["id"].as_str()).context("the server listed no models at /v1/models; pass --model")?.to_string();

Both reasoning fields are read. The two servers put Gemma's thinking in different places:

#[derive(Deserialize)]
struct Message {
    content: Option<String>,
    /// Where llama.cpp puts Gemma 4's thinking
    reasoning_content: Option<String>,
    /// Where vLLM puts it
    reasoning: Option<String>,
}

Server stats are optional and printed generically. llama.cpp returns a timings object; vLLM returns none. Both are Option<Value>, and whichever is present gets its own section.

Tip: Two Ways to Get an Empty Reply From a Healthy Server

Use /v1/chat/completions, never /v1/completions. On these instruction-tuned checkpoints the raw completions endpoint returns empty text, which looks exactly like a broken server. Give Gemma room to think. On llama.cpp, content stays empty until the thinking closes. Starve it and see:

./target/release/gemma-rust --max-tokens 32 "In one sentence, what is a TPU?"

Step 7 - Point It at Cloud Run

Same binary, different endpoint. If you deployed the Cloud Run rig, its Makefile prints the URL:

GEMMA_ENDPOINT=$(make -s -C ~/gemma4-dev/gpu-2B-cloudrun-devops-agent endpoint) ./target/release/gemma-rust "In one sentence, what is a TPU?"

What the Server Says About Itself?

./target/release/gemma-rust --status

A 404 is reported, not an error. /version is vLLM's; /props and /slots are llama.cpp's. Each server answers half the probes, and the list of which ones is itself useful.