Local AI Translation Benchmark: Gemma 4 Leads in Speed but Stumbles on Literary Prose
A test of four local GGUF models translating an English novel into Serbian reveals that high token throughput and 100% automated pass rates do not guarantee faithful or grammatical prose.
Deploying small language models on modest consumer hardware often forces a difficult tradeoff between execution speed, operational reliability, and linguistic accuracy. In an exploratory benchmark designed to test these limits, four quantized local models were tasked with translating complex literary English into Serbian on an entry-level laptop equipped with an NVIDIA GTX 1650 Ti GPU and 4 GB of VRAM. Running via a local llama.cpp server, the test evaluated TranslateGemma 4B, Gemma 4 E2B, Qwen3 4B, and Qwen3.5 4B across identical passages from Tobias Smollett's novel The Adventures of Ferdinand Count Fathom.
While Google DeepMind's Gemma 4 E2B emerged as the unambiguous frontrunner in throughput and pipeline stability, the qualitative findings exposed a stark reality: models that pass every automated validation check can still produce deeply flawed, unpublishable text.
Throughput and reliability under low-resource constraints
The experiment fed the first 20 non-empty lines—totaling 943 English words—of the 158,396-word novel to each model using Q4_K_M GGUF quantization, greedy decoding (temperature 0, seed 0), and a 512-token completion cap. The prompt instructed the models to generate fluent Latin-script Serbian while maintaining the original literary tone and avoiding extraneous commentary.
From an operational standpoint, Gemma 4 E2B outpaced the competition. It achieved a generation throughput of 59.65 output tokens per second and prompt ingestion speeds of 325.39 tokens per second, completing the entire 20-segment batch in 38.95 seconds of total wall-clock time. By comparison, TranslateGemma 4B generated 41.76 tokens per second (finishing in 55.48 seconds), Qwen3 reached 40.06 tokens per second, and Qwen3.5 clocked 34.62 tokens per second.
System reliability diverged significantly across the architectures. Both Gemma 4 and TranslateGemma registered a 100% technical acceptance rate (20 out of 20 passages). However, TranslateGemma defaulted to Cyrillic in 25% of its outputs, requiring programmatic transliteration to Latin characters. Meanwhile, Qwen3.5 accepted 19 lines across 22 calls after falling into an infinite degeneration loop on segment four—endlessly generating the Serbian word "ništa" ("nothing") until hitting the 512-token limit across three identical retries. Qwen3 suffered from character corruption and truncated completions, accepting only 16 lines while consuming 28 HTTP requests and taking 147.59 seconds.
When automated validation hides linguistic collapse
Despite pristine technical scores, human inspection of the Serbian text revealed severe stylistic, lexical, and grammatical breakdowns. Automated validation flags formatting syntax, line alignment, and script choices, but it remains blind to semantic drift and invented vocabulary.
For instance, while Gemma 4 correctly captured the meaning of "altogether impracticable" where TranslateGemma failed, it invented non-standard terms such as "samopoželju patrona". In another passage, Gemma 4 abruptly flipped grammatical gender mid-sentence ("Osmešila sam se … i tremeo sam …") and relied on stiff constructions like "ovaj adres". Both Gemma variants stumbled over basic literary idioms: "there is no medium" was mistranslated literally as lacking tools or mechanisms ("nema sredstva") rather than recognizing the absence of a middle ground.
Regional variety consistency also proved difficult. While the target was standard Ekavian Serbian, TranslateGemma continually drifted into Ijekavian forms ("svijet", "mjeri", "posljedice"), and Qwen3.5 mixed both variants within adjacent outputs. Although postprocessing can cleanly map Cyrillic letters to Latin equivalents, mechanical text transformation cannot fix garbled syntax or misinterpreted source prose.
What it means for developers
For engineers constructing multilingual processing pipelines, this benchmark highlights critical architectural lessons:
- Automated pass rates are deceptive: Valid JSON, matching line counts, and script conformity do not equate to translation fidelity. Quality assurance must include semantic evaluation and human-in-the-loop validation.
- Heuristic retries must be context-aware: Re-executing identical deterministic prompts (temperature 0) against infinite generation loops or script errors simply burns GPU cycles. Pipelines must deploy specific repetition penalties, decoding adjustments, or fallback paths when token caps are exhausted prematurely.
- Text cleaning upstream is non-negotiable: Irregular source formatting—such as collapsed sentence boundaries ("impracticable.Indeed")—substantially degrades small-model comprehension.
While compact open models like Gemma 4 provide fast draft assistance on edge hardware, mission-critical translation often demands larger reasoning models. When projects outgrow local hardware bounds, developers can try top AI models cheaply through one API at https://apixoai.online to benchmark advanced proprietary architectures without maintaining complex multi-vendor infrastructure.
Ultimately, small local models show real promise for assisted drafting workflows, but deploying them for unattended literary publication remains beyond the reach of 4-billion-parameter systems.
Source: I Tested Four Local AI Models on a Literary Translation. Gemma 4 Won — With a Catch. — Towards AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
