Agentic RAG Benchmark Highlights Latency Trade-Offs in Engineering Workflows
A new benchmark comparing Agentic RAG with plain hybrid RAG in small hydropower engineering reveals gains in retrieval precision alongside substantial latency penalties.
A newly published evaluation of Retrieval-Augmented Generation (RAG) applied to Small Hydropower Plant (SHP) engineering offers fresh insights into the trade-offs between agentic and plain hybrid architectures. The benchmark reveals that while multi-agent workflows enhance retrieval precision over complex, multi-format engineering documentation, they introduce substantial latency overhead without any guarantee of outperforming traditional pipelines on answer faithfulness.
Document Processing and Chunking Pipeline
Engineering documentation for small hydropower projects spans digital PDFs, scanned reports, spreadsheets, and presentation files. To prepare these varied sources for context retrieval without relying on cloud inference APIs, the author implemented a three-tiered parsing pipeline running locally via Ollama. Text-accessible files are processed through dedicated office parsers or pymupdf4llm, which targets Markdown extraction for digital PDFs. Documents failing an 80-character validation threshold trigger a fallback to Docling for OCR and layout recovery using EasyOCR or RapidOCR.
To maintain structural context, the system processes parsed Markdown through a four-pass rule-based hierarchical chunker instead of semantic segmentation models:
- Section Splitting: Structural headings (
#,##) and horizontal rules divide documents into logical sections while capturing embedded page or slide labels. - Targeted Chunking: Narrative text splits at paragraph and sentence boundaries using a soft target of 1,800 characters. Sliced tables retain column header rows to preserve numerical meaning across chunk boundaries.
- Fragment Merging: Leftover fragments under 150 characters merge into preceding chunks to prevent orphaned text.
- Metadata Enrichment: Each passage receives top-level citation banners detailing source filenames, page locations, and section hierarchies before calculating token estimates with
tiktoken.
Vector Indexing and Multi-Agent Architecture
The indexed passages are vectorized locally using qwen3-embedding:0.6b, yielding 1,024-dimensional vectors stored in a Qdrant collection named shp_knowledge. Chunks receive deterministic UUID5 identifiers to support clean overwrites during re-indexing. Search logic combines Qdrant's dense vector search with a local BM25 sparse keyword retriever using Reciprocal Rank Fusion (RRF), allowing exact equipment codes and technical terms to be weighted alongside semantic meaning.
The orchestration layer, built on LangGraph, routes incoming queries through specialized agent roles:
- Planner Agent: Classifies user input into direct responses, domain-boundary refusals, or technical RAG tasks, decomposing complex queries into sub-questions with metadata filters (such as
is_table: true). - Orchestrator & Workers: Uses LangGraph's dynamic
Send()API to dispatch requests to specialized narrative retrievers, tabular retrievers, and a FastAPI-based Model Context Protocol (MCP) server designed for physics calculations (power equations, specific speed, penstock sizing). - Quality Gates: Evaluates context sufficiency (Gate 1) and generation groundedness (Gate 2) prior to presenting answers, invoking fallback or feedback agents when context is missing or responses are ungrounded.
For developers seeking to test and benchmark multi-agent systems using diverse model architectures, trying out top AI models cheaply through one API at https://apixoai.online provides a streamlined option alongside local Ollama or free-tier setups like Gemini and Groq.
What it means for developers
The benchmark demonstrates clear architectural lessons for software engineers building document-driven AI applications:
- Precision vs. Latency: Agentic RAG provides tighter retrieval control for complex multi-step queries, but the iterative agent steps introduce significant response latency compared to plain hybrid RAG.
- Faithfulness Limits: Multi-agent routing does not automatically eliminate hallucinations or guarantee higher faithfulness over a well-constructed traditional hybrid pipeline.
- Local Accessibility: Complete engineering RAG pipelines—including document parsing, hybrid retrieval, tool execution, and quality checks—can be effectively operated without recurring cloud inference bills using local vector databases like Qdrant and open-weight models.
- Data Preparation Matters: Injecting explicit context banners and keeping table headers intact during chunking directly improves downstream retrieval quality regardless of agent complexity.
Source: Agentic RAG vs. Plain Hybrid RAG: What My Hydropower Benchmark Revealed — Towards AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
