Overview
A .NET 8 conversational agent on Bot Framework v4. Tenants upload their own knowledge base across eight file formats; an ingestion pipeline parses, chunks, embeds and indexes it into three specialised Azure AI Search indexes plus a Cosmos row store for spreadsheets. At query time an intent analyser classifies the question, a router decides whether the answer lives in documents or tables, and Semantic Kernel drives automatic function-calling over search plugins to fetch the exact rows, passages and figures.
The problem
Generic document chat fails on real enterprise knowledge bases in two specific ways: spreadsheets get flattened into prose and become unqueryable, and figures become invisible because nothing indexes what an image means. Meanwhile latency compounds — every retrieval hop and every history token is time the user spends watching a typing indicator.
Approach
- 01Three specialised search indexes instead of one — narrative text, extracted table rows, and figures with vision-generated captions — so each content type is retrieved by a strategy that suits it.
- 02A separate Cosmos DB row store for pure spreadsheets, making exact-value lookups a database query rather than a semantic guess.
- 03Cross-pipeline file ranking that scores every tenant file across both retrieval paths before choosing a source, so a knowledge base mixing a DOCX and an XLSX routes correctly.
- 04Semantic Kernel auto-function-calling over a multi-stage tabular plugin with schema discovery, column detection and numbered disambiguation — so the model selects from real options instead of inventing identifiers.
- 05A sustained latency programme: zero-LLM regex fast paths for follow-ups, value-level early exits, dynamic prompt right-sizing, and conversation-history compression.
- 06Full-stack platform work underneath — chunked file upload via GraphQL, per-tenant configuration in Table Storage, SignalR for real-time push, and Polly-backed resilience throughout.
Decisions & trade-offs
Compressing conversation history from ~10.8K tokens to ~500
Follow-up questions were carrying the entire history into every call and produced a 38-second latency spike. Compressing to a bounded summary fixed it, at the cost of some fidelity on very long conversations — an acceptable trade when the alternative is a user watching a typing indicator for half a minute.
Extracting follow-up fields with regex instead of an LLM
'Order id 10020' followed by 'what about 10015?' does not need a model to resolve. A pure-regex field extractor handles it with zero LLM calls above a confidence threshold, and only falls through to the expensive path below it. The cheapest call is the one you do not make — and this class of follow-up is most of real usage.
Pre-executing document search instead of trusting auto-function-calling
Semantic Kernel's automatic tool invocation proved unreliable for document queries — it would sometimes answer without searching. Text and figure searches are now pre-executed in parallel and injected, while tabular queries still route through the kernel where auto-calling works well. Two paths is more code than one; it is also the difference between grounded and plausible.
Right-sizing the system prompt per query type
A single full agentic prompt cost roughly 2–4K tokens on every turn regardless of need. Selecting the smallest sufficient prompt — a ~500-token context-reuse prompt for follow-ups, larger ones for tabular or document work — cut both latency and spend. The cost is more prompts to maintain and a selection step that can pick wrong; the fallback is the full prompt, so a wrong pick degrades to the old behaviour rather than failing.
Filtering on reranker score rather than returning top-k
Results below a reranker threshold are dropped even when that leaves fewer results than requested. Returning a weak match ranked first is worse than returning less, because a grounded-looking wrong answer is the failure mode users cannot detect.
What it does not do
- Worst-case queries still run 20–30 seconds, which is why the turn pipeline sends a typing indicator immediately and keeps it alive in the background. The sub-5-second target holds for the fast paths, not for every query.
- The reranker cutoff trades recall for precision — a genuinely relevant passage scoring below threshold is dropped rather than shown with a caveat.
- Tenants with several spreadsheets sharing column names need a disambiguation turn; the ranker narrows it, but the user is sometimes asked to choose.
- Auto-function-calling was not dependable enough to rely on for document retrieval, so that path carries hand-written pre-execution rather than being fully agentic.