Ask It Twice: The Rewrites Between a Convincing Demo and a System That Agrees with Itself

Our first natural-language analytics system was five LLM calls in a trench coat, and it demoed beautifully, until someone asked the same question twice and got two different answers. What followed was a chain of rewrites driven less by missing code than by flawed assumptions. Correctness moved out of the prompt and into a semantic layer. Queries shifted off the shared warehouse onto single-scope datasets built per conversation. The router between a "safe path" and a "smart path" gave way to a deterministic core called per sub-question. Others changed how a turn is recorded and how a report is rendered. Every stage of this evolution was judged by a simple evaluation metric you can implement in an afternoon: run a question N times against a frozen dataset and score agreement at three levels - the SQL it wrote, the numbers it got, and the conclusion it drew, because a system can agree on every number and still tell you two different stories. The costs and the payoffs never arrived together: the extractions that made each simplification possible landed weeks before it. While we did not discover these patterns in this order, this talk presents the sequence that would have saved us the most pain.
Format: TECHNICAL TALK
Location: ROOM A
Speakers: Milan Ignjic
Level: Beginner to Intermediate
Audience: Registered participants