What 283 Production Requests Taught Me About Assumptions
I analyzed 283 production API requests to understand how users actually interact with an AI platform. What I found contradicted several assumptions our team had made about how the system would be used.
Why I Analyzed Production Traffic
We had a problem. Our AI platform was designed for a specific use case: long-form question answering over a corpus of technical documents. You upload a document set, you ask questions, the system retrieves relevant context and generates an answer.
The product team had designed the interface around this use case. The engineering team had built the infrastructure for it. The documentation assumed it.
And then users started using it differently.
Queries were shorter than we expected. Users would send single-sentence questions where we expected paragraph-length context-setting. Users would ask follow-up questions in sequence - not the conversational flow we'd designed for, but something more like a rapid-fire interrogation. Users would ask about topics that weren't in any uploaded document - general knowledge questions that our retrieval layer had no corpus to answer.
We had built a system for a use case. Our users had found a different use case.
This is the story of what I learned from analyzing 283 production requests to understand the gap.
What The Data Showed
Query length distribution
We had assumed queries would average 50-100 words. The actual median query length was 12 words. The mode was a single sentence.
This matters because our prompt construction assumed some context-setting. We had prompt templates that started with "Given the following context, answer the user's question." For 12-word queries, this felt heavy. For single-sentence queries, it felt absurd.
More importantly: short queries retrieve differently than long queries. A 12-word query has fewer semantic dimensions than a 100-word query. The vector that represents "how do I configure the auth service?" is different from the vector that represents "I'm trying to set up authentication for a new service and I need to understand the difference between JWT and session-based auth, particularly for a microservices architecture." Our retrieval was optimized for longer queries because that's what we tested with. Our users were sending shorter queries because that's what felt natural.
Follow-up patterns
We had designed for a conversation model: you ask a question, you get an answer, you ask a follow-up. Each follow-up was treated as a new conversation turn.
In practice, users would send 4-8 messages in rapid sequence - essentially a monologue with the model as the recipient. The "follow-up" wasn't a clarification or a pivoting of the original question. It was an extension.
This meant our conversation context window was being used differently than we'd designed. Users expected continuity across a sequence of messages. We were treating each message as independent with a shared context that was being rebuilt each time.
Out-of-scope questions
About 23 percent of queries had no relevant context in any uploaded document. These weren't edge cases - they were a significant fraction of total usage.
Our system had no explicit handling for these. The retrieval layer would return empty or near-empty results. The model would either say "I don't know" or, more often, hallucinate an answer based on its training data.
This was the failure mode we had not designed for and had not tested.
What This Changed
Query preprocessing
We added query preprocessing that expanded short queries using an LLM to generate a richer query representation - without changing what the user saw. The user still typed 12 words. The retrieval layer received a 12-word query plus a 50-word expanded version that provided more semantic context for the vector search.
This is not a new technique. What's worth noting is that we only discovered we needed it because we analyzed production traffic. We had tested extensively with long queries. We had not tested with the actual query distribution our users sent.
Conversation modeling
We changed how we handled conversation context. Instead of treating each message as a separate retrieval event, we built a conversation memory that accumulated across a session and was used to enrich each subsequent retrieval.
This is conversation modeling 101. What's worth noting is that we built it in response to data, not in response to a design decision. We had designed for conversation. We had not designed for the conversation patterns our users actually had.
Out-of-scope detection
We added an explicit relevance check: before generating a response, the system checked whether the retrieved context was sufficient to answer the query. If the retrieved context had low relevance to the query, the system would say "I don't have information about that in the provided documents" rather than hallucinate.
This sounds obvious. It wasn't in the original design. It wasn't in the original design because we had not anticipated that 23 percent of queries would be out-of-scope. We had designed for a corpus that was always sufficient.
What This Taught Me About Assumptions
The assumption that hurt us most was the assumption of intended use. We had designed for a use case. We had tested with inputs that matched that use case. We had not tested with inputs that matched how users actually used the system.
This is not a unique story. It's a common pattern in platform engineering: you build for a designed use case, users find a different use case, you either adapt or lose users.
The more specific assumption: we assumed that the query distribution in production would match the query distribution in our testing. It didn't. Our testing queries were written by engineers who understood the system. Our production queries were written by users who had a different mental model of what the system did.
The lesson I took: production traffic is the only reliable signal of how your system is actually used. Testing with internal users is valuable. Testing with designed test cases is valuable. Neither replaces the information you get from watching what real users actually do.
What I'd Do Differently
Instrument before launch, not after
We added query analysis after we'd shipped and noticed anomalies. This meant we had weeks of production traffic we couldn't analyze because we hadn't been capturing the right data.
If I were building the system from scratch, I'd capture query length, retrieval precision, and response confidence scores from day one. Not because I knew I'd need them - because I knew I couldn't predict exactly what I'd need.
Test with adversarial users before launch
Internal testing tends to confirm intended use. External testing, or at least testing with users who don't know what the system is "supposed" to do, tends to surface how the system will actually be used.
This is why dogfooding has limits. When you dogfood your own product, you use it the way you designed it to be used. You don't discover the use cases you didn't design for.
Treat production traffic as a design input
The most valuable product decision we made after analyzing the traffic was changing the query preprocessing. The second most valuable was adding out-of-scope detection. Both came from data, not from intuition.
Production traffic is a design input that most teams underinvest in analyzing. We had dashboards. We didn't have analysis. There's a difference.
Final Thoughts
The thing about assumptions is that they're invisible until they're violated. We assumed our users would use the system the way we designed it. They didn't. The gap was only visible when we looked at what they actually did.
This is the argument for instrumentation that goes beyond availability and latency. Metrics that tell you whether the system is up are necessary. Metrics that tell you whether the system is being used the way you designed are valuable. The second set of metrics is what lets you catch the gap before it becomes a problem.
We fixed the query preprocessing and the conversation modeling. We added out-of-scope detection. The system got better. What I remember most clearly is that the fix started with going to look - not with having the right answer, but with admitting I didn't know how the system was actually being used.
Related Articles
Why Observability Comes Before Optimization
Every optimization I've seen fail started the same way: an engineer who was confident about where the problem was, before they'd measured anything. Here's what changed how I think about building systems.
The Hidden Cost of LLM Context
Every RAG system I've worked with had the same silent failure mode: retrieval that looked correct but wasn't. Here's what I learned about managing context quality in production AI systems.