Let's be honest: your current "AI feature" is likely a hidden API call to a proprietary model, a few dozen lines of Python, and a prayer. If you are a technical lead or an architect, you have likely felt that sinking sensation in your gut when a stakeholder asks why you can't just "plug in an LLM" to your existing SQL-backed product by Monday. The industry has conflated a successful local prototype with a production-ready system.
But a prototype is an experiment; an architecture is a commitment to reliability, cost, and maintenance. Calling a completions endpoint is a transaction, not a design. When you ship a text box that talks back without a defined retrieval strategy, an evaluation framework, or a fallback mechanism, you aren't shipping "AI." You are shipping a high-entropy liability that will eventually hallucinate a refund policy or leak system prompts to a curious user. Real architecture begins where the openai.ChatCompletion call ends. It is the connective tissue that ensures the model serves the business rather than just consuming its margins and reputation.
The Retrieval Illusion
The most common mistake in enterprise "AI" is treating the LLM as a database. It is not. It is a reasoning engine with a limited, fuzzy memory. If your strategy for providing context is "stuff everything into the prompt," you have already failed the architecture test. Context windows are expensive, and the more you cram in, the more the model's attention wanders.
A real architecture requires a sophisticated retrieval strategy. How do you chunk your data? Are you using vector embeddings, keyword search, or a hybrid? How do you handle stale data when your underlying SQL records change but your vector store hasn't re-indexed? If you cannot explain the "R" in your RAG (Retrieval-Augmented Generation) beyond "we use Pinecone," you don't have an architecture. You have a dependency you don't understand.
The Evaluation Void
In traditional software, we have unit tests. In AI, we have "vibe checks." You run five prompts, the answers look reasonable, and you declare it ready for production. This is professional negligence.
An LLM is stochastic. The same input can yield different outputs. Without a quantitative evaluation baseline — using frameworks like RAGAS or custom "golden datasets" — you are flying blind. You need to know your faithfulness score, your relevancy score, and your hallucination rate. If you "improve" a prompt to fix one edge case, how do you know you didn't just break ten others? Architecture requires a regression suite for the "un-testable."
The Economic Reality
Your SaaS margins were likely built on deterministic compute and tiered storage. LLMs break this. Every query has a variable cost based on token count, and "reasoning" models can be orders of magnitude more expensive than the APIs they replace.
Shipping a feature without a cost-per-query model and hard rate limits is like handing your customers a blank check drawn on your cloud budget. A technical architect must decide: at what point is the query too expensive to fulfill? When do we route to a smaller, cheaper, locally-hosted model? If the answer is "we'll figure it out in the next quarter's billing cycle," you are building a financial trap, not a product.
Ownership and the Fallback Contract
Who owns your "intelligence"? If you rely entirely on a third-party API, you are one policy change or model "optimization" away from a broken product. OpenAI, Anthropic, and Google frequently update their models, often leading to "model drift" where your carefully engineered prompts suddenly stop working.
Furthermore, what happens when the API is down? Or when the latency spikes to 30 seconds? A robust architecture demands a fallback behavior. Maybe it's a cached response, a simpler heuristic-based answer, or a graceful "I'm experiencing high latency" message. Leaving the user staring at a spinning loader while your backend waits for a 504 error is a failure of design.
Artifact: The LLM Architecture Audit
Before you merge that PR or sign off on the "AI roadmap," every architect should be able to answer these seven questions.
| Item | Question | Requirement for "Go" |
|---|---|---|
| 1. Retrieval Strategy | How is context selected and managed? | Hybrid search defined; re-indexing latency documented. |
| 2. Evaluation Baseline | What are the quantitative metrics for success? | A "golden set" of 50+ queries with human-verified scores. |
| 3. Fallback Behavior | What happens when the model is down or "confused"? | Pre-defined failure states and UI triggers (no infinite loops). |
| 4. Cost per Query | What is the maximum $ cost of a single user session? | Unit economics calculated at 90th percentile token usage. |
| 5. Latency Contract | What is the P99 response time and how is it mitigated? | Streaming enabled or background processing with polling. |
| 6. Ownership | Who owns the prompts and what happens if the provider changes? | Prompts version-controlled; abstraction layer in place for providers. |
| 7. Failure UX | How does the UI handle hallucinations or "I don't know"? | Clear attribution of "AI generated" content and "Report Error" button. |
Answering "we'll fix it later" to more than two of these means you are building a feature, not an architecture. And in the enterprise world, features without architecture are just debt with a better marketing budget.
Discuss an enterprise AI architecture or delivery challenge → techbuzzz.me.
