Instead, translations should happen automatically. This is the story of how we built that system, which… wasn’t simple at all!

The Problem

Over the years, we accumulated a large number of apps. Like most products, they were filled with buttons, labels, placeholders, tooltips, and messages written directly in English. Whenever a team wanted to support another language, the process was always the same:

1. Find the strings
2. Extract them
3. Create translation keys
4. Update the UI
5. Maintain translation files forever

It worked, but nobody enjoyed doing it. We started wondering whether the build process could do most of that work for us. That idea eventually became wick-ui-i18n.

Build-Time Translation Extraction

At the center of the system is a Vite plugin built specifically for our internal component library, wick-ui. The plugin scans application code during the build, finds text inside supported components, extracts it automatically, and generates a translation dictionary.

Developers keep writing components normally. No translation keys. No manual extraction.

How It Works

The plugin runs during Vite’s transform phase. Before doing any expensive parsing, it performs a quick check. If a file doesn’t contain a wick-ui component, it gets skipped immediately. Files that pass the check are parsed into a Babel AST (Abstract Syntax Tree), and the plugin walks through the tree looking for translatable content.

Handling Different Types of Text

One of the first challenges was that text can appear in several different forms:

<WuButton>Hello</WuButton>
<WuButton>{'Hello'}</WuButton>
<WuButton>Hello {name}</WuButton>
<WuButton>{`Hello ${name}`}</WuButton>
<WuButton children="Hello" />

Did you know? The expressions inside ${} in JavaScript template literals are formally called quasis.

Each case produces a different AST structure, so they need different handling. Plain JSX text is straightforward. Static string expressions are straightforward too. Template literals are more interesting. For example:

{
  `Hello ${name}, how are you doing?`;
}

Since part of the text is dynamic, it can’t become a single translation key. Instead, the plugin extracts the static portions and leaves runtime values untouched. The result looks like this:

<>
  <WuTranslate __i18nKey="Hello" />
  {name}
  <WuTranslate __i18nKey=", how are you doing?" />
</>

(Note: In this output, WuTranslate acts as a React fragment. More on this later.)

The same idea applies to translatable props such as placeholder, title, and aria-label. For example:

<WuInput placeholder="Search..." />

Becomes:

<WuInput placeholder={wt("Search...")} />

The HTML Entity Surprise

One of the more unexpected bugs appeared after the initial rollout. A string containing & displayed correctly in one place but showed the literal text & somewhere else.

The root cause was Babel.

When Babel parses JSX text, it automatically decodes HTML entities. By the time the plugin sees the value, & has already become &. Using that decoded value as a translation key caused inconsistencies when the code was rewritten.

The fix was to inspect the original source text instead of Babel’s decoded value. Whenever an entity is found, the plugin preserves it exactly as written and only extracts the surrounding text. This means:

<WuButton>Hello &amp; World</WuButton>

Becomes:

<WuButton>
  <WuTranslate __i18nKey="Hello" />
  &amp;
  <WuTranslate __i18nKey="World" />
</WuButton>

The entity remains untouched while the surrounding text stays translatable.

The wt() Helper

Not every string lives inside JSX. Sometimes translations are needed in configuration objects, event handlers, utility functions, or hooks. For those situations, we added wt():

placeholder={wt('Enter your name')}

The plugin records these keys during the build process but leaves the code itself unchanged. From a developer’s perspective, it’s just another function call.

However, a standard function is not reactive. To solve this, we also introduced the useWt() hook to make text reactive when translations change dynamically.

Generated Output

As files are processed, the plugin builds a dictionary of translation keys. At the end of the build, that dictionary is written to wick-ui-i18n.json.

During development, Vite serves this file dynamically so newly added keys appear immediately without requiring a rebuild. The plugin also injects any required imports automatically.

Runtime Translation

The build step only creates the dictionary. Runtime translation is handled by wick-ui-lib.

WuTranslateProvider

Applications wrap themselves with WuTranslateProvider:

<WuTranslateProvider defaultLocale="es_LA">
  <App />
</WuTranslateProvider>

When the app starts—or when the locale changes—the provider loads the generated dictionary. For English, that’s all that’s required.

For any other language, the provider sends the dictionary to our translation service and receives translated values in return. If the translation fails, the system falls back to the original English strings instead of breaking the UI.

With the hook, we can change the language dynamically. On change, we call the backend, sending the JSON file with the target language code, and it instantly returns the translation for the respective keys. The UI update is instantaneous. Although the performance tab might show multiple renders, they are lightweight text updates—so no worries!

WuTranslate

Text extracted by the plugin is rewritten into:

<WuTranslate __i18nKey="Hello" />

The component simply looks up the translated value and falls back to the original key when a translation is missing. This component actually replaces the text with a React Fragment. We could have used a span or another HTML node, but we chose <Fragment> so the plugin wouldn’t inject any unnecessary DOM nodes. It also makes debugging significantly cleaner.

Implementation Note: We used magic-string during the compilation phase to ensure that original line numbers persist for accurate debugging.

wt() and useWt()

For non-React code, wt() reads from a shared dictionary store and returns the translated value when available. When reactivity is required inside components, useWt() can be used instead. This allows us to fully support translations inside plain .ts files (like constant arrays we want to loop through).

Translation Management

Translations are managed through the Admin2 dashboard. The interface is split into two primary backend sections:

1. Automatic Translation Generation

Not every translation exists when an application first requests it. When the translation API receives a dictionary, any missing keys for the target language are automatically queued for processing.

A background service batches these missing entries and sends them through our AI translation pipeline, powered by our internal AI infrastructure and Google Translate. The generated translations are then stored in the database for future use.

While a translation is being generated, the API simply falls back to the original English text, ensuring the UI remains fully functional. Once generated, future requests are served directly from storage.

2. Translations & Languages Workflow

• Translations Section: This is where translation entries are reviewed and edited. Users can:
  • Search translation keys
  • Edit translations for individual languages
  • Identify missing translations
  • Save only modified values
  • Import translations in bulk
  • Test translations directly against the API
• Languages Section: This section manages supported languages. Each language stores metadata such as:
  • Language code & Display name
  • AI router code
    • Character set & RTL (Right-to-Left) configuration
    • Auto-translation support

The system currently supports ~200 languages, and the entire workflow—from key discovery to translation generation and storage—is fully automated.

End-to-End Flow

Here is exactly what the completed developer and user workflow looks like:

[Developer writes raw JSX]
       │
       ▼
 1. <WuButton>Save changes</WuButton>
       │
       ▼ (Vite Build Phase via Babel AST)
 2. Plugin extracts text to `wick-ui-i18n.json`
    Component is rewritten to:
    <WuButton><WuTranslate __i18nKey="Save changes" /></WuButton>
       │
       ▼ (Translations managed/automated through Orion)
 3. User opens the application in French
       │
       ▼ (WuTranslateProvider fetches keys)
 4. UI instantly renders:
    "Enregistrer les modifications"

The developer never created a translation key, never maintained a translation file, and never added any i18n plumbing. They simply wrote a button.

Why Plan Mode isn’t enough

For a long while, my default move with any non-trivial feature was Claude Code’s plan mode. Sketch the idea, let it draft a plan, skim it, hit go.

The cracks showed up fast.

• A fuzzy ask produces a fuzzy plan. The output looks structured and confident, but it’s quietly inheriting every gap in your thinking.
• The agent drifts mid-execution. You approve a plan, switch to execute, and it quietly adds files and patterns the plan never mentioned. You end up appending “implement exactly as written, nothing extra” to every run.
• It never challenges the idea itself. Plan mode answers how Claude would build your thing. It never asks whether the thing is worth building, or whether you’ve actually thought it through. You can burn three days producing a clean, well-structured mistake.

That last one is the real gap. Plan mode assumes your idea is already sound and your only problem is execution. Most of the time, that assumption is wrong — and that’s the exact gap /grill-me fills.

The skill that argues back

Matt Pocock — the TypeScript educator behind Total TypeScript — opened up his personal .claude directory as a public skills repo on February 3, 2026. Within three months it had crossed 50,000 GitHub stars, peaking near the top of GitHub’s trending charts. Not a framework, not a wrapper — just the actual markdown files he uses every day.

The repo targets one failure mode: misalignment. You think the agent understood you. Then you see what it built, and it clearly didn’t. The same communication gap you get with a human dev, now at machine speed.

The headline fix is /grill-me. It removes vagueness before any code exists, by interrogating your plan one decision at a time. The entire skill is three sentences:

Interview me relentlessly about every aspect of this
plan until we reach a shared understanding. Walk down
each branch of the design tree, resolving dependencies
between decisions one by one.

That’s it. No clever machinery — the value is entirely in being forced to answer.

Getting it running:

npx skills add mattpocock/skills

Then run /setup-matt-pocock-skills once per repo to wire up your issue tracker and config.

The questions you didn’t want to answer

Take a real feature — per-workspace billing limits in a multi-tenant SaaS. You sketch the idea and type /grill-me.

Claude: When a workspace hits its limit, does the whole workspace get blocked, or just new usage?

Me: Just new usage, I think.

Claude: Who gets notified — the user who triggered it, the workspace admin, or both? Through what channel?

Me: Admin. Probably email.

Claude: If the admin ignores that email and a user hits the wall mid-session, what’s the experience? Hard error? Degraded mode? Graceful message?

Me: …I hadn’t defined that.

Claude: That’s your entire degradation story. It shapes the API contract, the frontend, and what the billing page has to show. Let’s keep going.

One rough idea. A handful of decisions that reshape the architecture before implementation, instead of surfacing as bugs three days in.

Every answer you give in a grill session is a decision you won’t have to reverse in code review.

The pipeline

_{FIG. 01 · The four-step workflow.}

The whole thing runs as a single continuous session, each step feeding the next.

/grill-me → /to-prd → /to-issues → /afk

The rule that makes it work: never /clear between steps. The PRD skill leans on everything from the grill — your answers, your reasoning, the edge cases you named, the trade-offs you accepted. A PRD generated cold is a template. A PRD generated right after a grill is a spec with a point of view.

Tight PRD in, tight tickets out — /to-issues produces tasks with real context and acceptance criteria, not “implement auth.” Tight tickets mean /afk can execute without guessing.

There’s a second payoff that shows up later. The PRD and the issues are written artifacts — they live in your tracker, not just in a chat window. So when a session dies halfway, or you come back to the same feature a week later, you’re not reconstructing intent from memory. You point a fresh session at the PRD and the open issues, and it picks up with the full reasoning already in hand: what you decided, why, and what’s left. The grill happens once; the context it produces keeps paying out across every session that touches the feature.

Output quality at the end is a direct function of honesty at the beginning.

When the grill gets hard: /handoff

Deep grill sessions burn context. You’re well into one when you realize you need to prototype something or check how an existing schema behaves. The instinct is to do it right there in the same session.

Don’t.

Pocock also wrote /handoff, built for exactly this. It compresses the current session into a document a fresh agent can pick up — context, decisions, intent, suggested next steps. Your grill stays clean; the side quest gets its own window.

Two patterns:

• Fire and forget — something out of scope appears. Hand it to a fresh agent, return to the grill.
• Grill → handoff → prototype → handoff back — validate an assumption in a throwaway session, bring the learning home.

The harder the questions, the more context a grill accumulates — and the more valuable it becomes to protect it. /handoff is how you keep going hard without drowning the session.

The deeper the grill, the more there is to lose by derailing it. A handoff lets you chase a tangent and come back to a session that’s still sharp.

Skills worth adding to the stack

/tdd — a red-green-refactor loop that builds one vertical slice at a time: failing test, minimal implementation, refactor. If your grill and PRD were thorough, the acceptance criteria already exist — this turns them into tests Claude has to satisfy before it can claim something works.

/zoom-out — when you land in an unfamiliar stretch of code and can’t see how it fits the bigger picture, this pulls the agent up a level: it explains the section’s role, its dependencies, and how it connects to the rest of the system. Useful when you inherit a module, come back to old code, or need the lay of the land before changing anything.

/grill-with-docs — a /grill-me variant that tests your plan against the existing domain model and codebase. The right choice when you’re extending a live system rather than starting fresh; it ties new decisions to what already exists and updates context docs inline.

Beyond the grill: getting Claude Code to run lean

The grill is the star of the workflow, but it only pays off if the rest of your setup isn’t quietly bleeding tokens and context. These are the habits that keep Claude Code fast, cheap, and sharp around it.

Run grill sessions on the strongest model; implement on a cheaper one.

A grill is pure exploration — no file writes, no tool-call overhead — so a frontier model like Opus earns its cost here by asking sharper questions and catching contradictions a smaller model misses, and the token bill stays low because it’s just conversation. Once you cross into implementation — /afk, /tdd, execution — a model like Sonnet does the heavy lifting at a fraction of the price.

Spend your best model where decisions are made, not where code is typed.

.claudeignore cuts your context bill in one edit.

By default Claude pulls everything in your project into scope. A .claudeignore file (same syntax as .gitignore) stops the dead weight from auto-loading — excluding .next/ alone can trim 30–40% off context in a Next.js repo. It’s the highest-leverage two minutes you’ll spend.

node_modules/
dist/
build/
.next/
__pycache__/
*.lock
.git/
*.db

This doesn’t stop Claude from reading these files when you explicitly ask — it just keeps them out of automatic exploration.

Keep files under 200–300 lines.

Agents read whole files. When one reads a 1,500-line file to find a 20-line function, it pays for 1,480 lines of noise — and reasons worse for it. This isn’t a prompt trick; it’s a codebase discipline that compounds across every read in a session.

A large file is a tax every agent pays, every time it looks.

Teach Claude to grep, not read — in CLAUDE.md.

CLAUDE.md is the one file Claude reads at the start of every session, so it’s where your project’s standing rules live — stack, conventions, and the things you never want to re-explain. The highest-value rule you can put there: tell it which files to search instead of read. We have a translation-keys file with over 2,000 entries; reading it whole would torch the context budget on noise. So CLAUDE.md says:

grep in `packages/survey-type-defs/src/appTranslation/AppTranslationKeyEnum.ts`
— 2000+ keys, never read the full file

One line, and Claude stops pulling a giant file into context every time it needs a single key. Do this for every oversized-but-unavoidable file in your repo.

Disable MCP servers you aren’t using.

Historically, Claude Code would load MCP tool definitions directly into context, and a few heavy servers (Playwright, GitHub, Gmail, etc.) could quietly consume thousands of tokens each — and a sprawling setup, tens of thousands per turn. Recent Claude Code versions mitigate this with Tool Search, now enabled by default: tools are discovered and loaded on demand instead of shipping every schema upfront. It helps, but large MCP setups still add overhead and complexity. Run /context to inspect what each server is costing you, and prune unused servers with /mcp.

Compact early — around half-full, not when it breaks.

Quality degrades as the window fills; by the time responses feel hazy, you’re already deep in the rot zone. The practical sweet spot is compacting around 50–60% capacity, while Claude still has full, uncompressed context to summarize from. Waiting until 90% means you’re summarizing an already-degraded view.

Git worktrees for parallel agents.

Independent features don’t need to run one after another. Worktrees give each agent its own branch and isolated directory off the same repo, so three agents can do in an hour what would take three sequentially — no context collisions, no merge chaos.

Thinking is the new bottleneck

The shape of building software has flipped.

Writing code used to be the bottleneck — now it’s the part agents are best at. What’s left for us is deciding what’s actually worth building and being precise enough that the implementation doesn’t drift.

That’s why the highest-leverage skill in this workflow is /grill-me — a skill that generates no code at all, yet creates more leverage than anything else.

The leverage has moved upstream. Specificity, edge-case honesty, and tight context management matter more than raw output.

Everything else — lean context, smaller files, the right model for the right phase — is just there to protect that thinking from noise.

The grill comes first. The code comes second.

The original application earned its battle scars over time. Every quirky conditional, every oddly specific validation, every silent fallback — they exist because something went wrong once, someone fixed it, and the fix outlived the memory of why it was needed. When you rewrite, you ship clean code into a world that remembers none of that.

That’s not a failure of planning. It’s the nature of the work.

The Edge Case Problem

“You can’t engineer that risk away.”

Legacy systems accumulate institutional memory in the codebase itself. A rewrite starts from specs, from documentation, from what people think the system does — not from what it actually does in the 0.1% of cases that never made it into a ticket.

Those cases exist. They will surface. Usually at the worst time, usually for the most demanding customer.

The teams that pretend otherwise — that spend energy projecting confidence in a flawless launch — are the same teams that get caught flat-footed when the inevitable happens. No incident response playbook. No communication template. No clear owner. Just scrambling.

What Actually Makes the Difference

A bug report after a rewrite is not evidence that the rewrite was a mistake. It’s evidence that the system is being used — by real users, in real environments, doing things no test suite anticipated.

What matters is what happens next.

The teams that come out of these situations stronger share a few things:

1. They communicate fast, even when the answer is incomplete. “We’re aware, we’re on it, here’s what we know so far” is infinitely better than silence. Silence reads as ignorance or indifference. Neither is a good look.

2. They show genuine urgency — not performance. There’s a difference between a team that looks busy and a team that’s actually moving. Customers and support teams can tell. So can your engineering manager.

3. They support the people closest to the customer. Your support team is absorbing the heat directly. If they’re left without context, without a timeline, without someone to escalate to — you’ve made their job impossible. That erodes internal trust as much as it erodes customer trust.

4. They own it without hedging. Not “there may have been an issue with the legacy data migration path.” Just: “We got this wrong. Here’s what happened, here’s the fix, here’s what we’re doing so it doesn’t happen again.”

Trust Is the Long Game

A perfect release is a nice thing. It’s also rare, fragile, and mostly luck-dependent once you’re past a certain scale of complexity.

Trust isn’t built in the good moments. It’s stress-tested in the bad ones.

When your stakeholders, your customers, and your team watch how you respond to a post-release bug — they’re forming a judgment about you that will outlast that bug by years. Did you hide? Did you deflect? Did you communicate clearly and move fast?

The engineering teams worth working with, and the engineering leaders worth following, are the ones who’ve internalized this:

The bug is a moment. Your response to it is a reputation.

A Note on Rewrites Specifically

Rewrites deserve some defense here. They are often the right call — technically and organizationally. Legacy systems accrue complexity that eventually becomes load-bearing in ways that block every other improvement. Rewrites unlock velocity, maintainability, and the ability to hire engineers who don’t need six months of tribal knowledge before they’re productive.

The answer isn’t to avoid rewrites out of fear of post-launch bugs. The answer is to:

• Build a robust incident response process before you ship
• Prepare your customer-facing teams with context and escalation paths
• Communicate proactively with high-risk customer segments at launch
• Treat the first 30 days post-launch as a separate, high-attention phase

And when bugs happen — because they will — own them completely.

That ownership is what builds the trust that makes your next rewrite easier to get approved, easier to staff, and easier to land.

For about six weeks one of our services was sending every user’s entire interaction history into the prompt with no cap on size. For most users that was fine. For our power users it was three to five minutes per call, our upstream LLM gateway timing out, jobs piling up in the queue, and demos blowing up in real time.

We had a demo on Tuesday. On Monday we shipped this:

.slice(0, 10_000)

Latency went from minutes to two seconds. We made the demo. Six days later we replaced that one line with what should have been there all along: per-user semantic retrieval on Postgres and pgvector.

This post is about both halves of that story. Why the hotfix worked and was still wrong, and what the real solution looks like.

The shape of the problem

Briefly, so the details below make sense: we run an internal service that generates survey responses on behalf of users, for cases where we need realistic-looking respondents to seed a test, validate a new flow, or reproduce an edge case. To make a generated answer sound like a specific person and not a generic one, the service feeds the LLM that user’s prior question-and-answer history as context, then asks the model to fill the current survey page in their voice.

That detail aside, the lesson in this post is general. Any system that pipes a user’s full history into an LLM prompt has the same shape. The history is unbounded, the model is paid per token, and the data path is silent about what part of the history actually matters for the question being asked.

Worth naming what we were doing, because the rest of the post turns on it. We were doing context stuffing, not retrieval. Every Q&A pair the user had ever answered, joined together, sent to the model in one shot. The irony is not lost on us: the retrieval in “retrieval-augmented generation” is exactly what we did not have. We had stuffing. The rest of this post is the story of how we built the retrieval part, six weeks later, after the stuffing broke.

For a user with a small history (a handful of past surveys), the blob is a few KB and the model takes a second or two to come back. That was the steady state for most users. It was the state we shipped at, and for a while, it held.

How we ended up sending megabyte-sized prompts

The original implementation took the path of least resistance: dump everything the system knew about a user into the prompt and let the model figure out what to do with it. Every past Q&A pair, joined together with newlines, in whatever order the rows had landed in the database. No cap on size, no relevance filtering, nothing fancy.

For about six weeks it worked. Then we onboarded a long-tenured user group, with people who had answered hundreds of surveys each. Their context blobs blew past a megabyte. The largest one in production was 2.66 million characters.

You can probably guess what the LLM gateway did with that.

LLM calls on those users started taking three to five minutes. Some hit the upstream timeout. The background job queue feeding the LLM started backing up. And worst of all, we had product demos lined up where the customer was watching the queue drain in real time and asking why.

There was no real mystery about what was wrong. One log line was enough to confirm it:

this.logger.debug(`prompt length: ${prompt.length}`);
// prompt length: 2660142

2.66 million characters into a chat completion endpoint will take minutes, in any model, in any region. The model has to read it.

The cheapest debugging step for a slow third-party call is logging the size of what you are sending. It is also the one most people skip. Payload size belongs in the same log line as the URL and the response time, in every HTTP client wrapper you write. You will get it back, in saved time, the first time something goes wrong.

The hotfix

We had less than 24 hours before the demo. The hotfix went in around lunchtime:

const MAX_MEMBER_CONTEXT_CHARS = 10_000

private truncateContext(memberContext: string, surveyId: number): string {
  if (memberContext.length <= MAX_MEMBER_CONTEXT_CHARS) {
    return memberContext
  }
  this.logger.warn(
    `[CONTEXT-TRUNCATE] surveyId=${surveyId}: memberContext ${memberContext.length} chars truncated to ${MAX_MEMBER_CONTEXT_CHARS}`,
  )
  return memberContext.slice(0, MAX_MEMBER_CONTEXT_CHARS)
}

A slice, a warning log so we could measure how often it tripped, and a MAX_MEMBER_CONTEXT_CHARS constant so the magic number had a name. End-to-end LLM latency went from three to five minutes back down to roughly two seconds. The demo went out the door.

Nobody on the team thought this was the answer. The PR description said in writing that this was a stopgap to keep our LLM path responsive, and that a proper retrieval implementation was the next ticket. But it bought us the breathing room to build that properly instead of in a panic.

This is also where most teams stop. The fire is out, the metric is green, the velocity board moves. The thing that made us not stop is that we already knew what was wrong with the hotfix, because it was obvious the moment we wrote it.

Why slice(0, N) is not retrieval

The blob we were truncating was stitched together in insertion order: the order Q&A pairs happened to land in the database, which is roughly the chronological order in which the user took the surveys. For our heaviest user, the first 10 KB was answers from surveys taken years ago. The page we were generating a response for was a recent satisfaction tracker. The Q&A pairs that would have actually helped the model, that user’s recent answers on the same product surface, sat at character offset 1.4 million. The hotfix was clipping them off and feeding the model old, mostly unrelated history.

To be honest about what we know and don’t know here: we did not measure a quality regression in production. We do not have an automated eval on generated-answer quality yet. What we have is the structural argument, which is enough to take seriously. If the data you are clipping has no order with respect to the question being asked, the first N bytes of it are statistically no different from a random sample of N bytes. The model is not failing in that case. It is succeeding loudly on the wrong input. That is a worse failure mode than slow latency, because there is no graph that lights up when it happens.

The dangerous column is the right one. The hotfix turned a problem we could see into a problem we couldn’t. That is a fine trade for two days while you ship a demo. It is not a fine steady state.

Truncation is not a solution, it is an incident in disguise. It silences a metric you watch (latency) by creating a regression in something you don’t (answer quality). If the data you are clipping has no order relative to the query, slicing the first N bytes of it is no different statistically from picking N bytes at random. The fix is not a bigger N.

The right question

Under pressure on Monday we asked the small question: how do we make this prompt smaller, today, in time for the demo? Truncation answered it.

The bigger question was the one we walked into the design doc with: which part of this user’s history is actually relevant to the page we are filling right now? The honest answer to that is not “the first N bytes.” It is “the chunks that semantically match the question being asked.” The first is truncation. The second is retrieval. They look similar from outside. They behave nothing alike.

But before we describe the retrieval design, the more important framing: retrieval is not the default path. Most users in our system have small context blobs, a few KB of past answers. Sending the whole thing to the LLM works fine. There is no reason to embed, store, query, and reconstruct context for a user whose entire history would fit in a tweet.

The honest design is: keep the legacy blob path for everyone whose context fits inside the model’s effective attention window. Only build the retrieval pipeline for the heavy users where that path breaks. The threshold we landed on is 200K tokens of user context, measured with tiktoken rather than length / 4 since token counts vary 2-5× by content. Below that, the model gets the whole blob. Above it, retrieval kicks in.

This matters because it changes what the system looks like. We are not building a RAG system. We are building a gated escape hatch for the long tail of users whose history is too big for the cheap path. The 90%+ of users who stay under the threshold never touch the new infrastructure. Storage cost, embedding API cost, retrieval latency, all of it scales with the small fraction of heavy users, not with the whole user base.

The shape of the retrieval path itself, for heavy users:

1. At sync time, if the user’s blob crosses 200K tokens, split their Q&A pairs into chunks of 10 (chronological order). Store one row per chunk in the vector table.
2. A background worker fills in embeddings for each new chunk asynchronously.
3. At fill time, embed the question on the current page, run a cosine similarity search over that user’s chunks, take top 5, send those into the prompt.

We will come back to why 10 pairs per chunk and not 1 or 100 in a moment. That’s the part the design discussion actually argued about.

Why pgvector, not a new vector database

We already run Postgres. Postgres has pgvector. Pulling in a new vector database (Pinecone, Weaviate, Qdrant, whatever the current favourite is) for a single feature is the kind of architectural decision that looks free on a slide and is anything but free six months later in oncall. So we built on what we already operate.

Embeddings come from OpenAI’s text-embedding-3-small at 1536 dimensions, which is what the vector(1536) column type below is sized for. The model is cheap, fast, and accurate enough for our use case. We are not married to it; the column dimension is the only place the choice leaks into the schema, and a future migration to a different model is a backfill, not a rewrite.

CREATE TABLE panel_member_answer_vectors (
  id SERIAL PRIMARY KEY,
  synced_panel_member_id INT NOT NULL
    REFERENCES synced_panel_members(id) ON DELETE CASCADE,
  chunk_text TEXT NOT NULL,      -- concatenation of 10 Q&A pairs
  pair_count INT NOT NULL,        -- how many pairs are in this chunk
  embedding vector(1536),
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  UNIQUE (synced_panel_member_id, chunk_text)
);
CREATE INDEX idx_pmav_member ON panel_member_answer_vectors (synced_panel_member_id);

Three things in that schema are deliberate and worth calling out.

No HNSW index, on purpose. HNSW is the canonical pgvector index and the default advice in any RAG blog post. We did not use it. The reason is the shape of our query, not the size of our data. We never search across all users, we only search within one user. A B-tree on synced_panel_member_id narrows the result set to that one heavy user’s chunks, and then a sequential cosine comparison over that bounded set runs in single-digit milliseconds. At our heaviest user today the set is tiny; at the ceiling we are sizing for, it tops out around 2,000 chunks per user. Either way the cosine pass stays in memory and stays fast. HNSW would add build time, write amplification, and a real tuning surface (pgvector tuning is a topic of its own), and at our query shape it would not be faster. We will add it the day telemetry says we need it. Not before.

UNIQUE on (synced_panel_member_id, chunk_text), with a side benefit. When we ran the migration to populate the new table, the constraint deduped a sync bug we didn’t know we had: our heaviest user had 78 duplicate Q&A pairs out of 127 stored rows. The blob format had hidden it the entire time, because duplicates inside one giant joined string just look like one long string of repeated text. The constraint did the cleanup as a side effect of being correct. The upstream sync bug that produced the duplicates is filed separately and is being fixed at the source; the constraint stays regardless, because defensive layers like this are the difference between a one-time data wash and a recurring oncall page. A reminder that schema constraints are not just gates against bad input. They are also flashlights, pointed at dirty data that is already there.

Nullable embedding column. Chunks are inserted synchronously as part of the user-sync transaction. Embeddings are filled in asynchronously by a background worker that batches calls to the embedding provider. This separates two concerns that have no business being in the same transaction: the durable record of what the user answered, and the derived index that lets us search it. If the embedding provider is down, sync still works. Retrieval falls through to the legacy blob path until the worker catches up. The system degrades, it does not break.

A flag, not a query: how the system knows which path to take

Here is the question the gated design forces you to answer: at fill time, how does the code know whether this user is a small-blob user or a heavy retrieval user?

The naive answer is “query the vector table and see if rows exist.” That works, and it is wrong. It pays a DB roundtrip for every single response generation, including the 90%+ of calls that will immediately turn around and use the blob anyway. Multiply that by every page of every survey for every user in a run, and the “small users are free” claim quietly stops being true.

The honest answer is to store the state where it is needed. We added one column to the existing user row:

is_chunked: boolean; // default false

The flag is set to true at sync time, the moment the user’s blob crosses the 200K-token threshold. The row was already being loaded for the response generation anyway, so the flag arrives for free. Retrieval becomes a single in-memory check:

if (!member.isChunked) return legacyBlob(member.context);
// only heavy members reach this line
return semanticRetrieval(member.id, queryEmbedding);

One deliberate semantic: the flag is sticky-true. Once a user becomes chunked, they stay chunked, even if a future resync drops them below 200K tokens. The reasoning is part sunk cost (the chunks are already vectorized, the embedding bill is already paid), part state consistency (no edge case where rows exist in the vector table but the flag is telling fill-time to use the blob). The trade-off is that a user cannot be un-chunked through the normal resync flow. Only deleting and re-creating the user row resets the state. We made peace with that.

The bigger principle: store state explicitly, don’t recompute it. A boolean that is set once at write time beats a DB query on every read. Reading code that says if (!member.isChunked) tells the next engineer exactly what is going on at this branch in the design. Reading code that says if ((await countVectorRows(memberId)) === 0) tells them nothing about why the system is asking that question, and quietly costs an extra network roundtrip on every fill. The boolean is not faster by accident; it is faster because it is the right shape for the decision being made.

The grouping decision: per-pair, per-survey, or per-N

This was the longest discussion in the design. Three options were on the table, and the interesting part of the design is which one we picked and why the obvious choices were wrong.

Per-pair (one Q&A = one row)

The first instinct, and the one most RAG tutorials show you. Embed each Q&A pair on its own, retrieve top-K pairs.

It dies on the row count math at scale. We are not a 1000-user app. A user group can have 90K users, and a heavy user can have 20K answers across years of activity. Per-pair grouping puts that at 1.8 billion rows in a single Postgres table, with around 11 TB of vector data. Postgres can technically store that. We do not want to operate it. Vacuum behaviour, replication lag, backup windows, index rebuild times: every one of those gets unhealthy fast at that row count, and none of them is the failure mode you want for a feature that is supposed to be an “escape hatch.”

Per-pair also pays embedding cost per pair. At a million pairs across a user base, that is a million embedding API calls at sync time. The cost is real money even at OpenAI’s small-model pricing, and the throughput is bounded by the embedding API’s rate limit. None of this is unsolvable, but it is all overhead being paid in exchange for a granularity the retrieval doesn’t need.

Per-survey (one survey’s worth of answers = one row)

The next-most-obvious idea, and intuitively appealing. A survey is a natural topic boundary. NPS surveys are about loyalty, demographic surveys are about who the person is, product surveys are about features. Grouping by survey preserves topic. No magic number to tune.

It dies on two real problems:

1. Similarity score dilution. A survey covers many sub-topics. A satisfaction tracker might have a Net Promoter question, a product feedback question, a competitor comparison, and a demographic question, all in one survey. Embedding all of those into one vector averages the topic signal out. When the page asks something specific (“how satisfied with response time”), the per-survey chunk’s embedding is generic and doesn’t match strongly. The cosine search produces lower scores across the board, which is exactly the failure mode you don’t want from a retrieval system.

2. Survey size is unbounded. A QuestionPro survey can have 500 questions. Some have 5000. A 500-question survey at ~150 chars per answer is 75K characters in one row, which is past the 8192-token input limit of most embedding models. We’d have to chunk anyway. So we’re not actually grouping by survey, we’re chunking inside surveys and pretending the boundary matters. It doesn’t.

Per-N pairs (we landed on N=10)

The middle ground. Take a user’s pairs in chronological order and group them in fixed batches of 10. One row per batch. ~1500 characters per chunk on average. Bounded chunk size, bounded row count, embedding input always inside the model’s limit.

The trade-off is real: a chunk of 10 chronologically-adjacent pairs may contain mixed topics. Mixing two or three topics in 1500 characters still produces a meaningful embedding (the topics are at least adjacent in the user’s timeline), but it’s not as clean as one topic per chunk would be. We chose this anyway. The reason: the alternatives are worse. Per-pair scales catastrophically. Per-survey loses signal for surveys that mix topics, which is most of them, and breaks on size for the large ones.

10 is not magic. It is the smallest number where the embedding has enough text to carry topic signal and where the row count stays in single-digit millions instead of single-digit billions at scale. It is a tuning parameter, and we will revisit it. To revisit it well, we need something we do not have yet: an automated eval on generated-answer quality. The next thing this work unlocks is exactly that. Once retrieval is doing something other than “use the blob,” we finally have two variants to compare, and a reason to build the eval. That is its own ticket, and probably its own post.

The other rejections (briefer, because they’re the standard ones)

BM25 / Postgres full-text search. Free and tempting, but plainto_tsquery is AND-default and stems aggressively. A page asking “how satisfied with our services” tokenizes to 'satisfi' & 'servic', returns zero matches against a user who had answered five surveys about “AI advancement satisfaction.” Survey questions paraphrase across surveys (“how likely to recommend” → “would you suggest us to a friend”). Handling paraphrase is the reason to do semantic retrieval, and BM25 cannot.

JSONB array column on the existing users table. Tempting because it avoids a new migration. But pgvector’s <=> cosine operator works at the row level, not the array-element level. An array column forces UNNEST per query → no usable index → sequential scan of the entire users table on every retrieval.

Reuse the existing question_vectors table with a discriminator column. Polymorphic schema with nullable foreign keys and branching logic in every query. “One fewer table” is a benefit that lasts a sprint. Polymorphism is a maintenance cost that compounds for years.

HNSW from day one. Right index, wrong workload. Per-user B-tree filter + in-memory cosine over a few thousand vectors beats HNSW for our query shape until row count per user grows another order of magnitude. We’ll add it when telemetry says we need it.

Vector retrieval has a stack of canonical defaults: HNSW indexes, per-pair embeddings, recursive chunking, hybrid keyword-plus-semantic scoring. Each is correct for some scale and some query shape. None is correct for all of them. The canonical architecture is canonical for someone else’s problem. Measure your data, and run the row count math at the scale you actually operate at, before you copy it.

K=5, not K=10 or K=20

Once retrieval is working, the next instinct is to be generous with K. More context is more information, right?

Wrong, and it took some reading to convince me of it. The current production-RAG research (Lushbinary’s 2026 guide summarizes it well) keeps finding the same thing: LLM accuracy degrades as input grows. The model spreads its attention across whatever you give it. If five of your ten retrieved passages are noise, the five good ones do worse than they would alone, because the model is now also reasoning about the noise. Smaller K with higher-precision results beats larger K with looser thresholds.

That was the second time on this project we ran into the same fact in a different costume. Truncation hid the relevant data behind a length cap. Over-large K drowned it in distractors. The lesson is the same lesson: less input, more accurate output, as long as the input you keep is the right input.

What the new path costs

Because the retrieval pipeline only runs for heavy users, the cost numbers are nothing like “build RAG for the whole user base.” From the data we have in production today, roughly 5-10% of users cross the 200K-token threshold. That estimate will sharpen as we roll out and see real traffic distribution, but the order of magnitude is right.

The sizing below is for the ceiling we are designing for, not the heaviest user in production today. Today’s heaviest is small (a few dozen distinct Q&A pairs, after the UNIQUE constraint deduped them). The numbers that follow describe what happens when a fully active group of long-tenured users shows up. That is the case the design has to survive.

• Eligible users: for a 90K-user group, somewhere between 5K and 10K cross the threshold. The rest stay on the legacy blob path with zero infrastructure changes: no row in the vector table, no embedding call, no extra latency.
• Storage: at the ceiling we are sizing for, a heavy user with ~20K answers produces ~2K chunks at 10 pairs each. At 1536-dim float32, that is ~12 MB per heavy user, ~60-120 GB total for the heavy tail of a large user base. Postgres handles that easily on standard storage.
• Embedding API cost: one async call per new chunk at sync time, batched. At fill time, the query embedding is cached per page, so the same page being filled for N heavy users costs one embedding call total, not N.
• Retrieval latency: single-digit milliseconds per query. The B-tree filter narrows to one user’s chunks; the cosine pass runs over a few thousand vectors in memory.
• End-to-end LLM latency on heavy users: two to three seconds. Same as the truncation hotfix was giving us, except the model is now being handed the right context instead of an arbitrary prefix of the wrong context.
• End-to-end LLM latency on small users: unchanged. They never enter the retrieval path.

The legacy blob is the fallback for two different reasons. For small users it is the default, because it is cheaper, simpler, and has no embedding-worker dependency. For heavy users it is the safety net: if retrieval returns zero rows, or the embedding worker is behind, the call falls through to the old behaviour rather than failing. Worst case is what we shipped before the hotfix, which is a worst case we have already lived through. There is no regression path we can introduce by rolling this out. Only an upside path we can lose if rollout exposes something we missed.

Three things worth carrying out of this

If the rest is too long to remember, these are the three.

Log the payload size on every external call. Especially LLM calls. Especially the ones where the payload is “everything we know about this user.” The cost of always having the number is zero. The cost of not having it the first time something goes slow is hours.

Treat slice(0, N) on an LLM prompt as a hotfix, not a solution. It is a perfectly reasonable thing to ship on a Monday afternoon to keep a demo alive. It is not a thing to leave in place for a quarter. If the data you are cutting has no order relative to the question being asked, the first N bytes are statistically no better than a random N bytes, and the model will use them anyway, quietly, without telling you. The fix is not a bigger N, the fix is a smaller, more relevant prompt.

Measure your own data before copying a reference architecture. HNSW indexes, recursive chunking, hybrid keyword-plus-semantic scoring, large-K retrieval. Each of these is a correct default for somebody’s workload, but not automatically for yours. Our per-user B-tree filter followed by an in-memory cosine pass over a handful of vectors looks “wrong” against a textbook RAG diagram, and is the right answer for our query shape. Yours might be different. Read your data first, then pick the architecture.

Where this is going

The retrieval pipeline is rolling out behind a feature flag. The legacy truncation helper is still there underneath as a safety net. Once we have a few weeks of production data showing the fallback rate stays below our threshold, the blob column and the truncation helper both come out together.

The other consumer of the same context blob (a conversational AI surface elsewhere in our stack) currently has no truncation at all, and quietly exceeds the model’s token limit on long-tail users. Same retrieval call fixes it. The API is already built, it just needs to be wired in.

If your system pipes user history into an LLM prompt and your only defence against runaway prompt size is a length cap, you are probably shipping the same bug we were. The fix is not a bigger cap. The fix is a smaller, more relevant prompt.

🌱 What Is the Strangler Fig Pattern?

The name comes from the strangler fig tree — a plant that germinates in the canopy of a host tree and slowly grows downward, wrapping around the trunk until it becomes a self-supporting structure. The host tree doesn’t get cut down. It simply becomes less and less essential as the fig takes over, until one day it’s hollow and the fig is the only thing left standing.

Applied to software, the pattern works like this: you don’t rewrite the legacy system — you build alongside it. New features go into the new system. Old modules get migrated one at a time, with a routing layer directing traffic to whichever implementation is live for each surface. The legacy codebase shrinks with every migration. The new system grows. At no point is the old system taken offline mid-flight, and at no point does the business have to wait for a complete rewrite to ship. The risk is distributed across months of incremental work instead of concentrated in a single cutover event.

🔨 The “Whack-a-Mole” Reality

Before this migration, our legacy CX system was a massive Java Struts monolith with over 1,000 files and a hand-rolled JDBC layer. It lacked an ORM — every query was hand-mapped SQL with no type safety and no enforced schema ownership.

We lived in a “Whack-a-Mole” culture. — fix the reminder logic in Send Flow, watch the Analytics counts drift. Tighten a validation rule in Admin Setup, and a Deploy Survey action you’d never heard of starts throwing NullPointerException in production because it had silently depended on the looser behaviour for years. We weren’t slow because we were careless. We were slow because the system punished speed.

⚡ The Catalyst: Centralised Root Cause Module

Sometimes back, we hit a turning point. We needed to build a new feature: Centralized Root Cause (CRC) — a complex churn-risk analyser. CRC wasn’t uniquely complex — we’d built harder things in the monolith before. The feature was a natural seam: a new surface, no legacy entanglement, a clean API boundary. We used it as the forcing function to stop feeding the monolith and start replacing it. Adding them to the monolith was a recipe for disaster and more cleanups later.

Instead, we chose to break this pattern and launched a new repository - Version2 — with a new tech stack and a strategic architectural hard stop:

The Nav-Bar Mandate If a new feature requires a navigation bar, it must live in the new tech stack (NestJS + React) in the Version2 repo. No new nav bars are to be added in legacy code. This single rule became our architectural enforcement mechanism.

🏗️ Phase 1 — Wrapping the Legacy

We didn’t pivot to a complex microservices mesh. Instead, we adopted an App-Service architecture — isolating specific business domains into focused, manageable services that communicate clearly while sharing necessary infrastructure.

The Strangler Fig works by wrapping the old system. We used our Navigation Bar as the routing layer. To the user, the experience is seamless. Behind the scenes, we route traffic between V1 (the legacy monolith) and V2 (Version2). When a module has migrated, the corresponding Struts action redirects the request to Version2’s API. Version2 then handles authentication and authorisation independently — validating the session, verifying permissions, and serving the response.

The Three Phases

Phase 1 — V1 only The starting state. Every client request goes directly to the legacy system — no routing layer, no abstraction. Admin Setup, Send Flow, and Analytics all live inside the same deployable, sharing the same JDBC layer and the same database. There is no seam to exploit. The monolith is the entire system.

Phase 2 — Transition This is where the Strangler Fig actually begins. A Nav Bar built on the Struts facade is introduced as the routing layer — the single seam between legacy and new. The client never talks to V1 or V2 directly; it talks to the nav bar, which decides where to send the request. V1 continues running unchanged — Admin Setup, Send Flow, and Analytics still live in the legacy WAR and handle their existing traffic. Meanwhile, V2 comes online alongside it, initially serving only the new modules: Root Cause, Customer 360. The two systems coexist in production. This is intentional, not temporary — Phase 2 is the stable operating state for the bulk of the migration.

Phase 3 — V2 Target The end state isn’t fully realised yet — this is where we’re headed. The nav bar thins from a Struts facade into a lightweight proxy, its only job being to route requests to the right Version2 module. V1 is still running, but its surface area keeps shrinking. The work currently in flight: moving Deploy and Dashboard out of Analytics in the legacy WAR and into their own modules in Version2. Once those land, the module hierarchy in V2 reflects how the product actually works — Root Cause, Dashboard and Deploy as independent modules. The monolith doesn’t disappear in a single moment. It just runs out of things to do.

⚙️ The Technical Standard

Moving to a new stack wasn’t just a language change — it was a shift to a modern service standard that made the failure modes of the old system structurally impossible.

The 4-Layer Data Flow

No module may access another module’s Repository directly. All cross-module calls go through Services (Facade Pattern).

What this means in practice

• 4-Layer Data Flow: Controller → Service → Repository → Entity. Controllers handle HTTP only. All business logic lives in Services. All DB queries use parameterized TypeORM QueryBuilder — no raw SQL, no hand-mapped JDBC.
• Facade Pattern: No module may access another module’s repository directly. Cross-module interaction goes exclusively through the exposed Service. This is the structural fix for the Whack-a-Mole problem.
• DTO Validation: All inbound payloads validated using class-validator at every entry point. Magic string bugs and unvalidated inputs never reach business logic.

📊 A Tale of Two Eras

🚀 Impact: Speed, Stability, and Sanity

By adopting this pattern, we didn’t just clean up our code — we unlocked the business.

⚡ Velocity From monthly “Big Bang” releases to fast, continuous builds. Each service ships on its own schedule.

🛡️ Reliability With 100% E2E coverage using Testcontainers, we no longer “deploy and pray.” Real MySQL + Redis in every test run.

📊 Customer360 Built a high-performance churn analysis tool in record time — proving the new architecture handles complex data aggregation far faster than the monolith ever could.

🔭 What We’ve Shipped — and What’s Next

✅ Already shipped in V2

• Centralized Root Cause (CRC) — greenfield module, zero legacy dependency. New tables, new NestJS module, new React frontend. V1 untouched.
• Customer360 — AI-powered churn risk analysis across all workspace surveys. Aggregated NPS scoring, top 10 root causes, per-customer severity and action items. Shares some V1 data inputs during transition, fully owned by V2.

🔄 Currently in progress

• Deploy (Send flow) — moving from a fragile, rigid V1 design to a reliable, independent V2 service. Deep dependency on shared campaign infrastructure being wrapped behind clean interfaces.
• Analytics Dashboard — swapping old UX and slow queries for a modern React frontend and optimised NestJS backend to deliver insights in milliseconds.

🗺️ Strategic Comparison

💡 The Lesson

The most important thing we learned is that you don’t fix a legacy system by refactoring it forever. You fix it by building a better future right next to it. This approach has moved us from a system that limits our potential to one that empowers it.

One nav tab at a time. That’s how you strangle a monolith without breaking what’s already working.

Prologue — Why “We’ll Add It Later” Is a Trap

Every system starts with the same quiet assumption: we’ll add limits later.

And at the start, that’s a fair call. Data is small. Users are few. Edge cases are theoretical. Adding limits up front feels like premature optimization — or worse, a bad user experience you’re imposing on people who haven’t done anything wrong yet.

So you skip them.

Then, months or years in, the system starts answering back. Not with errors. Not with crashes. With something far more dangerous: it gets slow. Inconsistent. Unpredictable. Users file vague tickets like “the app feels weird today” and your dashboards look almost fine.

That’s the moment you realize limits were never optional. They were just deferred — and deferred limits always come due with interest.

This is the story of the incidents that reshaped how we think about limits — and why stress testing is the cheapest way to find them before your users do.

🚨 Incident 01 — The Save That Stopped the World

The first incident wasn’t a crash. It was a plateau. CPU had climbed, stuck, and stayed there. Latency tails were drifting up across the platform — every user a little slower, with no single request to blame. The graphs were red, but they weren’t telling us anything specific.

After enough time in logs and traces, the pattern emerged. This wasn’t one rogue survey — it was a whole category of them. Surveys with large conjoint questions, where editing features and levels meant bulk-writing hundreds of related entities at once. Every one of those edits funneled into the same hot path.

A quick primer: a conjoint question is built from features (like price or color) and levels (specific values per feature). The math is combinatorial — 10 features × 20 levels is 200 entities for a single question, and surveys often have several.

Every bulk edit touched all of them in one transaction. That matched the blow-up pattern exactly — and it was baked into the feature itself, not a misuse of it.

The entire hot path came down to one call:

await levelRepository.saveAll(levels);

It looks innocent. It wasn’t.

repository.save() in TypeORM is an upsert. For every entity in the array, it checks whether the row exists, compares it field-by-field against the database version, and deep-merges nested relations before persisting. On small arrays, it’s fine. On arrays of a few thousand related entities, the comparison logic degrades into O(n²) — and all of it runs as synchronous JavaScript on the main thread.

While that code ran, nothing else on the Node process could make progress.

_{FIG. 01 · Event loop blocked by synchronous CPU work on a single request.}

We fixed the immediate problem quickly — an optimized bulk-write strategy that skips the per-entity comparison — and latency recovered within hours. We’re also moving CPU-heavy work to worker threads, so no future slow operation can starve the event loop. But the fix wasn’t the lesson.

The lesson was everything that had to be true for this to happen.

1. No structural limits. A conjoint question could grow to hundreds of features and levels, and a survey could contain several — with nothing to stop it. A cap on features and levels per question would have ended the story right there. One user’s edit could never have become everyone else’s problem.

2. No stress testing. Even with limits in place, we wouldn’t have known whether they were safe — because the write path had never been tested at scale. The ORM’s O(n²) cliff was invisible at small data sizes, which is exactly the kind of thing stress testing exists to find.

Limits decide what the system allows. Stress testing proves the system can handle what it allows. With neither, production ended up being the first real test — the hard way to find out.

Without limits, one user can degrade the system for everyone. Without stress testing, you won’t know until they do.

🚨 Incident 02 — The Limit We Didn’t Set (So Browser Did)

A customer opened a survey with 800+ questions in a single block. Many of those questions had <audio> tags embedded in their content, and when the survey loaded, the browser dutifully started creating audio players for every one of them.

It didn’t finish. Somewhere past the first few dozen, Chrome stopped cooperating. The page froze. The console filled with errors:

_{FIG. 02 · Chrome's WebMediaPlayer limit hit silently when rendering too many audio elements.}

Chrome has a hard cap on the number of simultaneous WebMediaPlayer instances per page. It’s baked into the browser — you can’t raise it, you can’t bypass it. Exceed it, and every subsequent media element silently fails to initialize.

The fix was virtualization — render only what’s in the viewport, defer the rest. But virtualization wasn’t the lesson. It was the workaround.

The lesson was that Chrome already had a limit. We just hadn’t matched it. We allowed effectively unbounded questions per block, each free to embed any number of media tags. Chrome allowed a few dozen concurrent players. One of those numbers was fixed; the other wasn’t. They were always going to meet — and the browser was always going to win.

A product-level cap, set well below Chrome’s, would have kept the collision from ever happening. Users would see a clear message from us, instead of a silent failure from the browser. We didn’t have that cap. So Chrome drew the line for us.

Every layer of your stack has limits. The only question is whether you set yours, or discover someone else’s.

🚨 Incident 03 — The Limit on What’s Awake

The first two incidents failed loudly. One froze a backend thread, the other froze a browser tab. The third one didn’t fail at all. It just stopped being usable.

A customer with a 200-question survey opened the editor, expanded a few blocks to work across them, and the UI began to crawl. Clicks took a beat. Scrolling stuttered. Typing lagged behind the keystrokes.

We assumed it was the question count. It wasn’t — not on its own. A block with 200 questions rendered fine if nothing else was expanded. A block with 50 questions rendered fine if three others were also open. The slowness only showed up when enough of the UI was active at the same time.

We hadn’t put a limit on that. Any number of blocks could be expanded, each rendering all of its questions, inputs, and validations. A single user could, with a few clicks, ask the browser to keep hundreds of stateful React subtrees alive and reactive at once.

_{FIG. 03 · Same survey, different activation patterns. Cost scales with what's awake, not with what exists.}

The fix was to bound the active state. Above a threshold, only one block stays expanded at a time; opening another collapses the previous. The UI recovered immediately — not because rendering got faster, but because we stopped asking the browser to keep the whole survey hot in memory.

Cost scales with what’s awake, not with what exists. If you don’t cap what’s active, the browser will — by slowing down until users stop.

💡 Limits You Don’t Have to Enforce

After three incidents, we were convinced: limits needed to be first-class.

The most obvious limit to add was a cap on the total number of answer entities a survey can hold. The enforcement seemed straightforward — on every update, count the answers, reject if over the cap.

Before we shipped it, we did the thing we hadn’t done in any of the previous incidents. We stress-tested it.

It caught the problem immediately.

Counting total answers meant walking the entire survey tree on every update. The walk was cheap on small surveys. On large ones, it grew expensive — and the larger the survey, the more often it ran, because larger surveys see more edits.

A naive reading of “enforce the limit” would have placed a growing traversal directly in the hot path of every write. We were about to recreate Incident 01 on purpose.

_{FIG. 04 · Same limit, two ways to enforce it. One costly. One free.}

So we stepped back and asked a different question: do we need this check at all?

We already had limits at lower layers — a cap on answers per question, and a cap on questions per survey. Multiply those together and the total answer count is already bounded, without any runtime check. The survey-level cap we wanted to enforce was already enforced, for free, by the composition of the limits beneath it.

We removed the check from the design. The final set of limits stayed at the layers where they naturally belonged — per question, per row, per column — and the aggregate took care of itself.

This was the turning point for how we thought about limits. The previous incidents had taught us that limits needed to exist. This one taught us something subtler: where you enforce a limit matters more than the limit itself. A cap enforced at the wrong layer is just a new bottleneck. A cap guaranteed by composition is free.

The best limits are the ones the system already guarantees. If a cap can be derived from lower-level constraints, enforcing it again at runtime is overhead with no benefit.

🧠 A Better Mental Model for Limits

After three incidents and one near-miss, limits stopped being ad-hoc reactions and became a design vocabulary.

Two of these matter most. Composition saves you the most work — the best limits are the ones you don’t have to enforce. Enforcement is the easiest to get wrong — the right limit in the wrong place is still a bottleneck.

And underneath all of them: stress testing is how you find out whether your limits actually hold. Without it, production does the testing for you.

✅ Limits Are Guarantees

By the time we were done, we understood limits weren’t a constraint we had to add to the product. They were a set of guarantees we already needed:

• One user’s actions can’t slow down the experience for everyone else.
• The browser doesn’t silently enforce its limits on your users.
• The interface stays responsive no matter how much data it holds.
• Your safety checks don’t become the bottleneck they were meant to prevent.

That reframing changed how we designed features. “What’s the limit?” moved from one of the last questions in a design doc to one of the first.

If you don’t define limits deliberately, your system will discover them accidentally — in production, under load, at the worst possible time.

Introduction

Remember when you thought building an AI agent would be easy? “Just throw some prompts at GPT-4 and call it a day,”.

“What could go wrong?” you said.

Narrator: Everything went wrong.

Large Language Models have revolutionized intelligent applications, but here’s what nobody tells you at those fancy AI conferences: scaling an AI agent from “cool demo” to “production system that doesn’t bankrupt your company” is… challenging. Token costs spiral like a startup’s cloud bill, context windows overflow faster than your coffee cup on Monday morning, and response times make dial-up internet look speedy.

This is our journey (currently in progress, bugs included) from a basic implementation to building a production-ready, cost-efficient multi-agent system for QuestionPro’s BI platform. We’re exploring three key patterns, making mistakes in real-time, and documenting everything so you don’t have to. Think of this as “Mythbusters” but for AI architecture, with 100% more token optimization and slightly fewer explosions.

Part 1: The Basic Agent - Or “How Hard Could It Be?”

The Innocent Beginning

Like every developer who’s just discovered LangGraph, we started with the “obvious” approach, THE MONOLITHIC AGENT: one agent to rule them all, one agent to find them, one agent to bring them all, and in the darkness bind them (to a $10,000/month OpenAI bill… roughly).

// Single agent with all capabilities
const agent = new ChatOpenAI({ model: "gpt-4" }).bindTools([
  createDashboard,
  getDashboard,
  createWidget,
  listSurveys,
  getQuestions,
  // ... 40+ more tools
]);

const systemPrompt = `
You are a dashboard analytics assistant.

Available dashboards: [... 500 lines of context ...]
Survey schemas: [... 2000 lines of schemas ...]
Widget configurations: [... 1500 lines of configs ...]
`;

The “Oh No” Moment

You know that feeling when you check your AI bill and your heart skips a beat? That was us, thinking about going into “production”.

Our “Everything is Fine” Metrics:

📊 The Uncomfortable Truth:
- Avg tokens per request: 15,000 (narrator: this is not good)
- Cost per conversation: $1.50 (multiply by users... *sweats*)
- P95 latency: 8 seconds (users hate us)
- Projected monthly costs: $4,500 (CFO hates us more)
- Success rate: 89% (11% of the time, it works every time!)

🔴 The "We Need to Talk" Problems:
- Context overflow (GPT-4 politely hanging up on us)
- Tool selection paralysis (like Netflix, which movie to watch)
- Exponential cost growth
- LLM having an existential crisis (overwhelmed with 40+ tools)

The Exponential Growth Problem (Or: Math is Unforgiving)

Turn 1:  12,500 tokens  ✅ "This is fine"
Turn 5:  22,000 tokens  ⚠️ "This is less fine"
Turn 10: 37,000 tokens  ❌ "This is fire, everything is fire"

We were literally pricing ourselves out of business, one helpful conversation at a time.

The real kicker? We kept asking “Can we just add more features?” while totally unaware of the bill that might get generated.

Note: These numbers are rough estimates from our experiments and paper napkin calculations. We’re still actively developing and iterating, but gives an idea to think in the right direction.

Part 2: The Skills Pattern - Progressive Disclosure

The “Aha!” Moment

Or: How We Discovered That Laziness is Actually a Virtue

After three existential crises where we admitted our agent was hemorrhaging money, we stumbled upon a life-changing concept:

“What if… and hear me out… we DON’T load everything into the prompt like we’re packing for a 2-week vacation?”

Revolutionary, I know. We felt like we’d discovered fire, except fire was actually “reading the documentation properly.”

Enter progressive disclosure - a fancy term for “only fetch stuff when you actually need it,” which is basically how every efficient human operates but somehow felt groundbreaking when applied to AI. (Yes, we’re aware of the irony.)

“Don’t load everything upfront. Load only what you need, when you need it.”

Instead of including all context in every request, we implemented progressive disclosure - the agent loads specialized knowledge through the skills pattern.

Implementation

// Lightweight system prompt with skill metadata only
const systemPrompt = `
You are a dashboard analytics assistant.

Available Skills (load when needed):
- survey_schema: Get survey questions and field types
- widget_config: Get chart-specific settings  
- dashboard_filters: Get filter options
- styling_themes: Get appearance options

Current context:
- Workspace: 1
- Dashboard: ${state.dashboardId || "none"}
`;

// Skills load on-demand
const loadSkillTool = tool(async ({ skillName, resourceId }) => {
  switch (skillName) {
    case "survey_schema":
      return await fetchSurveySchema(resourceId);
    // Returns ~2,000 tokens only when needed

    case "widget_config":
      return await fetchWidgetConfig(resourceId);
    // Returns ~1,500 tokens only when needed
  }
});

Skills in Action

User: “Create a gender chart from Customer Survey”

Request 1: Agent sees lightweight prompt
├─ System: 1,000 tokens (was 8,000)
├─ Tools: 1,500 tokens
└─ Messages: 2,000 tokens
   Total: 4,500 tokens ✅

Agent decides: "I need the survey schema"
└─ Calls: load_skill("survey_schema", 12345)

Request 2: Agent receives survey details
├─ Previous context: 4,500 tokens
├─ Survey schema: 2,000 tokens
└─ Total: 6,500 tokens ✅

Agent creates chart with correct data mapping

The Results (Or: When Theory Meets Reality and Actually Works)

📊 Skills Pattern Metrics (We Were Shocked Too):

- Avg tokens: 6,500 (was 15,000) 🎉
- Cost per conversation: $0.65 (was $1.50) 💰
- P95 latency: 4s (was 8s) ⚡
- Monthly costs: $1,950 (was $4,500) 🎊

✅ 57% cost reduction (CFO can sleep better)
✅ 50% faster responses (users will hate us a little less now)
✅ Stable token usage (no more exponential nightmares)
✅ Team morale: Significantly improved

As this was my late night hustle when no one was around, I just high fived my wall.

### But Wait... (The Plot Thickens)

Just when we thought we'd solved AI, reality decided to humble us. Again.


❌ The "Not So Fast" Problems:

1. Agent handling 40+ tools → Like asking someone to pick
   their favorite child, but with 40 children
2. Mixed concerns → Debugging became "which of these
   40 things broke?"
3. No parallelization → Everything sequential because
   apparently we hate speed
4. Some skills returned 3,000+ tokens → The token diet
   didn't last long

The Comedy of Errors:

User: "Style my dashboard in dark mode."
(A simple, reasonable request.)

Agent's internal monologue:
"Hmm, I have 40 tools. Let me read each description...
create_dashboard? No...
update_dashboard? Maybe?
create_widget? Probably not...
update_theme? OH WAIT THAT'S THE ONE! ✅
...only took me 3 seconds to figure that out"

User: _has already rage-quit_

Part 3: Multi-Agent Architecture - Specialized Experts

The “Why Didn’t We Think of This Sooner?” Moment

The Breakthrough (Cue Dramatic Music)

Picture this: It’s 2 AM, you’re on your third coffee, scrolling through LangGraph documentation for the millionth time, when suddenly…

“What if… what if… we don’t make one agent do EVERYTHING? What if we have specialized agents? Like… real companies?”

🤯 Mind. Blown.

We felt like we’d invented the wheel, despite the fact that humans have been organizing into specialized roles since, like, the dawn of civilization. But hey, better late than never!

Our New Squad:

• Dashboard Agent: The organized one who actually reads the manual
• Widget Agent: The creative type, probably went to art school
• Datasource Agent: The data nerd (affectionately), speaks fluent SQL
• Styling Agent: Fashion police of the digital world

Basically, we went from having one overworked, stressed-out agent having a breakdown, to a healthy work environment with proper delegation. Revolutionary.

The Breakthrough

“What if instead of one agent doing everything, we had specialized experts?”

Like a company with departments (Sales, Engineering, Design), we created specialized agents:

• Dashboard Agent: Dashboards and tabs expert
• Widget Agent: Visualizations expert
• Datasource Agent: Data and surveys expert
• Styling Agent: Themes and appearances expert

Federal Router Implementation

// Lightweight orchestrator
const federalAgent = new ChatOpenAI({ model: "gpt-4" }).bindTools([
  routeToDashboardAgent,
  routeToWidgetAgent,
  routeToDatasourceAgent,
  routeToStylingAgent,
]);

const federalPrompt = `
You are a routing assistant. Delegate to specialists:

- Dashboard operations → Dashboard Agent
- Widget creation/editing → Widget Agent
- Data selection → Datasource Agent
- Styling/themes → Styling Agent

You route and synthesize - you don't execute tasks.
`;

Specialized Agent Example

// Dashboard Agent - Only 6 focused tools
const dashboardAgent = new ChatOpenAI({ model: "gpt-4" }).bindTools([
  createDashboard,
  getDashboard,
  updateDashboard,
  createTab,
  updateTab,
  deleteDashboard,
]);

Conversation Flow

User: “Create sales dashboard with gender chart from Customer Survey”

┌────────────────────────────────────────┐
│ 1. Federal Router                      │
│ Tools: 4 routing (400 tokens)          │
│ Decision: Dashboard → Data → Widget    │
│ Cost: 1,400 tokens                     │
└────────────────────────────────────────┘
                ↓
┌────────────────────────────────────────┐
│ 2. Dashboard Agent                     │
│ Tools: 6 dashboard (600 tokens)        │
│ Creates: "Sales" dashboard (ID 100)    │
│ Cost: 1,400 tokens                     │
└────────────────────────────────────────┘
                ↓
┌────────────────────────────────────────┐
│ 3. Datasource Agent                    │
│ Tools: 10 data (1,000 tokens)          │
│ Finds: Survey 12345, Question 67890    │
│ Cost: 2,200 tokens                     │
└────────────────────────────────────────┘
                ↓
┌────────────────────────────────────────┐
│ 4. Widget Agent                        │
│ Tools: 8 widget (800 tokens)           │
│ Creates: Pie chart widget              │
│ Cost: 2,800 tokens                     │
└────────────────────────────────────────┘

Total workflow: 8,200 tokens (vs 60,000)

The Results (Holy Sh— I Mean, Wow!)

📊 The Numbers Don't Lie (But We Triple-Checked Anyway):

- Avg tokens per workflow: 8,200 (was 60,000) 📉
- Cost per workflow: $0.082 (was $0.60) 💸
- P95 latency: 3.5s (was 8s) 🚀
- Projected monthly costs: $780 (was $4,500) 🎯
- Success rate: 97% (was 89%) ✨

✅ 86% token reduction (not a typo!)
✅ 86% cost savings (CFO wants to buy us lunch)
✅ 56% faster responses (users think we upgraded servers)
✅ Clear debugging (we now know WHICH agent to blame)
✅ Team happiness: Through the roof
✅ Sleep quality: Significantly improved

At this point, we were pretty sure we’d reached peak performance. We were wrong.

Part 4: The Hybrid Approach - Maximum Efficiency

Peak Engineering

Or: What Happens When You Combine Your Two Good Ideas

After achieving what we thought was AI nirvana with multi-agents, I had a dangerous thought:

“Hey… what if we combine multi-agents WITH the skills pattern?”

It was an idea that precedes either genius or disaster. (In software engineering, these are often the same thing.)

But then we realized: Why choose between our two good ideas when we can have BOTH?

Multi-agents solved the “too many tools” problem, but we could still use skills to optimize even further. It’s like discovering that peanut butter AND jelly make a better sandwich together. Revolutionary? No. Delicious? Absolutely.

Combining the Best of Both

Multi-agents solved tool confusion, but skills can optimize even further. Each specialized agent uses skills for deep context.

// Widget Agent with Skills
const widgetAgent = new ChatOpenAI({ model: "gpt-4" }).bindTools([
  // Core tools (lightweight, always bound)
  createWidget,
  updateWidget,
  deleteWidget,

  // Skill loader (heavy context on-demand)
  loadSkill,
]);

const widgetPrompt = `
You are the Widget Agent - visualization expert.

When you need details:
- Chart config → load_skill("bar_chart_config")
- Widget styling → load_skill("widget_styling", widgetId)
- Data mapping → load_skill("data_mapping_rules")

Current: Dashboard ${state.dashboardId}, Tab ${state.tabId}
`;

Hybrid in Action

User: “Create a pie chart showing age distribution”

Federal Router → Datasource Agent → Widget Agent

Datasource Agent:
├─ Loads: survey_schema skill (2,000 tokens)
├─ Finds age question
└─ Returns structured data

Widget Agent:
├─ Loads: pie_chart_config skill (1,500 tokens)
├─ Creates widget with proper mapping
└─ Total: 6,200 tokens ✅

Vs. loading everything upfront: 18,000 tokens ❌

The Metrics

After implementing the hybrid approach and running it through our test suite (read: throwing everything at it to see what breaks), we got these numbers. We didn’t believe them at first either.

📊 The "Are We Sure This Is Right?" Results:

Token Usage (The Good Kind of Reduction):
├─ Simple queries: 2,500 (was 12,000) → 79% reduction 🎯
├─ Medium queries: 6,000 (was 35,000) → 83% reduction 💪
└─ Complex workflows: 12,000 (was 80,000) → 85% reduction 🚀

Cost Per User/Month:
├─ Light users: $0.25 (was $1.20) → 79% savings
├─ Regular users: $1.50 (was $8.75) → 83% savings
└─ Power users: $6.00 (was $35.00) → 83% savings
(Power users can stay, we can afford them now!)

Performance (Users Think We're Wizards):
├─ P50 latency: 1.8s (was 4.2s) → 57% faster ⚡
├─ P95 latency: 3.5s (was 8.7s) → 60% faster ⚡⚡
└─ P99 latency: 5.2s (was 15s) → 65% faster ⚡⚡⚡

Reliability (Actually Impressed Ourselves):
├─ Success rate: 97% (was 89%) → +8% improvement
├─ Tool selection: 99% (was 85%) → +14% improvement
└─ Context overflow: 0.1% (was 8%) → Basically extinct

💰 Projected Monthly Costs: $780 (was $4,500)
💰 Annual Savings: $44,640
💰 Engineer Stress Levels: Down 90%
☕ Coffee Consumption: Actually decreased
😴 Sleep Quality: Markedly improved

Choosing Your Pattern: The Decision Tree

The “Which One Do I Actually Need?” Decision Tree

Or: Saving You From Our Mistakes

Look, we tried ALL the things so you don’t have to. Here’s our hard-won wisdom, gained through tears, tokens, and too much coffee:

Final Architecture Overview

    ┌─────────────────────────────────────────────┐
    │ Federal Router Agent (GPT-3.5)              │
    │ 4 routing tools (400 tokens)                │
    │ Fast, cheap classification                  │
    └─────────────────────────────────────────────┘
                      │
          ┌───────────┼───────────┬──────────┐
          ↓           ↓           ↓          ↓
    ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
    │Dashboard │ │ Widget   │ │Datasource│ │ Styling  │
    │ (GPT-4)  │ │ (GPT-4)  │ │ (GPT-4)  │ │(GPT-3.5) │
    ├──────────┤ ├──────────┤ ├──────────┤ ├──────────┤
    │6 tools   │ │8 tools   │ │10 tools  │ │6 tools   │
    │+ Skills: │ │+ Skills: │ │+ Skills: │ │+ Skills: │
    │ filters  │ │ configs  │ │ schemas  │ │ themes   │
    │ layouts  │ │ mapping  │ │ datasets │ │ fonts    │
    │ rules    │ │ styling  │ │ stacks   │ │ a11y     │
    └──────────┘ └──────────┘ └──────────┘ └──────────┘

Lessons From the Trenches

What Actually Worked (Surprisingly)

1. Baby Steps Don’t Just Work, They’re Essential

We wanted to rebuild everything overnight. Our tech lead said “no.” We’re glad he did.

┌─────────────────────┬──────────────┬────────────────┐
│ Your Situation      │ Pattern      │ Expected Gain  │
├─────────────────────┼──────────────┼────────────────┤
│ < 10 tools          │ Basic Agent  │ Keep simple    │
│ Simple context      │              │                │
├─────────────────────┼──────────────┼────────────────┤
│ 10-25 tools         │ Skills       │ 50-60% savings │
│ Large context       │              │                │
│ Single domain       │              │                │
├─────────────────────┼──────────────┼────────────────┤
│ 25-50 tools         │ Multi-Agent  │ 70-80% savings │
│ Multiple domains    │              │                │
│ Clear separation    │              │                │
├─────────────────────┼──────────────┼────────────────┤
│ 50+ tools           │ Hybrid       │ 80-85% savings │
│ Complex domains     │              │                │
│ Deep context needs  │              │                │
└─────────────────────┴──────────────┴────────────────┘

Hard Truths We Learned (The Expensive Way)

1. Tool Descriptions: Write Them Like Your Job Depends On It

Because your token budget literally does.

❌ Bad (Our First Attempt):

createWidget: {
  description: "Creates a widget";
}
// LLM: "Cool story bro, but WHEN do I use this?"

✅ Good (After Many Painful Iterations):

createWidget: {
  description: `Creates a visualization widget.
  
  Use when: User asks to "add chart/graph/visualization"
  Don't use for: Updating widgets (use update_widget)
  
  // Yes, we're treating the LLM like a junior dev.
  // Yes, it works.`;
}

Pro tip: Think of tool descriptions as writing documentation for the world’s most literal intern. Because that’s essentially what it is.

2. Monitor What Matters

Track more than just cost:

logger.info({
  event: "agent_execution",
  workflowName: "dashboard_creation",
  metadata: {
    agentType: "widget_agent",
    skillsLoaded: ["bar_chart_config"],
    tokensUsed: 3800,
  },
});

3. Progressive Skill Rollout

Don’t implement all skills at once. Start with the most-used:

Priority 1: survey_schema (80% of requests)
Priority 2: bar_chart_config (60% of requests)
Priority 3: widget_styling (40% of requests)
Priority 4: dashboard_filters (30% of requests)

4. Don’t Over-Specialize (Learn From Our Hubris)

In our enthusiasm, we initially created 8 agents. EIGHT. We were basically creating an AI bureaucracy.

Original (Too Many):
├─ Dashboard Agent
├─ Tab Agent ❌ (seriously, tabs needed their own agent?)
├─ Widget Agent
├─ Widget Styling Agent ❌ (why did we do this to ourselves)
├─ Survey Agent ❌
├─ Dataset Agent ❌
├─ Analytics Agent ❌
└─ Export Agent ❌

After Therapy (Better):
├─ ✅ Dashboard Agent (includes tabs, we're not savages)
├─ ✅ Widget Agent (includes styling, it's fine)
├─ ✅ Datasource Agent (all data sources, one happy family)
└─ ✅ Styling Agent (themes + accessibility)

Sweet spot: 4-6 agents

More than that and you’re just creating coordination overhead. It’s like having too many group chats - eventually nobody knows what’s happening where.

Quick Wins That’ll Make You Look Like a Hero

(Seriously, Do These Today)

1. Stop Writing Novellas in Your System Prompts (30 minutes)

Your system prompt is not the place to explain your life story, company history, and philosophical stance on data visualization.

2. Return Structured Data from Skills

❌ Bad (text):

"The survey has gender as multiple choice and age as numeric...";

✅ Good (JSON):

{
  questions: [
    { id: 67890, text: "Gender?", type: "multiple_choice" },
    { id: 67891, text: "Age?", type: "numeric" },
  ];
}

3. Don’t Over-Specialize

We initially had 8 agents, consolidated to 4:

• ✅ Dashboard Agent (includes tabs)
• ✅ Widget Agent (includes styling)
• ✅ Datasource Agent (all data sources)
• ✅ Styling Agent (themes + accessibility)

Lessons Learned: What We’ve Learned (So Far)

The Real Truth About Building AI Agents

Here’s what they don’t tell you in the glossy blog posts and conference talks:

Building scalable AI agents isn’t about having the most sophisticated architecture from day one. It’s about:

1. Start simple - Basic agent for MVP (it’s okay, we all started here)
2. Measure everything - If you’re not tracking tokens, you’re flying blind
3. Fail fast, learn faster - We made EVERY mistake so you don’t have to
4. Evolve incrementally - Skills → Multi-Agent → Hybrid (this is the way)
5. Focus on user value - Fast, accurate, reliable (novel concept, we know)

Our Journey in Numbers (The Before/After You’ve Been Waiting For)

Where We Started (The Dark Times):
├─ Cost: $4,500/month (gulp)
├─ Tokens: 15k per request (yikes)
├─ Success: 89% (not great, Bob)
└─ Team morale: Low

Where We're At Now (The Good Times):
├─ Cost: $780/month (manageable!)
├─ Tokens: 6.6k per request (sustainable)
├─ Success: 97% (actually good!)
└─ Team morale: High

What’s Next (Or: What We’re Learning Next)

As we continue building and scaling QuestionPro’s BI agent, here’s what we’re watching:

• Context management is king - Progressive disclosure isn’t optional, it’s survival
• Specialization is inevitable - One agent doing everything is like one person doing all jobs at a company (recipe for disaster)
• Hybrid is the sweet spot - Why choose between good ideas?
• Cost optimization is non-negotiable - It’s literally the difference between “cool AI feature” and “bankrupt company”
• Testing is hard - LLMs are non-deterministic. Our test suite has trust issues.
• Documentation matters - Future you will thank present you (we learned this the hard way)

Resources (That Actually Helped Us)

LangGraph Documentation:

• Multi-Agent Patterns (bookmark this, seriously)
• Progressive Disclosure / Skills Pattern (game changer)
• Examples that actually work (rare, treasure them)

Monitoring Tools We Actually Use:

• Langfuse (our current favorite)
• LangSmith (also good)
• Custom OpenTelemetry (for the brave)
• Coffee (monitors our alertness)

Calculate Your Potential Savings (Do This Right Now)

Seriously, take 2 minutes and see what you could be saving:

// Your current situation
const currentCost = ((monthlyRequests * avgTokens) / 1_000_000) * $10;

// What you COULD be saving
const potentialSavings = {
  withSkills: currentCost * 0.5, // 50% reduction
  withMultiAgent: currentCost * 0.7, // 70% reduction
  withHybrid: currentCost * 0.85, // 85% reduction
};

// Now imagine what you could do with that money
// (We're thinking coffee budget + GPU upgrades)

Final Thoughts (The Honest Ones)

Look, we’re still figuring this out. We’re still learning. We’re still making mistakes (just more expensive ones now that we know better). But that’s the point… nobody has this completely figured out. AI is moving faster than documentation can keep up.

What we DO know:

• Start simple, evolve based on actual problems
• Measure EVERYTHING (seriously, token tracking is non-negotiable)
• Don’t over-engineer (we did this so you don’t have to)
• Community knowledge is invaluable (thank you, internet strangers)
• It’s okay to not know everything (we certainly don’t)

The Future

As agents scale:

• Context management is king - Progressive disclosure wins
• Specialization is inevitable - One agent can’t do it all
• Hybrid is the sweet spot - Combines best of all patterns
• Cost optimization is non-negotiable - Make or break for production

The Early Days

When I joined QuestionPro, the company had just moved its office from Nashik to Pune. Our first workspace was small but cozy—more like a collaborative lab than a traditional office. There were only eight of us then, wearing multiple hats and learning as we went.

I started as a Java Developer, primarily focused on bug fixes, feature testing, and writing or updating help documentation. Those early tasks might seem small, but they taught me the importance of precision, patience, and understanding the product from the user’s perspective.

Learning, Mentorship, and Confidence

The next two years were a turning point. Under the mentorship of Shri and Anish, I began to take on larger challenges, own modules, and develop new features. Their guidance shaped not only my technical skills but also my professional mindset.

They taught me that leadership isn’t just about giving directions—it’s about owning outcomes and continuously learning. Even today, they remain my role models, and I still find myself learning from their approach to problem-solving and innovation.

Stepping into Leadership

As QuestionPro grew, so did my responsibilities. I transitioned into a Team Lead (TL) role, managing a small team while continuing to code and contribute hands-on. This phase was both exciting and demanding—it pushed me to develop skills in people management, communication, and strategic thinking.

However, what truly made it memorable were the moments outside of work—our Friday night gatherings, table tennis matches, and cricket games. These experiences built lasting friendships and strengthened our culture of collaboration and trust.

Working with Visionary Leaders

One of the highlights of my career has been the opportunity to work directly with Vivek Bhaskaran, our Founder & CEO, and Erik Koto, our COO/Ex-CEO. Having direct access to the leadership team gave me a unique perspective on how decisions are made and how innovation is driven at scale.

The management team at QuestionPro has always been accessible, supportive, and empathetic. They’ve built a culture where everyone’s ideas are valued, and every challenge is seen as an opportunity to grow.

Growth, Ownership, and Empowerment

Over the years, my focus has shifted from purely technical work to engineering leadership. I’ve learned the art of hiring, mentoring, and helping team members chart their own career paths. Watching people I’ve mentored grow into confident engineers and leaders has been one of the most rewarding aspects of my journey.

Today, as an Engineering Manager, I handle three different products, each with its own challenges and learning curves. The responsibility is immense, but so is the satisfaction that comes with building scalable, user-centered products that impact thousands of users worldwide.

What Makes QuestionPro Special

If I were to summarize what makes QuestionPro unique, it would be its open culture and growth mindset. Here, you’re encouraged to take initiative, challenge assumptions, and experiment with new ideas.

Failures aren’t punished—they’re treated as learning opportunities. That philosophy has allowed me to evolve continuously, both as an engineer and as a leader.

Looking Back—and Ahead

From debugging my first piece of code to managing multiple teams and products, every step of this journey has taught me something valuable.

• QuestionPro has given me more than a career—it has given me a community.

• It has shown me that growth doesn’t happen by chance; it happens by staying curious, embracing challenges, and never stopping learning.

• And most importantly, it has reinforced my belief that great products are built by great people who trust and learn from each other.

As I look to the future, I’m excited to continue this journey of building, leading, and learning—together with a team that feels more like family.

Correctness by Construction (CbC) is the shift from the chaotic energy of “Let’s see if this runs” to the calm confidence of “I know this runs because I made it impossible for it not to.”

p.s. “No, not Complete blood Count (CbC) :3”

Let me talk about potential tradeoffs before jumping into how we could achieve the CbC mindset,

Pros

• Peace: You spend 90% less time in the debugger (which is just a crime scene where you are both the detective and the murderer).
• Sleep: Yeah.

Cons

• Slower Start: You have to think before you type. It feels less like “hacking” and more like “engineering.”
• Boredom: You won’t look as busy to your peers because you aren’t “putting out fires” constantly.

Here is the toolkit for your CbC journey, a guide to replacing ‘hope’ with the certainty of flawless, intentional design.

The Prologue: Strong Specifications

Before you write a single line of code, write the story. Define exactly what the function does,

• The Vibe: Don’t build a house based on a napkin sketch.
• The Habit: If you can’t explain the constraints in English, you certainly can’t explain them in JavaScript ;)

The Bouncer: Safe Languages & Types

Use your type system to forbid “impossible” situations. A bug cannot happen if the code literally refuses to compile it.

Don’t: Build a “Frankenstein” object full of optional flags

• Bad: A request state like

{ isLoading: boolean, error?: string, data?: string }.

• The Trap: You can accidentally create a state where isLoading: true and error: “Failed”. Is it loading? Is it broken? The code is confused, and so are you.

Do: Use Discriminated Unions. Force the data to exist only when the state allows it.

• Good: Define distinct states

type State = { status: "loading" } | { status: "success"; data: string };

If you check status === ‘loading’, TypeScript physically prevents you from accessing .data. It doesn’t exist yet. The compiler catches the bug before you even run the code.

The Handshake: Design-by-Contract

Every function is a business deal.

• Preconditions: “You promise to give me a positive number.”
• Postconditions: “I promise to return its square root.”
• Invariants: “I promise the database never catches fire.” If the contract is broken, the program shouldn’t try to limp along, it should shout immediately!

The Haiku: Modularity & Simplicity

Complexity is where bugs hide to reproduce. Keep your components small, pure, and simple.

• The Philosophy: Write code like a Haiku, not a dissertation. If a function does three things, it’s doing two things too many.

Code Hygiene: Avoid Error-Prone Patterns

Some patterns are just slippery floors waiting for a lawsuit.

• Don’t: Use shared mutable state (global variables that everyone touches). That’s like sharing a toothbrush.
• Do: Use Pure Functions (same input always equals same output) and Result types instead of throwing random Exceptions.

The Epilogue: Tests Confirm, They Don’t Fix

Testing is your safety net, not your construction plan.

• The Shift: If you rely on tests to find all your bugs, you’ve already lost.
• The Goal: Build it so solid that the tests are just a formality, a victory lap to prove you were right all along.

The Grand Finale

Adopt the mindset of a sculptor. You don’t chip away stone and hope it looks like a horse later. You verify every angle as you go.

QuestionPro is excited to announce TechCommit, a focused technical meetup designed to accelerate your growth and provide actionable insights into building high-performance, intelligent systems. Join us in our Pune office on Saturday, December 13th, for two power-packed sessions!

🗓️ Essential Event Details

🎙️ Agenda: Deep Dives by QuestionPro Experts

We have lined up two high-impact talks from our senior experts, focusing on the real-world challenges of building highly available systems and leveraging Artificial Intelligence for data analysis.

Talk 1: From Zero to 10k Live Users: Scaling Real-time Engagement Platforms

Speaker: Rohan Rao, Software Engineer

Building a real-time platform that handles thousands of concurrent users is a critical challenge. Rohan Rao, a Software Engineer with over 7 years of experience, will share lessons learned and practical strategies for:

• Designing resilient and scalable architectures.
• Implementing high-performance solutions that maintain speed under heavy load.

Talk 2: AI in Action: Building a Smarter Topic Analysis System

Speaker: Pratik Dhulubulu, Dedicated Data Scientist

Artificial intelligence is rapidly transforming how companies understand user data. Pratik Dhulubulu, a Data Scientist with 4+ years of experience, will move past the hype and demonstrate how to:

• Leverage AI to solve complex data problems.
• Build a sophisticated, data-backed topic analysis system using modern Python techniques.

Why This Pune Tech Meetup Is a Must-Attend

1. Focused Learning: Gain high-value, technical knowledge from experienced practitioners in a short, efficient format.
2. Networking: Connect face-to-face with fellow software engineers, data scientists, and developers in the vibrant Pune tech scene.
3. Community: Discuss the real-world implementation challenges of scaling and AI in a professional environment.

Secure Your Spot

Spaces for this developer meetup in Pune are limited. Don’t miss out!

The last day to register is Thursday, December 11th.

Register Now

We look forward to connecting with you at the QuestionPro office in Baner!