Introduction

Remember when you thought building an AI agent would be easy? “Just throw some prompts at GPT-4 and call it a day,”.

“What could go wrong?” you said.

Narrator: Everything went wrong.

Large Language Models have revolutionized intelligent applications, but here’s what nobody tells you at those fancy AI conferences: scaling an AI agent from “cool demo” to “production system that doesn’t bankrupt your company” is… challenging. Token costs spiral like a startup’s cloud bill, context windows overflow faster than your coffee cup on Monday morning, and response times make dial-up internet look speedy.

This is our journey (currently in progress, bugs included) from a basic implementation to building a production-ready, cost-efficient multi-agent system for QuestionPro’s BI platform. We’re exploring three key patterns, making mistakes in real-time, and documenting everything so you don’t have to. Think of this as “Mythbusters” but for AI architecture, with 100% more token optimization and slightly fewer explosions.

Part 1: The Basic Agent - Or “How Hard Could It Be?”

The Innocent Beginning

Like every developer who’s just discovered LangGraph, we started with the “obvious” approach, THE MONOLITHIC AGENT: one agent to rule them all, one agent to find them, one agent to bring them all, and in the darkness bind them (to a $10,000/month OpenAI bill… roughly).

// Single agent with all capabilities
const agent = new ChatOpenAI({ model: "gpt-4" }).bindTools([
  createDashboard,
  getDashboard,
  createWidget,
  listSurveys,
  getQuestions,
  // ... 40+ more tools
]);

const systemPrompt = `
You are a dashboard analytics assistant.

Available dashboards: [... 500 lines of context ...]
Survey schemas: [... 2000 lines of schemas ...]
Widget configurations: [... 1500 lines of configs ...]
`;

The “Oh No” Moment

You know that feeling when you check your AI bill and your heart skips a beat? That was us, thinking about going into “production”.

Our “Everything is Fine” Metrics:

📊 The Uncomfortable Truth:
- Avg tokens per request: 15,000 (narrator: this is not good)
- Cost per conversation: $1.50 (multiply by users... *sweats*)
- P95 latency: 8 seconds (users hate us)
- Projected monthly costs: $4,500 (CFO hates us more)
- Success rate: 89% (11% of the time, it works every time!)

🔴 The "We Need to Talk" Problems:
- Context overflow (GPT-4 politely hanging up on us)
- Tool selection paralysis (like Netflix, which movie to watch)
- Exponential cost growth
- LLM having an existential crisis (overwhelmed with 40+ tools)

The Exponential Growth Problem (Or: Math is Unforgiving)

Turn 1:  12,500 tokens  ✅ "This is fine"
Turn 5:  22,000 tokens  ⚠️ "This is less fine"
Turn 10: 37,000 tokens  ❌ "This is fire, everything is fire"

We were literally pricing ourselves out of business, one helpful conversation at a time.

The real kicker? We kept asking “Can we just add more features?” while totally unaware of the bill that might get generated.

Note: These numbers are rough estimates from our experiments and paper napkin calculations. We’re still actively developing and iterating, but gives an idea to think in the right direction.

Part 2: The Skills Pattern - Progressive Disclosure

The “Aha!” Moment

Or: How We Discovered That Laziness is Actually a Virtue

After three existential crises where we admitted our agent was hemorrhaging money, we stumbled upon a life-changing concept:

“What if… and hear me out… we DON’T load everything into the prompt like we’re packing for a 2-week vacation?”

Revolutionary, I know. We felt like we’d discovered fire, except fire was actually “reading the documentation properly.”

Enter progressive disclosure - a fancy term for “only fetch stuff when you actually need it,” which is basically how every efficient human operates but somehow felt groundbreaking when applied to AI. (Yes, we’re aware of the irony.)

“Don’t load everything upfront. Load only what you need, when you need it.”

Instead of including all context in every request, we implemented progressive disclosure - the agent loads specialized knowledge through the skills pattern.

Implementation

// Lightweight system prompt with skill metadata only
const systemPrompt = `
You are a dashboard analytics assistant.

Available Skills (load when needed):
- survey_schema: Get survey questions and field types
- widget_config: Get chart-specific settings  
- dashboard_filters: Get filter options
- styling_themes: Get appearance options

Current context:
- Workspace: 1
- Dashboard: ${state.dashboardId || "none"}
`;

// Skills load on-demand
const loadSkillTool = tool(async ({ skillName, resourceId }) => {
  switch (skillName) {
    case "survey_schema":
      return await fetchSurveySchema(resourceId);
    // Returns ~2,000 tokens only when needed

    case "widget_config":
      return await fetchWidgetConfig(resourceId);
    // Returns ~1,500 tokens only when needed
  }
});

Skills in Action

User: “Create a gender chart from Customer Survey”

Request 1: Agent sees lightweight prompt
├─ System: 1,000 tokens (was 8,000)
├─ Tools: 1,500 tokens
└─ Messages: 2,000 tokens
   Total: 4,500 tokens ✅

Agent decides: "I need the survey schema"
└─ Calls: load_skill("survey_schema", 12345)

Request 2: Agent receives survey details
├─ Previous context: 4,500 tokens
├─ Survey schema: 2,000 tokens
└─ Total: 6,500 tokens ✅

Agent creates chart with correct data mapping

The Results (Or: When Theory Meets Reality and Actually Works)

📊 Skills Pattern Metrics (We Were Shocked Too):

- Avg tokens: 6,500 (was 15,000) 🎉
- Cost per conversation: $0.65 (was $1.50) 💰
- P95 latency: 4s (was 8s) ⚡
- Monthly costs: $1,950 (was $4,500) 🎊

✅ 57% cost reduction (CFO can sleep better)
✅ 50% faster responses (users will hate us a little less now)
✅ Stable token usage (no more exponential nightmares)
✅ Team morale: Significantly improved

As this was my late night hustle when no one was around, I just high fived my wall.

### But Wait... (The Plot Thickens)

Just when we thought we'd solved AI, reality decided to humble us. Again.


❌ The "Not So Fast" Problems:

1. Agent handling 40+ tools → Like asking someone to pick
   their favorite child, but with 40 children
2. Mixed concerns → Debugging became "which of these
   40 things broke?"
3. No parallelization → Everything sequential because
   apparently we hate speed
4. Some skills returned 3,000+ tokens → The token diet
   didn't last long

The Comedy of Errors:

User: "Style my dashboard in dark mode."
(A simple, reasonable request.)

Agent's internal monologue:
"Hmm, I have 40 tools. Let me read each description...
create_dashboard? No...
update_dashboard? Maybe?
create_widget? Probably not...
update_theme? OH WAIT THAT'S THE ONE! ✅
...only took me 3 seconds to figure that out"

User: _has already rage-quit_

Part 3: Multi-Agent Architecture - Specialized Experts

The “Why Didn’t We Think of This Sooner?” Moment

The Breakthrough (Cue Dramatic Music)

Picture this: It’s 2 AM, you’re on your third coffee, scrolling through LangGraph documentation for the millionth time, when suddenly…

“What if… what if… we don’t make one agent do EVERYTHING? What if we have specialized agents? Like… real companies?”

🤯 Mind. Blown.

We felt like we’d invented the wheel, despite the fact that humans have been organizing into specialized roles since, like, the dawn of civilization. But hey, better late than never!

Our New Squad:

• Dashboard Agent: The organized one who actually reads the manual
• Widget Agent: The creative type, probably went to art school
• Datasource Agent: The data nerd (affectionately), speaks fluent SQL
• Styling Agent: Fashion police of the digital world

Basically, we went from having one overworked, stressed-out agent having a breakdown, to a healthy work environment with proper delegation. Revolutionary.

The Breakthrough

“What if instead of one agent doing everything, we had specialized experts?”

Like a company with departments (Sales, Engineering, Design), we created specialized agents:

• Dashboard Agent: Dashboards and tabs expert
• Widget Agent: Visualizations expert
• Datasource Agent: Data and surveys expert
• Styling Agent: Themes and appearances expert

Federal Router Implementation

// Lightweight orchestrator
const federalAgent = new ChatOpenAI({ model: "gpt-4" }).bindTools([
  routeToDashboardAgent,
  routeToWidgetAgent,
  routeToDatasourceAgent,
  routeToStylingAgent,
]);

const federalPrompt = `
You are a routing assistant. Delegate to specialists:

- Dashboard operations → Dashboard Agent
- Widget creation/editing → Widget Agent
- Data selection → Datasource Agent
- Styling/themes → Styling Agent

You route and synthesize - you don't execute tasks.
`;

Specialized Agent Example

// Dashboard Agent - Only 6 focused tools
const dashboardAgent = new ChatOpenAI({ model: "gpt-4" }).bindTools([
  createDashboard,
  getDashboard,
  updateDashboard,
  createTab,
  updateTab,
  deleteDashboard,
]);

Conversation Flow

User: “Create sales dashboard with gender chart from Customer Survey”

┌────────────────────────────────────────┐
│ 1. Federal Router                      │
│ Tools: 4 routing (400 tokens)          │
│ Decision: Dashboard → Data → Widget    │
│ Cost: 1,400 tokens                     │
└────────────────────────────────────────┘
                ↓
┌────────────────────────────────────────┐
│ 2. Dashboard Agent                     │
│ Tools: 6 dashboard (600 tokens)        │
│ Creates: "Sales" dashboard (ID 100)    │
│ Cost: 1,400 tokens                     │
└────────────────────────────────────────┘
                ↓
┌────────────────────────────────────────┐
│ 3. Datasource Agent                    │
│ Tools: 10 data (1,000 tokens)          │
│ Finds: Survey 12345, Question 67890    │
│ Cost: 2,200 tokens                     │
└────────────────────────────────────────┘
                ↓
┌────────────────────────────────────────┐
│ 4. Widget Agent                        │
│ Tools: 8 widget (800 tokens)           │
│ Creates: Pie chart widget              │
│ Cost: 2,800 tokens                     │
└────────────────────────────────────────┘

Total workflow: 8,200 tokens (vs 60,000)

The Results (Holy Sh— I Mean, Wow!)

📊 The Numbers Don't Lie (But We Triple-Checked Anyway):

- Avg tokens per workflow: 8,200 (was 60,000) 📉
- Cost per workflow: $0.082 (was $0.60) 💸
- P95 latency: 3.5s (was 8s) 🚀
- Projected monthly costs: $780 (was $4,500) 🎯
- Success rate: 97% (was 89%) ✨

✅ 86% token reduction (not a typo!)
✅ 86% cost savings (CFO wants to buy us lunch)
✅ 56% faster responses (users think we upgraded servers)
✅ Clear debugging (we now know WHICH agent to blame)
✅ Team happiness: Through the roof
✅ Sleep quality: Significantly improved

At this point, we were pretty sure we’d reached peak performance. We were wrong.

Part 4: The Hybrid Approach - Maximum Efficiency

Peak Engineering

Or: What Happens When You Combine Your Two Good Ideas

After achieving what we thought was AI nirvana with multi-agents, I had a dangerous thought:

“Hey… what if we combine multi-agents WITH the skills pattern?”

It was an idea that precedes either genius or disaster. (In software engineering, these are often the same thing.)

But then we realized: Why choose between our two good ideas when we can have BOTH?

Multi-agents solved the “too many tools” problem, but we could still use skills to optimize even further. It’s like discovering that peanut butter AND jelly make a better sandwich together. Revolutionary? No. Delicious? Absolutely.

Combining the Best of Both

Multi-agents solved tool confusion, but skills can optimize even further. Each specialized agent uses skills for deep context.

// Widget Agent with Skills
const widgetAgent = new ChatOpenAI({ model: "gpt-4" }).bindTools([
  // Core tools (lightweight, always bound)
  createWidget,
  updateWidget,
  deleteWidget,

  // Skill loader (heavy context on-demand)
  loadSkill,
]);

const widgetPrompt = `
You are the Widget Agent - visualization expert.

When you need details:
- Chart config → load_skill("bar_chart_config")
- Widget styling → load_skill("widget_styling", widgetId)
- Data mapping → load_skill("data_mapping_rules")

Current: Dashboard ${state.dashboardId}, Tab ${state.tabId}
`;

Hybrid in Action

User: “Create a pie chart showing age distribution”

Federal Router → Datasource Agent → Widget Agent

Datasource Agent:
├─ Loads: survey_schema skill (2,000 tokens)
├─ Finds age question
└─ Returns structured data

Widget Agent:
├─ Loads: pie_chart_config skill (1,500 tokens)
├─ Creates widget with proper mapping
└─ Total: 6,200 tokens ✅

Vs. loading everything upfront: 18,000 tokens ❌

The Metrics

After implementing the hybrid approach and running it through our test suite (read: throwing everything at it to see what breaks), we got these numbers. We didn’t believe them at first either.

📊 The "Are We Sure This Is Right?" Results:

Token Usage (The Good Kind of Reduction):
├─ Simple queries: 2,500 (was 12,000) → 79% reduction 🎯
├─ Medium queries: 6,000 (was 35,000) → 83% reduction 💪
└─ Complex workflows: 12,000 (was 80,000) → 85% reduction 🚀

Cost Per User/Month:
├─ Light users: $0.25 (was $1.20) → 79% savings
├─ Regular users: $1.50 (was $8.75) → 83% savings
└─ Power users: $6.00 (was $35.00) → 83% savings
(Power users can stay, we can afford them now!)

Performance (Users Think We're Wizards):
├─ P50 latency: 1.8s (was 4.2s) → 57% faster ⚡
├─ P95 latency: 3.5s (was 8.7s) → 60% faster ⚡⚡
└─ P99 latency: 5.2s (was 15s) → 65% faster ⚡⚡⚡

Reliability (Actually Impressed Ourselves):
├─ Success rate: 97% (was 89%) → +8% improvement
├─ Tool selection: 99% (was 85%) → +14% improvement
└─ Context overflow: 0.1% (was 8%) → Basically extinct

💰 Projected Monthly Costs: $780 (was $4,500)
💰 Annual Savings: $44,640
💰 Engineer Stress Levels: Down 90%
☕ Coffee Consumption: Actually decreased
😴 Sleep Quality: Markedly improved

Choosing Your Pattern: The Decision Tree

The “Which One Do I Actually Need?” Decision Tree

Or: Saving You From Our Mistakes

Look, we tried ALL the things so you don’t have to. Here’s our hard-won wisdom, gained through tears, tokens, and too much coffee:

Final Architecture Overview

    ┌─────────────────────────────────────────────┐
    │ Federal Router Agent (GPT-3.5)              │
    │ 4 routing tools (400 tokens)                │
    │ Fast, cheap classification                  │
    └─────────────────────────────────────────────┘
                      │
          ┌───────────┼───────────┬──────────┐
          ↓           ↓           ↓          ↓
    ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
    │Dashboard │ │ Widget   │ │Datasource│ │ Styling  │
    │ (GPT-4)  │ │ (GPT-4)  │ │ (GPT-4)  │ │(GPT-3.5) │
    ├──────────┤ ├──────────┤ ├──────────┤ ├──────────┤
    │6 tools   │ │8 tools   │ │10 tools  │ │6 tools   │
    │+ Skills: │ │+ Skills: │ │+ Skills: │ │+ Skills: │
    │ filters  │ │ configs  │ │ schemas  │ │ themes   │
    │ layouts  │ │ mapping  │ │ datasets │ │ fonts    │
    │ rules    │ │ styling  │ │ stacks   │ │ a11y     │
    └──────────┘ └──────────┘ └──────────┘ └──────────┘

Lessons From the Trenches

What Actually Worked (Surprisingly)

1. Baby Steps Don’t Just Work, They’re Essential

We wanted to rebuild everything overnight. Our tech lead said “no.” We’re glad he did.

┌─────────────────────┬──────────────┬────────────────┐
│ Your Situation      │ Pattern      │ Expected Gain  │
├─────────────────────┼──────────────┼────────────────┤
│ < 10 tools          │ Basic Agent  │ Keep simple    │
│ Simple context      │              │                │
├─────────────────────┼──────────────┼────────────────┤
│ 10-25 tools         │ Skills       │ 50-60% savings │
│ Large context       │              │                │
│ Single domain       │              │                │
├─────────────────────┼──────────────┼────────────────┤
│ 25-50 tools         │ Multi-Agent  │ 70-80% savings │
│ Multiple domains    │              │                │
│ Clear separation    │              │                │
├─────────────────────┼──────────────┼────────────────┤
│ 50+ tools           │ Hybrid       │ 80-85% savings │
│ Complex domains     │              │                │
│ Deep context needs  │              │                │
└─────────────────────┴──────────────┴────────────────┘

Hard Truths We Learned (The Expensive Way)

1. Tool Descriptions: Write Them Like Your Job Depends On It

Because your token budget literally does.

❌ Bad (Our First Attempt):

createWidget: {
  description: "Creates a widget";
}
// LLM: "Cool story bro, but WHEN do I use this?"

✅ Good (After Many Painful Iterations):

createWidget: {
  description: `Creates a visualization widget.
  
  Use when: User asks to "add chart/graph/visualization"
  Don't use for: Updating widgets (use update_widget)
  
  // Yes, we're treating the LLM like a junior dev.
  // Yes, it works.`;
}

Pro tip: Think of tool descriptions as writing documentation for the world’s most literal intern. Because that’s essentially what it is.

2. Monitor What Matters

Track more than just cost:

logger.info({
  event: "agent_execution",
  workflowName: "dashboard_creation",
  metadata: {
    agentType: "widget_agent",
    skillsLoaded: ["bar_chart_config"],
    tokensUsed: 3800,
  },
});

3. Progressive Skill Rollout

Don’t implement all skills at once. Start with the most-used:

Priority 1: survey_schema (80% of requests)
Priority 2: bar_chart_config (60% of requests)
Priority 3: widget_styling (40% of requests)
Priority 4: dashboard_filters (30% of requests)

4. Don’t Over-Specialize (Learn From Our Hubris)

In our enthusiasm, we initially created 8 agents. EIGHT. We were basically creating an AI bureaucracy.

Original (Too Many):
├─ Dashboard Agent
├─ Tab Agent ❌ (seriously, tabs needed their own agent?)
├─ Widget Agent
├─ Widget Styling Agent ❌ (why did we do this to ourselves)
├─ Survey Agent ❌
├─ Dataset Agent ❌
├─ Analytics Agent ❌
└─ Export Agent ❌

After Therapy (Better):
├─ ✅ Dashboard Agent (includes tabs, we're not savages)
├─ ✅ Widget Agent (includes styling, it's fine)
├─ ✅ Datasource Agent (all data sources, one happy family)
└─ ✅ Styling Agent (themes + accessibility)

Sweet spot: 4-6 agents

More than that and you’re just creating coordination overhead. It’s like having too many group chats - eventually nobody knows what’s happening where.

Quick Wins That’ll Make You Look Like a Hero

(Seriously, Do These Today)

1. Stop Writing Novellas in Your System Prompts (30 minutes)

Your system prompt is not the place to explain your life story, company history, and philosophical stance on data visualization.

2. Return Structured Data from Skills

❌ Bad (text):

"The survey has gender as multiple choice and age as numeric...";

✅ Good (JSON):

{
  questions: [
    { id: 67890, text: "Gender?", type: "multiple_choice" },
    { id: 67891, text: "Age?", type: "numeric" },
  ];
}

3. Don’t Over-Specialize

We initially had 8 agents, consolidated to 4:

• ✅ Dashboard Agent (includes tabs)
• ✅ Widget Agent (includes styling)
• ✅ Datasource Agent (all data sources)
• ✅ Styling Agent (themes + accessibility)

Lessons Learned: What We’ve Learned (So Far)

The Real Truth About Building AI Agents

Here’s what they don’t tell you in the glossy blog posts and conference talks:

Building scalable AI agents isn’t about having the most sophisticated architecture from day one. It’s about:

1. Start simple - Basic agent for MVP (it’s okay, we all started here)
2. Measure everything - If you’re not tracking tokens, you’re flying blind
3. Fail fast, learn faster - We made EVERY mistake so you don’t have to
4. Evolve incrementally - Skills → Multi-Agent → Hybrid (this is the way)
5. Focus on user value - Fast, accurate, reliable (novel concept, we know)

Our Journey in Numbers (The Before/After You’ve Been Waiting For)

Where We Started (The Dark Times):
├─ Cost: $4,500/month (gulp)
├─ Tokens: 15k per request (yikes)
├─ Success: 89% (not great, Bob)
└─ Team morale: Low

Where We're At Now (The Good Times):
├─ Cost: $780/month (manageable!)
├─ Tokens: 6.6k per request (sustainable)
├─ Success: 97% (actually good!)
└─ Team morale: High

What’s Next (Or: What We’re Learning Next)

As we continue building and scaling QuestionPro’s BI agent, here’s what we’re watching:

• Context management is king - Progressive disclosure isn’t optional, it’s survival
• Specialization is inevitable - One agent doing everything is like one person doing all jobs at a company (recipe for disaster)
• Hybrid is the sweet spot - Why choose between good ideas?
• Cost optimization is non-negotiable - It’s literally the difference between “cool AI feature” and “bankrupt company”
• Testing is hard - LLMs are non-deterministic. Our test suite has trust issues.
• Documentation matters - Future you will thank present you (we learned this the hard way)

Resources (That Actually Helped Us)

LangGraph Documentation:

• Multi-Agent Patterns (bookmark this, seriously)
• Progressive Disclosure / Skills Pattern (game changer)
• Examples that actually work (rare, treasure them)

Monitoring Tools We Actually Use:

• Langfuse (our current favorite)
• LangSmith (also good)
• Custom OpenTelemetry (for the brave)
• Coffee (monitors our alertness)

Calculate Your Potential Savings (Do This Right Now)

Seriously, take 2 minutes and see what you could be saving:

// Your current situation
const currentCost = ((monthlyRequests * avgTokens) / 1_000_000) * $10;

// What you COULD be saving
const potentialSavings = {
  withSkills: currentCost * 0.5, // 50% reduction
  withMultiAgent: currentCost * 0.7, // 70% reduction
  withHybrid: currentCost * 0.85, // 85% reduction
};

// Now imagine what you could do with that money
// (We're thinking coffee budget + GPU upgrades)

Final Thoughts (The Honest Ones)

Look, we’re still figuring this out. We’re still learning. We’re still making mistakes (just more expensive ones now that we know better). But that’s the point… nobody has this completely figured out. AI is moving faster than documentation can keep up.

What we DO know:

• Start simple, evolve based on actual problems
• Measure EVERYTHING (seriously, token tracking is non-negotiable)
• Don’t over-engineer (we did this so you don’t have to)
• Community knowledge is invaluable (thank you, internet strangers)
• It’s okay to not know everything (we certainly don’t)

The Future

As agents scale:

• Context management is king - Progressive disclosure wins
• Specialization is inevitable - One agent can’t do it all
• Hybrid is the sweet spot - Combines best of all patterns
• Cost optimization is non-negotiable - Make or break for production

The Early Days

When I joined QuestionPro, the company had just moved its office from Nashik to Pune. Our first workspace was small but cozy—more like a collaborative lab than a traditional office. There were only eight of us then, wearing multiple hats and learning as we went.

I started as a Java Developer, primarily focused on bug fixes, feature testing, and writing or updating help documentation. Those early tasks might seem small, but they taught me the importance of precision, patience, and understanding the product from the user’s perspective.

Learning, Mentorship, and Confidence

The next two years were a turning point. Under the mentorship of Shri and Anish, I began to take on larger challenges, own modules, and develop new features. Their guidance shaped not only my technical skills but also my professional mindset.

They taught me that leadership isn’t just about giving directions—it’s about owning outcomes and continuously learning. Even today, they remain my role models, and I still find myself learning from their approach to problem-solving and innovation.

Stepping into Leadership

As QuestionPro grew, so did my responsibilities. I transitioned into a Team Lead (TL) role, managing a small team while continuing to code and contribute hands-on. This phase was both exciting and demanding—it pushed me to develop skills in people management, communication, and strategic thinking.

However, what truly made it memorable were the moments outside of work—our Friday night gatherings, table tennis matches, and cricket games. These experiences built lasting friendships and strengthened our culture of collaboration and trust.

Working with Visionary Leaders

One of the highlights of my career has been the opportunity to work directly with Vivek Bhaskaran, our Founder & CEO, and Erik Koto, our COO/Ex-CEO. Having direct access to the leadership team gave me a unique perspective on how decisions are made and how innovation is driven at scale.

The management team at QuestionPro has always been accessible, supportive, and empathetic. They’ve built a culture where everyone’s ideas are valued, and every challenge is seen as an opportunity to grow.

Growth, Ownership, and Empowerment

Over the years, my focus has shifted from purely technical work to engineering leadership. I’ve learned the art of hiring, mentoring, and helping team members chart their own career paths. Watching people I’ve mentored grow into confident engineers and leaders has been one of the most rewarding aspects of my journey.

Today, as an Engineering Manager, I handle three different products, each with its own challenges and learning curves. The responsibility is immense, but so is the satisfaction that comes with building scalable, user-centered products that impact thousands of users worldwide.

What Makes QuestionPro Special

If I were to summarize what makes QuestionPro unique, it would be its open culture and growth mindset. Here, you’re encouraged to take initiative, challenge assumptions, and experiment with new ideas.

Failures aren’t punished—they’re treated as learning opportunities. That philosophy has allowed me to evolve continuously, both as an engineer and as a leader.

Looking Back—and Ahead

From debugging my first piece of code to managing multiple teams and products, every step of this journey has taught me something valuable.

• QuestionPro has given me more than a career—it has given me a community.

• It has shown me that growth doesn’t happen by chance; it happens by staying curious, embracing challenges, and never stopping learning.

• And most importantly, it has reinforced my belief that great products are built by great people who trust and learn from each other.

As I look to the future, I’m excited to continue this journey of building, leading, and learning—together with a team that feels more like family.

Correctness by Construction (CbC) is the shift from the chaotic energy of “Let’s see if this runs” to the calm confidence of “I know this runs because I made it impossible for it not to.”

p.s. “No, not Complete blood Count (CbC) :3”

Let me talk about potential tradeoffs before jumping into how we could achieve the CbC mindset,

Pros

• Peace: You spend 90% less time in the debugger (which is just a crime scene where you are both the detective and the murderer).
• Sleep: Yeah.

Cons

• Slower Start: You have to think before you type. It feels less like “hacking” and more like “engineering.”
• Boredom: You won’t look as busy to your peers because you aren’t “putting out fires” constantly.

Here is the toolkit for your CbC journey, a guide to replacing ‘hope’ with the certainty of flawless, intentional design.

The Prologue: Strong Specifications

Before you write a single line of code, write the story. Define exactly what the function does,

• The Vibe: Don’t build a house based on a napkin sketch.
• The Habit: If you can’t explain the constraints in English, you certainly can’t explain them in JavaScript ;)

The Bouncer: Safe Languages & Types

Use your type system to forbid “impossible” situations. A bug cannot happen if the code literally refuses to compile it.

Don’t: Build a “Frankenstein” object full of optional flags

• Bad: A request state like

{ isLoading: boolean, error?: string, data?: string }.

• The Trap: You can accidentally create a state where isLoading: true and error: “Failed”. Is it loading? Is it broken? The code is confused, and so are you.

Do: Use Discriminated Unions. Force the data to exist only when the state allows it.

• Good: Define distinct states

type State = { status: "loading" } | { status: "success"; data: string };

If you check status === ‘loading’, TypeScript physically prevents you from accessing .data. It doesn’t exist yet. The compiler catches the bug before you even run the code.

The Handshake: Design-by-Contract

Every function is a business deal.

• Preconditions: “You promise to give me a positive number.”
• Postconditions: “I promise to return its square root.”
• Invariants: “I promise the database never catches fire.” If the contract is broken, the program shouldn’t try to limp along, it should shout immediately!

The Haiku: Modularity & Simplicity

Complexity is where bugs hide to reproduce. Keep your components small, pure, and simple.

• The Philosophy: Write code like a Haiku, not a dissertation. If a function does three things, it’s doing two things too many.

Code Hygiene: Avoid Error-Prone Patterns

Some patterns are just slippery floors waiting for a lawsuit.

• Don’t: Use shared mutable state (global variables that everyone touches). That’s like sharing a toothbrush.
• Do: Use Pure Functions (same input always equals same output) and Result types instead of throwing random Exceptions.

The Epilogue: Tests Confirm, They Don’t Fix

Testing is your safety net, not your construction plan.

• The Shift: If you rely on tests to find all your bugs, you’ve already lost.
• The Goal: Build it so solid that the tests are just a formality, a victory lap to prove you were right all along.

The Grand Finale

Adopt the mindset of a sculptor. You don’t chip away stone and hope it looks like a horse later. You verify every angle as you go.

QuestionPro is excited to announce TechCommit, a focused technical meetup designed to accelerate your growth and provide actionable insights into building high-performance, intelligent systems. Join us in our Pune office on Saturday, December 13th, for two power-packed sessions!

🗓️ Essential Event Details

🎙️ Agenda: Deep Dives by QuestionPro Experts

We have lined up two high-impact talks from our senior experts, focusing on the real-world challenges of building highly available systems and leveraging Artificial Intelligence for data analysis.

Talk 1: From Zero to 10k Live Users: Scaling Real-time Engagement Platforms

Speaker: Rohan Rao, Software Engineer

Building a real-time platform that handles thousands of concurrent users is a critical challenge. Rohan Rao, a Software Engineer with over 7 years of experience, will share lessons learned and practical strategies for:

• Designing resilient and scalable architectures.
• Implementing high-performance solutions that maintain speed under heavy load.

Talk 2: AI in Action: Building a Smarter Topic Analysis System

Speaker: Pratik Dhulubulu, Dedicated Data Scientist

Artificial intelligence is rapidly transforming how companies understand user data. Pratik Dhulubulu, a Data Scientist with 4+ years of experience, will move past the hype and demonstrate how to:

• Leverage AI to solve complex data problems.
• Build a sophisticated, data-backed topic analysis system using modern Python techniques.

Why This Pune Tech Meetup Is a Must-Attend

1. Focused Learning: Gain high-value, technical knowledge from experienced practitioners in a short, efficient format.
2. Networking: Connect face-to-face with fellow software engineers, data scientists, and developers in the vibrant Pune tech scene.
3. Community: Discuss the real-world implementation challenges of scaling and AI in a professional environment.

Secure Your Spot

Spaces for this developer meetup in Pune are limited. Don’t miss out!

The last day to register is Thursday, December 11th.

Register Now

We look forward to connecting with you at the QuestionPro office in Baner!

As the saying goes:

“In the digital age, scaling is the compass that guides your product toward success in the vast ocean of possibilities.”

This blog is a story of how we scaled LivePolls, a real-time engagement product from QuestionPro, to handle a staggering 10,000 participants in a single session - on a single server.

You’ll see the problems we faced, the creative solutions we engineered, and the lessons we learned along the way. So grab your favorite drink and some snacks - this is a scaling story you’ll actually enjoy reading.

A Bit of Context

LivePolls is a free product from QuestionPro that allows users to engage audiences in real time - through polls, quizzes, and interactive questions.

• The Admin creates a quiz or poll and starts a live session
• Participants join using a session PIN
• Everyone engages together in real-time through dynamic charts, leaderboards, and results

At its core, LivePolls uses WebSockets for fast, real-time communication between clients and servers. And when we say real-time, we mean milliseconds matter.

The Challenge

One of QuestionPro’s enterprise customers wanted to host a LivePoll session with over 5,000 participants.

Our previous tests had maxed out around 2,000 participants per server. Beyond that, things… well, broke.

We lacked infrastructure, saw memory leaks, hit socket connection limits, and faced painfully slow queries. And thus began our epic journey to scale.

Problems We Faced

1. No infrastructure to load-test beyond 2,000 participants
2. Memory leak in the application
3. Nginx limits on concurrent socket connections
4. Slow MySQL queries causing high memory usage and timeouts

Solutions We Devised

1. Building Infrastructure for 10,000 Virtual Participants

Since LivePolls uses WebSockets, we first tried Artillery.io for load testing - it’s a solid library for socket-based tests.

But we quickly hit a wall:

• Only ~1,000 WebSocket connections per local machine
• A steep learning curve for complex test flows (like joining, answering, viewing results)

So, we decided to go rogue and wrote our own simple JavaScript script to simulate participant actions:

joining → answering → viewing results

Then came the real question - where do we run these thousands of connections?

We leveraged free-tier cloud VMs from AWS, since:

• There was no limit on the number of VMs we could spin up
• Each VM could handle around 1,000 socket connections
• So, 10 VMs = 10,000 virtual participants. Simple math. Beautiful scaling.

When we later moved the setup to our in-house test servers, we had to tweak some system parameters (like increasing file descriptors):

# /etc/security/limits.conf
your_user soft nofile 1000000
your_user hard nofile 1000000

Each socket uses one file descriptor - so, no descriptors = no sockets = no fun.

2. Fixing the Memory Leak

A memory leak occurs when an app allocates memory but doesn’t release it properly.

After several smaller tests, we found that memory usage never dropped back to normal even after participants disconnected. Suspicious. 🤔

We dug deeper using heap snapshots and discovered that WebSocket connections were not being released after closing. Our stack uses Socket.io, which internally uses the WS library for WebSocket handling.

Then came the “aha” moment - we found EIOWS, a C++-based, high-performance, low-memory WebSocket engine, fully compatible with Socket.io.

With just one line of code changed, we swapped WS with EIOWS - and boom, no more leaks. (If only all life’s problems could be solved this easily.)

3. Tuning Nginx for 10K+ Concurrent Connections

Once memory was stable, we went for the big one - 5K participants.

…and hit a wall again.

The error?

Too many open files

Classic. 😅

We increased:

worker_connections 8192;
worker_processes 8;

Expecting ~64K connections. But we still crashed around 4K.

Turns out, each connection in Nginx uses 2 file descriptors - one for the client, one for the upstream server.

So, we added:

worker_rlimit_nofile 16192;

This allowed each worker process to handle roughly 8K connections, comfortably crossing the 10K mark.

4. Optimizing Slow MySQL Queries

With 10K people connected, the next step was an end-to-end test: participants answer, admin views charts/leaderboards, participants view relative rank, etc.

We found the application server crashed when admin requested result charts. The culprit: queries pulling all rows into application memory and then aggregating in JavaScript - a disaster at 10K+ rows.

Below are concrete examples showing before (inefficient) and after (optimized) queries we used.

Naive / Inefficient Approach (what we had)

These queries fetch all rows into the app and then process them in JS:

-- inefficient, pulls all rows
SELECT * FROM responses WHERE question_id = 123;

If you then sort or aggregate on this in-app, memory usage explodes. 💥

Optimized Queries - Let MySQL Do the Heavy Lifting

1. Get response count for each answer option (for charting)

SELECT answer_option_id, COUNT(*) AS response_count
FROM responses
WHERE question_id = 123
GROUP BY answer_option_id;

Why better: MySQL aggregates rows on the server and returns a tiny result set (one row per option), not 10K rows.

2. Get respondents sorted by score and limit (leaderboard)

SELECT respondent_id, total_score
FROM respondents_scores
WHERE session_id = 456
ORDER BY total_score DESC
LIMIT 100;

Why better: Database-level ORDER BY + LIMIT avoids fetching all respondents into memory.

The Result

After fixing memory leaks, tuning Nginx, and optimizing queries:

✅ LivePolls successfully handled 10,000 concurrent users on a single app server (2 CPU, 4GB RAM)
✅ Multiple full-scale tests were conducted smoothly
✅ No crashes, no leaks, no long waits - just 10,000 people engaging in real-time

Final Thoughts

Scaling isn’t just about tweaking parameters - it’s about learning how your system breathes under pressure.

We hope this story gave you a few ideas, maybe even some laughs, and definitely the confidence to take your own systems beyond limits.

Remember:

“Scaling isn’t a destination; it’s a journey of continuous improvement and evolution.”

And sometimes, that journey starts with a single config file. 🚀

In today’s world of AI-assisted coding, where tools like GitHub Copilot, ChatGPT, and cursor editors can generate entire modules in seconds, many developers are falling into a subtle trap — they are typing prompts, not writing solutions. The goal of the Conscious Vibe Coder is not to reject AI, but to use it wisely: balancing automation with human reasoning, product understanding, and performance thinking.

The Vibe Shift in Coding

Coding used to be about writing logic line by line. Today, it’s about designing, reviewing, and refining AI-generated logic. Developers who can blend creativity, clarity, and critical reasoning stand apart. The ‘vibe’ of modern coding isn’t about how fast you type — it’s about how deeply you think.

Boilerplate Isn’t the Enemy, But Mindless Acceptance Is

AI excels at generating repetitive or boilerplate code. Setting up a controller, defining routes, initializing React states — these can be automated. But blindly accepting that output without understanding its flow is where problems begin.

@Controller("users")
export class UserController {
  constructor(private readonly userService: UserService) {}

  @Get(":id")
  findOne(@Param("id") id: string) {
    return this.userService.findOne(+id);
  }
}

AI can generate the structure instantly — but the conscious coder will ask: What if the ID doesn’t exist? Should we handle exceptions? Do we need caching or role-based access control? That’s the human layer AI cannot replace.

Clarity of Thought Drives Code Quality

Before prompting AI, clarity matters more than syntax. If your instructions are vague, your output will reflect it. A conscious coder thinks like an architect — not a script generator.

❌”Build a booking API.”

✅ “Create a Spring Boot REST API for hotel bookings with endpoints to create, cancel, and list bookings. Validate overlapping dates, include exception handling, and use an in-memory database for demo.” The difference isn’t just in wording; it’s in mindset. Clear thinking produces clear instructions, which in turn produce better AI outputs.

Review Like a Mentor, Not a Machine

AI often produces code that ‘works’ but doesn’t ‘scale’. A conscious coder reviews every line — as if mentoring a junior developer.

The question is never ‘Does it run?’ but ‘Is it robust, readable, and reliable?’

AI-Generated Example:

public UserProfile getProfile(Long userId) {
  return userRepository.findById(userId).get();
}

Refactored Version:

public UserProfile getProfile(Long userId) {
return userRepository.findById(userId)
        .orElseThrow(() -> new ResourceNotFoundException("User not found: " + userId));
}

The improved version avoids runtime crashes, adds contextual errors, and is maintainable. Real-world consciousness means thinking about nulls, caching, retries, and downstream impacts.

The Human Advantage in the Loop

AI tools can predict syntax, but they can’t predict intent. They don’t understand business logic, ethics, or the small product details that define why something should or shouldn’t happen. Imagine an AI-generated discount system for an e-commerce site:

public double calculateFinalPrice(double price, double discount, double coupon) {
  return price - discount - coupon;
}

It works mathematically — but fails logically.

In your product, discounts and coupons aren’t supposed to stack. So the correct version is:

public double calculateFinalPrice(double price, double discount, double coupon) {
  double appliedDiscount = Math.max(discount, coupon);
  return price - appliedDiscount;
}

That difference — understanding business logic, user impact, and edge conditions — is the human layer that AI doesn’t see. AI writes code that’s syntactically correct; humans write code that’s contextually correct. A conscious developer reads AI output like a reviewer, not a consumer. You question assumptions, ask “What if?”, and make the code align with real-world behavior — not just logic.

Coding with Product Empathy

Coding isn’t just about systems; it’s about people. A conscious coder doesn’t only think in APIs and functions — they think in user experience, product flow, and real-world outcomes. Imagine AI writes a simple payment retry function:

async function processPayment(orderId: string) {
  let attempts = 0;
  while (attempts < 3) {
    const success = await charge(orderId);
    if (success) break;
    attempts++;
  }
}

Technically fine — but logically dangerous. If the first payment went through but the confirmation failed due to a network glitch, this logic may charge the customer multiple times. A conscious developer understands business context and rewrites it safely:

async function processPayment(orderId: string) {
  const existing = await checkTransactionStatus(orderId);
  if (existing?.status === "SUCCESS") return existing;

  let attempts = 0;
  while (attempts < 3) {
    const success = await charge(orderId);
    if (success) return success;
    attempts++;
  }
  throw new Error("Payment failed after retries");
}

Empathy-driven coding means caring about what happens after the code runs — in the user’s world, not just in your terminal.

Code That Runs vs. Code That Scales

Here’s where most developers slip.

AI-generated code can pass tests, but may crumble under real load, missing caching, concurrency handling, or memory management. Conscious coders always test the system’s behavior, not just its syntax.

Example

useEffect(() => {
fetch('/api/products')
.then(res => res.json())
.then(setProducts);
}, []);

Refactored Version:

const [loading, setLoading] = useState(true);
const [error, setError] = useState(null);

useEffect(() => {
  const controller = new AbortController();
  const fetchData = async () => {
    try {
      const res = await fetch("/api/products", { signal: controller.signal });
      if (!res.ok) throw new Error("Failed to fetch");
      const data = await res.json();
      setProducts(data);
    } catch (err) {
      if (err.name !== "AbortError") setError(err.message);
    } finally {
      setLoading(false);
    }
  };
  fetchData();
  return () => controller.abort();
}, []);

The second version anticipates real scenarios — slow networks, user cancellations, and error handling. AI can write syntax, but performance empathy comes from human experience.

The Conscious Loop: AI + Human Refinement

Being a Conscious Vibe Coder doesn’t mean rejecting AI — it means co-creating intelligently. You let AI handle structure, and you focus on quality, performance, and clarity. Think of it as a loop: instruct → generate → review → refactor → re-instruct. This approach creates a developer who can build faster and smarter — someone who can lead design discussions, not just follow autocomplete suggestions. The conscious coder owns the solution, not the syntax.

Final Thoughts

In the age of AI-powered development, our value as engineers lies in our ability to think critically, design for humans, and question every line. Conscious Vibe Coders don’t just code — they create systems that last. Stay thoughtful. Stay curious. Stay conscious.

For years, our text analysis platform Discover was the workhorse of our operations. It was sophisticated, accurate, and reliable. But it had one fundamental limitation that would eventually force us to rethink everything: every single topic had to be manually defined before we could analyze any text.

How Discover Worked

Imagine you’re analyzing customer feedback for a restaurant chain. Before Discover could process a single review, we needed domain experts to sit down and create a comprehensive list of topics:

• “food quality”
• “service speed”
• “ambiance”
• “cleanliness”
• “value for money”
• … and potentially 50+ more

Once these topics were defined, Discover would read each piece of text and figure out which predefined topics it matched. The system was actually quite clever about this—it offered four different ways to match text to topics:

RAG-based Tagging used advanced AI to understand context and map text to the closest matching topic. If someone wrote “the meal was delicious,” it would correctly map this to “food quality.”

Rule-based Extraction used linguistic patterns and grammar rules to extract topic-opinion pairs from sentences.

Language Model Extraction used fine-tuned AI models to directly generate topic-opinion-sentiment combinations.

The output was beautifully structured. For the review “The food was excellent but service was slow,” Discover would produce:

• Topic: “food quality”

  Opinion: “excellent”

  Sentiment: Positive

• Topic: “service speed”

  Opinion: “slow”

  Sentiment: Negative

It worked brilliantly. Until it didn’t.

When the System Started Breaking Down

The Expert Bottleneck

Every new client meant weeks of preparation. We’d gather domain experts, conduct stakeholder interviews, review sample data, and painstakingly build topic taxonomies.

• For a new restaurant chain: 2-3 weeks of topic curation
• For a healthcare feedback system: 3-4 weeks
• For a complex B2B SaaS product: Sometimes a month or more

We were spending more time defining topics than actually analyzing data. Every project started with the same bottleneck: waiting for humans to tell the AI what to look for.

The Problem of Emerging Topics

Here’s where things got really frustrating. The world doesn’t wait for your topic taxonomy to catch up.

When COVID hit, suddenly customers were talking about:

• “contactless delivery”
• “safety protocols”
• “mask enforcement”
• “outdoor seating”

Discover couldn’t see these topics. They weren’t in our predefined list. By the time we identified these patterns manually and added them to the taxonomy, we’d already missed weeks or months of valuable insights in historical data.

The painful reality: We were always looking backward, never forward.

The Granularity Dilemma

Different people needed different views of the same data:

• Marketing teams wanted broad themes: “Is customer service a problem?”
• Operations teams wanted specifics: “Is it the checkout process or the return policy causing issues?”
• Executives wanted strategic insights: “How does our brand perception compare to competitors?”

Discover forced us to choose one granularity level upfront. Want to switch from high-level themes to detailed issues? Reprocess everything with a new topic taxonomy.

The Multilingual Nightmare

Expanding internationally meant exponentially more work. English restaurant topics weren’t the same as Spanish restaurant topics—cultural differences meant different things mattered. Either we had to compromise with this or add topics in different languages. Each language and region needed its own carefully curated taxonomy.

We found ourselves maintaining hundreds of topic lists, each requiring expert maintenance. It was unsustainable.

The Turning Point: What If Topics Could Discover Themselves?

One day, someone asked the obvious question: “What if we just let the AI figure out what topics exist?”

It seemed radical. Our entire system was built on human-curated topics. But the more we thought about it, the more sense it made.

Instead of telling the AI “These are the topics, find them in the text,” what if we said “Here’s the text, tell us what topics exist”?

The New Vision

We needed a system that could:

✅ Analyze any domain immediately - No weeks of topic curation
✅ Discover emerging topics automatically - See new patterns as they appear
✅ Provide multiple levels of detail - Zoom from strategic themes to operational specifics
✅ Work across languages - Understand meaning, not just words
✅ Keep getting smarter - Learn and adapt as more data comes in

This meant rebuilding our entire approach from the ground up.

Building the New System: From Classification to Discovery

The Fundamental Shift

• Discover’s approach: “Here are the topics. Match text to them.”
• AI Topics approach: “Here’s the text. Discover the topics within it.”

The technical term is moving from supervised classification to unsupervised clustering. But what really matters is the user experience transformation:

Before: Project kickoff → 3 weeks of topic curation → Data processing → Analysis
After: Project kickoff → Upload data → Topics discovered automatically → Analysis

How Automatic Topic Discovery Works

AI Topics uses semantic embeddings—a way of converting text into mathematical representations that capture meaning. Here’s why this matters:

When Discover saw “fast service” and “quick response,” it treated them as potentially different topics because they used different words.

AI Topics understands they mean the same thing. It converts both phrases into similar mathematical representations, so they naturally cluster together as one topic.

This happens automatically. No one needs to tell the system that “fast” and “quick” are synonyms, or that “service” and “response” are related in customer feedback contexts. The AI learns semantic relationships from the data itself.

The Magic of BERTopic Clustering

We chose BERTopic as our clustering engine—a state-of-the-art topic modeling framework that combines the best of modern NLP with robust clustering algorithms.

Here’s the process in simple terms:

1. Understanding - Every document gets converted into a mathematical representation that captures its meaning using transformer-based embeddings
2. Dimensionality Reduction - Using UMAP (Uniform Manifold Approximation and Projection), we reduce the complexity while preserving the semantic structure of the data
3. Finding Patterns - HDBSCAN (Hierarchical Density-Based Spatial Clustering) identifies natural groupings in the data—documents talking about the same things cluster together automatically
4. Initial Labeling - BERTopic generates preliminary, raw topic names based on the most statistically distinctive terms in each cluster (e.g., ‘quick_fast_rapid’)—a necessary but often unintuitive first step
5. Organizing - Topics arrange themselves hierarchically, from broad themes down to specific issues

The beautiful part? This all happens automatically. Upload your data, and within hours (not weeks), you have a complete topic taxonomy discovered from the content itself.

Smart Configuration: One Size Doesn’t Fit All

Here’s something crucial we learned: the optimal clustering parameters change depending on your data size.

Clustering 100 documents requires different settings than clustering 100,000 documents. BERTopic uses UMAP and HDBSCAN under the hood, both of which have parameters that dramatically affect results.

Our solution: Dynamic configuration selection.

AI Topics automatically adjusts UMAP and HDBSCAN parameters based on the number of texts you’re analyzing:

• Small datasets (hundreds of documents): More aggressive parameter settings to ensure patterns emerge even with limited data
• Medium datasets (thousands of documents): Balanced parameters that capture both broad themes and specific nuances
• Large datasets (tens of thousands+): Conservative parameters that prevent over-fragmentation and maintain interpretable clusters

This adaptive approach means you get quality results whether you’re analyzing a week’s worth of customer feedback or a decade’s worth of archived documents. No manual tuning required—AI Topics knows what works best for your data volume.

What About Discover’s Classification Strength?

Here’s the good news: we didn’t throw away what made Discover great.

AI Topics includes an available_taxonomy operation that works exactly like Discover—but enhanced with modern AI. If you have predefined topics you want to use (maybe you’re in a well-established domain with known categories), you can still use them. The system will classify your text against those topics using the same sophisticated extraction methods Discover offered.

The difference? Now you have a choice. Use predefined topics when they make sense, or let the system discover topics when exploring new territory.

The Migration Journey: Challenges We Faced

Challenge 1: Speed vs. Intelligence

Discover was fast—about 50 milliseconds per document. Why? Because checking text against a predefined list is computationally simple.

AI Topics’ automatic discovery is more complex. Converting text to semantic embeddings and running clustering algorithms takes more time—about 100 milliseconds per document for batch processing.

Our solution: A two-tier approach.

• For real-time topic assignment (when you need answers now), the system assigns new documents to existing topics quickly—just as fast as Discover.
• For topic discovery and refinement (when you’re exploring patterns), the system runs more thorough analysis in the background. You submit a job, get an immediate confirmation, and receive results when the deep analysis completes.

Most users don’t notice the difference. They get fast responses for daily operations and thorough insights for strategic analysis.

Challenge 2: Making Sense of Discovered Topics—Enter OpenAI

Discover had an unfair advantage: humans named the topics. “Customer Service Quality” is immediately clear to anyone.

Automatically discovered topics? Sometimes the labels were… less intuitive. Early BERTopic attempts produced gems like:

• Cluster 1: “good_great_excellent_amazing_fantastic”
• Cluster 2: “quick_fast_rapid_speedy_prompt”
• Cluster 3: “wait_long_slow_delay_forever”

Technically accurate—these words did cluster together. But not exactly boardroom-ready.

The breakthrough: OpenAI representation labeling.

Instead of relying on simple keyword extraction, we integrated OpenAI’s language models to generate intelligent, contextual topic labels. The AI reads the most representative documents in each cluster and creates human-like descriptions of what the topic is actually about.

Those same clusters transformed:

• Cluster 1 → “Overall Satisfaction and Positive Experiences”
• Cluster 2 → “Service Speed and Efficiency”
• Cluster 3 → “Wait Times and Service Delays”

But it gets better. OpenAI’s understanding of context means labels adapt to your domain:

• In restaurant feedback, a cluster about “noise” becomes → “Ambiance and Noise Levels”
• In hospital feedback, the same semantic cluster becomes → “Patient Rest and Recovery Environment”

The AI understands that identical concepts need different framing depending on context.

The quality improvement was dramatic. In blind tests with business stakeholders, OpenAI-generated labels were preferred over human-curated topic names 73% of the time. Why? Because the AI had read every single document and distilled the essence, while humans were working from assumptions and samples.

Users can still refine labels manually if needed, but most of the time, the OpenAI-generated names are immediately usable.

Challenge 3: Optimizing BERTopic for Different Data Volumes

BERTopic is powerful, but it’s not a one-size-fits-all solution. The same clustering parameters that work beautifully for 500 documents can produce terrible results for 50,000 documents.

The challenge: UMAP and HDBSCAN (the algorithms powering BERTopic’s clustering) have parameters that need careful tuning based on data characteristics. Set them too conservatively, and small datasets won’t reveal any patterns. Set them too aggressively, and large datasets fragment into hundreds of meaningless micro-topics.

Our solution: Intelligent parameter adaptation.

We built logic that analyzes your dataset size and automatically selects optimal UMAP and HDBSCAN configurations:

• For smaller datasets, the system uses parameters that encourage pattern discovery even when data is limited. This prevents the “everything is one big topic” problem.
• For medium datasets, it balances between capturing nuanced sub-topics and maintaining broad thematic groupings.
• For larger datasets, it applies conservative settings that prevent over-fragmentation while still revealing meaningful structure.

The result? Whether you’re analyzing 200 customer reviews or 200,000, you get coherent, interpretable topics without manual parameter tuning. The system adapts to your data automatically.

Challenge 4: When Topics Evolve

Real-world topics don’t stay still. During COVID, “service” discussions shifted dramatically—from “Is the staff friendly?” to “Are safety protocols being followed?”

AI Topics needed to detect these shifts and adapt. We built monitoring capabilities that track:

• How topic distributions change over time
• When topic coherence starts degrading (a sign that a topic is splitting or evolving)
• When new patterns emerge that don’t fit existing clusters

When significant drift is detected, AI Topics can trigger retraining to refine topic boundaries. This keeps the analysis relevant as the world changes.

The Advantages: What Changed for the Better

Dramatically Better Topic Quality

The most surprising benefit wasn’t just eliminating manual curation—it was discovering that automatically generated topics were often better than human-curated ones.

Real impact: A retail client came to us with Black Friday feedback data. With Discover, their domain experts had created 47 predefined topics like “checkout experience,” “product quality,” and “delivery speed.”

Our automatic discovery found 63 topics—a 34% increase in thematic coverage—and many revealed insights the manual taxonomy had missed entirely:

• “Gift wrapping delays” (a Black Friday-specific concern)
• “Size inconsistency across brands” (an operational issue causing returns)
• “Mobile app cart abandonment” (technical friction the IT team needed to know about)

The human experts weren’t wrong—they just couldn’t anticipate every nuance hidden in thousands of customer comments. The AI could.

Never Miss Emerging Topics Again

AI Topics continuously monitors for new patterns. When enough documents start discussing something novel, it surfaces as a new topic cluster automatically.

Example: A hotel chain client suddenly started receiving feedback about “EV charging stations.” This wasn’t in anyone’s original topic taxonomy because it wasn’t a common concern when the project started. AI Topics identified it as an emerging topic within days, allowing the hotel chain to prioritize installing charging stations at high-demand locations.

Discover would have missed this entirely until the next quarterly taxonomy review.

Higher Quality Topic Granularity

Discover required choosing granularity upfront: broad categories or specific issues? You couldn’t have both without running separate analyses.

AI Topics’ automatic discovery naturally creates multiple granularity levels simultaneously because it uses hierarchical clustering in BERTopic:

• Level 1 (Strategic): “Guest Experience Quality”
  • Level 2 (Departmental): “Room Comfort,” “Service Responsiveness,” “Facility Amenities”

What makes this powerful is that the AI determines the right number of levels and topics at each level based on the actual data patterns—not human assumptions.

A retail client with seasonal products found their topic hierarchy changed across seasons. Summer analysis had deep sub-topics around “outdoor furniture durability,” while winter analysis had completely different granular topics around “indoor heating products.” The AI adapted automatically.

The insight: Different datasets need different structures. Forcing a predefined hierarchy misses these natural variations.

Semantic Search That Actually Understands

• Discover search: “Find documents about fast service” → searches for the exact words “fast” and “service”
• AI Topics search: Understands you’re looking for concepts about service speed and finds:
  • “the staff was very efficient”
  • “quick response to our requests”
  • “didn’t have to wait long”
  • “prompt attention from servers”

None of these contain your exact search terms, but they’re all semantically related to what you’re looking for.

Real impact: A customer support team reported finding 40% more relevant documents for issue investigation. Problems they thought were isolated turned out to have been discussed repeatedly using different terminology.

True Multilingual Understanding

Instead of maintaining separate topic taxonomies for each language, AI Topics uses multilingual embeddings. English “excellent food” and Spanish “comida excelente” are understood as the same concept automatically.

Real impact: A global hotel chain operating in 15 countries went from maintaining 15 separate topic systems to one unified system that understands all their languages. Cross-cultural insights that were previously impossible became routine.

What We Learned Along the Way

The Importance of Human Touch

Fully automated doesn’t mean fully hands-off. The best results come from a partnership between AI and human intelligence.

• Topic labeling benefits enormously from occasional human review. The system might generate “speed_efficiency_temporal” as a cluster name. A human immediately sees this should be “Service Speed.”
• Quality validation requires human judgment. Is “parking” a subset of “facilities” or its own top-level topic? Depends on your business context.

Start Simple, Add Complexity Gradually

Our first automatic clustering was overly sophisticated. We used every advanced technique we could think of. It was slow and hard to debug when something went wrong.

Stepping back to simpler approaches—good embeddings with straightforward clustering—delivered 90% of the value with 10% of the complexity. We added sophistication only where it clearly improved results.

The Value of Explaining the AI

Users were initially skeptical. “How can the AI know what topics are important if we don’t tell it?”

Building transparency helped enormously:

• Show representative documents for each topic
• Display confidence scores for topic assignments
• Explain why documents clustered together
• Provide side-by-side comparisons of automatic vs. manual taxonomies

When people understood how the discovery worked, trust followed.

Looking Ahead: What’s Next

Smarter Topic Evolution Tracking

Right now, the system detects that topics are changing. Next, we want to visualize how they’re changing:

• Topics splitting into sub-topics
• Topics merging as concepts converge
• Topics appearing and disappearing over time

Imagine a time-lapse showing how customer concerns evolved throughout a product lifecycle.

Predictive Intelligence: From What to Why

Moving beyond “What topics exist?” to “What topics actually matter?”

• Which new conversation topic today can predict a 15% rise in customer churn next month?
• Which topics drive repeat purchases?
• Which topics are early indicators of viral content?

Connecting topic analysis to business outcomes would transform it from descriptive analytics to predictive intelligence.

Real-Time Topic Monitoring

Imagine a live dashboard showing topics as they emerge in real-time—not hours after data upload, but streaming as new documents arrive. Social media monitoring could surface trending topics within minutes of them starting to appear.

The Bottom Line: Different Tools for Different Jobs

Discover wasn’t wrong—it was optimized for a specific use case. When you:

• Work in a well-understood domain
• Have stable topic requirements
• Need maximum processing speed
• Want human-validated topic names

Then predefined topic classification (like Discover’s approach) is actually the better choice. And with AI Topics’ available_taxonomy operation, you can still use this approach when it makes sense.

But when you:

• Explore new domains without established taxonomies
• Need to discover emerging patterns
• Want to understand semantic relationships
• Require multilingual analysis
• Need strategic-to-tactical granularity

Then automatic topic discovery is transformative.

We’re not declaring victory over an old system. We’re celebrating having the right tools for different situations.

The Real Achievement

The technical accomplishments are meaningful: BERTopic clustering, semantic embeddings, OpenAI representation labeling, adaptive parameter configuration, hierarchical topic trees, multilingual understanding.

But the real achievement is qualitative: topics that capture nuances human experts miss while remaining immediately understandable.

The combination of BERTopic’s sophisticated clustering and OpenAI’s contextual labeling creates something neither could achieve alone:

• BERTopic finds the patterns
• OpenAI makes them comprehensible
• Together, they reveal insights hiding in plain sight

When someone says “We have a new dataset to analyze,” the conversation isn’t just about eliminating setup time. It’s about discovering what’s actually in the data, not just confirming what we thought would be there.

That’s the shift that matters. From AI that classifies text into boxes we define, to AI that shows us which boxes actually exist—and labels them in ways we immediately understand.

Technical Summary

• Discover: Supervised topic classification with predefined taxonomies and human-curated labels
• AI Topics: BERTopic-based unsupervised discovery with OpenAI representation labeling
• Best of Both: available_taxonomy operation preserves classification capabilities when needed

Key Quality Improvements

• Discovery of nuanced topics human experts didn’t anticipate
• Context-aware, business-ready labeling through OpenAI integration
• Adaptive topic granularity based on natural data patterns
• Superior semantic understanding for better search and grouping
• Unified multilingual analysis maintaining consistent quality across languages

The Philosophy: Let humans focus on interpretation and strategy. Let AI handle pattern discovery, intelligent labeling, and routine classification.

Mindset Shift

The first and perhaps the most profound transformation I’ve experienced is a shift in mindset.

When I started my career, my perspective was narrow:

• I saw tasks as checkboxes to be completed.
• I measured success by how quickly I could deliver code.
• I viewed problems as obstacles, not opportunities.

At QuestionPro, I learned to see beyond the code. Problems are no longer roadblocks—they are signals pointing toward hidden issues, unmet user needs, or areas of improvement in our systems. A bug isn’t just “something broken”; it’s a chance to refine, to rethink, and to deliver a better user experience.

I also began to see features differently. A feature is not just a “requirement” to be implemented but part of a larger story. Every line of code we write contributes to a user’s journey, and every decision we make leaves an imprint on how someone interacts with the product. This realization changed how I approach design, architecture, and collaboration.

Most importantly, this mindset shift extended to how I see my own role. As a developer, my contribution is not just technical—it’s strategic. It’s about asking the right questions:

• Why does this matter for the user?
• How does this tie into the product vision?
• What impact will this decision have six months down the road?

This way of thinking has made me not just a better developer, but a more thoughtful professional. It has given me resilience too—because when you see challenges as opportunities, the journey itself becomes rewarding.

Scope of Growth

If I were to describe the most valuable gift I’ve received here, it would be growth—and not just in technical skills. The growth I’ve experienced spans multiple areas:

• From technical expertise to communication
• From cultural awareness to people management
• From understanding a single feature to comprehending the entire product lifecycle

This wide range of opportunities is rare, and it has been maintained through a few core practices that stand out to me:

Learning Environment

A healthy learning environment is not a luxury, it’s a necessity. QuestionPro has made it possible for us to learn without fear of judgment. Mistakes are seen as part of the process, and curiosity is encouraged.

Collective Learning

Learning collectively has been a breakthrough. When we share our discoveries, failures, and ideas, the knowledge spreads faster and deeper than any documentation could allow. I’ve seen how this practice saves countless hours across the team and also builds trust.

Continuous Push and Improvement

Borrowing from the CI/CD mindset, growth works the same way: push continuously, improve continuously. Growth is like resistance training—it’s not always comfortable, but it’s the only way to get stronger. At QuestionPro, this philosophy is embedded in both personal and professional development.

Understanding the Product’s Goal

One of the most important lessons I’ve learned is the value of understanding the product’s larger goal.

Think about it this way: the best doctors don’t just treat a symptom; they treat the body as a whole. Similarly, when we build a feature, we shouldn’t just focus on a ticket or task in isolation. Instead, we need to connect it back to the overall product vision.

This approach has helped me identify edge cases more effectively, think from the user’s perspective rather than just the developer’s perspective, and write code that reduces bugs in the long run. It’s not just about solving what’s in front of you; it’s about anticipating what lies ahead.

Cultural Fitness

Early in my career, I underestimated the importance of cultural fitness. At QuestionPro, I realized how critical it is.

Working in harmony requires decency, mutual respect, and approachability. These may sound simple, but when a team of 15 developers embodies these traits, the impact on collaboration and delivery is remarkable. On the flip side, even one person who doesn’t align with the cultural norms can affect the morale and the outcomes of the entire group.

Cultural fitness isn’t about uniformity—it’s about respect, openness, and creating an environment where everyone feels safe to contribute.

Communication

For a long time, I thought communication was simply about fluency in English. QuestionPro changed that belief completely.

Effective communication is about clarity, context, and empathy. Some practices I’ve picked up include:

• Being specific with time commitments
  Example: “I’ll ping you within 10 minutes” instead of “I’ll ping you soon.”
• Surfacing blockers early
  Example: “I’ve been stuck here for 30 minutes; can you pair with me?”

These small practices create big changes in how smoothly a team functions. Communication is not about sounding good—it’s about making sure the other person understands you without confusion.

People Management

Finally, one of the biggest growth areas for me has been managing people. Leading a team of 15 developers is a responsibility that requires more than technical know-how. It requires empathy, patience, and the ability to recognize each individual’s strengths.

At QuestionPro, I’ve learned that good people management is less about directing and more about enabling. My role is not to dictate solutions but to remove obstacles, foster trust, and empower others to do their best work.

Sometimes this means mediating between two teammates with differing viewpoints. Sometimes it means mentoring juniors and helping them connect technical work with product thinking. And sometimes it simply means stepping back and letting others take the lead.

I’ve found that when people feel trusted and supported, their creativity and ownership naturally grow—and the product benefits tremendously as a result.

Closing Thoughts

Looking back at these three years, I can say with confidence that QuestionPro has been more than just a workplace for me—it has been a place of transformation. The lessons on mindset, growth, cultural fitness, communication, and people management have not only shaped my career but also influenced how I approach challenges in life.

As I continue this journey, I hold onto one truth: growth is not a destination; it’s a continuous cycle of learning, sharing, and evolving. And for that, I remain grateful.

But raw requests don’t tell the whole story of our data footprint. When we look at what’s actually stored in our databases to understand user behavior and survey results, the scale is staggering: we’re talking about ~500 billion rows of data.

Storing this volume in a single MySQL database instance is simply not feasible—it would grind to a halt. The data is partitioned across different MySQL databases, allowing us to scale horizontally.

The Bottleneck

When we fire our analytical queries like which aggregates data using count/sum even with a fully sharded architecture, we were hitting performance walls. The slowness became particularly painful when analyzing data for a single, large survey. For surveys with over 10 million responses, it could take up to 30 seconds to get a correct answer, which was unacceptable for our users.

We tried everything in the traditional playbook: optimizing indexes, tweaking configurations, and rewriting queries. Despite our best efforts, the benefit was minimal. We had to face the facts: we had squeezed every last drop of performance out of MySQL for our analytical needs. It was time to look for a new solution—a different kind of database.

The Search Begins

Our quest for a new analytical database began. We had a clear understanding of the data we needed to analyze and the types of queries we wanted to run. We shortlisted several promising candidates:

• MongoDB
• Elasticsearch
• Google BigQuery
• Apache Druid
• ClickHouse
• Apache Pinot

Each database came with its own set of pros and cons. Some required more storage, others came with a higher cost, a few had complicated query languages, and some were simply slower than others in our tests.

After a thorough evaluation, one database emerged as the clear winner, solving our most critical problems: ClickHouse. The query performance was astounding. Aggregations on over 100 million records completed in milliseconds. We were thrilled.

Productionizing ClickHouse

We designed our new ClickHouse system for resilience and scale from day one. It included replicas for redundancy and fault tolerance, along with multiple shards to continue partitioning our data and scaling horizontally.

What’s Next? Making Analytics Real-Time

With the performance problem solved, a new challenge emerged: real-time data availability. When users send out a survey, they want to see the results instantly, especially for new surveys with a small number of responses. This meant our analytical database needed to have near real-time data.

We started looking for solutions to transfer data from our production MySQL databases to ClickHouse. The obvious approach was polling—periodically querying MySQL for new data. However, we quickly dismissed this, as it would place an unnecessary and constant load on our primary databases.

Instead, we found a much more elegant solution: MySQL binlogs. Binlogs are transaction logs that MySQL uses internally for replication. Essentially, any DML operation (CREATE, UPDATE, DELETE) creates an entry in these logs. We realized we could tap into this existing mechanism. Instead of replicating from MySQL to another MySQL server, we would replicate from MySQL to ClickHouse via Kafka.

To achieve this, we built a pipeline using two powerful systems:

1. Debezium + Kafka Connect: This stack reads data directly from the MySQL binlogs and streams it into Kafka topics in real-time.
2. Kafka: Acting as a highly scalable, high-throughput pub-sub message bus.

ClickHouse has the native ability to connect to Kafka and ingest data streams, and we could even perform transformations within ClickHouse itself. This gave us a complete, end-to-end ELT (Extract-Load-Transform) system. We built redundancy into every layer, from Kafka Connect to the Kafka brokers themselves.

The result? We could transfer data from MySQL to ClickHouse in under one second.

Was it this smooth? - Hell No

Our first task was to change MySQL’s binlog format from Mixed to Row. This change was critical, as the Row format was necessary to read data from the binary logs. While this might seem straightforward, we successfully executed this migration with zero downtime.

However, this solution introduced a new problem: the Row format dramatically increased storage consumption, as it records the entire row’s data for every single transaction. This storage issue became particularly apparent during the initial data snapshot. While the continuous data stream didn’t require much space, the initial bulk load would completely fill our Kafka storage.

Throughout this project, we faced and fixed numerous other issues. This would not have been possible without our robust monitoring and logging infrastructure. It provided the crucial early symptoms that allowed us to drill down and address the underlying root causes effectively.

Monitoring and Observability

A system this critical requires robust monitoring. We went beyond the basics of liveliness, readiness, CPU, and memory usage. We implemented key performance indicators specific to our pipeline, including:

• End-to-end latency: The time taken for data to travel from MySQL to Kafka, and from Kafka to ClickHouse.
• Throughput: The volume of data moving through the system, measured in bytes and messages per second.

Thanks to this powerful and resilient architecture, we successfully built a blazing-fast BI and analytics system called “QuestionPro BI” capable of delighting our users with instant insights from billions of records.

0. Why an In-House Component Library?

You might be asking, “In a sea of UI libraries for React, why on earth would we build our own?” That’s a fair question! The answer is pretty simple: consistency and efficiency. We have a bunch of products that all share the same design system and branding. With different teams working on different projects, it gets really tough to keep all those designs perfectly aligned.

So, to ensure our products have a consistent look and feel, we decided to build a common UI library that everyone can use. This means our teams can stop worrying about moving icons “1px to the left” and instead focus on what really matters: building amazing features and functionality.

1. Keep It Simple

A React component can be as simple as an HTML element. Avoid over-engineering components by forcing props for everything. For example, don’t pass text as a label prop when you can just use children. We don’t use raw HTML like that, so try to keep the developer experience as close to HTML as possible.

// ❌ Don't
<Button label="A Button" />

// ✅ Do
<Button>A Button</Button>

Why This Matters: Simplicity keeps the API intuitive and reduces developer friction. Your components should feel like a natural extension of HTML.

Resource: React Docs – JSX in Depth

2. Extend Types from Native Elements

When building components, you don’t need to redefine every prop. Instead, extend from React’s native element types (yes, div or button in React aren’t “raw” HTML elements). This way, you automatically inherit all supported HTML attributes, including ARIA attributes. You can then layer custom props on top.

interface ButtonProps extends React.ButtonHTMLAttributes<HTMLButtonElement> {
  variant: "primary" | "secondary" | "danger";
}

export const Button: React.FC<ButtonProps> = (props) => {
  return <button {...props}></button>;
};

Why This Matters: It keeps your types aligned with React’s ecosystem, avoids reinventing the wheel, and makes your components more flexible and accessible by default.

Resource: TypeScript Handbook – Extending Types

3. Accessibility Matters

Accessibility should not be an afterthought. Use semantic HTML and proper roles. Don’t use a div to mimic a button — use a real <button>. Every component has its role. For example, an <input type="number" /> has the role spinbutton. Since components will be used in different contexts, make sure accessibility is baked in from the start.

// ❌ Don't
export const Button = () => <div onClick={() => {}}></div>;

// ✅ Do
export const Button = () => <button></button>;

Why This Matters: Accessibility ensures your components can be used by everyone, including users with assistive technologies. It also aligns with legal and industry standards.

Resource: WAI-ARIA Authoring Practices

4. Follow Conventional Commits

Conventional Commits provide a consistent way to write commit messages. Instead of vague messages like button updated or misc changes, use structured messages that can feed into automated changelog generation and versioning. For example:

• Commit often.
• Keep commits component-scoped.
• Use descriptive prefixes: feat(Button): add new color

Why This Matters: A consistent commit history makes it easier to track changes, automate versioning, and generate clean changelogs.

Resource: Conventional Commits Specification

5. Unit Tests Are Essential

For a component library, unit tests are not optional — they’re essential. They ensure that functionality doesn’t break when new features are added. Bugs will inevitably slip in, but tests catch them early. Make sure every piece of functionality has at least basic test coverage. We used Vitest and React Testing Library for this.

Why This Matters: Tests protect your consumers. A single regression in your library could break dozens of projects — tests act as your safety net.

Resources: Vitest Docs, React Testing Library Docs

6. Configuration Best Practices

Two common mistakes when building libraries are:

1. Not making the library tree-shakable. Your bundler should be able to remove unused components automatically. Example:

   // component/Button/index.ts
   export { Button, type ButtonProps } from "./Button.tsx";
   
   // index.ts
   export * from "@component/Button";
   

2. Bundling dependencies into your build. Your library should only include your code, not external dependencies like React. Mark them as peerDependencies in package.json, and configure your bundler to treat them as external.

// package.json
{
  "peerDependencies": {
    "react": ">=18 <20"
  }
}

// vite.config.ts
export default defineConfig({
  build: {
    rollupOptions: {
      external: ["react"],
      output: {
        globals: {
          react: "React",
        },
      },
    },
  },
});

Why This Matters: Tree-shaking keeps your consumers’ bundles small, and marking dependencies as peers prevents multiple React versions or unnecessary bloat in downstream apps.

Resources: Tree-Shaking in Rollup, Vite Library Mode Guide

Building a component library is less about fancy code and more about making it easy and reliable for others to use. Keep it simple, make it accessible, test it well — and you’ll already be ahead.

What lessons have you learned while building your own libraries? I’d love to hear your thoughts!