Product Decisions About AI: Quality, Cost, Latency, Risk

The Three-Way Tradeoff That Governs Every AI Feature

Almost every product decision about an AI feature reduces to three axes:

Quality: How good is the output? How often is it right, helpful, appropriate? Cost: How much does it cost to run? (LLM API costs, compute costs, human review costs) Latency: How fast does it return a result?

These three axes are in fundamental tension. You can usually improve one by sacrificing another:

• Better quality → higher cost (more expensive model, more processing steps)
• Lower cost → lower quality (cheaper model, fewer retrieval steps)
• Lower latency → lower quality (skip the reranker step, use a faster but less accurate model)

Product decisions about AI features are choices about where to land on this three-dimensional tradeoff.

Defining "Good Enough"

The most important product decision is not "how good can we make this" but "how good does this need to be to be useful." These are different questions with different answers.

A customer support FAQ bot does not need to be right 99.9% of the time. It needs to be right often enough that it reduces ticket volume and users are not frustrated. 85% accuracy with a clear "contact a human" fallback may be good enough.

A medical diagnostic aid may need 99%+ accuracy on specific claim types to be deployable. No amount of cost or latency benefit justifies a lower bar.

The product role in AI development is to define what "good enough" means for the specific context - not to leave that to the ML team.

Risk: The Fourth Dimension

Risk is the dimension most often missing from AI product discussions.

Failure mode risk: When the model is wrong, what happens? A classification error in a spam filter has low consequences. A classification error in a fraud detection system has significant financial and reputational consequences.

Bias risk: Does the model perform differently across demographic groups? What are the consequences if it does?

Adversarial risk: Can the model be manipulated by users who know how it works? A customer-facing LLM can be prompted to produce off-brand or harmful content if not properly constrained.

Regulatory risk: For some domains (financial advice, medical information, hiring), AI-assisted decisions carry legal compliance requirements.

Risk assessment should happen before deployment, not after.

The Build vs. Buy vs. Integrate Decision

Three options for adding AI capability:

Buy: Purchase a pre-built AI product. Fastest, lowest maintenance, least customizable. Best when the problem is generic enough that off-the-shelf solutions work.

Integrate: Use an AI API and build the integration layer yourself. Moderate speed, moderate customization, moderate cost. Best when you need specific behavior but do not need to train your own model.

Build: Train or fine-tune your own model. Slowest, highest maintenance, most customizable, potentially lowest inference cost at scale. Best when the problem is genuinely unique and off-the-shelf solutions fail.

Most AI features at most companies are "integrate" decisions, not "build" decisions. Starting with "build" when "integrate" would work wastes months and money.

When Not to Build an AI Feature

Some problems should not be solved with AI:

• When the rules are simple and stable - write code
• When 80% accuracy is not good enough and you cannot achieve more - do not deploy
• When the failure modes are unacceptable regardless of overall accuracy - do not deploy without human review
• When the cost of running the AI exceeds the value it creates - do not build it

The most sophisticated product judgment in AI is knowing when not to build the AI feature.

Where to Go Next

The final content module of this path maps AI-related roles: what each one does, what it expects, and how to choose a sensible next step.

What to Practice Next

• Pick a product that uses AI (a search engine, recommendation feed, or writing assistant) and write a one-page product spec for one improvement: define the user problem, the AI component, the success metric, and one risk of getting it wrong.
• Find a publicly reported AI product failure (news article or post-mortem) and identify what decision - data, model, UX, or policy - was the root cause; explain what a different decision would have changed.
• Interview a non-technical person who uses an AI-powered product and ask them to describe what they think the system is doing - compare their mental model to the technical reality and note the gaps.