ML Interview Questions: What Actually Gets Asked

This is not a list of definitions to memorize. It is the questions that interviewers at top companies actually ask, what they are trying to learn from each one, and what a strong answer looks like.

Foundations

Q: What is the bias-variance tradeoff?

What they are testing: can you connect theory to practice, not just recite the definition.

Strong answer: "Bias is error from wrong assumptions - a linear model applied to a nonlinear problem will always be biased regardless of data size. Variance is error from sensitivity to training data - a 1000-node decision tree will fit training data perfectly but may not generalize. The total expected test error is bias² + variance + irreducible noise.

In practice, this means: if your model underfits (high train error), the problem is bias - add model complexity, add features, or try a more powerful model. If it overfits (low train error, high val error), the problem is variance - add regularization, get more data, or reduce model complexity. Neither pure bias nor pure variance is the goal - you want the sweet spot."

---

Q: How do you choose between L1 and L2 regularization?

What they are testing: deep understanding, not "L1 does feature selection."

Strong answer: "Both penalize large weights, but in different geometries. L2 adds λ·Σw² to the loss, which penalizes all weights proportionally - it shrinks them toward zero but rarely to zero. L1 adds λ·Σ|w|, which can drive weights exactly to zero because the gradient of |w| is ±1 regardless of magnitude, creating a constant pull toward zero.

So: use L1 when you believe your feature space is sparse - many features are irrelevant and you want the model to identify them. Use L2 when you believe most features contribute something. Use Elastic Net (L1 + L2 combined) when you have correlated features: L1 tends to pick one from a correlated group arbitrarily, while Elastic Net can include both with smaller weights."

---

Q: Explain gradient descent and its variants.

Strong answer: "Gradient descent updates parameters by moving in the direction of the negative gradient: θ ← θ - α·∇L. Three main variants differ in how much data they use per update:

• Batch GD: compute gradient over the entire dataset per step. Most accurate estimate, but prohibitively slow for large datasets.
• Stochastic GD: compute gradient over a single example. Fast but noisy - the estimate bounces around, which can help escape local minima but makes convergence unstable.
• Mini-batch GD: compute gradient over a batch (typically 32–256). The practical standard - balances speed and stability, vectorizes well on GPUs.

Beyond the variant, the learning rate schedule matters significantly. A constant learning rate is rarely optimal. Adaptive methods like Adam adjust the learning rate per parameter based on gradient history, which works well for most deep learning tasks."

---

Q: What is cross-validation, and when would you not use it?

Strong answer: "Cross-validation estimates model performance on unseen data by training on k-1 folds and validating on the held-out fold, repeating k times. The result is k test scores averaged together - a lower-variance estimate than a single train-test split.

When I would not use it: for time series data, where the future cannot inform the past. Randomly shuffled k-fold would allow future data into training, creating data leakage. Instead, use time-series CV: train on all data before time t, validate on data in window [t, t+window], advance the window.

Also: for very large datasets (>10M examples), k-fold is computationally expensive for minimal benefit - a single large holdout set is usually sufficient."

Machine Learning in Practice

Q: Walk me through how you would approach a new ML problem from scratch.

What they are testing: engineering process and judgment, not memorization.

Strong answer: "I start with the business problem, not the model. What decision is being made? What is the cost of a wrong decision? That drives the choice of metric and the error budget.

Then I look at the data before writing any code: understand the distribution, check for class imbalance, identify obvious leakage risks, look at label quality. Bad data kills good models; time spent here saves weeks later.

Next: start with the simplest model that could plausibly work - often a logistic regression or gradient boosted tree. This gives you a baseline to beat and lets you validate the data pipeline quickly. Do not start with a neural network unless you have a specific reason.

Then iterate: feature engineering, model selection, hyperparameter tuning - but always with held-out data guiding decisions, never the training set.

Finally, think about how the model gets to production: the serving latency requirements, how predictions will be used, how the model will be monitored. A model that cannot meet latency requirements or cannot be monitored is not production-ready."

---

Q: Your model works well in offline evaluation but degrades in production. What do you check?

Strong answer: "I think about this systematically in three buckets.

Data issues first: Is the production data distribution similar to training data? Check key feature means and distributions. Is there a data pipeline bug - wrong transformations, missing imputation, wrong feature ordering? Is there a label leakage issue I missed in offline evaluation that does not exist in production?

Model issues second: Is the model threshold set correctly for the production class distribution? Did the training-serving skew - features computed differently at training vs. serving time?

Environmental issues third: Are there any serialization or version mismatches - different scikit-learn version loading the model, different preprocessing pipeline?

In practice, 80% of these issues are data pipeline problems, not model problems. I would start by logging raw features at prediction time and comparing them to training features before anything else."

Deep Learning

Q: Explain the attention mechanism in transformers.

Strong answer: "Attention lets each token in a sequence look at all other tokens and decide which ones are most relevant to its representation.

For a token at position i, we compute three vectors using learned weight matrices: Query (what I am looking for), Key (what I have), and Value (what I provide if relevant). The attention score between token i and token j is Q_i · K_j / √d_k - scaled dot product to prevent vanishing gradients with large embedding dimensions. These scores are softmaxed to get attention weights that sum to 1. The output for token i is a weighted sum of all Value vectors.

Self-attention does this across all tokens in the same sequence - the model learns which words attend to which other words. Multi-head attention runs h parallel attention operations with different weight matrices, then concatenates - this lets the model attend to multiple types of relationships simultaneously (syntactic structure, semantic similarity, coreference) rather than just one.

The transformer is fast because attention is parallelizable across positions, unlike RNNs which must process sequentially."

---

Q: What is the difference between fine-tuning and prompt engineering?

Strong answer: "Both are ways to adapt a pretrained LLM to a specific task, but they operate at different layers.

Prompt engineering changes the input without touching the model weights. It works by activating patterns the model learned during pretraining. It is cheap (no training), instant to iterate, and requires no data. The downside: it only works when the model already knows how to do the task - prompting cannot teach a model to do something it genuinely cannot do.

Fine-tuning updates the model weights on task-specific data. It can teach the model new behaviors, styles, or domain knowledge not in pretraining. It produces a model that is consistently better on the target task. The cost: requires labeled data, compute, and careful evaluation to avoid catastrophic forgetting.

The decision tree: try prompting first - it is free and fast. If prompting hits a ceiling after careful iteration, consider fine-tuning. If the task requires deep domain knowledge not in pretraining, or a specific output format the model consistently struggles with, fine-tuning is worth the investment."

LLM and AI Engineering

Q: How would you evaluate an LLM application?

Strong answer: "LLM evaluation has no single right metric - it depends on the task. I think about it in layers.

First, define what 'good' means for the specific use case. For a support bot: accuracy, helpfulness, tone, escalation rate. For code generation: correctness (does it run?), efficiency, readability.

Second, build an evaluation set representative of production inputs - not just easy cases. Include edge cases, adversarial prompts, and cases from each major user intent.

Third, use a combination of: automated metrics (BLEU/ROUGE for text tasks, pass rate for code), LLM-as-judge (a separate LLM scores the output on defined dimensions - surprisingly effective), and human evaluation for high-stakes decisions.

Fourth, set up online evaluation: log production requests, sample and score them with your evaluator, track metrics over time. A model that performs well on your eval set but degrades in production has an eval coverage problem.

The biggest mistake is treating evaluation as a one-time step before launch. It needs to be continuous - model versions change, prompts change, and the world changes."

Agents, Harnesses, and Evals

These questions did not exist in interview loops three years ago. Now they appear in most AI engineer loops, usually in the system design or "tell me about a project" round.

"What is the difference between the model and the agent?" The model proposes actions; the harness (the code around it) decides which tools exist, validates and authorizes each call, executes it, logs it, and decides what the model sees next. Reliability, security, and cost control all live in the harness. A good answer names at least three harness responsibilities and gives an example of moving a fix from the prompt into the harness.

"How would you stop an agent from doing something destructive?" Scope tools per task (a research task has no write tools), require an explicit approval state for irreversible actions, put a turn and token budget on every run, sandbox code execution, and log every proposed call including refused ones. Mention that prompt injection is structurally unsolved, so the goal is containing what a successful injection can do.

"How do you evaluate an agent?" Three layers: final answer (did it complete the task), trajectory (did it call the right tools in a sensible order, without looping or ignoring errors), and per-turn production signals (user corrections, escalations, retries). Describe building a task suite from real failures, grading tool-call correctness programmatically, using an LLM judge only where you have validated it against human labels, and wiring the suite into CI as a regression gate.

"When would you use a reasoning model?" When the task is multi-step with a checkable answer and the accuracy gain is worth the latency and cost. Answer with a method: measure accuracy versus thinking budget on your own eval set, find where the curve flattens, and route only the requests that need it.

"What is MCP and why does it matter?" An open protocol for exposing tools, resources, and prompts to a model through a server that any compatible client can use. It matters because it decouples integrations from providers and agent frameworks: you write the tool server once. A strong answer mentions tool description quality as the main lever on whether the model uses the tools correctly.

"How does prompt caching change how you write prompts?" Stable content first (system prompt, tool definitions, reference docs), variable content last, nothing per-request in the cached prefix. Cache hit rate becomes an operational metric. For agents with large tool lists it is often the biggest single cost lever.

"Explain GRPO or RLVR in two minutes." RLVR: post-train with a reward computed by a checker (tests pass, answer matches) instead of human labels, which scales for math and code. GRPO: sample a group of responses per prompt, score them, use each one's reward relative to the group mean as the advantage, no value model. Together they are the recipe behind reasoning models. Then say when you would not use it: any task without a cheap verifier.