Neural Networks and Transformers Basics

Why This Matters Before You Use a Library

You can call transformers.pipeline("sentiment-analysis") without knowing anything about how neural networks work. But you cannot debug it, evaluate it honestly, fine-tune it well, or explain its failure modes. This post builds the mechanical understanding that makes you a practitioner rather than a user.

The Perceptron: Where It Started

A perceptron is the simplest neural network - a single computation unit. It takes a set of inputs, multiplies each by a weight, sums the results, adds a bias, and passes the result through an activation function.

output = activation(w₁x₁ + w₂x₂ + ... + wₙxₙ + b)

In vector notation: output = activation(W · x + b)

The activation function introduces nonlinearity. Without it, stacking multiple layers would just be stacking matrix multiplications - which is equivalent to one matrix multiplication, making depth pointless.

Common Activation Functions

Activation	Formula	Used For
ReLU	max(0, x)	Hidden layers (most common)
Sigmoid	1 / (1 + e^(-x))	Binary output (probability)
Softmax	eˣⁱ / Σeˣʲ	Multiclass output (probability distribution)
GELU	x × Φ(x)	Transformer hidden layers
Tanh	(eˣ - e⁻ˣ) / (eˣ + e⁻ˣ)	RNNs, some hidden layers

Feedforward Networks: Stacking Layers

A feedforward neural network (also called a multilayer perceptron, or MLP) stacks multiple perceptron layers. Each layer transforms its input into a new representation. The final layer produces the output (logits for classification, a scalar for regression).

pythonCopy
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(input_dim, 256),
    nn.ReLU(),
    nn.Linear(256, 128),
    nn.ReLU(),
    nn.Linear(128, num_classes),
)

The intermediate layers learn to represent the input in progressively more abstract ways. Early layers detect low-level patterns; later layers combine them into higher-level concepts.

Training: Forward Pass, Loss, Backpropagation

Training a neural network is iterative:

1. Forward pass: Feed input through the network to get predictions
2. Compute loss: Measure how wrong the predictions are
3. Backward pass (backpropagation): Compute gradients of the loss with respect to every weight
4. Update weights: Nudge weights in the direction that reduces loss

pythonCopy
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

for batch_X, batch_y in dataloader:
    optimizer.zero_grad()
    outputs = model(batch_X)          # Forward pass
    loss = criterion(outputs, batch_y) # Compute loss
    loss.backward()                    # Backward pass
    optimizer.step()                   # Update weights

Why Adam Works Better Than Vanilla Gradient Descent

Adam (Adaptive Moment Estimation) maintains separate learning rates for each weight, adapting them based on how much that weight has been updated historically. Weights that get consistent gradient signals move faster; noisy gradients move slower. This makes training much more stable than a single global learning rate.

Regularization: Preventing Memorization

A neural network with enough capacity can memorize the training data - achieving low training loss but high validation loss. This is overfitting.

Dropout: Randomly zeros out a fraction of activations during training. Forces the network to learn redundant representations rather than relying on specific neurons.

pythonCopy
nn.Sequential(
    nn.Linear(256, 128),
    nn.ReLU(),
    nn.Dropout(p=0.3),  # 30% of neurons dropped during training
    nn.Linear(128, 64),
)

Weight decay (L2 regularization): Adds a penalty proportional to the sum of squared weights to the loss. Discourages large weights, which correspond to the network being too sensitive to specific input features.

Early stopping: Monitor validation loss during training. Stop when it starts increasing even if training loss is still decreasing.

Batch normalization: Normalizes layer activations to have zero mean and unit variance. Stabilizes training, allows higher learning rates, provides mild regularization.

From RNNs to Attention: Why Sequential Models Struggled

Before transformers, sequential data (text, time series) was handled by recurrent neural networks (RNNs). RNNs process data one step at a time, maintaining a hidden state that carries information forward.

The fundamental problem: the hidden state is a bottleneck. To predict the last word in a paragraph, the hidden state at that point must somehow encode everything relevant from 500 words ago. For long sequences, this fails - early information is compressed and lost (the vanishing gradient problem).

LSTMs and GRUs improved this with gating mechanisms, but the fundamental bottleneck remained.

Attention: The Key Idea in Transformers

Attention solves the bottleneck by allowing every position in the sequence to directly look at every other position, rather than routing information through a sequential hidden state.

The Query-Key-Value Framework

For each position in the input sequence, attention computes:

• Query (Q): "What am I looking for?"
• Key (K): "What information do I contain?"
• Value (V): "What will I contribute if selected?"

Attention weights are computed by how well each key matches the query:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) × V

The result: each position gets a weighted combination of all values, with higher weights for positions whose keys match its query. This allows long-range dependencies to be captured directly.

Multi-Head Attention

Real transformers use multi-head attention - running the attention mechanism in parallel with different learned projections. Each "head" can attend to different types of relationships (syntactic, semantic, positional). The outputs are concatenated and projected.

The Transformer Architecture

A transformer layer consists of:

1. Multi-head self-attention
2. Layer normalization
3. Feedforward network (two linear layers with activation)
4. Another layer normalization

Positional encoding is added to token embeddings to give the model information about sequence order (since attention itself is permutation-invariant - it has no notion of position).

Encoder-only transformers (BERT): attend to all positions in both directions. Used for classification, named entity recognition.

Decoder-only transformers (GPT, Claude, Llama): attend only to previous positions (causal masking). Used for text generation.

Encoder-decoder transformers (T5, original transformer): encoder processes input, decoder generates output while attending to encoder. Used for translation, summarization.

What You Need to Know Before Using Pre-Trained Models

Context window: The maximum sequence length the model can process. Longer inputs get truncated. Newer models have context windows of 128K–1M+ tokens.

Tokenization: Input text is split into tokens (subwords), not characters or words. "Unbelievably" might become ["un", "believ", "ably"]. Token count, not word count, determines what fits in the context window.

Temperature and sampling: LLMs generate tokens probabilistically. Temperature controls sharpness of the distribution. Temperature=0 is greedy (always picks the most probable token). Temperature>1 makes outputs more random.

Embeddings vs. completions: Pre-trained models can be used for producing embeddings (fixed-size representations of text) or for generating completions (producing new text). These use cases require different API calls and different evaluation approaches.

Common Mistakes and Bad Instincts

Not normalizing inputs. Neural networks are sensitive to input scale. Features with very different ranges will cause unstable training. Always normalize or standardize input features.

Using sigmoid outputs for multiclass problems. Sigmoid produces independent probabilities per class. Softmax produces a probability distribution over all classes (they sum to 1). Use softmax for mutually exclusive classes.

Ignoring learning rate schedules. A fixed learning rate is almost always suboptimal. Use warmup followed by cosine decay or step decay for stable, efficient training.

Using all available compute for one big model. Smaller models trained with more compute often outperform larger models trained with less. Optimal compute allocation is a research area; in practice, start smaller than you think you need.

Treating pre-trained model outputs as ground truth. Pre-trained models are trained on population-average data. Their outputs reflect statistical patterns in that data, not verified facts. Evaluate them rigorously before trusting them.

Where to Go Next

Neural network fundamentals are the focus of Module 16 (Neural Networks From Scratch) and Module 17 (Deep Learning With PyTorch) in the College Student path. The SWE path covers the same terrain in Modules 10 and 11 with a practitioner emphasis. The attention mechanism and transformer architecture are covered deeply in Modules 19 and 20 of the College Student path (Sequence Models and Transformers) and Module 14 of the SWE path.

From Linear Models to Neural Networks

A linear model learns one weighted combination of inputs. A neural network stacks many learned transformations with nonlinear functions between them. That lets the model learn interactions automatically.

For tabular data, this is not always an advantage. Gradient boosted trees often beat neural networks on small and medium structured datasets. Neural networks shine when data is high-dimensional and pattern-rich: images, audio, text, code, and large-scale recommendation signals.

What a Layer Learns

A neural network layer projects input data into a new representation. Early layers learn simple patterns. Later layers combine them into more abstract patterns.

For images:

• Early layers detect edges and textures
• Middle layers detect shapes and parts
• Later layers detect objects

For text:

• Early representations capture token identity and local context
• Middle representations capture syntax and phrase structure
• Later representations capture task-relevant meaning

This is why representation learning matters. The model is not only fitting labels. It is building intermediate features.

Training Loop Anatomy

Every neural network training loop has the same skeleton:

1. Take a batch of examples
2. Run a forward pass to produce predictions
3. Compute loss
4. Run backpropagation to compute gradients
5. Update weights with an optimizer
6. Repeat
7. Evaluate on validation data

Most bugs happen around data shape, loss choice, train/eval mode, learning rate, and leakage between train and validation.

Transformers: The Key Ideas

Transformers introduced a more parallel way to process sequences. The important pieces are:

• Tokenization: turn text into model-readable units
• Embeddings: represent tokens as vectors
• Positional information: preserve order
• Attention: let tokens gather information from other tokens
• Feed-forward layers: transform each position
• Stacking: repeat the process many times

Attention is the distinctive piece. It lets the model decide which other tokens matter for the current token. In "the trophy did not fit in the suitcase because it was too small", attention helps resolve what "it" refers to.

Fine-Tuning, Prompting, and Retrieval

Modern transformer use usually falls into three patterns:

• Prompting: steer a pretrained model through instructions and examples
• Retrieval: provide external context at request time
• Fine-tuning: update model weights using task-specific examples

Use prompting first when the task is mostly formatting, reasoning style, or instruction following. Use retrieval when the model needs private or changing knowledge. Use fine-tuning when you need consistent behavior that cannot be achieved through prompts and retrieval alone.

Debugging Neural Networks

When training fails, check in this order:

1. Can the model overfit a tiny batch?
2. Are labels aligned with inputs?
3. Is the loss appropriate for the task?
4. Is the learning rate reasonable?
5. Are train and validation preprocessing identical?
6. Are gradients exploding, vanishing, or zero?
7. Is the evaluation code correct?

Most failures are not mysterious. They are plumbing, data, or objective mismatches.

Closing Thought

The practical standard is not memorization. It is whether you can use the idea to make a better engineering decision, explain that decision to someone else, and notice when reality disagrees with your assumptions.

What to Do Next

Turn this article into a small artifact. Write a checklist, run a tiny experiment, sketch the architecture, or review an old project using the concepts above. Learning becomes durable when it changes what you inspect before you trust a result.

For a portfolio or team setting, save that artifact next to the code or decision memo. Future reviewers should be able to see not only what you built, but how you reasoned about correctness, risk, and tradeoffs.

Evidence Habit

When in doubt, prefer evidence over confidence. Keep the smallest repeatable test that proves the idea works, and revisit it whenever data, users, models, or requirements change.

Team Review Prompts

Before treating this work as complete, ask a teammate to review it using three prompts:

1. What assumption is most likely to break in production?
2. What evidence would make you trust the result?
3. What simpler approach should we compare against?

These questions are deliberately plain. They work because they force the discussion away from tool enthusiasm and back toward judgment, evidence, and maintainability.

Final Rule

Neural networks are powerful because they learn representations, not because they remove the need for judgment. The engineer still chooses data, objective, architecture, evaluation, and deployment constraints. The model learns within the world you define for it.

Interpretability

Neural networks are often harder to explain than linear models or trees, but they are not impossible to inspect. Useful tools include:

• Saliency maps for vision
• Attention inspection with caution
• Embedding nearest neighbors
• Activation analysis
• Counterfactual examples
• Probing classifiers
• Error clustering

Do not oversell interpretability. A pretty heatmap is not proof of reasoning. Interpretability tools are debugging aids. They help you form hypotheses about behavior, then test those hypotheses with data.

Model Size Is a Product Choice

Larger models often improve quality but increase latency, cost, memory use, and operational complexity. Smaller models can be faster, cheaper, easier to deploy at the edge, and good enough for narrow tasks.

The right model is the smallest model that meets the quality bar with acceptable safety margins. This principle matters for both classic deep learning and modern LLM systems.

Regularization and Generalization

Neural networks can memorize. Regularization helps them learn patterns that transfer beyond the training set.

Common tools:

• Weight decay: discourages very large weights
• Dropout: randomly disables activations during training
• Data augmentation: creates useful variation in inputs
• Early stopping: stops training when validation performance worsens
• Smaller models: reduce capacity when data is limited

Regularization is not a magic switch. It encodes a preference for simpler, more stable behavior.

Batch Size, Learning Rate, and Optimizers

Batch size controls how many examples contribute to each update. Larger batches are computationally efficient but may require learning-rate adjustment. Smaller batches add noise that can sometimes help generalization.

The learning rate controls update size. Too high and loss may explode. Too low and training crawls. Optimizers such as Adam adapt update sizes per parameter, which is why they are common defaults for deep learning.

Still, defaults are not destiny. Plot training and validation loss. If training loss does not decrease, suspect optimization. If training loss decreases but validation loss worsens, suspect overfitting or data mismatch.

Transfer Learning

Most practical neural network projects do not train from scratch. They start from pretrained models:

• ImageNet-pretrained CNNs or vision transformers for images
• Transformer language models for text
• Sentence embedding models for retrieval
• Multimodal models for image-text tasks

Transfer learning works because the pretrained model has already learned reusable representations. Your job is to adapt those representations without destroying them.

Deployment Implications

Neural networks introduce serving constraints:

• Larger models cost more to run
• GPU availability may affect architecture
• Quantization can reduce cost with some quality tradeoff
• Batch inference can improve throughput
• Model warmup and cold starts matter
• Monitoring must include both technical and quality signals

Understanding model internals helps you make these tradeoffs instead of treating the model as an opaque artifact.