What Neural Networks Are (and Aren't)

The Name That Misleads Everyone

"Neural network" evokes the human brain: biological neurons, consciousness, something that thinks. This framing is almost entirely wrong, and it causes people to reason badly about AI.

Neural networks are named for a loose analogy to the structure of the brain - they have simple units (artificial neurons) that connect to each other. But the analogy ends there. A neural network does not think, does not understand, does not have any of the properties of the brain that matter for consciousness or genuine reasoning. It is a mathematical function. A sophisticated one, but a function.

Getting this right changes how you interpret AI behavior and failure.

What a Neural Network Actually Does

A neural network takes in numbers and produces numbers. Everything else is detail.

The inputs are numbers representing something you care about: pixel brightness values in an image, numerical encodings of words in a sentence, numerical features describing a transaction. The outputs are numbers representing a prediction: the probability this image contains a cat, the most likely next word, the probability this transaction is fraud.

In between, the network passes the input numbers through layers of computation. Each layer transforms its input into a new representation - reshaping the numbers in ways that make the final prediction easier. An image recognition network might transform raw pixels into edges, then shapes, then object parts, then object identities. The final layer maps the last representation to the output.

None of this requires understanding. It is arithmetic, done in a very large number of steps.

Why Depth Matters

A "deep" neural network has many layers. More layers allow the network to learn more abstract representations. A shallow network can only detect simple patterns. A deep network can combine simple patterns into complex ones.

The analogy: Lego bricks. Each layer is like adding a level of assembly. The first layer detects individual bricks. The second layer connects bricks into simple shapes. The third layer combines shapes into structures. The tenth layer is capable of representing an entire building. None of this requires the Lego bricks to "understand" architecture - it just requires enough layers for the patterns to compose.

What Training Actually Adjusts

A neural network has millions or billions of numerical parameters (weights). Training adjusts these weights by showing the network examples and measuring how wrong its outputs are.

The feedback signal is the loss - a number measuring how far the network's output is from the correct output. Training adjusts the weights to reduce the loss. After seeing millions of examples, the weights settle into values that produce good outputs on most inputs.

What the network learns is entirely determined by what the training examples reward. If the training examples reward predicting cats correctly, the weights adjust to do that. The network does not decide to learn cats - it learns whatever the loss function rewards.

The Three Things Neural Networks Cannot Do

Understanding what neural networks cannot do is at least as important as understanding what they can do.

1. They cannot reason from principles they were not trained on. A neural network that has never seen examples of a certain situation cannot reason about it from first principles the way a human can. It will either generalize poorly from related examples or fail entirely. This is why AI medical tools trained in one country may perform poorly in another - the population and disease patterns are different.

2. They cannot know what they do not know. Neural networks produce outputs for every input, including inputs outside their training distribution. They do not have a reliable "I don't know" response unless explicitly trained to produce one. A language model will confidently generate a plausible-sounding answer to a question it has no basis to answer. This is the root cause of hallucinations.

3. They cannot explain their reasoning. When a neural network classifies an image or generates text, there is no logical chain of reasoning you can inspect. The output is the result of billions of parameter interactions. This opacity is a fundamental property of the architecture, not a fixable bug.

What This Means for AI Tools You Use

When you use an AI tool powered by a neural network:

• Confident output does not mean correct output. The network produces the most statistically likely output given its training data. Confidence and accuracy are different things.

• Plausible does not mean true. A language model generates text that sounds like something a knowledgeable person would say. It sounds knowledgeable because it was trained on text written by knowledgeable people. It is imitating the style of knowledge without necessarily having the underlying facts.

• Strange failures are expected, not surprising. A network that performs brilliantly on typical inputs may fail spectacularly on slightly unusual ones. This is not a defect - it is a fundamental property of learned pattern matching.

A Calibrating Example

In 2020, a widely used medical imaging AI achieved dermatologist-level accuracy in detecting skin cancer in photos - but failed disproportionately on images of darker skin tones because the training data was predominantly images of lighter skin. The network learned what it was shown. It had no understanding of skin biology or of the ethical requirement to perform equally across demographic groups.

This is not a story about evil AI. It is a story about neural networks doing exactly what they are designed to do: learn from data. The failure is in the data, not the architecture. Understanding this distinction makes you a better evaluator of every AI system you encounter.

Where to Go Next

The next module applies this understanding specifically to large language models - the neural networks behind ChatGPT, Claude, Gemini, and every other conversational AI tool.

What to Practice Next

• Build a two-layer neural network from scratch in NumPy (no frameworks) that learns XOR - implement the forward pass, compute the loss, back-propagate gradients manually, and confirm the network converges.
• Use the TensorFlow Playground (playground.tensorflow.org) to experiment with hidden layer count and activation functions on a non-linearly separable dataset - screenshot the decision boundary before and after adding a hidden layer.
• Explain in plain English to someone non-technical what a weight is, what training does to the weights, and why a deeper network can represent more complex patterns than a shallow one - use an analogy, not math.