A neural network can be viewed as a chain of simple calculations.
Numbers arrive at the input. Each neuron multiplies them by its weights, adds the results, adds a bias, and produces a new number. The final neuron turns the resulting values into an answer, such as 0 or 1.
In the simplest network, the inputs connect directly to one output neuron:
two inputs → one neuron → answerSuch a neuron learns AND and OR without difficulty. But it cannot learn XOR, no matter how long training continues.
Add one more step between the inputs and the answer:
two inputs → two intermediate neurons → one output neuronThose two intermediate neurons form a hidden layer.
It is called hidden because we do not provide correct answers for it. We know only what the entire network should return; during training, the network chooses its internal values by itself.
For each input, the hidden layer calculates two new numbers. For example:
(0, 1) → (0.004, 0.903)The output neuron receives this new pair rather than the original 0 and 1.
This is what makes XOR solvable. In the original coordinates, no single straight line can separate the four points. The hidden layer moves them into a different arrangement in which such a line does exist.
XOR is convenient to study because it has only four possible inputs:
0 XOR 0 = 0
0 XOR 1 = 1
1 XOR 0 = 1
1 XOR 1 = 0We will first train one neuron and see why it stops near an answer of 0.5. Then we will add two hidden neurons, print the values they calculate, and see exactly how they changed the problem. The complete source is included in the appendix.
One Neuron Draws One Boundary
Take the simplest binary classifier.
It receives two values, multiplies them by weights, adds a bias, and passes the result through a sigmoid:
for _ in 0..EPOCHS {
for (input, target) in &data {
let output = sigmoid(w0 * input[0] + w1 * input[1] + bias);
let delta = (output - target) * output * (1.0 - output);
w0 -= RATE * delta * input[0];
w1 -= RATE * delta * input[1];
bias -= RATE * delta;
}
}Before the sigmoid, the neuron calculates an ordinary linear expression:
w₀·a + w₁·b + biasIf the result is large and positive, the sigmoid produces a number close to one. If the result is negative, it produces a number close to zero.
The points for which the expression equals zero form a straight line:
w₀·a + w₁·b + bias = 0This line divides the plane into two regions. On one side, the neuron answers closer to 0; on the other, closer to 1.
Two classes are called linearly separable if one straight line can separate them completely. A single sigmoid neuron can learn only such a boundary.
To verify the training code itself, run it on more than XOR. Using the same initialization, epoch count, and learning rate, train AND, OR, and XOR in sequence:
$ cargo run --release --quiet
one sigmoid unit, 20000 epochs, learning rate 0.5
function final loss correct learned line
AND 0.000228 4/4 +8.02·a +8.02·b -12.12 = 0
OR 0.000119 4/4 +8.67·a +8.67·b -4.10 = 0
XOR 0.250397 2/4 -0.13·a -0.06·b +0.06 = 0AND and OR are learned completely. All four inputs are classified correctly, and the final loss is almost zero.
XOR produces a different result. The loss remains near 0.25, and only two of the four rows are classified correctly.
Let us inspect what the trained neuron actually returns:
XOR, what the trained unit actually answers:
0 XOR 0 -> 0.516 (target 0)
0 XOR 1 -> 0.500 (target 1)
1 XOR 0 -> 0.484 (target 1)
1 XOR 1 -> 0.468 (target 0)It returns approximately 0.5 for all four inputs.
The neuron has nearly stopped distinguishing the data. Its weights have also collapsed toward zero:
-0.13·a -0.06·b +0.06 = 0This is not an accidental stop at a bad point. For a model that cannot express the correct boundary, an answer near 0.5 is a reasonable compromise.
The XOR table contains two zero targets and two one targets. If they cannot be separated, the model reduces its total error by answering roughly halfway for every input.
Gradient descent did not miss a solution. It found the best solution available to one neuron.
Why AND and OR Are Easier
Place the four possible inputs on a plane. The first coordinate is a, and the second is b.
For AND, only this point belongs to class one:
(1, 1)The remaining three should receive zero. One diagonal line can easily separate the upper-right corner from the rest.
For OR, only this point belongs to class zero:
(0, 0)The other three points can remain on the opposite side of another line.
For XOR, these points belong to class one:
(0, 1)
(1, 0)and these belong to class zero:
(0, 0)
(1, 1)Points from the same class occupy opposite corners of the square.
No matter where we draw a straight line, at least one side contains points from both classes. We cannot separate (0, 1) and (1, 0) from the other two corners at the same time.
More epochs therefore cannot fix the problem. Training can move and rotate the line and change its distance from the points, but it cannot turn one boundary into two separate regions.
The problem is not training quality. The required line does not exist.
Adding a Hidden Layer
Now two neurons appear between the inputs and the output:
pub struct Network {
/// hidden[j] = (w_a, w_b, bias)
pub hidden: [[f64; 3]; 2],
/// output = (w_h0, w_h1, bias)
pub output: [f64; 3],
}Each hidden neuron receives the original a and b, but uses its own weights and bias.
The first calculates h₀, and the second calculates h₁:
(a, b) → (h₀, h₁)The output neuron then works with these two new numbers:
h₀, h₁ → answerTraining still uses ordinary gradient descent. First, the output error is calculated; then it is passed one layer backward:
let delta_out = (out - target) * out * (1.0 - out);
let grad_h = [
self.output[0] * delta_out,
self.output[1] * delta_out,
];
for j in 0..2 {
let delta_h = grad_h[j] * h[j] * (1.0 - h[j]);
self.hidden[j][0] -= RATE * delta_h * input[0];
self.hidden[j][1] -= RATE * delta_h * input[1];
self.hidden[j][2] -= RATE * delta_h;
}The output neuron updates its weights from its own error. Each hidden neuron then receives part of that error, in proportion to how strongly it affected the final answer.
The code contains no special rule for XOR. We do not tell one hidden neuron to calculate OR and the other to calculate NAND. The network sees only the inputs, correct answers, and error magnitude.
Let us watch training progress:
$ cargo run --release --quiet
2-2-1 network, 20000 epochs, learning rate 0.5
epoch loss correct
0 0.252969 2/4
2500 0.168668 3/4
5000 0.167458 3/4
7500 0.167008 3/4
10000 0.059079 4/4
12500 0.001075 4/4
15000 0.000517 4/4
17500 0.000339 4/4
20000 0.000251 4/4At first, the network still makes mistakes.
By 2,500 epochs, it answers three of the four inputs correctly, but then remains near a loss of 0.167 for a long time. It has found an intermediate solution resembling OR: three points are placed correctly, while the fourth remains on the wrong side.
Between 7,500 and 10,000 epochs, the network leaves this plateau. All four answers become correct, after which the loss quickly falls almost to zero.
The final values look like this:
what the trained network answers:
0 XOR 0 -> 0.013 (target 0)
0 XOR 1 -> 0.985 (target 1)
1 XOR 0 -> 0.985 (target 1)
1 XOR 1 -> 0.019 (target 0)The network now distinguishes all four inputs confidently.
But successful training alone does not explain what the hidden layer did.
The Final Neuron Still Draws Only a Straight Line
The output neuron did not become more complex. It still calculates a linear expression:
w₀·h₀ + w₁·h₁ + biasIts boundary is still a straight line:
w₀·h₀ + w₁·h₁ + bias = 0It did not learn to draw two separate regions or gain a special XOR operation.
Only the data it works with changed.
Previously, it saw the original coordinates:
(a, b)Now it sees the hidden layer's activations:
(h₀, h₁)Print both coordinate sets and check whether a separating line exists.
In the original space:
input coordinates (a, b) — the space XOR lives in:
(0, 0) -> class 0
(0, 1) -> class 1
(1, 0) -> class 1
(1, 1) -> class 0
a straight line separating the classes exists: falseNo straight line separates the classes.
Now inspect the numbers calculated by the hidden layer:
hidden coordinates (h0, h1) — the space the output unit sees:
(0, 0) -> (0.956, 0.999) class 0
(0, 1) -> (0.004, 0.903) class 1
(1, 0) -> (0.005, 0.903) class 1
(1, 1) -> (0.000, 0.079) class 0
a straight line separating the classes exists: trueThe required line exists in the new coordinates.
That is why the network can solve XOR.
Two Different Inputs Become Almost Identical
The hidden layer's work is especially clear for the two class-one inputs:
(0, 1)
(1, 0)In the original coordinates, they are opposite corners of the square.
After the hidden layer, they become:
(0, 1) → (0.004, 0.903)
(1, 0) → (0.005, 0.903)They differ by roughly one thousandth. To the output neuron, they are nearly the same point.
The hidden layer joined two different inputs that should lead to the same answer.
For XOR, it does not matter which bit is one. What matters is that exactly one bit is one. The hidden layer has almost removed the difference between "one on the left" and "one on the right."
The class-zero inputs were transformed differently:
(0, 0) → (0.956, 0.999)
(1, 1) → (0.000, 0.079)They did not end up close to each other, but both landed on the same side of the output boundary.
The hidden layer does not need to gather each class into a single point. It only needs to arrange the data so that the final neuron can separate it with one straight line.
The Boundary Learned by the Output Neuron
After the transformation, the output layer learned this line:
the line the output unit learned, in hidden coordinates:
-9.91·h0 +9.91·h1 -4.71 = 0
signed distance of each point
(positive means class 1):
(0.956, 0.999) score -4.293 -> 0.013 class 0
(0.004, 0.903) score +4.186 -> 0.985 class 1
(0.005, 0.903) score +4.185 -> 0.985 class 1
(0.000, 0.079) score -3.925 -> 0.019 class 0The weight of h₀ is almost equal to the weight of h₁ but has the opposite sign. The rule can therefore be rewritten approximately as:
h₁ - h₀ > 0.475For class-one inputs, h₁ is much larger than h₀, so the score is positive.
For (0, 0), both coordinates are large and close together; for (1, 1), h₁ is too small. The score remains negative in both cases.
After the sigmoid, positive values become answers near 0.985, while negative values become 0.013 and 0.019.
The final neuron once again drew a straight line.
But this time it did so in the space prepared by the hidden layer.
Why Another Linear Layer Is Not Enough
It may seem that the problem was solved simply because the network gained more weights.
But if we remove the sigmoids from the hidden layer, two consecutive layers remain one linear transformation.
The first layer calculates:
h = W₁x + b₁The second calculates:
y = W₂h + b₂Substitute the first expression into the second:
y = W₂(W₁x + b₁) + b₂After expanding the parentheses, this is again an ordinary linear function of the original input.
No matter how many linear layers we place in sequence, they can always be replaced by one. No new geometry appears, and XOR remains inseparable.
The sigmoid between layers prevents this collapse. Each hidden neuron transforms its weighted sum nonlinearly, so the overall transformation can no longer be replaced by one linear function.
That nonlinearity makes it possible to move the points.
But saying only that "the network became nonlinear" is still too general. Our experiment shows the concrete result: two opposite class-one inputs ended up almost at the same point, after which the classes became linearly separable.
What Hidden Layers Do in Larger Networks
For XOR, the hidden layer creates only two new coordinates. We can print them and understand how every point moved.
A larger neural network performs a similar process, but with many more intermediate numbers.
For example, a model receives raw data:
image pixelsThe first layer calculates new values that may respond to simple lines and brightness changes. The next layer combines them into more complex shapes. The final layer receives not the original pixels but the result of several successive transformations.
The same applies to text, audio, and tabular data. Each layer passes the next one not the original input, but new numbers that proved useful for reducing the error.
It is not always possible to give each number a simple name. But the underlying process remains the same as for XOR:
raw data
→ new intermediate values
→ a simpler problem for the next layerA hidden layer does not need to produce the final answer itself. Its job is to prepare the data for the next part of the network.
Why AND and OR Do Not Need a Hidden Layer
AND and OR are already separable by one straight line in the original coordinates.
A single neuron therefore learned them without an intermediate transformation. The original a and b values were sufficient.
This matters: a hidden layer is not useful merely because more layers are always better.
It is needed when the original data is arranged inconveniently for a simple output layer and an intermediate transformation can make the problem easier.
For AND and OR, a suitable boundary already exists. For XOR, it must first be created in a new space.
The Single Neuron Did Not Train Poorly
Answers near 0.5 are easy to mistake for failed training.
But the AND and OR results show that the code, learning rate, and epoch count work. The same neuron finds an almost perfect solution for those functions.
XOR differs neither in the number of examples nor in computational complexity. It simply has different geometry.
One neuron can choose only one straight line in the original coordinates. No such line exists for XOR.
After adding a hidden layer, the output neuron remained the same. It still combines two weighted values and divides the plane with one boundary.
The problem was solved earlier, when the hidden neurons transformed the original inputs.
The points (0, 1) and (1, 0) ended up in almost the same place. The two zero inputs remained on the other side of the output line. What could not be separated in (a, b) coordinates became separable in (h₀, h₁) coordinates.
The hidden layer did not memorize the four rows of the XOR table.
It changed the data so that the final neuron could once again solve the problem with one straight line.
Appendix: Full Source Files
perceptron/src/main.rs — 108 lines
//! Scene 01: one layer, honestly trained, failing at XOR.
//!
//! A single sigmoid unit over two inputs — the whole model is a line
//! plus a squash. It learns AND and OR in a few thousand epochs and
//! never learns XOR, no matter how long it runs. The run below trains
//! all three so the failure is a comparison, not a claim.
use anyhow::Result;
const EPOCHS: usize = 20_000;
const RATE: f64 = 0.5;
/// Deterministic small weights, so every run of this article agrees.
fn seeded(count: usize, seed: u64) -> Vec<f64> {
let mut state = seed;
(0..count)
.map(|_| {
state ^= state << 13;
state ^= state >> 7;
state ^= state << 17;
(state >> 11) as f64 / (1u64 << 53) as f64 * 2.0 - 1.0
})
.collect()
}
fn sigmoid(x: f64) -> f64 {
1.0 / (1.0 + (-x).exp())
}
/// The four points of a two-bit truth table, with the target for the
/// requested function.
fn dataset(function: &str) -> Vec<([f64; 2], f64)> {
[[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]]
.into_iter()
.map(|input| {
let (a, b) = (input[0] as u8 == 1, input[1] as u8 == 1);
let target = match function {
"AND" => a && b,
"OR" => a || b,
_ => a ^ b,
};
(input, if target { 1.0 } else { 0.0 })
})
.collect()
}
struct Trained {
loss: f64,
correct: usize,
weights: [f64; 2],
bias: f64,
}
fn train(function: &str) -> Trained {
let start = seeded(3, 0x14_11_2026);
let (mut w0, mut w1, mut bias) = (start[0], start[1], start[2]);
let data = dataset(function);
for _ in 0..EPOCHS {
for (input, target) in &data {
let output = sigmoid(w0 * input[0] + w1 * input[1] + bias);
// d(MSE)/d(pre-activation) for a sigmoid unit.
let delta = (output - target) * output * (1.0 - output);
w0 -= RATE * delta * input[0];
w1 -= RATE * delta * input[1];
bias -= RATE * delta;
}
}
let mut loss = 0.0;
let mut correct = 0;
for (input, target) in &data {
let output = sigmoid(w0 * input[0] + w1 * input[1] + bias);
loss += (output - target).powi(2);
if (output >= 0.5) == (*target >= 0.5) {
correct += 1;
}
}
Trained {
loss: loss / data.len() as f64,
correct,
weights: [w0, w1],
bias,
}
}
fn main() -> Result<()> {
println!("one sigmoid unit, {EPOCHS} epochs, learning rate {RATE}\n");
println!("function final loss correct learned line");
for function in ["AND", "OR", "XOR"] {
let result = train(function);
println!(
"{function:<9} {:>10.6} {}/4 {:+.2}·a {:+.2}·b {:+.2} = 0",
result.loss, result.correct, result.weights[0], result.weights[1], result.bias
);
}
println!("\nXOR, what the trained unit actually answers:");
let xor = train("XOR");
for (input, target) in dataset("XOR") {
let output = sigmoid(xor.weights[0] * input[0] + xor.weights[1] * input[1] + xor.bias);
println!(
" {} XOR {} -> {output:.3} (target {target})",
input[0] as u8, input[1] as u8
);
}
Ok(())
}mlp/src/lib.rs — 104 lines
//! Scene 02: the same trainer with one hidden layer of two units.
//!
//! Nothing else changes — same data, same sigmoid, same gradient
//! descent, same deterministic initialization. The only addition is
//! two units between input and output, and with them XOR becomes
//! learnable. Scene 03 imports this network to look at what those two
//! units are actually computing.
pub const EPOCHS: usize = 20_000;
pub const RATE: f64 = 0.5;
pub fn sigmoid(x: f64) -> f64 {
1.0 / (1.0 + (-x).exp())
}
/// The XOR truth table.
pub fn xor_data() -> Vec<([f64; 2], f64)> {
vec![
([0.0, 0.0], 0.0),
([0.0, 1.0], 1.0),
([1.0, 0.0], 1.0),
([1.0, 1.0], 0.0),
]
}
/// A 2-2-1 network: two inputs, two hidden units, one output.
pub struct Network {
/// hidden[j] = (w_a, w_b, bias)
pub hidden: [[f64; 3]; 2],
/// output = (w_h0, w_h1, bias)
pub output: [f64; 3],
}
fn seeded(count: usize, seed: u64) -> Vec<f64> {
let mut state = seed;
(0..count)
.map(|_| {
state ^= state << 13;
state ^= state >> 7;
state ^= state << 17;
(state >> 11) as f64 / (1u64 << 53) as f64 * 2.0 - 1.0
})
.collect()
}
impl Network {
pub fn new(seed: u64) -> Self {
let w = seeded(9, seed);
Self {
hidden: [[w[0], w[1], w[2]], [w[3], w[4], w[5]]],
output: [w[6], w[7], w[8]],
}
}
/// Hidden activations for one input — the new coordinates.
pub fn hidden_of(&self, input: [f64; 2]) -> [f64; 2] {
let mut activations = [0.0; 2];
for (j, unit) in self.hidden.iter().enumerate() {
activations[j] = sigmoid(unit[0] * input[0] + unit[1] * input[1] + unit[2]);
}
activations
}
pub fn forward(&self, input: [f64; 2]) -> f64 {
let h = self.hidden_of(input);
sigmoid(self.output[0] * h[0] + self.output[1] * h[1] + self.output[2])
}
/// One epoch of plain gradient descent over the whole table.
pub fn train_epoch(&mut self, data: &[([f64; 2], f64)]) {
for (input, target) in data {
let h = self.hidden_of(*input);
let out = sigmoid(self.output[0] * h[0] + self.output[1] * h[1] + self.output[2]);
// Output unit.
let delta_out = (out - target) * out * (1.0 - out);
let grad_h = [self.output[0] * delta_out, self.output[1] * delta_out];
self.output[0] -= RATE * delta_out * h[0];
self.output[1] -= RATE * delta_out * h[1];
self.output[2] -= RATE * delta_out;
// Hidden units: the same rule, one layer back.
for j in 0..2 {
let delta_h = grad_h[j] * h[j] * (1.0 - h[j]);
self.hidden[j][0] -= RATE * delta_h * input[0];
self.hidden[j][1] -= RATE * delta_h * input[1];
self.hidden[j][2] -= RATE * delta_h;
}
}
}
pub fn loss(&self, data: &[([f64; 2], f64)]) -> f64 {
data.iter()
.map(|(input, target)| (self.forward(*input) - target).powi(2))
.sum::<f64>()
/ data.len() as f64
}
pub fn correct(&self, data: &[([f64; 2], f64)]) -> usize {
data.iter()
.filter(|(input, target)| (self.forward(*input) >= 0.5) == (*target >= 0.5))
.count()
}
}mlp/src/main.rs — 35 lines
//! Trains the 2-2-1 network on XOR and prints the loss curve.
use anyhow::Result;
use mlp::{EPOCHS, Network, RATE, xor_data};
fn main() -> Result<()> {
let data = xor_data();
let mut network = Network::new(0x14_11_2026);
println!("2-2-1 network, {EPOCHS} epochs, learning rate {RATE}\n");
println!("epoch loss correct");
for epoch in 0..=EPOCHS {
if epoch % 2_500 == 0 {
println!(
"{epoch:>5} {:>9.6} {}/4",
network.loss(&data),
network.correct(&data)
);
}
if epoch < EPOCHS {
network.train_epoch(&data);
}
}
println!("\nwhat the trained network answers:");
for (input, target) in &data {
println!(
" {} XOR {} -> {:.3} (target {target})",
input[0] as u8,
input[1] as u8,
network.forward(*input)
);
}
Ok(())
}coordinates/src/main.rs — 85 lines
//! Scene 03: what the hidden layer is actually for.
//!
//! The same trained network, but instead of its answers we print the
//! two hidden activations for each input — the coordinates the output
//! unit sees. In the original coordinates no line separates XOR; the
//! program checks that claim by brute force over every line on a grid.
//! In the hidden coordinates the output unit's own line separates them,
//! and the margin is printed.
use anyhow::Result;
use mlp::{EPOCHS, Network, sigmoid, xor_data};
/// Can ANY line separate these four labelled points? Brute force over
/// a dense grid of directions and offsets — a claim worth checking.
fn separable(points: &[([f64; 2], f64)]) -> bool {
let steps = 400;
for i in 0..steps {
let angle = std::f64::consts::TAU * i as f64 / steps as f64;
let (wa, wb) = (angle.cos(), angle.sin());
for j in 0..=steps {
let bias = -3.0 + 6.0 * j as f64 / steps as f64;
let ok = points
.iter()
.all(|(p, target)| ((wa * p[0] + wb * p[1] + bias) > 0.0) == (*target >= 0.5));
if ok {
return true;
}
}
}
false
}
fn main() -> Result<()> {
let data = xor_data();
let mut network = Network::new(0x14_11_2026);
for _ in 0..EPOCHS {
network.train_epoch(&data);
}
println!("input coordinates (a, b) — the space XOR lives in:");
for (input, target) in &data {
println!(
" ({:.0}, {:.0}) -> class {}",
input[0], input[1], *target as u8
);
}
println!(
" a straight line separating the classes exists: {}",
separable(&data)
);
println!("\nhidden coordinates (h0, h1) — the space the output unit sees:");
let hidden: Vec<([f64; 2], f64)> = data
.iter()
.map(|(input, target)| (network.hidden_of(*input), *target))
.collect();
for ((input, _), (h, target)) in data.iter().zip(&hidden) {
println!(
" ({:.0}, {:.0}) -> ({:.3}, {:.3}) class {}",
input[0], input[1], h[0], h[1], *target as u8
);
}
println!(
" a straight line separating the classes exists: {}",
separable(&hidden)
);
println!("\nthe line the output unit learned, in hidden coordinates:");
println!(
" {:+.2}·h0 {:+.2}·h1 {:+.2} = 0",
network.output[0], network.output[1], network.output[2]
);
println!(" signed distance of each point (positive means class 1):");
for (h, target) in &hidden {
let score = network.output[0] * h[0] + network.output[1] * h[1] + network.output[2];
println!(
" ({:.3}, {:.3}) score {score:+.3} -> {:.3} class {}",
h[0],
h[1],
sigmoid(score),
*target as u8
);
}
Ok(())
}