Rectified Linear Unit
The name ReLU is a metonymy—a figure of speech where a thing is called by the name of something associated with it. In the neural network literature of the 1980s, individual neurons were called “units”—processing units, hidden units, output units. When Nair and Hinton (2010) introduced a unit whose activation function was the rectified linear function max(0, x), they named the unit: a Rectified Linear Unit.
Through metonymy, the name of the unit became the name of the function. Today, “ReLU” refers to max(0, x) itself, and the word “unit” in the acronym has become invisible—absorbed into the proper noun. When we write “ReLU activation,” we are saying “Rectified Linear Unit activation”—redundant in the same way as “ATM machine.” The unit is the activation; the activation is the unit.
In the diagram above, “Rectified” labels the zero-clamped region (x < 0) and “Linear” labels the identity region (x ≥ 0). The “Unit” is not a region of the graph—it is the neuron that uses this curve as its activation function.
| Operation | ReLU (bpd_relu) |
|---|---|
| Formula | f(x) = max(0, x) |
| Category | Piecewise linear |
This is the backend-neutral op_expr fact that specifies the operation. The same term is lowered to LLVM IR, CUDA C, MLIR, and every other backend — one specification, many targets.
op_expr(bpd_relu,
sel(is_nan(var), var,
sel(ge(var, const(0.0)), var, const(0.0)))).
| Backend | Rust/Oxide (rust_oxide) |
|---|---|
| Target | gpu |
| Description | Rust/cuda-oxide. Compiled with rustc + cuda-oxide runtime. |
This (operation, backend) pair has not yet been measured. When the backend emitter and verification infrastructure are ready, this cell will be tested against the PyTorch reference.
