Rectified Linear Unit
The name ReLU is a metonymy—a figure of speech where a thing is called by the name of something associated with it. In the neural network literature of the 1980s, individual neurons were called “units”—processing units, hidden units, output units. When Nair and Hinton (2010) introduced a unit whose activation function was the rectified linear function max(0, x), they named the unit: a Rectified Linear Unit.
Through metonymy, the name of the unit became the name of the function. Today, “ReLU” refers to max(0, x) itself, and the word “unit” in the acronym has become invisible—absorbed into the proper noun. When we write “ReLU activation,” we are saying “Rectified Linear Unit activation”—redundant in the same way as “ATM machine.” The unit is the activation; the activation is the unit.
In the diagram above, “Rectified” labels the zero-clamped region (x < 0) and “Linear” labels the identity region (x ≥ 0). The “Unit” is not a region of the graph—it is the neuron that uses this curve as its activation function.
| Operation | ReLU (bpd_relu) |
|---|---|
| Formula | f(x) = max(0, x) |
| Category | Piecewise linear |
This is the backend-neutral op_expr fact that specifies the operation. The same term is lowered to LLVM IR, CUDA C, MLIR, and every other backend — one specification, many targets.
op_expr(bpd_relu,
sel(is_nan(var), var,
sel(ge(var, const(0.0)), var, const(0.0)))).
| Backend | LLVM NVPTX (llvm_nvptx) |
|---|---|
| Target | nvptx |
| Description | LLVM IR targeting NVIDIA PTX (GPU). Compiled with llc for nvptx64, loaded via CUDA driver API. |
This is the actual LLVM IR (NVPTX) emitted by our Prolog/Logtalk toolchain for this operation. Every instruction was generated from a declarative op_expr specification — no hand-written assembly.
; GENERATED from op_expr(bpd_relu) via lower_llvm — backend 4 (shared AST).
define void @relu(float* noalias %src, float* noalias %dst, i64 %n) {
entry:
br label %loop
loop:
%i = phi i64 [0, %entry], [%inext, %body]
%cond = icmp slt i64 %i, %n
br i1 %cond, label %body, label %done
body:
%sp = getelementptr float, float* %src, i64 %i
%v = load float, float* %sp, align 4
%l0 = fcmp uno float %v, %v
%l1 = fadd float 0.0, 0.0
%l2 = fcmp oge float %v, %l1
%l3 = fadd float 0.0, 0.0
%l4 = select i1 %l2, float %v, float %l3
%l5 = select i1 %l0, float %v, float %l4
%dp = getelementptr float, float* %dst, i64 %i
store float %l5, float* %dp, align 4
%inext = add i64 %i, 1
br label %loop
done:
ret void
}
| Tier | SILVER — 0 ULP, robustness testing pending |
|---|---|
| ULP Divergence | 0 |
| Source Commit | b03bc23f7 |
| Deployment | deployed |
| Measured | 2026-09-25 |
| Hardware | NVIDIA Tesla P4 (enclave GPU) |
| Gate | Output compared to PyTorch reference on identical input tensor. 0 ULP = every floating-point bit matches. |
