Every number on this page was measured on a Tesla P4 (sm_61), launched and profiled from Prolog, with hardware counters. No vendor whitepapers, no estimates — probe kernels and counted sectors. Provenance at the foot of the page.
A noop-copy kernel — load every element, store it back, compute nothing — sets the floor under every elementwise op. The bars show measured read/write time at four working-set sizes:
Memory-bandwidth defines the no-op copy kernel. 100% of its runtime is spent moving bytes, as it performs no compute function. We use this kernel as a measurement reference, and to measure some basic parameters of our GPU’s performance.
To measure some basics about more realistic kernels, which perform computation, we study ReLU (Rectified Linear Units) because it is one of the simpler, element-wise computations. We do know that element-wise computations are typically limited by memory bandwidth. When you consider what kernel fusions to perform, where the kernel you are studying fits on this spectrum will decide which optimizations can make a difference:
Standalone elementwise — ALU invisible behind memory
Back-to-back kernels — cache warm, ALU emerges
In-place loop — pure compute, 42% Kolmogorov delta
In the register-resident regime the instruction-count delta between strict and fast relu emission — invisible behind memory in regime one — becomes a measured 42% wall-time difference.
An elementwise op can write its result back into its input buffer (halving cache footprint) or into a second buffer. Below the L2 boundary it makes no difference. AT the boundary (N=300K, ~2.4MB two-buffer footprint vs 2MB L2), the two-buffer version spills — and pays:
| N | layout | time μs | L2 read misses | DRAM read sectors |
|---|---|---|---|---|
| 6144 | two_buffer | 4.44 | 48 | 103 |
| 6144 | inplace | 4.43 | 48 | 107 |
| 125000 | two_buffer | 4.6 | 976 | 2167 |
| 125000 | inplace | 4.59 | 976 | 2202 |
| 300000 | two_buffer | 15.83 | 2336 | 5130 |
| 300000 | inplace | 6.13 | 2336 | 5166 |
At the L2 boundary, in-place is 2.6× faster — its read-floor sits at ZERO because every read hits cache the writes just warmed.
Fair comparison: L2 flushed between measurements, both orderings tested, result order-independent.
L1: 48KB per SM. L2: 2MB shared across the chip. DRAM: 8GB. Timing a size-sweep under two cache configurations (30 runs each, fresh flush between):
| N elements | 6144 | 12288 | 24576 | 32768 | 65536 | 131072 |
|---|---|---|---|---|---|---|
| PREFER_L1 μs | 6.5 | 6.7 | 6.75 | 6.97 | 7.42 | 8.43 |
| PREFER_NONE μs | 7.18 | 7.29 | 7.5 | 7.57 | 7.76 | 8.56 |
Caveat, honestly held: the PREFER_NONE penalty (~9%) is driver decision overhead at launch, not cache behavior — the hardware cache allocation is the same.
L1 partially survives kernel boundaries: 64 cache lines persisted between two launches — 7% reuse, measured by read-miss deltas.
The steady-state 7.5 μs launch+sync, decomposed by measurement:
For a 6K-element kernel, ~98% of wall time is asking the GPU to do something. ~2% is the GPU doing it.
Caveat, honestly held: the 4.7 μs sync-poll term is a host-side polling artifact (the CPU waiting to notice completion), not GPU-intrinsic cost. First-launch costs 51 μs unconfigured, 23.9 μs with cache-config set; steady-state 7.5 μs.
Same bytes, same op — only the access pattern changes. L2 sector counts show WHY: stride-32 touches 8× the sectors for the same data:
| pattern | direction | time μs | L2 sectors |
|---|---|---|---|
| coalesced | read | 6.04 | 976 |
| stride_2 | read | 5.77 | 976 |
| stride_32 | read | 15.45 | 7826 |
| reversed | read | 5.24 | 976 |
| coalesced | write | 4.44 | 976 |
| stride_2 | write | 4.45 | 986 |
| stride_32 | write | 24.27 | 7820 |
| reversed | write | 4.48 | 978 |
Stride-32 writes run 5.5× slower. Reversed order runs at coalesced speed — the hardware coalesces by address-set per warp, not by thread order.