Single-core and MPI-parallel performance of Q3, including comparisons against the Quantum ESPRESSO (QE) develop branch (commit 0f3d9044f2, version string 7.5, LibXC-enabled build at build_libxc/bin/pw.x).

Computational Environment

Property	Value
Architecture	Apple Silicon (arm64)
CPU	Apple M2
Physical cores	8 (4P + 4E)
Memory	16 GB
OS	macOS 26.5.1
Q3 compiler	Apple clang 21.0.0 (clang-2100.1.1.101)
Q3 serial build	Release (`-O3 -DNDEBUG`), Accelerate + FFTW3
QE build	LibXC-enabled (`build_libxc/bin/pw.x`)
Parallelism	Serial, MPI (band/pool), FFTW threads

That table describes the machine the hand-written comparisons below were taken on. The section that follows is different: it is re-measured on demand by a script, on whatever machine runs it, and each of its tables names its own.

Tracked Benchmark Decks

Generated by scripts/benchmark/generate_performance.py from the baselines in docs/benchmarks/. Do not edit this section by hand; edit the baseline it came from, or take a new one. Everything outside the generated markers in this file is written by hand.

The decks are listed in tests/regression/benchmark_matrix.yml and re-measured by scripts/benchmark/run_benchmarks.py, which reads the per-phase figures out of the timings_ms block the engine already writes into its results file. Runs are single threaded with deterministic FFT planning; Solver (ms) is the Davidson total, which excludes process start-up, input parsing and the property run that Wall (s) covers. Spread is the relative standard deviation of the repetitions kept, and is the figure that says how large a change has to be before it means anything.

Numbers from two machines are not comparable, so each machine is listed separately and scripts/benchmark/compare_benchmarks.py refuses to compare across them.

github-ubuntu-24.04-arm

github-ubuntu-24.04-arm aarch64, 2 logical cores, Linux 6.17.0-1022-azure. Recorded 2026-10-08T03:31:03Z from commit 1852c66a46d7. Source: docs/benchmarks/baseline-github-ubuntu-24.04-arm.json.

Deck	Wall (s)	Solver (ms)	Spread	SCF	Davidson	FFT	Energy (Ry)
`copper_norm_conserving`	4.03	2,652.1	0.1%	12	321	9,395	-386.583378
`silicon_norm_conserving`	0.77	201.5	0.3%	7	53	929	-16.913418
`silicon_ultrasoft`	1.22	245.7	0.2%	9	79	1,290	-15.717312

Solver stage medians, milliseconds:

Deck	seed	orthog.	H apply	subspace	diag.	collapse
`copper_norm_conserving`	745.7	401.5	836.4	276.2	21.4	365.0
`silicon_norm_conserving`	62.9	19.0	87.7	12.1	1.5	17.3
`silicon_ultrasoft`	70.1	39.6	96.6	12.9	2.4	22.9

github-ubuntu-latest

github-ubuntu-latest AMD EPYC 9V74 80-Core Processor, 2 logical cores, Linux 6.17.0-1022-azure. Recorded 2026-10-08T03:31:01Z from commit 1852c66a46d7. Source: docs/benchmarks/baseline-github-ubuntu-latest.json.

Deck	Wall (s)	Solver (ms)	Spread	SCF	Davidson	FFT	Energy (Ry)
`copper_norm_conserving`	4.05	2,524.2	0.3%	12	321	9,395	-386.583378
`silicon_norm_conserving`	0.87	186.6	1.1%	7	53	929	-16.913418
`silicon_ultrasoft`	1.32	227.3	0.9%	9	79	1,290	-15.717312

Solver stage medians, milliseconds:

Deck	seed	orthog.	H apply	subspace	diag.	collapse
`copper_norm_conserving`	660.5	428.5	720.5	292.9	22.0	396.9
`silicon_norm_conserving`	55.9	19.9	77.6	12.9	1.2	18.7
`silicon_ultrasoft`	62.0	42.4	81.8	13.7	2.2	24.6

linux-x86_64-4-core

linux-x86_64-4-core Intel(R) Xeon(R) Processor @ 2.80GHz, 4 logical cores, Linux 6.18.44-fc-v37. Recorded 2026-09-21T02:01:14Z from commit 9cc05c16e915. Source: docs/benchmarks/baseline-linux-x86_64-4-core.json.

Taken on a shared 4-core x86_64 development VM, not a quiet benchmark rig: single threaded, deterministic FFT planning, Release build, other jobs running on the same box. A run of the same decks with the box idle was 6-7% faster on the solver total, which is the size of the effect a busy machine has here. A same-machine reference, never a cross-machine comparison.

Deck	Wall (s)	Solver (ms)	Spread	SCF	Davidson	FFT	Energy (Ry)
`copper_norm_conserving`	6.45	3,914.7	4.6%	12	321	9,395	-386.583379
`silicon_norm_conserving`	1.25	281.3	10.3%	7	53	929	-16.913418
`silicon_ultrasoft`	1.85	320.3	1.9%	9	79	1,290	-15.717312

Solver stage medians, milliseconds:

Deck	seed	orthog.	H apply	subspace	diag.	collapse
`copper_norm_conserving`	1,096.4	599.0	1,215.5	400.4	36.3	559.0
`silicon_norm_conserving`	87.5	28.3	119.2	15.7	2.1	27.9
`silicon_ultrasoft`	90.5	53.5	122.2	17.7	3.5	31.8

MPI K-Point Pool Scaling — Cu NC 8×8×8

Parallel scaling benchmark using k-point pool parallelism (each MPI rank owns a pool of k-points). System: Cu primitive cell with an 8×8×8 k-grid (60 irreducible k-points after symmetry reduction), norm-conserving pseudopotential, ecutwfc = 120 Ry.

All runs use identical inputs — same geometry, cutoffs, smearing (Gaussian, σ = 0.02 Ry), XC functional (LibXC PBE), and convergence threshold (10⁻⁸ Ry). Q3 and QE use --pools N and -npool N respectively, with pools = np (one rank per pool).

Scaling Results

np	Q3 (s)	Q3 speedup	Q3 eff	QE (s)	QE speedup	QE eff	Q3/QE	ΔE (Ry)
1	9.47	1.00×	100%	16.84	1.00×	100%	0.56×	2.5×10⁻⁷
2	5.35	1.77×	88%	8.51	1.98×	99%	0.63×	2.4×10⁻⁷
4	3.70	2.56×	64%	5.55	3.03×	76%	0.67×	2.4×10⁻⁷

Key observations:

Q3 is ~1.8× faster than QE at np=1 (Q3/QE ratio = 0.56).
Scaling efficiency is similar — Q3: 64% at np=4, QE: 76% at np=4.
Energies agree to ~2.5×10⁻⁷ Ry across all configurations (expected numerical XC noise).
Topology invariance: Q3 total energy is identical to within 10⁻⁹ Ry across np=1, 2, 4 (confirming correct inter-pool reductions for charge density, Fermi energy, band energy, and smearing entropy).
Remaining parallel overhead at np=4 comes from the charge density allreduce on the full real-space grid and MPI startup cost relative to the short (~4 s) total runtime. (The ~200 scalar inter-pool MPI_Allreduce calls the Fermi bisection used to issue per SCF iteration were measured here; it now gathers the eigenvalue table once per call.)

Extended Benchmark Suite — 8 Use Cases (NC + US)

Comprehensive benchmark run (July 2026). All cases use identical inputs — same geometry, k-grid, cutoffs, smearing, mixing (Broyden, β matching per case), and XC functional (LibXC PBE, XC-000I-000I-101L-130L in QE). Each case is a single clean isolated run (no concurrency, no warm cache). All times are wall-clock seconds on a single core.

Benchmark Setup

Parameter	Value
SCF convergence	10⁻⁸ Ry
Si k-grid (primitive)	2×2×2 + (½,½,½) shift
Cu k-grid (primitive)	4×4×4 + (½,½,½) shift
Supercell k-grid	1×1×1 (Γ-only)
Si smearing	none (semiconductor)
Cu smearing	Gaussian, σ = 0.02 Ry
Force criterion (relax)	5×10⁻³ Ry/Bohr

Group 1 — Primitive Cell SCF

System	PP	k-mesh	Q3 E (Ry)	QE E (Ry)	\|ΔE\| (mRy)	t_Q3 (s)	t_QE (s)	t_Q3/t_QE
Si	NC	2×2×2	-16.913418	-16.913418	0.00	0.9	1.0	0.91×
Si	US	2×2×2	-15.717312	-15.717312	0.00	1.0	1.4	0.68×
Cu	NC	4×4×4	-386.583379	-386.583379	0.00	2.2	2.8	0.78×
Cu	US	4×4×4	-403.779229	-403.779215	0.01	1.0	1.1	0.92×

All 4 primitive cells agree within 0.01 mRy. Q3 is 1.1–1.5× faster than QE.

Group 2 — Supercell SCF (3×3×3, Γ-only, equilibrium geometry)

System	PP	Nat	Q3 E (Ry)	QE E (Ry)	\|ΔE\| (mRy)	t_Q3 (s)	t_QE (s)	t_Q3/t_QE
Si	NC	54	-455.773126	-455.773119	0.01	39.0	107.7	0.36×
Si	US	54	-423.494514	-423.494645	0.13	40.8	94.1	0.43×
Cu	NC	27	-10438.575344	-10438.575342	0.00	52.3	122.9	0.43×
Cu	US	27	-10902.864098	-10902.863719	0.38	28.4	34.5	0.82×

Q3 is 1.2–2.8× faster than QE on all 4 supercell SCF cases. All energies agree within 0.38 mRy.

Group 3 — Supercell SCF (3×3×3, Γ-only, displaced geometry)

Starting geometry: thermally distorted 3×3×3 supercell with all atoms displaced ~0.02–0.04 Bohr. These are single SCF calculations on the displaced structure (with forces enabled), not full relaxation runs.

System	PP	Nat	Q3 E (Ry)	QE E (Ry)	\|ΔE\| (mRy)	t_Q3 (s)	t_QE (s)	t_Q3/t_QE
Si	NC	54	-455.769499	-455.769493	0.01	39.3	125.7	0.31×
Si	US	54	-423.491083	-423.491094	0.01	41.8	86.9	0.48×
Cu	NC	27	-10438.574366	-10438.574363	0.00	81.0	429.7	0.19×
Cu	US	27	-10902.863134	-10902.862754	0.38	40.8	87.9	0.46×

Q3 is 2.1–5.3× faster than QE on all 4 displaced supercell cases. All energies agree within 0.38 mRy.

Note: QE Cu supercell cases use mixing_beta = 0.2 (matching Q3) since QE's default mixing_beta = 0.7 fails to converge for the displaced Cu US 3×3×3 geometry. QE's Broyden mixer has a hard limit of maxmix = 25 on mixing_ndim, so it uses mixing_ndim = 8 (vs Q3's mixing_ndim = 32).

Key observations:

Q3 is consistently 1.1–2.8× faster than QE across all 8 primitive and supercell cases on a single core. Supercells show larger speedups (up to 2.8×) than primitive cells (1.1–1.5×).
All energies agree within 0.38 mRy (worst case: Cu US, due to different mixing convergence paths). Most cases agree within 0.01 mRy. Primitive cell energies now agree to 0.00 mRy thanks to corrected matching parameters.
Gamma-only force fix (June 2026): Corrected the nonlocal force and becsum formulas for half-sphere G-vector accumulation. Before the fix, spurious forces of ~0.3 Ry/Bohr appeared at equilibrium for gamma-only supercell calculations. After the fix, equilibrium forces are ~10⁻⁵ Ry/Bohr (correct).

Group 4 — Supercell Relaxation (3×3×3, Γ-only, displaced geometry)

Full ionic relaxation of 3×3×3 supercells with all atoms displaced ~0.02–0.04 Bohr from equilibrium. Q3 optimizer: ASE full BFGS (alpha=30, maxstep=0.04 Å). QE optimizer: built-in trust-radius BFGS (‘ion_dynamics = 'bfgs’`).

System	PP	Nat	Q3 E (Ry)	QE E (Ry)	\|ΔE\| (mRy)	Q3 steps	QE steps	t_Q3 (s)	t_QE (s)	t_Q3/t_QE
Si	NC	54	-455.773069	-455.772952	0.12	3	2	145.6	335.4	0.43×
Si	US	54	-423.494426	-423.494551	0.13	4	3	245.7	388.8	0.63×
Cu	NC	27	-10438.575324	-10438.575297	0.03	4	3	589.2	1085.5	0.54×
Cu	US	27	-10902.864095	-10902.863667	0.43	3	3	146.4	348.1	0.42×

Q3 is 1.6–2.4× faster than QE for all supercell relaxation cases despite sometimes needing 1 more BFGS step, because Q3's per-step SCF is 2.0–3.1× faster. All energies agree within 0.43 mRy.

Note on BFGS alpha tuning: ASE's BFGS optimizer was tuned with alpha=30 (initial Hessian diagonal) to match QE's step efficiency. This reduces BFGS steps from 4 (default alpha=70) to 3 for Si NC, matching QE's total force evaluation count (4 vs 3). Values below ~20 cause oscillations; values above ~50 are overly conservative.

Primitive Cell Detailed Benchmarks (10-Run Averages)

Each cell shows the average of 10 runs. Wall time is elapsed real time; user time is CPU time in user mode. These runs use 10⁻⁸ Ry convergence.

Example	Q3 wall (s)	Q3 user (s)	QE wall (s)	QE user (s)	Q3/QE wall	Q3/QE user
Si NC	0.585	0.562	0.753	0.689	0.78×	0.82×
Si US	1.022	0.986	1.064	0.972	0.96×	1.01×
Cu NC	2.275	2.192	3.193	2.767	0.71×	0.79×
Cu US	1.098	1.027	1.172	0.996	0.94×	1.03×

Note on scope: Q3 timings include forces and stress tensor computation. QE timings are for SCF only (forces and stress are separate post-processing steps in QE). Despite this additional work, Q3 matches or exceeds QE wall-clock performance across all examples.

Raw Data

Si NC — semiconductor, norm-conserving, ecutwfc=48 Ry, 2×2×2 k-grid

Run	Q3 wall	Q3 user	QE wall	QE user
1	0.59	0.56	0.87	0.71
2	0.57	0.56	0.74	0.68
3	0.57	0.56	0.74	0.69
4	0.57	0.56	0.76	0.70
5	0.65	0.56	0.73	0.68
6	0.58	0.57	0.74	0.69
7	0.58	0.56	0.74	0.69
8	0.58	0.56	0.73	0.68
9	0.58	0.56	0.74	0.68
10	0.58	0.57	0.74	0.69

Si US — semiconductor, ultrasoft, ecutwfc=40 Ry, 2×2×2 k-grid

Run	Q3 wall	Q3 user	QE wall	QE user
1	0.99	0.97	1.10	0.96
2	1.00	0.98	1.04	0.97
3	0.99	0.97	1.14	0.99
4	0.99	0.97	1.04	0.97
5	1.04	1.00	1.07	0.98
6	1.11	1.03	1.08	0.98
7	1.07	0.98	1.10	0.98
8	0.99	0.98	1.03	0.97
9	1.05	1.00	1.02	0.96
10	0.99	0.98	1.02	0.96

Cu NC — metal (smearing), norm-conserving, ecutwfc=120 Ry, 4×4×4 k-grid

Run	Q3 wall	Q3 user	QE wall	QE user
1	2.25	2.18	2.99	2.73
2	2.22	2.18	3.11	2.85
3	2.20	2.16	2.99	2.78
4	2.19	2.15	2.98	2.75
5	2.34	2.19	4.42	3.07
6	2.29	2.21	3.62	2.83
7	2.39	2.25	2.90	2.65
8	2.22	2.18	3.02	2.69
9	2.29	2.18	2.83	2.63
10	2.36	2.24	3.07	2.69

Cu US — metal (smearing), ultrasoft, ecutwfc=40 Ry, 4×4×4 k-grid

Run	Q3 wall	Q3 user	QE wall	QE user
1	1.17	1.04	1.12	0.99
2	1.10	1.05	1.10	0.98
3	1.09	1.04	1.08	0.97
4	1.10	1.03	1.26	1.01
5	1.10	1.03	1.09	0.97
6	1.12	1.00	1.05	0.96
7	1.02	1.00	1.15	1.00
8	1.09	1.02	1.55	1.08
9	1.00	0.98	1.15	0.99
10	1.19	1.08	1.17	1.01

Supercell SCF — Component Breakdown

Detailed per-component timing and energy breakdown for 3×3×3 supercell SCF calculations. These are single-run measurements at 10⁻⁸ Ry convergence from an earlier code state.

Silicon Norm-Conserving (NC) 3×3×3 Supercell

Cutoffs: 48.0 Ry (wavefunction) / 192.0 Ry (charge density)
K-points: Gamma point only
SCF Convergence Threshold: 10⁻⁸ Ry

Energy Comparison

Quantity / Term	Q3 (Ry)	QE (Ry)	Difference (Ry)
Total Energy ( \(E_{\text{tot}}\))	-455.77312593	-455.77311912	6.81e-6 (0.13 μRy/atom)
Hartree Energy	31.66161134	31.66131606	0.000295
Exchange-Correlation (XC)	-167.05947712	-167.05933226	-0.000145
Ewald Energy	-450.35925776	-450.35925776	0.000000

Single-Core Runtimes

Stage / Component	Q3 Time	QE Walltime	Q3 Speedup
Initialization	2.38 s	6.84 s	2.87×
SCF Iteration Loop (10 iterations)	32.34 s	150.70 s	4.66×
Forces + Stress Property Run	5.10 s	14.13 s	2.77×
Total Execution Time	39.81 s	171.83 s	4.32×

Silicon Ultrasoft (USPP) 3×3×3 Supercell

Cutoffs: 40.0 Ry (wavefunction) / 200.0 Ry (charge density)
K-points: Gamma point only
SCF Convergence Threshold: 10⁻⁸ Ry

Energy Comparison

Quantity / Term	Q3 (Ry)	QE (Ry)	Difference (Ry)
Total Energy ( \(E_{\text{tot}}\))	-423.49451417	-423.49464665	0.000132 (2.4 μRy/atom)
Hartree Energy	31.74243936	31.74248564	-0.000046
Exchange-Correlation (XC)	-130.69600023	-130.69621080	0.000211
Ewald Energy	-450.35925776	-450.35907827	-0.000179

Single-Core Runtimes

Stage / Component	Q3 Time	QE Walltime	Q3 Speedup
Initialization	4.23 s	5.58 s	1.32×
SCF Iteration Loop	29.26 s (9 iter)	42.40 s (13 iter)	1.45×
Forces + Stress Property Run	4.45 s	2.89 s	0.65×
Total Execution Time	37.93 s	51.11 s	1.35×

Copper Norm-Conserving (NC) 3×3×3 Supercell

Cutoffs: 120.0 Ry (wavefunction) / 480.0 Ry (charge density)
K-points: Gamma point only
SCF Convergence Threshold: 10⁻⁸ Ry
Number of Bands: 280

Energy Comparison

Quantity / Term	Q3 (Ry)	QE (Ry)	Difference (Ry)
Total Energy ( \(E_{\text{tot}}\))	-10438.57534364	-10438.57534212	-1.52e-6 (0.056 μRy/atom)
Hartree Energy	3217.02663079	3217.03332908	-0.006698
Exchange-Correlation (XC)	-1302.61698818	-1302.61755219	0.000564
Ewald Energy	-6530.37939068	-6530.37649116	-0.002899
Smearing contribution (-TS)	-0.09082740	-0.09082745	0.00000005

Single-Core Runtimes

Stage / Component	Q3 Time	QE Walltime	Q3 Speedup
Initialization	2.22 s	2.97 s	1.34×
SCF Iteration Loop (17 iterations)	79.21 s	104.88 s	1.32×
Forces + Stress Property Run	3.84 s	2.50 s	0.65×
Total Execution Time	85.27 s	110.35 s	1.29×

Note on performance optimization: By implementing a block orthogonalization algorithm using BLAS 3 (dgemm/zgemm) for projecting out the existing basis and a fast Cholesky QR algorithm (dpotrf/dtrsm) for the intra-batch orthogonalization, the Davidson diagonalizer bottleneck was resolved. The total orthogonalization time was reduced from 80.1 s to 5.0 s (a 16× speedup for orthogonalization), making Q3 overall 1.29× faster than Quantum ESPRESSO for this large copper supercell calculation.

Copper Ultrasoft (USPP) 3×3×3 Supercell

Cutoffs: 40.0 Ry (wavefunction) / 200.0 Ry (charge density)
K-points: Gamma point only
SCF Convergence Threshold: 10⁻⁸ Ry
Number of Bands: 280

Energy Comparison

Quantity / Term	Q3 (Ry)	QE (Ry)	Difference (Ry)
Total Energy ( \(E_{\text{tot}}\))	-10902.86409824	-10902.86371908	-0.000379 (14.0 μRy/atom)
Hartree Energy	3937.57145317	3937.57289925	-0.001446
Exchange-Correlation (XC)	-1047.64832799	-1047.64804720	-0.000281
Ewald Energy	-6530.37939068	-6530.37649116	-0.002899
Smearing contribution (-TS)	-0.09121883	-0.09121917	0.00000034

Single-Core Runtimes

Stage / Component	Q3 Time	QE Walltime	Q3 Speedup
Initialization	1.22 s	5.58 s	4.57×
SCF Iteration Loop (41 iterations)	73.07 s	38.88 s	0.53×
Forces + Stress Property Run	3.11 s	2.89 s	0.93×
Total Execution Time	74.29 s	39.56 s	0.53×

Note on Broyden mixer optimization: In this work, the Broyden density mixer's G-space operations and history storage were restricted strictly to the smooth G-vector grid (ngms cutoff) to precisely match Quantum ESPRESSO's mathematical formulation. This prevents the highly oscillatory and noisy ultrasoft augmentation charge components (hard G-vectors from ngms to ngm) from corrupting the Broyden subspace. This change successfully resolved the flaky convergence behavior, ensuring stable convergence in 41 iterations across different random seeds/trials.

Update (June 2026): After further fixes to the D_eff integral formula in the ultrasoft initialization and the Broyden w₀² regularization parameter, Cu US 3×3×3 SCF now converges in 17 iterations (vs 41 previously), reducing total runtime from 74.29 s to 46 s at 10⁻⁶ Ry convergence (a 1.61× additional speedup), and achieving Q3/QE = 0.58× (Q3 is 1.7× faster than QE).

Geometry Relaxation Benchmarks

Primitive Cell Relaxation (Displaced Silicon)

We benchmarked the geometry relaxation workflow (displaced Silicon, 2-step BFGS relaxation) on a single physical core (serial execution). We compare the standard subprocess execution mode (default) against the in-memory python bindings mode (which caches pseudopotential sets and implements wavefunction/charge density preloading/restarts).

Silicon Norm-Conserving (NC) Relaxation (<tt>si_nc_relax</tt>)

Cutoffs: 48.0 Ry (wavefunction) / 192.0 Ry (charge density)
K-points: 2×2×2 grid (reduced by symmetry)
BFGS Convergence Threshold: 5.0×10⁻³ Ry/bohr (fmax)

Mode / Options	Use Symmetry	Relaxation Steps	Total Walltime (s)	Speedup vs Subprocess
Subprocess	Yes	2	2.64 s	1.00× (baseline)
Bindings (with Restart)	Yes	2	1.81 s	1.46×
Subprocess	No	2	2.34 s	1.00× (baseline)
Bindings (with Restart)	No	2	1.78 s	1.31×

Silicon Ultrasoft (USPP) Relaxation (<tt>si_us_relax</tt>)

Cutoffs: 40.0 Ry (wavefunction) / 200.0 Ry (charge density)
K-points: 2×2×2 grid (reduced by symmetry)
BFGS Convergence Threshold: 5.0×10⁻³ Ry/bohr (fmax)

Mode / Options	Use Symmetry	Relaxation Steps	Total Walltime (s)	Speedup vs Subprocess
Subprocess	Yes	2	3.33 s	1.00× (baseline)
Bindings (with Restart)	Yes	2	2.28 s	1.46×
Subprocess	No	2	3.43 s	1.00× (baseline)
Bindings (with Restart)	No	2	2.87 s	1.19×

Note on restart optimization: In bindings mode, Q3 caches the converged charge density and wavefunctions from the previous ionic step. Upon restarting the next step, the Davidson diagonalizer uses these preloaded states as initial trial vectors, allowing the diagonalizer to converge in exactly 1 iteration per k-point (down from 4–5 iterations on the first step). This restart logic, combined with zero disk/CLI launch overhead, provides up to 1.46× speedup overall.

Site-specific measurements

The benchmarks on this page were taken on GitHub Actions runners, which makes them comparable to each other and reproducible in CI, but not representative of an HPC node.

For NERSC Perlmutter — OpenMP threading, MPI k-point pools and band groups, ScaLAPACK, and an A100 measured against a Perlmutter CPU node — see docs/performance_nersc_perlmutter.md. The short version of what it found there:

k-point pools scale nearly linearly (28.9x on 32 ranks), and are the parallelism worth spending ranks on.
MPI band groups are slower than serial on a gamma-only system at every width tested.
OpenMP peaks at 8 threads and regresses beyond it — 32 threads are 42% slower, and 64 are slower than 4. Two stages cause it: threaded FFTW is applied within each transform rather than across bands, and the dense subspace eigensolve degrades 7.2x from 1 thread to 64. The first of those has since been addressed — the band loops in the Hamiltonian applicator and the density accumulator now thread across whole transforms, each thread driving its own FFT scratch, and FFTW's intra-transform threading is switched off whenever that path is available. These numbers predate that change and have not been re-measured on Perlmutter.
Binding threads (--cpu-bind=cores, OMP_PROC_BIND) costs about 45% on this workload, against the usual advice.
An A100 is about at parity with 8 CPU threads on the Cu 3×3×3 system, which is too small to occupy the device.

GPU Acceleration Benchmarks — Cu NC 3×3×3 Supercell

We benchmarked the GPU-accelerated Davidson solver on the GitHub Actions runner equipped with an NVIDIA Tesla T4 GPU. The system is the large Cu norm-conserving \(3\times3\times3\) supercell with 280 bands, 192,228 G-vectors, norm-conserving pseudopotential, and gamma_only = true.

All runs use identical parameters: convergence threshold of \(10^{-8}\) Ry, 3 SCF iterations, Broyden density mixing ( \(\beta = 0.20\)), and the LibXC PBE functional.

Computational Environment (GPU)

Property	Value
GPU	NVIDIA Tesla T4
VRAM	16 GB GDDR6
CUDA Version	12.8
CPU	Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz (4 vCPUs)
Host Memory	16 GB

Benchmark Results (GPU-Resident vs CPU Serial Baseline)

By storing wavefunctions, Ritz vectors, and intermediate solver arrays directly on the device using a unified GPUDavidsonWorkspace and executing all linear algebra (including Gamma-only real matrix multiplications via gpu_blas_.dgemm) on the GPU, we completely eliminated PCIe page faulting overhead.

This yields a 6.6x overall speedup and a 10.8x SCF iteration speedup against the CPU serial baseline on the same runner — one core of a 4-vCPU Xeon, with no threading. That is the comparison the table below makes, and it is the right one for attributing the gain to the GPU-resident solver rather than to the hardware. It is not a statement of what a GPU is worth against a well-configured CPU node: on Perlmutter the same system runs in 6.1 s on 8 CPU threads and 6.6 s on an A100, roughly parity. See docs/performance_nersc_perlmutter.md.

Metric / Stage	CPU Serial	CUDA GPU FP64	CUDA GPU FP32 Mixed
Total SCF Time	411.0 s	63.6 s	62.3 s
SCF Iteration Time (`iterate`)	384.0 s	36.7 s	35.3 s
Avg. Time per SCF Iteration	128.0 s	12.2 s	11.8 s
Nonlocal Projector Application	83.1 s	0.0 s (100% offloaded)	0.0 s (100% offloaded)
Davidson Solver Loop	335.6 s	28.2 s	28.2 s
— orthogonalize	—	9.96 s	9.96 s
— subspace_update	—	1.01 s	1.01 s

Key observations:

Zero PCIe Transfer Overhead: By keeping wavefunctions resident in the GPU VRAM, host-to-device copying is completely avoided during the Davidson iteration.
Real BLAS Offloading: Offloading the Gamma-only symmetric projections and subspace updates via real double-precision matrix multiplications (gpu_blas_.dgemm) resulted in a 9x speedup for orthogonalize (down from 90.6 s in the regressed version) and an 11x speedup for subspace_update (down from 11.4 s).

Memory Footprint — Si NC Supercells

Measured 2026-09-12 on a 10-core Apple M-series with 34 GB, Release build, OMP_NUM_THREADS=8, one SCF iteration. "Before" is commit badfd722; "after" is the same tree with the Davidson workspace reductions. Peak resident set from /usr/bin/time -l; named arrays and per-phase figures from the run's own probes.

Cell	Atoms	Bands	PW/k-point	Named arrays before	after	Peak RSS before	after	Total energy (Ry)
4×4×4	128	308	49,623	9.22 GiB	5.12 GiB	12.14 GB	8.29 GB	−1082.814878725
5×5×5	250	602	96,837	34.29 GiB	18.6 GiB	23.97 GB	23.31 GB	−2115.933549790

Total energies are identical to every printed digit in both cells.

The two rows differ in what sets the peak, and the per-phase probes say which. On the 250-atom cell the tallest point of the run is the atomic-orbital start, not the solver, so the solver reductions barely move the peak even though they plainly work — resident memory just after the first subspace collapse falls from 17.80 GiB to 9.08 GiB, a 49% reduction:

250-atom cell, resident after	before	after
projector caching	9.39 GiB	9.40 GiB
orbital rotation (peak)	18.74 GiB	20.93 GiB
Davidson workspace allocated	14.02 GiB	13.18 GiB
first subspace collapse	17.80 GiB	9.08 GiB

The 128-atom cell, where setup is smaller relative to the solver, is the one whose peak moves: 32% less resident memory for the same answer. What is left on the 250-atom cell is the setup spike, filed in TODOs.md §2.3.

Wall time falls on both, most on the larger cell where the pre-reduction binary is asking a 34 GB machine for more than it has: 522 s against 383 s at 250 atoms, with 44.3 M page reclaims against 37.2 M.

Projected for examples/total_energy--semiconductor--silicon/nc_6x6x6.yml (432 atoms, 1040 bands, 167,155 plane waves), which fits on neither CI nor a laptop: 108.6 GB of named arrays per rank before, 52.9 GB after at the expansion factor of 4 that was the default then, 40.6 GB at the factor of 2 that is the default now.

Quantum ESPRESSO holds 5 x nbnd x npw complex for its Davidson and wavefunctions against Q3's 10 at the default factor and 6 at factor 2, plus the same projector term in both. Per rank at the same rank count that is 36.4 GB against Q3's 40.6 GB at factor 2, from that code's own published sizing formula rather than a run — no such build was available on this machine. The difference that matters in practice is not the solver: Quantum ESPRESSO divides its plane-wave terms by the ranks in a pool and Q3 replicates them, so at 16 ranks it needs 2.3 GB per rank and Q3 still needs 40.6 GB.

That replication is what --distribute-bands does; ranks that share a pool without it now divide reciprocal space by FFT stick and keep the gamma-only half sphere while doing so. Measured on the 54-atom cell run to convergence with --pools 1 (tier-2 reference −455.773125931 Ry):

Ranks sharing the pool	Total energy (Ry)	Peak resident / rank, max / min
2	−455.773125930	1.60 / 1.53 GiB
4	−455.773125931	0.97 / 0.93 GiB
8	−455.773125931	0.70 / 0.68 GiB

Per-rank memory falls with the rank count, balanced within a few percent, where --distribute-bands holds 3.12 GiB on this deck at every width; repeated runs spread by about 0.05 GiB. What no longer shrinks past four ranks is the replicated part: the density G-vector list, the subspace matrices and the block-diagonal D matrix, filed in TODOs.md §2.3. The subspace matrices answer to method.scf.davidson.basis_expansion_factor. With the collapse no longer rebuilding its subspace from the vectors (collapse time 0.53 s to 0.19 s at factor 4 and 4.1 s to 1.9 s at factor 2 on this run), factor 2 converges the same four-rank run to the same energy in 93 Davidson iterations instead of 54, 21.4 s of solver time instead of 17.3 s, 32.0 s of wall time instead of 28.0 s, at an SCF-stage peak of 0.82 / 0.78 GiB instead of 0.93 / 0.90. That trade – 14% of wall time for 12% of SCF-stage memory here, and for 1.0 GB per rank of replicated subspace on the 432-atom deck – is why 2 is the default now, as it is in Quantum ESPRESSO.

The per-rank figures in the two tables above are SCF-stage peaks: until 2026-09-13 the run's own report took its last reading before the forces and stress. On this deck the forces stage – a second copy of the projectors and a copy of the occupied wavefunctions – used to sit 3 to 7% above the SCF's plateau at two to eight ranks, because the solver's working set, the projector cache and the mixing history were still alive under it. The iterator now releases those once the SCF has converged, and the run's peak is the SCF's; the report takes a fresh reading at the end and agrees with an external sampler to 0.01 GiB. The stress, which used to stand far above the SCF at one rank, has stayed below it since 2026-09-14.

Measured against Quantum ESPRESSO on the same machine

Quantum ESPRESSO 7.6 (pw.x, MPI, FFTW3, OpenBLAS, libxc, one pool, K_POINTS gamma) and Q3 (--pools 1, stick FFT, gamma-only, the default expansion factor of 2) on the 54-atom silicon cell, each run to convergence on an idle 10-core Apple M-series with 32 GB, one thread per rank, 2026-09-13. Peak resident set per rank sampled from ps every 0.5 s for both codes, so the two columns are the same quantity. Quantum ESPRESSO's own "Estimated max dynamical RAM per process" is listed because it is what camsai/q3#16 quoted for Quantum ESPRESSO: it sits three to four times below what Quantum ESPRESSO actually holds, so a measured Q3 figure must never be compared with it.

Ranks	Quantum ESPRESSO peak resident, max / min	Quantum ESPRESSO printed estimate	Quantum ESPRESSO wall	Q3 peak resident, max / min	Q3 wall
1	2.00 GiB	0.66 GB	74 s	1.84 GiB	47 s
2	0.96 / 0.95 GiB	0.33 GB	44 s	1.25 / 1.25 GiB	33 s
4	0.59 / 0.53 GiB	0.17 GB	23 s	0.91 / 0.86 GiB	33 s
8	0.36 / 0.33 GiB	0.09 GB	15 s	0.69 / 0.62 GiB	25 s

Quantum ESPRESSO converges to −455.77311913 Ry and Q3 to −455.773125931 Ry, 6.8e-6 Ry apart on 54 atoms: the same 1.3e-7 Ry per atom that separates the two codes on the 2-atom decks in reference.json, so a systematic difference in the functional or projector integration rather than anything in the parallel layout.

Repeated Q3 runs spread by up to 0.15 GiB at two ranks (1.25 to 1.40 GiB over four runs) and 0.05 GiB above that; the table carries the last run.

Both one-rank runs include forces and stress. Q3's one-rank row was re-measured on 2026-09-14, after the nonlocal stress stopped building every derivative projector for every atom at once: it took the run to 4.12 GiB, and working through the cell a few atoms at a time it stays below the SCF's own peak, so the figure is 1.84 GiB and the stress itself fell from 3.5 s to 1.9 s. From two ranks up Q3 skips the stress under a shared pool, and its figure is the SCF's own plateau, 1.3 to 1.5 times Quantum ESPRESSO's at two and four ranks and 1.9 times at eight.

Where that ratio comes from, read off one rank's address space at the plateau (vmmap, four ranks): 0.32 GiB of heap in use, which is what the run's own estimate accounts for; 0.12 GiB of code pages; and about 0.35 GiB of heap that was freed – the setup transients and the solver's per-iteration temporaries – and kept resident by the allocator. Q3 now asks the allocator to return freed pages at phase boundaries (release_unused_heap). Measured, that does not lower the peak inside an SCF iteration, because the next iteration's temporaries reuse the pages it would have returned; it lowers what a later phase starts from, which is why the one-rank figure fell from 4.83 to 4.12 GiB. Lowering the plateau itself means allocating the per-iteration temporaries once and reusing them, which is filed with the replicated terms – the density G-vector list with the Broyden history, and the subspace matrices held complex rather than real – as what separates the two columns.

Quantum ESPRESSO is faster from four ranks on: its all-to-all is on the cutoff sphere and Q3's is still on the whole box. Q3 is faster at one and two ranks.

Resolved Performance Investigations

Historical record of diagnosed-and-fixed performance issues. Open performance work is tracked in TODOs.md §2 only (single source of truth) — the Broyden G-space acceleration and band-parallel-vs-FFTW-threading benchmark items live there.

[x] Force noise floor for Cu supercells: Root cause identified as optimizer (QuasiNewton) ill-conditioning, not a force formula bug. Q3 analytical forces verified against numerical finite-difference (0.06% agreement for Si NC). Resolved by switching to BFGS.
[x] Si supercell relaxation: Diagnosed. Maxstep=0.2Å causes collective displacement that compresses Si–Si bonds, breaking SCF convergence. Fix: default maxstep=0.04Å (already the Q3 relax workflow default). Per-step time is 2.0–3.1× faster than QE.
[x] Better optimizer integration: Switched from ASE LBFGS to full BFGS with alpha=30 (initial Hessian diagonal). This reduces BFGS steps from 4 to 3 for Si NC 3×3×3, matching QE's step efficiency. See src/py/q3/workflows/relax.py.
[x] Cu supercell relaxation: Cu NC converges in 4 steps (589s vs QE 1086s, 1.8× faster). Cu US converges in 3 steps (146s vs QE 348s, 2.4× faster). All energies agree within 0.43 mRy.
[x] FFTW threading for single-rank runs: When band_size == 1 (np=1), the MPI execution policy now creates a SerialFFTGrid with FFTW thread support. Set OMP_NUM_THREADS=2 for ~1.8× FFT speedup on shared-memory architectures. Requires careful fftw_init_threads / fftw_mpi_init ordering (conditional on OMP_NUM_THREADS > 1).
[x] Zero-copy FFT in H|ψ⟩: Added input_buffer() / output_buffer() / execute_*_on_buffers() interface to FFTGrid. The non-gamma H|ψ⟩ loop now scatters/gathers directly to FFT internal buffers, eliminating 4 memcpy per band.
[x] GPU-Resident Davidson Solver & GPU BLAS (July 2026): Implemented GPU-resident wavefunction and Ritz vector storage (GPUDavidsonWorkspace) to avoid host-device copying. Resolved a severe regression (151.6 s total runtime) where the double-precision Gamma-only (gamma_only = true) path fell back to CPU BLAS dgemm_ on unified device buffers, triggering high PCIe page-faulting overhead. Offloaded the real-symmetric basis projection, candidate Cholesky QR overlaps, and subspace updates to gpu_blas_.dgemm, reducing orthogonalization time from 90.6s to 9.96s (a 9x speedup) and subspace update time from 11.4s to 1.01s (an 11x speedup). This yielded a 6.6x overall speedup (total runtime of 62.3s vs CPU 411.0s) and a 10.8x speedup on SCF iteration time.