main_mpi.cpp File Reference

#include "layer_0_core/profiling/counters.hpp"
#include <algorithm>
#include <chrono>
#include <filesystem>
#include <stdexcept>
#include <string>
#include "io/addon_registration.hpp"
#include "io/cli.hpp"
#include "io/input/parser/factory.hpp"
#include "io/logger_mpi.hpp"
#include "io/output/companion_outputs.hpp"
#include "layer_4_computation/flavors/factory.hpp"
#include "layer_4_computation/iterator/kohn_sham/planewave/execution/mpi.hpp"
#include "utils/time.hpp"

Go to the source code of this file.

Functions
int	main (int argc, char *argv[])

Function Documentation

main()

int main	(	int	argc,
		char *	argv[]
	)

Answered before the input is resolved, so --version works in a directory holding no input – which is where it is usually asked. MPI is already up, so init_mpi() has silenced every rank but the root and the line is printed once rather than once per rank; returning here runs the policy's destructor, which finalizes MPI.

Apply the requested log level only after init_mpi(): init replaces the default logger, so applying it earlier would be silently discarded.

Capture the rank now, while MPI is guaranteed live. The catch block below needs it to attribute a fatal error, and asking there would mean asking whether MPI is still up first – MPI_Comm_rank after MPI_Finalize is itself undefined.

Every rank reads it and applies its verbosity; a non-root rank's console sink is already off, so that reaches only the rank's own log file.

An explicit CLI value overrides the input manifest. Otherwise honor control.number_of_kpoint_pools; zero in either source selects the documented one-pool default. Every rank parses the same input and configures the topology before the first physics collective.

Switch the pool from splitting reciprocal space to splitting bands.

This must happen before the basis is generated: it is what makes band_size() report 1, which in turn keeps the full G-sphere on every rank, selects the serial FFT grid, and leaves stress enabled, which does not survive the reciprocal-space split.

It also has to happen after configure_topology, which builds the inter-pool communicator by splitting on the pool-local rank index. That pairing – rank i of every pool with rank i of the others – is what makes several pools work alongside distributed bands: within a pool every rank ends up holding the same whole density, because the bands were gathered back, so summing along the inter-pool axis gives every rank the total over all k-points. The pairing was written for the slab split, where rank i owns a matching slab in each pool; whole grids are the degenerate case of that and reduce just as correctly.

Report the topology before the first physics collective, so a run that stops later still says how its ranks were arranged. Without this the log is silent about rank count, pool count and ranks per pool, which makes "it hangs above N ranks" impossible to diagnose from output alone. The pool count printed here is what was asked for; the factory may still clamp it to the irreducible k-point count.

Report the true ranks per pool rather than band_size(). The two agree for the reciprocal-space split, where band_size() is how many ranks share a pool – but enabling band distribution sets it to 1 on purpose, to signal that each rank now holds whole coefficient vectors. Printing it here would then claim one rank per pool on a run using four, which is exactly the thing this line exists to make diagnosable.

Every flavor builds its iterator on this rank's policy, so a unit of a workflow – the SCF, the non-self-consistent pass of a band structure or a DOS, each step of a relaxation – runs on the same ranks and topology. The policy stays owned here rather than by the iterator: it outlives the flavor, for the reductions after the run and for MPI_Finalize, which its destructor calls at the end of main.

Reduce counters across all ranks so the summary reflects the total work done across all pools, not just rank 0's contribution.

Memory is reported as a spread rather than a sum. Every rank holds its own copy of the workspace, so the interesting figures are the largest rank (what a node must fit) and the smallest (whether the ranks are in fact symmetric, which under –distribute-bands they are meant to be).

Report from whichever rank threw, on stderr, rather than through log::error(): init_mpi() silences every non-root rank, so a failure raised there would say nothing at all. That is not hypothetical – the factory's shared-pool refusal is raised identically on every rank, and the first to exit nonzero takes the whole job down with it, including rank 0 part-way through printing. The run then exited nonzero with no reason given, intermittently and only under load. Every rank that reaches here now says so, flushed, before it goes.

Abort so that a rank failing alone cannot leave its peers blocked in a collective forever. The execution policy is still alive here – it is owned by main, outside the try block – so nothing has finalized MPI on the way: MPI_Finalize is collective, and reached from a rank-local failure it would block until peers that are still doing physics arrived, which is never. The abort ends the process before the policy's destructor runs.

The guard remains because MPI_Abort after MPI_Finalize is undefined – in practice OpenMPI buries the message above under a wall of standard-violation warnings and exits 14 instead of the 1 asked for here – and a finalized MPI means every rank reached the collective MPI_Finalize, so none is left hanging and returning normally is safe.

MPI_Finalize is called by PlanewaveMPIExecutionPolicy's destructor, as policy goes out of scope on this return, after every rank has finished the run and the reductions above.

Definition at line 23 of file main_mpi.cpp.

References q3::create_flavor(), q3::create_kohn_sham_iterator(), q3::ProfilingCounters::davidson_iteration_count, q3::log::fatal(), q3::ProfilingCounters::fft_call_count, q3::ResidentMemorySample::format_gibibytes(), q3::ProfilingCounters::hamiltonian_application_count, q3::log::info(), q3::log::init_mpi(), q3::ProfilingCounters::instance(), q3::cli::parse_options(), q3::ProfilingCounters::peak_resident_bytes_now(), q3::cli::print_footer(), q3::cli::print_header(), q3::cli::print_version(), q3::cli::read_input(), q3::addons::register_all(), q3::log::set_level(), q3::write_companion_outputs(), and q3::write_results_json().