Rainbow — CPU and Multi-machine Execution

Parent: Rainbow

Execution model

Use C++20/OpenMP across cores within a process. Support multiple sockets by allowing a process per NUMA domain with disjoint jobs or particle shards; benchmark this against a single process spanning sockets. Extend the same independent-job interface to user-selected machines. Do not assume multiple processors means multiple hosts, and do not launch remote workers without an identified authorized host list.

Pipeline

Resolve configuration -> acquire/build immutable optics tables -> generate deterministic particle batches -> evaluate optics and project -> bin projected footprints into image tiles -> render tiles with exclusive pixel ownership -> checkpoint linear results -> merge -> convert colour and tone-map preview. Stream batches under a memory budget; retain or regenerate particles by stable ID as appropriate.

Shared-memory performance

Positions, radii and weights use contiguous arrays; velocity is optional. No per-particle heap allocation or virtual dispatch in hot loops. Batch optical lookup coherently by radius where useful, then bin compact projected records by tile. Evaluate optics once per record, not per overlapping tile. Each tile has one writer; include footprints crossing tile boundaries without duplicating deposited energy. Large PSFs and long streaks need a measured alternative to excessive bin duplication.

Expose OpenMP thread count, scheduling and affinity. Use vectorization reports and profiling before architecture-specific intrinsics. Benchmark physical cores and hardware threads separately. Avoid nested thread-pool oversubscription, shared counters in hot loops and false sharing. On NUMA systems initialize data near its workers and consider per-socket copies of small read-mostly tables. Use double precision for reference optics and accumulation; lower precision requires quantified validation.

Independent jobs

Parallel work units include radius/wavelength scattering-table blocks, parameter-sweep configurations, frames, independent realizations and disjoint particle-ID shards. Prefer complete independent runs for sweeps; this avoids image merging and often uses multiple machines most efficiently. One-image acceleration uses particle shards producing complete linear foreground accumulators. The coordinator supplies global particle population/sample counts and weights; a worker must not renormalize its shard into a full cloud.

Merge and reproducibility

Every shard carries config/table hashes, global sample definition, shard interval, seed, exposure and linear colour-space metadata. Merge checks compatibility, rejects missing/duplicate/overlapping shards, and adds each foreground contribution exactly once. Background is added once after merging. Realization averages and Monte Carlo sample estimates use declared normalization rather than blindly summing complete images. Checkpoints preserve sums and sample counts.

Stable particle IDs and separate random streams for position, size, motion and numerical integration make physical realizations independent of thread scheduling. Fixed logical batches and canonical merge order provide a reproducibility mode. Floating-point agreement across CPU architectures/compilers is tolerance-based, not promised bitwise. Optical kernels use strict floating-point semantics initially; relaxed math requires accuracy comparison.

Multi-machine operation

A job manifest and ordinary CLI workers are the initial distribution interface; MPI is not required. Each worker consumes a self-contained task referencing checksummed tables/configs and writes results atomically after completion. Completed compatible jobs are resumable. No shared writable framebuffer or remote access inside particle loops. Host concurrency and thread allocations must fit the user-specified CPU and memory budget.

Measurement

Record table-build time, particles evaluated per second, projection/binning/render/merge timings, memory high-water mark, worker load imbalance and speedup against one thread on an identical scene. Quality settings remain fixed during performance comparisons. Inspect socket topology and available hardware before selecting tuning presets; no core counts or scaling factors have been assumed.

Dependencies

Required: C++20 compiler, CMake, OpenMP and toml++. Ninja is the preferred build runner. OpenEXR provides linear HDR output and a preview image writer will be selected when implementation starts. No Python runtime is required for the core and no extra test framework is required initially; CTest can run numerical verification executables.

All configuration controls

Validation strategy

version 1  ·  created 2026-10-16  ·  updated 2026-10-16