--- title: "Why error bars can miss the largest error" description: "Separate weight concentration, time correlation and sampling bias with block bootstrap, independent seeds and held-out validation." canonical_url: "https://sandboxphysics.com/stories/free-energy/uncertainty-and-validation/" language: "en" page_type: "story" alternate_languages: zh-CN: "https://sandboxphysics.com/zh/stories/free-energy/uncertainty-and-validation/" last_modified: "2026-10-28" --- [Free energy and rare events](https://sandboxphysics.com/stories/free-energy/) L3 · Chapter 5 # Why error bars can miss the largest error Sandbox Physics · 2026-10-28 An error bar describes variation under a particular sampling procedure. A procedure that repeatedly misses the same valley can be precisely wrong. A useful assessment checks concentrated weights, time correlations and agreement between independent runs. ## Before you begin Correlation factors from Chapter 2 and target weights from Chapter 4. Variance measures repeated-result scatter; bootstrap approximates repetition by resampling existing data. [Review the preceding chapter](https://sandboxphysics.com/stories/free-energy/multistate-reweighting/) - Distinguish weight effective size, time effective size and sampling bias. - Design a per-window block bootstrap and challenge reconstruction with quadrature and held-out checks. ## Ask how many observations carry the answer Normalized target weights $W_n$ sum to one. Ten observations sharing the answer equally differ from one carrying ninety percent. Weight effective sample size $N_{\mathrm{eff},w}$ measures this concentration through the sum of squared weights. Worked example: among one hundred observations, one has weight $0.5$ and the other ninety-nine share the remainder. The squared-weight sum is $0.25+99(0.5/99)^2=25/99$, giving effective size $3.96$. The answer can depend strongly on that single observation. This measure ignores time order and unvisited valleys. Duplicating one observation many times adds entries without adding independent information. Check weight concentration and time dependence separately. $$ N_{\mathrm{eff},w}=\frac{1}{\sum_n W_n^2} $$ This is a concentration diagnostic, not a universal independent-sample count. Correlated trajectories require blocks or another correlation treatment. ## Resample contiguous pieces of motion An ordinary bootstrap treats frames as independent draws. Breaking apart a slowly wandering path destroys its correlations and generally understates uncertainty. A block bootstrap draws contiguous pieces and preserves the motion inside them. Resample each window separately, retain its sample amount, and solve MBAR again for each replicate. Compare basin populations or free-energy differences across replicates. Keeping the old weights fixed and shuffling only the final plot misses uncertainty in window normalization. Blocks should cover correlation decay yet leave several nearly independent pieces. Increase block length and look for a stable error estimate; save the lengths and replicate count. A record containing one residence period supports only its visited region; another run must supply the missing exchange information. Source: PLUMED Masterclass 21.2 · Michael R. Shirts and John D. Chodera ## Build a reference with a different calculation The two-coordinate model permits direct integration of $e^{-\beta U}$. Refine the grid and check basin populations and projected free energies. Symmetry gives another check: with zero tilt, $(x,y)\mapsto(-x,-y)$ leaves the potential unchanged, so left and right half-planes have equal probability. Quadrature reads no random trajectory and can expose missed sampling. Its own errors come from finite grids, projection bins and boundary treatment. Record those separately from sampling error to state how closely two curves actually agree. Source: PLUMED Masterclass 21.2 ## Reserve a region for a genuine test Before designing windows, choose a validation region such as an interval between the valleys. Freeze the design and analysis, then evaluate its probability with new seeds and compare with quadrature. The trajectory may enter this region; the restriction is on repeatedly using its reference answer to tune the same reported method. Once a validation result changes the window design, keep that result as development history and reserve fresh seeds or another region. Otherwise apparent improvement can reflect repeated tuning to the answer. Distinct checks address distinct errors: step refinement tests the integrator; seeds reveal random scatter; blocks examine correlation; different initial valleys challenge equilibration. Each error estimate should name the question it addresses. ## Try the reasoning in an experiment ### [Explain the uncertainty of one free-energy difference](https://sandboxphysics.com/labs/umbrella-mbar/) 1. Fix model and windows. Save reconstruction, sample weights and block settings for the default seed. 2. Repeat with new seeds. Compare their scatter with block errors and inspect hidden-coordinate exchange. 3. Choose a validation interval in advance and compare its reweighted probability with quadrature. Identify the relative heights that remain undetermined when the chain disconnects. ## Exercises and explanations 1. One thousand equally weighted consecutive frames have weight effective size one thousand. Are they independent? Read the explanation The weight diagnostic only measures concentration. Similar neighboring frames can contain much less independent information, and a trapped sequence can miss the target distribution. Check correlations, block lengths and different starts. 2. A small block error accompanies clear disagreement with a high-precision integral. What should be checked next? Read the explanation Challenge systematic errors: different hidden-valley starts, actual overlap, step refinement and the division by bin width. More bootstrap replicates only describe the scatter of the available data more accurately. [Previous: How MBAR joins windows into one equilibrium distribution](https://sandboxphysics.com/stories/free-energy/multistate-reweighting/) [Next: How metadynamics gradually fills a visited valley](https://sandboxphysics.com/stories/free-energy/well-tempered-metadynamics/) ## Sources and reproduced scope Worked examples and schematic figures are original teaching material. The numerical research slice compares specified estimator equations on our seeded two-coordinate records with independent quadrature and PyMBAR. The final chapter reads Figure 1 of the MBAR paper and states which experimental data remain outside that calculation. [Model definitions, scripts and numerical records](https://github.com/yangzichao/sandboxphysics/tree/main/docs/frontier/fr15) 1. [PLUMED Masterclass 21.2, Statistical errors in MD (documentation v2.9)](https://www.plumed.org/doc-v2.9/user-doc/html/masterclass-21-2.html) Background and Exercises 2, 7–9: marginal probabilities, correlated samples, blocks and bootstrap. The tutorial reports PLUMED 2.7.0 for its original exercises. Our examples use an independently defined two-coordinate model. 2. [Michael R. Shirts and John D. Chodera, Statistically optimal analysis of samples from multiple equilibrium states, J. Chem. Phys. 129, 124105 (2008)](https://arxiv.org/abs/0801.1426v3) Frozen arXiv v3: Sections II–IV, Eqs. (9), (11)–(14), Section V and Figure 1, Appendices A–C. We implement the equilibrium estimating equations on our synthetic model. Figure 1 is read as an experimental example; its DNA-hairpin traces and plotted values are not reproduced.