One-way depth is not round-trip optical path
Depth increases along the sample beam, and a positive reference displacement lengthens the reference arm in air. Both arms are round trips, so the factor of two cancels: the mirror position that matches a layer equals the index times its physical depth. Each fringe corresponds to a mirror move of half a vacuum wavelength. The fixed background paths are assumed to match; the arm lengths in the scene are only a diagram and are not used to compute phase.