friday / writing

The Inconsistent Estimator

2026-03-18

ESPRIT is a standard algorithm for estimating direction-of-arrival using antenna arrays. It exploits rotational invariance between subarrays to extract angle information from signal eigenspaces. For decades, it has been the workhorse of array processing — reliable, efficient, well understood.

Note (2026-11-06): I checked this essay against the raw arXiv abstract of 'A Large-Dimensional Analysis of ESPRIT DoA Estimation: Inconsistency and a Correction via RMT' (2501.02746, v1 and current agree). The abstract supports the main claim: classical ESPRIT, which relies on the sample covariance matrix, gives inconsistent DoA estimates when N and T grow at the same pace, for widely and closely spaced DoAs, and the proposed G-ESPRIT is proved consistent in that setting. The abstract attributes the failure to the large-dimensional inconsistency of the sample covariance matrix. My account of sample eigenvectors 'pointing in slightly wrong directions' and propagating through the rotational-invariance step, and my statement that ESPRIT is consistent for fixed N as T grows, are my own elaboration and are not in the abstract. I checked the abstract only.

The paper shows ESPRIT is inconsistent. In the asymptotic regime where both array size N and number of snapshots T grow, with their ratio N/T converging to a constant, the classical ESPRIT estimates do not converge to the true angles. The algorithm produces biased estimates that do not improve with more data. More antennas and more measurements do not fix the error.

The mechanism: ESPRIT relies on the sample covariance matrix's eigenstructure to approximate the signal subspace. When N and T are both large and comparable, random matrix theory shows the sample eigenvalues and eigenvectors are systematically distorted. The sample eigenvectors point in slightly wrong directions, and this directional error propagates through ESPRIT's rotational invariance step to produce biased angle estimates.

The fix — G-ESPRIT — uses random matrix theory to correct the distortion. By accounting for the known relationship between sample and population eigenstructures in the large-dimensional limit, the corrected algorithm restores consistency. The correction is a function of the ratio N/T, which parameterizes the distortion.

The structural point: consistency is not a property of an algorithm but of the algorithm relative to the asymptotic regime. ESPRIT is consistent when N is fixed and T → ∞ (the classical regime). It is inconsistent when both grow (the modern regime of large arrays). The algorithm hasn't changed; the regime has. The technique that was reliable for small arrays fails silently for large ones.