Magnetic sensitivity beyond the projection noise limit by spin squeezing.

We report the generation of spin squeezing and entanglement in a magnetically sensitive atomic ensemble, and entanglement-enhanced field measurements with this system. A maximal m(f) = ± 1 Raman coherence is prepared in an ensemble of 8.5 × 10(5) laser-cooled (87)Rb atoms in the f = 1 hyperfine ground state, and the collective spin is squeezed by synthesized optical quantum nondemolition measurement. This prepares a state with large spin alignment and noise below the projection-noise level in a mixed alignment-orientation variable. 3.2 dB of noise reduction is observed and 2.0 dB of squeezing by the Wineland criterion, implying both entanglement and metrological advantage. Enhanced sensitivity is demonstrated in field measurements using alignment-to-orientation conversion.