A new rotation function for molecular replacement by using both the self and cross Patterson vectors.

The rotation function in molecular replacement is traditionally computed in reciprocal space. The common practice is to use a Patterson vector radius of about the size of the target molecule to limit the interatomic vectors to match only the Patterson self-vectors. In real space, the molecular image of a search model can be searched and matched to the Patterson function vector space. Depending on which asymmetric unit the matched Patterson vectors are in, both self or cross Patterson vectors can be matched to the search model. The algorithm described here for computing the rotation function is to average the signals from all images found by the image-seeking functions in the Patterson vector search. Tests show that a search model consisting of a two-helix fragment can be found in a myoglobin crystal using experimental data in a global rotation search. This demonstrates that the new rotation function is a potentially useful approach.