Anomalous shell effect in the transition from a circular to a triangular billiard.

We apply periodic orbit theory to a two-dimensional nonintegrable billiard system whose boundary is varied smoothly from a circular to an equilateral triangular shape. Although the classical dynamics becomes chaotic with increasing triangular deformation, it exhibits an astonishingly pronounced shell effect on its way through the shape transition. A semiclassical analysis reveals that this shell effect emerges from a codimension-2 bifurcation of the triangular periodic orbit. Gutzwiller's semiclassical trace formula, using a global uniform approximation for the bifurcation of the triangular orbit and including the contributions of the other isolated orbits, describes very well the coarse-grained quantum-mechanical level density of this system. We also discuss the role of discrete symmetry for the large shell effect obtained here.