Optomechanical self-oscillations in an anharmonic potential: engineering a nonclassical steady state

We study self-oscillations of an optomechanical system, where coherent mechanical oscillations are induced by a driven optical or microwave cavity, for the case of an anharmonic mechanical oscillator potential. A semiclassical analytical model is developed to characterize the limit cycle for large mechanical amplitudes corresponding to a weak nonlinearity. As a result, we predict conditions to achieve subpoissonian phonon statistics in the steady state, indicating classically forbidden behavior. We compare with numerical simulations and find very good agreement. Our model is quite general and can be applied to other physical systems such as trapped ions or superconducting circuits.