Unraveling nonclassicality in the optomechanical instability

Conditional dynamics due to continuous optical measurements has successfully been applied for state reconstruction and feedback cooling in optomechanical systems. In this paper, we show that the same measurement techniques can be used to unravel nonclassicality in optomechanical limit cycles. In contrast to unconditional dynamics, our approach gives rise to nonclassical limit cycles even in the sideband-unresolved regime, where the cavity decay rate exceeds the mechanical frequency. We predict a significant reduction of the mechanical amplitude fluctuations for realistic experimental parameters.