Hybrid atom-optomechanical system in the quantum regime

This thesis reports on light-mediated coupling between the collective spin of a cold atomic ensemble and a mechanical oscillator. In our experiments, we engineer coherent interactions between these distinct systems over a macroscopic distance of 2 metres. Thus, the system presented in this work establishes a versatile platform for implementing remote, coherent, and hybrid quantum mechanical coupling experiments. Achieving quantum coherent coupling between the systems will enable the execution of quantum protocols on this platform, such as quantum state swaps or entanglement generation. Quantum coherent coupling is possible when both the atomic spin and the mechanical oscillator interact with light with high cooperativity and their noise is limited by quantum backaction noise rather than thermal or technical noise. The first part of this thesis presents a series of improvements and characterisation measurements of both systems, demonstrating that in the current setup both systems operate in the regime of large quantum cooperativity. Our first system is the collective spin of dipole-trapped cold Rubidium atoms, a well-established platform in quantum optics with a large toolbox for quantum initialisation and manipulation. This thesis presents a theoretical and experimental characterisation of the Faraday spin-light interface, extending beyond the simplified one-dimensional model of two-level atoms. We find that for the relevant experimental parameters, the spin-light interaction is limited by quantum backaction noise and exhibits large cooperativity. Our second system is a nanomechanical membrane oscillator placed in an optical cavity. Nanomechanical membranes are macroscopic systems that can act as sensitive force sensors. In this work, the optomechanical system is upgraded by exchanging the membrane oscillator; replacing the previously used membrane with a phononic bandgap shield by a state-of-the-art nano-pillar membrane that has a mechanical Q-factor of $5.1\times 10^7$. With this enhancement and operation at cryogenic temperatures we experimentally demonstrate that the membrane-light interface enters the regime of high quantum cooperativity, showing cooling to the quantum backaction noise limit and ponderomotive squeezing of the light. In a first series of hybrid coupling experiments, strong coupling between the two systems is used to demonstrate coherent cooling of the mechanical oscillator with the atomic spin. The setup can be understood as a coherent feedback platform where the atomic spin acts as the controller. By leveraging the full control provided by the hybrid system, we perform spin-membrane state swaps, combined with stroboscopic spin pumping, to cool the membrane in a room-temperature environment to ${T}=\SI{216}{\milli\kelvin}$ ($\bar{n}_{m} = 2.3\times 10^3$ phonons) in $\SI{200}{\micro\second}$. The high cooperativity of the interaction of each system with light theoretically enables quantum coherent coupling of the two systems. Preliminary experiments demonstrate strong coupling between the systems at cryogenic temperatures, which has not been achieved in previous work. Future experiments can leverage this coupling to demonstrate quantum coherent control, e.g. by inducing entanglement between the two systems through coherent parametric-gain interactions.