Deep learning and defocus imaging for determination of three-dimensional position and orientation of microscopic objects

We present a method to determine the three-dimensional position and orientation of microscopic, non-spherical objects in microfluidic and laboratory-on-a-chip systems observed through conventional optical microscopes. The method is based on the combination of the General Defocusing Particle Tracking technique [Barnkob et al., “General defocusing particle tracking,” Lab Chip 15, 3556–3560 (2015)] and deep learning. It requires minimal input from the user, is suitable for real-time applications, and can be applied to any microscopic object with an approximately ellipsoidal shape, such as unicellular swimming organisms, red blood cells, or spheroidal colloids. The main challenge is linked to the construction of suitable training datasets for the neural network. We provide a procedure generally valid for active microswimmers and discuss possible strategies for other types of objects. An implementation using the Visual Geometry Group convolutional neural network (VGG-16) is presented and tested on synthetic images with different backgrounds and noise levels. The same implementation is used to track the position and orientation of different specimens of the heterotrophic ciliate Euplotes Vannus in free-swimming motion. The measurements were performed with a 10x objective over a depth of 800 µm with an average estimated uncertainty in the orientation angles of 9.0%.