Analysis of protein dynamics via Deep Learning

Session

DHA.2 - Advances in 3-dimensional image reconstruction

Authors

Mr Gabriele Marchello (1), Mr Cesare De Pace (1), Dr Lorena Ruiz-Perez (1), Prof Giuseppe Battaglia (1)

Affiliations

1. University College London

Keywords

3D reconstruction, deep learning, in-situ imaging, liquid-phase electron microscopy, protein dynamics

Abstract text

Electron microscopy (EM) is a technique used to image matter down to the atomic level, exploiting electrostatic interactions between the samples and an accelerated electrons beam focussed by magnetic lens. EM requires the beam to be kept under high vacuum to avoid undesired scattering in the electron path, thus limiting imaging to samples in solid state. Therefore, samples in liquid state require special fixation protocols, which can remove the liquid or vitrify it to a solid state. Both approaches inevitably introduce preparation artefacts and most importantly limit the imaging process to static snapshots.  However, these limitations have been overcome by liquid-phase electron microscopy (LP EM), which images structures in liquid. LP EM opens the way to countless new discoveries in many different research fields, such as electrochemical reaction, nanocrystals growth and most importantly imaging soft and biological matter in four dimensions (N. De Jonge et al. 2011). Imaging liquid samples however introduces new challenges as the intrinsic dynamic nature does not allow long exposure time or image averaging techniques with consequent low signal-to-noise ratio. Moreover, the resolution of the outcome is lowered by the presence of the liquid media and the fast dynamic of the samples. The former blurs the images proportionally to its thickness, while the latter requires a camera with a very high acquisition rate to avoid the motion blur. Consequently, conventional denoising algorithms results to be inadequate for LP EM images. The fast growth of deep learning-based approach set its roots in the difficulties of the conventional algorithms (A. Dey 2016). Deep learning heavily modifies the working conditions, as the need for strict and detailed list of instructions was replaced by the need of a big dataset. The impact of deep learning on the whole computer vision field – and especially on the image analysis subfield – was impressive. Conventional limits were taken down one by one, and new applications and technologies were made possible. Object recognition, image classification and noise reduction – amongst many others - are the fields that mainly benefit of the application of deep learning (A. Voulodimos et al. 2018). The bottleneck limiting the LP EM image analysis is definitely the noise lowering the resolution of images. Therefore, a deep learning-based approach to suppress the noise corrupting the images results to be fundamental for any analysis. The approach guaranteeing the best result was proven to be the Noise2Void (N2V) (Krull et al. 2018), a deep learning approach that recovers the signal hidden by noise in our LP EM images. A comparison of the application of the Image Denoising Pipeline (Marchello et al. 2020) - a conventional algorithm - and deep learning denoising algorithms are depicted in Fig. 1. The N2V was designed to denoise images without any knowledge of the clean target image, which is unavailable in the case of LP EM.  Even though the results of the two approaches seem comparable, the processing time is very different; the PID takes 100 minutes to process a single image, while the N2V took only a few seconds. 

Figure 1 – Comparison of both conventional and deep learning-based approach. From left to right, (left) the raw noisy image, (center) the Image Denoising Pipeline processed image and (right) the N2V denoised image. 

Therefore, this technique was stably adopted by our group before performing any further analysis on LP EM images. With the denoised images we were able to distinguish structures inside images and track their motion across the image sequence. Moreover, we were able to analyse the dynamical processes of samples in their liquid native state, reconstruct the 3D structure of proteins and ultimately monitor and analyse protein denaturation in real time. Moreover, we were able to increase the acquisition rate of the camera, imaging faster dynamical processes. 

The application of deep learning results to be fundamental to any analysis of LP EM image. It significantly increased the resolution of the images, unveiling features lying under the noise, whilst incredibly shortening the processing time if compared to conventional algorithms. In the future, the number of deep learning-based approaches will significantly increase, in order to replace and increase the performances of time-consuming tasks based on conventional algorithms. 

References

[1] De Jonge N. and Ross F. M. “Electron microscopy of specimens in liquid”, Nature Nanotechnology. 2011

[2] Dey A. “Machine learning algorithms: a review”, International Journal of Computer Science and Information Technologies. 2016

[3] Voulodimos A., Doulamis N., Doulamis A., and Protopapadakis E. “Deep learning for computer vision: a brief review”, Computational Intelligence and Neuroscience. 2018

[4] Krull A., Buchholz T.O., and Jug F. “Noise2Void – Learning denoising from single noisy images”, The IEEE Conference on Computer Vision and Pattern Recognition. 2019

[5] Marchello G., De Pace C., Duro-Castano A., Battaglia G., and Ruiz-Perez L. “End-to-end image analysis pipeline for liquid-phase electron microscopy”, under revision. 2020