Breaking the diffraction limits using neuromorphic microscopy

Work done in the lab of Prof. Deepak Nair at Centre for Neuroscience, Indian Institute of Science, Bangalore

About author

Rohit Mangalwedhekar is a master’s student at the Department of Atomic and Molecular Physics, Manipal Academy of Higher Education. He is currently working on his master’s project with the Nano-org Lab headed by Dr. Deepak Nair at the Centre for Neuroscience, Indian Institute of Science. His work involves improving the spatial and temporal resolution of microscopy using neuromorphic sensors and image processing algorithms. Outside the lab, he likes to photograph food, wildlife, and the night sky. He loves playing chess with his lab mates and having conversations over sushi and ramen.

Interview

How would you explain your research outcomes to the non-scientific community?

We have developed a new microscopy technique that utilises neuromorphic vision sensors to achieve a localisation precision below the diffraction limit. A neuromorphic camera or a “silicon retina” is a brain inspired vision sensor that functions similar to the eye. Every pixel of a neuromorphic sensor records events independently when it sees an increase(ON) or decrease(OFF) in brightness, unlike conventional cameras wherein the integrated intensity is recorded over the same time window for all the pixels to create an image. This asynchronous sampling allows us to reconstruct image frames at varying frame rates post-acquisition. These two aspects of neuromorphic cameras allow for imaging at high speeds, and imaging only dynamic phenomena while neglecting the static background.

We first determined the camera’s response to a time varying signal by studying periodically illuminated stationary fluorescent nanoparticles and correlating the ON and OFF events with the laser’s switching period. The spatial characteristics were determined by the localisation precision of these immobilised nanoparticles. For this, we used a wavelet algorithm, and a convoluted neural network algorithm trained on one and a half million images to predict the position of the particles. Using many predictions, we generated a probability density function for the ON and OFF events and combined them by mathematical means to improve the precision of locating the particle to about 10 nm! Once we characterised it in the spatial and temporal regime, we imaged freely diffusing particles and studied their fractalized random motion by reconstructing images in varying timescales.

Graphic description of the workflow Fluorescent beads were periodically illuminated and imaged by a neuromorphic(NM) camera and the data were reconstructed into frames containing ON(green) and OFF(red) events. The ON and OFF events frames were sorted and the particles were tracked using a deep learning and a wavelet algorithm(shown above). Frames can be reconstructed at different frame rates by accumulating events in suitable time windows for analysis at varying timescales(shown below). Created with help of BioRender.com.

How do these findings contribute to your research area?

Neuromorphic cameras have two key attributes that give them an edge over other cameras. One is that they record a change in brightness, rather than the actual intensity of light itself. And the second is that they record events asynchronously. This proves to be a huge advantage over standard cameras wherein the frame rate or fps is set before the acquisition, whereas for neuromorphic events, one can reconstruct the data as many times and in as many timescales (as low as sub-millisecond) as required. Furthermore, since they record only a change in intensity, only dynamic phenomena such as motion or switch on/off of fluorescence are recorded.

The advantages of this microscopy technique coupled with particle tracking and localisation algorithms will allow scientists to view and analyse molecular interactions, mechanisms, and organisation at the nanoscale within cells with high temporal precision. It could also find applications in studying chemical reactions and nanotechnology.

“The advantages of microscopy technique coupled with particle tracking and localisation algorithms will allow scientists to view and analyse molecular interactions, mechanisms, and organisation at the nanoscale within cells with high temporal precision.”

What was the exciting moment during your research?

I don’t think there was one big, exciting eureka moment. Instead, there were many little exciting moments throughout. When the deep learning algorithm was close to optimisation, and it was able to predict the position of particles for the first time, it was extremely exciting to me. There were multiple errors and many predictions which were wrong, but it was proof that it was working and could be further optimised. Given that the training, validation, and prediction processes together would take several hours, I’d often leave the program running overnight, and I’d know the results only the next morning after further analyses. All of this had to be repeated several times before I got it right. So, it was especially exciting to see that it was finally working, at least in principle. The first time the neuromorphic camera detected the fluorescence was also cool where we rejoiced on seeing the fluorescent beads blink red and green. The first time we saw the correlation between the fluorescence and laser switching was exciting too, I remember sitting and staring at the graph that showed a nice sharp peak at the expected frequency! I always had (and continue to have) tiny celebratory moments in my head every time a new piece of code runs without error!

What do you hope to do next?

I am excited to see what else is possible with these technological marvels! Neuromorphic imaging is incredibly exciting just because of the sheer amount of data that is collected and how it can be manipulated. We look forward to pushing the limits even further and finding new applications for this technique, but I don’t think I want to say anymore at the moment😉.

Where do you seek scientific inspiration from?

As a kid, it was mostly books, documentaries, and talk shows by Carl Sagan, Neil deGrasse Tyson, and other science communicators that drew me to take up scientific research. But today, as someone in research, I feel I am lucky that I don’t have to actively seek inspiration. I think the best of our work happens through a lot of discussions of what is happening in the lab, what people are working on, and what is to come next. And in this aspect, Dr. Deepak Nair is the source of inspiration in the group. He’s always brimming with ideas, always thrilled at the sight of new results, and that kind of energy is very contagious, and it just permeates through the rest of us in the lab! And everyone else just follows suit and we end up discussing a lot of the science happening in the lab (and outside). Given that each of us come from different scientific backgrounds and are working on different projects, there is a lot of learning and teaching happening among the lab members constantly. So I think I am mostly inspired by going about my daily life in the lab!

How do you intend to help Indian science improve?

In all honesty, I am quite unsure right now. Given that I am new to research myself, I am still learning my way around the community. There’s a lot of cutting-edge research happening in India, and I think interdisciplinary collaborations are the way ahead. It’s even more exciting to see the growth of science communication parallelly. I know that I want to be a part of the Sci-Comm happening here. As a start, I plan on explaining the work we have published through a short series of tweets and/or Instagram posts, both in English and Kannada. I hope I find the time to do this really soon!

Reference

Mangalwedhekar, R., Singh, N., Thakur, C.S. et al. Achieving nanoscale precision using neuromorphic localization microscopy. Nat. Nanotechnol. (2023). https://doi.org/10.1038/s41565-022-01291-1

Copy Editor

Nikita Nimbark

PostGrad in Biotechnology

Nikita Completed her PostGrad in Biotechnology. She has interest in Bioinformatics. Her hobbies include travelling and calligraphy. She is always up for new challenges.

For interview related queries, write to us at interview.biopatrika@gmail.com