“Houston, we have a question”: Martin Jones on Correlative Light and Electron Microscopy, AI in science, and open collaboration

This time 'Houston, we have a question' interview is joined by Martin Jones from The Francis Crick Institute - world famous biomedical research institute based in London.

Martin Jones is the Deputy Head of Microscopy Prototyping in the Electron Microscopy STP at the Francis Crick Institute. Martin holds PhD in physics, and his area of expertise is around developing hardware and software solutions for imaging and image analysis.

This interview is led by Dr Bruno Sanguinetti, Dotphoton’s CTO - the two met during their studies as quantum physicists.

Martin talks about his journey from physics to building image analysis software for electron microscopy systems such as Correlative Light and Electron Microscopy (CLEM). He explains why he is fascinated by CLEM and its benefits. Together with Dr Bruno Sanguinetti, Dotphoton’s CTO, they discuss AI and quantum computing 20 years ago and now, the real contribution AI is making in science and specifically electron microscopy today.

At the end Martin shares his opinion on what he believes to be the theme for the society for the next few years (hint: the need for open collaboration) - our traditional question to all the 'Houston, we have a question' guests.
Bruno:
We met quite a long time ago, 20 years or 21 years ago. The cool thing was quantum physics and quantum computers - that's what we were working on. At the conference the other day there were 3 or 4 of us who come from the quantum physics background, who are now more in the imaging and biology side of things.
From quantum physics to imaging and microscopy - was it an intentional shift?
Martin:
A bit of both, I suppose. One of the unfortunate things about academia is when you are a postdoc - they're short term contracts and you're always looking for the next job and you have some element of luck in what's available at different times.

I was coming to the end of my contract in Leeds (University of Leeds) and I was looking around for other physics jobs and I was also keeping an eye on other options. I was always interested in a range of things. So I was emailing friends that I knew from various times during studies. And I emailed somebody called Katie Bentley who I did my MSc with - we did MSc in Evolutionary and Adaptive Systems at the University of Sussex in about 2001 ish.

She did Maths undergrads, Computer Science Masters and PhD. She ended up working at Cancer Research UK (CRUK), the London research Institute. I just sent a speculative message asking if there were any interesting jobs for physicists around there.

And it just so happened that she and her boss had been to Germany a few weeks before to look at one of these amazing new microscopes, actually it was a light-sheet microscope. I know your previous interview was around this light-sheet microscopy area.

It was around 2010. They were enthused about having a physicist who could build stuff. And also image analysis has always been a major problem. Having people who can do custom coding in biology outside of normal bioinformatics is a bit rare. So it was hard to build the analysis pipelines they really wanted to do.
So I got in through the side door at CRUK and I had a nice introduction to the team and we chatted about what might be possible. It was a lucky move, happened to message somebody at exactly the right time. So it defined a whole career direction for me.
Bruno:
There are a lot of physics in optics and electron microscopy. Has your physics background actually helped you at the imaging part of it?
Martin:
I don't do so much of the actual imaging myself. I joined CRUK in one of the research groups. Then the group leader moved to Germany and I moved over to the electron microscopy core facility at the CRUK which is now migrated to the Francis Crick Institute where I am now.

The sorts of things I’ve been doing in the electron microscopy team have been around designing hardware to do imaging. Particular type of imaging called Correlative Light Electron Microscopy - we combine fluorescence and electron microscopes together.

Again it was a lucky piece of timing that my boss was about to leave to Germany and I wasn't in a position to be able to go. And then I went to a talk which my current boss had organised, Lucy Collinson. She organised a talk for an external speaker to come in to talk about this thing I'd never heard of: correlative light electron microscopy. They were talking about building a hardware to do all these clever things. And I looked at it and thought: well, that looks quite similar to what I did my PhD in.

It was a big vacuum chamber. So electron microscopes all in vacuum chamber. Getting light down to a sample somewhere in the depths of this vacuum chamber. We have tiny bits of space, a few millimeter gaps here and there and we have to put things in it.

And that was actually pretty similar to one of the main elements of the hardware development I was involved in during my PhD. And I just emailed Lucy and said: “I have an idea. I reckon it would be possible to do this, this and this”. And then she replied: “yeah come downstairs and let's talk”. So I ended up joining the team that way.

So in that sense knowing the physics experimental methods you'd use to perform a certain task - that was the real benefit. And all of these imaging things are subject to the laws of physics. So you have diffraction limits, and a particular size lens has a particular ability to capture light, and so on. You deal with a lot in experimental physics, but maybe not so much in experimental biology.
So that was really a useful thing. From an engineering side, knowing the difference between the different cameras, the quantum efficiency of the cameras and all of these technical details, which might not be very obvious what the benefits of different things might be to somebody who hasn't worked with these sorts of devices before.

Generally, in physics I always tried to straddle theory and experiment. So I didn't like this idea of pigeonholing to one or the other - sometimes you get these two camps.

Previously in my Masters and PhD, I tried to use some of these adaptive computation methods to solve certain problems. As AI has come into image analysis, in particular a lot recently, I've been dealing with systems like that for quite a few years. Knowing the backgrounds to that and the pros and cons of those things has really helped in a lot of the work I'm doing at the moment.
Bruno:
I do remember us talking a long time ago, more than 20 years ago, about these adaptive systems, which were basic AI systems. A lot of people chose to go instead of further research in that field, to further research in quantum computing and quantum physics, because indeed it was evolving at quite a slow pace at the time. But I have an impression that a few years ago, maybe 10 years ago, the speed at which AI started evolving overtook the quantum physics evolution speed.

On the other hand, it could just be that we're quite excited about everything in general.

So I was wondering if it's just an excitement about AI or if AI is the real thing, if it really will help to solve some real problems?
Martin:
Yeah. An important distinction to make.... The phrase 'AI' is used a lot and it means different things to different people.
Bruno:
So does 'quantum' :)
Martin:
Yeah, exactly. Add 'AI' to anything and you get extra money, right? That's how the funding works (laughing). Not really.

You can look at it in two ways. One, maybe the original idea about AI back in the days of cybernetics and things like that - was to understand how brains work. What is intelligence and how does it work? That's clearly a very complex thing. And we're very far from understanding that. I'm not a specialist in that, but it feels like these kinds of what some people might call artificial general intelligence systems that somehow are able to inform us on how brains work. It's not even a question how close we are to that.

But the other aspect, what people are more familiar with is more on the engineering side. I have a problem and I have some data, and I want to get to this solution or I want to optimise this thing. And then you can do it via this kind of iterative optimisation process, just keep making it better and better and better until it gets hopefully good enough for what you need. But in that process, maybe you leave to one side the understanding of what that thing is doing. So interpretability of AI is a very hot topic at the moment. And often it can be quite hard to understand what the systems are doing.

But in the engineering sense of solving this problem in a way that I hopefully convince myself is robust and not open to various errors or artefacts - that's a perfectly valid use as any research tool to get to answering any question you're asking.

In the last five years in particular, but more than that generally, particularly convolutional neural networks, deep learning methods coming into bioimage analysis has really revolutionised the field. Where things have been moving quite slowly in terms of how quickly methods were improving. I think that's really made it a bit of a step change in the last few years. Things that were effectively impossible before are now starting to be possible, and in some cases even routine. I think that's really exciting.

This background behind me - we applied a deep learning algorithm, we have an amazing scientific computing team at the Crick who wrote the system, this is a huge chunk of electron microscopy data that has been imaged over days or weeks, hundreds of gigabytes, raw data I expect. These colourful objects behind me, each different colour is a different mitochondria from within a piece of tissue. I think it's a piece of brain tissue. So the mitochondria and lots of different cells, and there are thousands of them in this image and you see they're really nice and clean - it's astounding.

Having tried to do this with non deep learning methods in the past and you can't do it really is. It doesn't really work. One of the most exciting things about AI is that it's cumulative as well. I built a system that does this, and somebody has a slightly different dataset. They don't need to start from scratch. They can take that system, do a little bit of work, and now we'll work on their data. Whereas the more traditional methods you have to dig into the code and you have to change all sorts of parameters yourself - this hand tuning is a bit of a dark art and it's quite difficult to justify what you're doing. Whereas if you let the machine - it's just an optimization problem. I think that's a very exciting area for AI.
Bruno:
But it looks like if I were to have to label that data by hand myself, it'd be really difficult. Has AI actually overtaken what humans can do? Not just in the volume of how much data you can do it on, but actually having a really complex image analysis?
Martin:
Yeah I would say. There's a slight of hand, which I've glossed over here, but at some point for this dataset... this was actually a conference competition datasets that people put out. And somebody in the team who made the competition has sat down and manually annotated this, probably several people, manually gone through this to provide the training data that allows us to build the network that performs the analysis and then to compare against. Of course, I could show you this picture and say it worked very well, but without checking and validating that this is good - you just have to take my word for that. And that's not how we do science,right? So this supervised methods - guess we're often implicitly talking about supervised methods - at the moment you still need fairly significant quantities of manually annotated ground truth data to train the systems.

But the hope is that as you bootstrap different methods you can use things like transfer learning, where you can use a smaller amount of data to adjust an already existing neural network model. I think that's really the next steps in the field - trying to move away from having to have huge amounts of manual annotations.
One thing that we've tried to alleviate this bottleneck of somebody having to sit down and manually annotate all of these objects. There was a really fascinating project called Galaxy Zoo, which is a citizen science project. Astronomers have a similar problem to microscopists - they can generate millions of images really easily now, just tons and tons of data. The throughput of the analysis is too low to deal with it. So these Galaxy Zoo teams led by Chris Lintott came up with this method: cropping images and sharing them out on the internet. And then you explain to people: "this is a scientific project, if you want to help us - come along and help us annotate these images".

It worked amazingly well in Galaxy Zoo. And a few years ago, maybe five years ago we set up our own citizen science project where we use this power of the crowds to help us generate the ground truth. Not for this data in particular, but for the Etch-a-Cell set of projects on the Zooniverse platform. We've got several projects now and we're collaborating with lots of people. While we still need to get large amounts of these manual annotations to do deep learning, we found a really good way of getting large quantities of annotations to train these networks. It's worked very well so far and we're looking forward to lots more exciting things from the project.
Bruno:
What do you think will be the most exciting thing in microscopy in the next few years?

I noticed that equipment wise, even in the last three or four years, cameras have become very close to perfect, optics have become very close to perfect. And they reached quantum limits - you can't have more than a hundred percent efficiency, you can't have less than zero noise. In the evolution of microscopy, where do you see it going? Both technically and data analysis.
Martin:
Yeah you're right, certain limits appear to be being reached, although people are very inventive and find clever ways of getting around these things. For example, super resolution microscopy, which got around the diffraction limit, which was this hard limit that you couldn't beat before.

There's a lot of interesting stuff going on in multimodal imaging. I mentioned earlier this microscope that got me into joining the EM team, a particular type of this multimodal imaging called Correlative Light and Electron Microscopy. The idea is that each imaging method, might be x-ray, or fluorescence microscopy or super resolution, or electron microscopy, or histopathology - they all give you slightly different information.

For example, fluorescence and electron microscopy. In fluorescence microscopy, I might label only the things I'm interested in. Like mitochondria and the nucleus, and endoplasmic reticulum in particular. And those things glow in whichever colour I've tagged on them. Maybe using green fluorescent protein and things like that. And everything else is dark. So you don't see anything else. You can pick out very specific functions. You can tell: this green colour is associated with something that's performing this function. So I know the function of the thing I see, but you're limited in resolution. So without going into super resolution and so on, the diffraction limits you to a couple of hundred nanometers, something like that, of these fluorescent images. A lot of the interesting stuff happens below that scale. You can become resolution limited in that sort of imaging.

In electron microscopy we can now image huge amounts of data. Maybe a single cell - if you image at the highest possible resolution - might be a terabyte of data. It might take you some days. For example, something like a focused ion beam scanning electron microscope can image at a few nanometers isotropic resolution per voxel. That's amazing, right? You see almost everything that you might be interested in seeing. But you also see everything. So there's no specificity in electron microscopy. You maybe have some heavy metal staining, so the membranes stand out, but every membrane will stand out. So you will see these images and there's a gray scale image and you've got a whole bunch of membranes. If you're not an expert - you don't know what's what, and it's very hard to tell things apart. And even if you are an expert - there are certain structures which you can't really identify just by looking at them. So you need extra information. Combining the fluorescence microscopy and electron microscopy you can use these sort of complementary sets of information.
You can tell from the fluorescence microscopy: the thing I'm interested in is over there, maybe I can save a bit of data on my electron microscope by only imaging over there where I know that object is, or where you can identify things that you couldn't identify in your electron microscopy. That's just two imaging modalities, but we also have x-ray microscopy, also known as micro-CT where we can do kind of micro resolution imaging, but over centimeter scales.

That can be linked to all sorts of other types of imaging. So these things, they all give you different information and combining them in a smart way, I think is going to be one of the next revolutions in how this imaging is done.

If you just blindly acquire all the data that you can possibly acquire - you have too much data and you can't analyze it. You need to somehow be a bit more nuanced in how you get the data and then how you use what data you have in the analysis.
Bruno:
One of our first customers was interested in correlating the two aspects of spectrum and shape. If you had some multimodal imaging, what you could try and do - and in that case it worked well - was to train with that multimodal data your machine learning network. And then by an inference from just one of the two modes, just the shape or just the spectrum tell what the other one would be. For example, from the fluorescence, you could get a super resolution generative image or vice versa, i.e. from a very high resolution electron microscope image you could try and infer what fluorescence that thing would have.

Is there anything going in that direction?
Martin:
Yeah, so I think the combination of AI and multimodal imaging is super interesting because deep learning methods are associated with the whole kind of big data movement.

A lot of this is an idea of bringing together disparate types of data and recognising patterns across things that maybe a human can't recognise, because maybe there's too much data or the different datasets are so different that it's hard to see how everything fits together, but as purely a multi-dimensional optimisation thing - they're just numbers to machine.

I think that's a really interesting area and I think there are definitely people exploring all sorts of these methods. The generative methods, for example, or the sort of mapping methods from one dataset to another, to try and shortcut having to do certain things. There are certain imaging steps, which may be very tricky to do, but if you can infer with good confidence, it can be quite hard to convince a reviewer that it is a robust thing to do and you're not just producing artefacts. Because a network that's trained to predict certain things can predict them inappropriately very easily. So if you google around some of the types of errors that generative methods can make - it can be quite amusing, but also a bit worrying and hard to auto extract.

So this all needs to be coupled with a thorough analysis of how we're confident that this is a meaningful thing to predict, and often it is. You just want to be belted and braced on making sure that there's no possible holes that might come back and bite you in the future.
Bruno:
I saw that there's quite a lot of work in that direction. We also try to understand better how these things work, especially if your input space that you use during inference doesn't match statistically the one that you used during training, then you get the wrong image effectively, and you can of course have some errors because the network is not used to seeing that type of image.

There's still very little technology to characterise what space this input actually lives in. Actually there's quite a big parallel with quantum physics. Because in quantum physics you have all these huge multi-dimensional spaces, where the dimensions are not necessarily orthogonal and you're always trying to find what is the best representation of that space. In machine learning, if you look at that input space, it's very interesting to know what the real underlying dimensions are and if they are orthogonal or not.
Martin:
Yeah. I think that's one of the things having come from quantum physics training - you get used to throwing around these multidimensional things, you come to accept them. Whether you understand them or not, you get used to it.

Actually it's conceptually a pretty hard thing. When somebody is just faced with: “Okay, this problem has that many dimensions” - it can be hard to interpret that or at least to deal with that.
Bruno:
In experimental physics, it helps that you have an actual physical thing in the lab and it’s entangled, then you see what happens and suddenly you get an intuition for what these dimensions are. But it is not so obvious.

The bonus question we ask all ‘Houston, we have a question’ guests. If you think about the next few years, what do you think the main theme that defines society will be?
Martin:
That's a good question. I suppose both in the science that I do and more generally, this idea of integrating things across barriers is super important. Because of industries and so on, things get verysiloed. So development happens behind closed doors in certain things.

There are probably people within these silos, all solving the same problem over and over again. Just some sort of integration and communication between those separate fields is really powerful. We see it in that sort of analogy of this core correlative electron microscopy. You get something that's more than the sum of the parts when you combine them. I think that feels like something that could happen more generally. There are things going on that for whatever reasons, people are kind of keeping hidden and people have profit margins and so on to look after.

But actually, if the purpose is to make life better for more people, then it kind of seems that somehow combining these knowledge bases would help that.
Dotphoton
Dotphoton provides innovative image compression solutions for big image data. Dotphoton’s unique set of algorithms and cutting edge approach to camera calibration enables file size reduction by a factor of 6—10, while preserving the raw quality of images. Dotphoton's deep understanding of latest insights from the quantum information field ensures it stays ahead as the highly reliable partner, trusted by the European Space Agency, Bosch, and the leading biomedical centres across the world.
Contact
Eugenia Balysheva
Dotphoton AG
news@dotphoton.com
+41 0 41 552 50 00