 Hello and welcome to today's seminar. My name is Paul Roberts and I'm a post-doctoral research fellow in the Barden lab at the University of Sussex where I'm researching the functional connectivity of the zebrafish retina. Before introducing our speaker, E.J. Chisholiski, I'd like to bring a few brief announcements to make. A few brief announcements to make. Firstly, our thanks go to Worldwide Neuro for instigating this platform for sharing Euro scientific research. Secondly, I'd like to remind our listeners that you can send your questions in by the YouTube chat. So our speaker, E.J. is happy to take some questions requiring brief answers during the talk and we will save questions requiring longer answers for the Q&A session after the talk. So the talk will last for about 50 minutes and this will be followed by about 10 minutes for Q&A in which I'll relay your questions to E.J. And after this, we'll invite all who are interested to join us on Zoom for an informal chat with E.J., posting the Zoom link on the YouTube chat at that time. So it gives me great pleasure to introduce our speaker, E.J. Chisholiski. So E.J. is the John R. Adler Professor of Neurosurgery and Professor of Ophthalmology at Stanford University where he has worked since 2013. Previously, he worked at the Salk Institute for Biological Studies for 15 years. He received his BA in mathematics from Princeton University and has his MS in mathematics and PhD in neuroscience from Stanford University. His research is focused on understanding the spatiotemporal patterns of electrical activity in the retina that convey visual information to the brain and their origins in retinal circuitry using large-scale multi-electrode recordings. His ongoing work now focuses on using basic science knowledge along with electrical stimulation to develop a novel high fidelity artificial retina treating incurable blindness. He has a recipient of an Alfred P. Sloan Research Fellowship, a McKnight Scholar Award, a McKnight Technological Innovation and Neuroscience Award and an RPB Stein Innovation Award. E.J. heads the Stanford Artificial Retina Project which aims to develop an electronic implant which will restore vision to people blinded by incurable retinal disease. And you can learn more about that project by visiting med.stanford.edu forward slash artificial dash retina. That's med.stanford.edu forward slash artificial dash retina. And you can find that link on the YouTube page. So without further ado, I'll hand over to E.J. to present his talk which is entitled Toward a High Fidelity Artificial Retina, The Vision Restoration. Great, thank you very much, Paul. And also I wanna thank Tom for the kind invitation and all to all my European colleagues, thank you for showing up at the end of your work days to attend this talk, I would appreciate that. I also wanna thank the organizers for doing something very special by launching this series. If there's any silver lining to the COVID pandemic that we're experiencing for science, it's that it's allowing us to really focus on staying connected in this important way. And in addition to saving all the trouble of travel and money and carbon footprint and all that, I think the potential for expanding access to people with limited financial resources and also to those with family responsibilities which often translates into women with young children is a big deal for science. It's a very positive development to increase diversity and access in science. And so kudos to the organizers for making this kind of series happen. It's a big deal for science overall. I'm gonna start sharing my screen now to introduce my talk. So what I'll be telling you about today is our research toward development of a high fidelity artificial retina for treating incurable blindness. And let me just zoom here. So the problem I think is fairly obvious which is that incurable blindness is a very debilitating disorder. And although we have wonderful assistive technologies, even simple ones that have been around for quite some time, it would be great if we could use what we've learned in neuroscience about the retina and the visual system to provide more acute sight to people who have lost their sight. So to think about that problem, let's consider the function of the retina at the beginning of the visual system. As many of you know and hope maybe all of you know, light from the outside world is imaged onto the surface of the retina which is a thin sheet of neural tissue with the rear of the eye that transduces light, processes the information and sends information to the brain. And that's neural circuitry is shown at the bottom. What happens in certain blinding diseases notably including age related macular degeneration and retinitis pigmentosa is that for various reasons the photoreceptor cells which capture the light are lost. And consequence of the loss of the photoreceptors there's considerable remodeling or changing of the inner retinal circuitry that processes those visual signals. Interestingly, however, the retinal ganglion cells, the output neurons of the retina that send the visual information to the brain remain alive and functional in large numbers which opens up the opportunity for a technological intervention that can be summarized this way. A camera that captures the visual image that the photoreceptors no longer capture, circuitry that can process that visual image and then electrodes to electrically stimulate the remnant retinal ganglion cells causing them to send spikes to the brain. If this is done well, this has the potential to restore vision to those who have lost it under these conditions. Now, this is certainly not our idea. It's an idea that people have discussed and worked on and made serious progress on for several decades. The first device that functions in this way is shown here. It's a device called the Argus II retinal implant from second site medical products. On the right is shown a grid of 60 electrodes spaced by a couple of hundred micrometers. And on the left is shown the overall system architecture of the camera mounted on a pair of goggles, a video processing unit electronics and an electrode array mounted on the eye. And here's the overall overview of the system. So this was the first device that was commercially available for treating blindness by electrical stimulation in this way. And there's a glass half empty glass half full story here. The half full aspect is that indeed electrical stimulation of the retina can provide rudimentary visual sensations to individuals who have been profoundly blind for decades. So that's a big deal. It means that this whole enterprise may be possible. On the other hand, these devices provide very limited vision with basic visual sensations consisting of arcs and blobs that together don't do a whole lot in terms of providing useful vision to the patient. In fact, blinded individuals who have these devices restored would not give up their cane or their guide dog for this kind of technology. Now, you can see some right away, some potential advances that are possible with this technology, such as having more electrodes and denser electrodes for stimulation. In fact, there are multiple groups around the world trying various approaches to improving the technology. Perhaps the leading or one of the most exciting approaches that's available today is in clinical trials. It's a sub-retinal approach. This is the one developed by Daniel Polanker and colleagues across the hall from us at Stanford University, a very exciting technology with smaller pixels, more of them, hundreds of pixels. And a device for transmitting the image via infrared imaging onto the device. And these photovoltaic cells convert that into electrical energy and stimulate neurons. So this technology is an exciting one. Higher density, more electrodes, and natural coupling to human vision and eye movements are important aspects. And we'll learn soon in clinical trials how well these do, but it's an exciting prospect for the near future. However, I want to point out something that in all the existing technologies and the technologies in development that is missing. And that is a fundamental aspect of the basic science, which we can illustrate very simply by thinking about what the retina is and what it isn't. The retina is not a camera. So obviously a camera is a device that takes a visual image and translates it into a grid of pixels that if they are done well, represent the image intensities and colors in an efficient and accurate fashion that relates directly to the incident light in the image. As we know, the retina doesn't do that. The retina transduces the visual image and turns it into eventually spikes in optic nerve fibers that are transmitted to a number of different brain targets. And although these spikes are related to the image intensities, obviously, they don't directly represent the pixel intensities at each location in the scene. It's much more complex than that. Indeed, we know that the retina consists of many different cell types of various major categories, including many different interneuron types and many different retinal ganglion cell types that send the visual image to the brain. In fact, in macaques and in humans, there are about 20 different retinal ganglion cell types. And these cell types differ in their morphology. They differ in their particular connectivity to retinal interneurons. They differ in their light response properties as far as it's been measured and they differ in exactly where they project centrally in the brain. Furthermore, each of the different retinal ganglion cell types appears to represent the entire visual image, telling us that this distinct cell type representation is kind of a big deal for vision. Now, the striking fact is that none of what I just told you is incorporated in any of the existing technologies for replacing vision. All of the existing technologies treat the problem as a problem of registering the visual image in a grid of pixels. And yet, for those of us who've spent many years studying neural coding and the retina and its relationship to this complex and beautiful neural circuitry, it's somewhat disappointing to realize that all of this information is not part of the neuroengineering efforts that have been developed so far. So this brings up the question, can we use what we know about the neural code to improve the neuroengineering and develop better devices for restoring vision? And a side point or maybe a slightly philosophical question that goes along with that is that if we can't do this in the retina, then where can we do this? I mean, really the retina is one of the best understood neural circuits in the brain. And it seems like we should be able to use what we know here to improve the neuroengineering efforts. And hopefully that we can lead the way there in terms of how many aspects of understanding brain function will be helpful for neuroengineering efforts in the future. So to think about that a little bit more deeply now, I'm gonna take you through, I'm gonna quickly summarize a couple of the main points and then talk a little bit about the functional requirements for such a device. Just to summarize things that most of you in the audience know very briefly, let me just point out a couple of things. So there are multiple different retinal ganglion cell types that transmit visual information to the brain. And these are some summaries of retinal ganglion cell types in the monkey and in the mouse. And the very distinctive morphology of the different ganglion cell types strongly suggests differences in visual function that have been observed as far as been measured. Also, I would like to point out that the collection of cell types in the monkey and in the mouse seem to be quite different from one another. And that's again, born out in what we know about their functional properties. So that's an important issue to think about in terms of the translational impact. We also know that the different ganglion cell types, notably these three classes of retinal ganglion cells, midget parasol and small vestratified, project to different layers of the lateral geniculate nucleus, which in turn have different projections into areas of the visual cortex that are responsible for various aspects of vision. So the projection patterns are different and the properties of those cells as summarized by the receptive field properties are different. Furthermore, as far as we know from almost every study that's looked at it, the anatomical organization of the different cell types indicates that each cell type covers the visual scene forming a tidy uniform representation of the image that's transmitted to the brain, indicating that these different cell types and different functional properties are not just epiphenomena, but they seem to be something fundamental about how the visual image is represented. Now, what functional aspects do need to think about to incorporate what we know about the retina into neuroengineering? Well, to get a little bit deeper into that, let's look at the functional aspects from the point of view of physiology. And in order to get into that, I will summarize some work on this and I'll be discussing the work in my lab on this, but really we've built on the work of many other investigators who've done decades of beautiful retinal research in many species. So I'm just gonna summarize it, our work because it's easiest for me, frankly, and it summarizes an important aspect of things. So choice of species is fundamental and although several species have been useful in understanding the functional properties of the retina, we're gonna focus on the retina of the macaque monkeys. And we do that because, as we mentioned, the macaque monkey retina has a parts list or set of cell types that are extremely similar to the human retina. And that's just not true for these other species. We obtain macaque monkey eyes from other investigators who are using animals in the course of their research. And when the animal is euthanized, we take the eyes and bring them back to our lab and are able to use them to perform electrophysiological recordings for periods of 24 to 48 hours. This allows us to collect a lot of data on the species of greatest interest. The physiological experiments we do to characterize what's going on in the retinal circuitry are performed using a setup that's indicated here where briefly a piece of retina's X size placed against a multi-electrode array with small electrodes, 30 to 60 microns apart and five to 10 microns in diameter. The retina's superfuse with an oxygenated saline solution and stimulated from above with the optically reduced image of a computer display. So this allows us to stimulate the retina and record its naturalistic activity. We do this using a custom large-scale multi-electrode recording system with several different kinds of electrode arrays shown at the top of the image. Custom analog BLSI circuitry developed in close collaboration with my colleague Alan Licky at BC Santa Cruz. This allows us to record the spikes from hundreds of cells simultaneously on 512 electrodes. And when we collect those spikes, of course, the voltage recordings are complex as a function of time. We're able to isolate the individual spikes and identify their spike times. I won't be discussing that in detail in the interest of time. When we have a recording like this, we then proceed to use now standard visual stimulation approaches with so-called white noise visual stimuli to characterize the light response properties of the cells recorded, collect up the spikes of the cells, and then reverse correlate those spikes with the white noise visual stimulus to identify the receptive field location of the cells recorded, which we can summarize this way, as well as the time course and chromatic properties of the cells that are recorded. And recordings like this allow us to simultaneously characterize responses of hundreds of cells and separate out the distinct cell types and an example of such a recording is shown here summarizing all these techniques. It's a busy slide, so please forgive that, but basically we can separate out the different cell types according to their functional properties as shown by these distinct clusters. And we can summarize the locations of the receptive fields of each of the types by each of these little regions here. So this is a whole collection of cells of one type, the so-called off-parasol cells recorded in one recording. Here's another cell type, another cell type, and so on. These are the five major cell types in the cat monkey retina that collectively constitute about 75 or 80% of the entire visual signal transmitted to the brain. We also identify the chromatic and temporal response properties of the cells, which allows us to cluster them and separate out the cell types. So we've got a pretty good situation in terms of seeing the patterns of activity and the spatial and temporal and chromatic aspects of light response. Now I wanna point out some, a couple of key aspects of the response of an individual cell that really should guide the way we think about the neuroengineering. And the first is the precision of timing of spikes in each individual cell. If we put on a visual stimulus that in this case flickers up and down in time and we record the spikes from a given cell of one of the types and then represent it in a raster format shown here where each tick is a spike and each row represents a different repeat of the identical stimulus. What we see is a very precise elicited pattern of electrical activity in this particular cell. Very, very reproducible. It's a little machine. And in fact, the variability in the timing can be as low as about one millisecond. So what this tells us is each individual cell has very precise spike timing properties. But perhaps even more importantly, what we see is that the different cell types have radically different properties while individual cells of a type are very similar to one another. So here's an example where we have a visual stimulus and we have a raster responses like I just show you. This is for a particular on parasol cell in the particular retina. It's one of the five major cell types that I mentioned. And you can see it fires at these particular instances in time relative to this visual stimulus. Now, if we record from a second on parasol cell in the exact same retina, what we see is that it's just a little carbon copy of the first parasol cell, really essentially identical visual response properties. Whereas if we go to a different cell type, in this case, the off parasol cell, it couldn't be more different. And another off parasol cell again is exactly the same. And this goes on as we march through the different cell types in the primate retina, the five major cell types that are most of the visual signal. These cells are radically different from one another, but within a cell type, the cells are extremely precisely similar to one another in their response properties. So clearly the neural code is very diverse across these different cell types and each cell type is a very precise representation. Finally, of course, many of you know that models have been developed in literature worked by us and by many others before us to explain the light response properties of each of these cell types. And along in the short of it is we have pretty good models to explain the responses of each cell types. We're not gonna go into detail on that, but an example of a case where a white noise visual stimulus is presented and the data from the cell are shown here and here are the predictions of the model show that in some cases, you can do a rather good job of explaining the responses of a given cell to the visual stimulus. So that's a quick summary of where we're at in terms of the basic science of the retina. And I'm sorry if that was a repeat for some of you, but I just wanted to make sure we're all on the same page. Now, so where does this leave us? So I'm gonna put a schematic of where this leaves us conceptually. We have multiple different cell types in the retina, each that has very particular response properties responding to the changing visual image over time. We have pretty good models of what these different patterns of activity are. And it's clear that the patterns of activity in the different cell types are radically different from one another and very precise within each cell type. We take this to be likely important that the visual images represented so distinctly in these different cell types that project the different centers in the brain. And we assume or I wanna put forward the hypothesis that it's a big deal for reproducing the function of the visual system to get close at least to reproducing these patterns of activity. Now, the challenge of course is that the cell types are not neatly laid out from one another in these separate sheets, but instead they're all mixed up with one another on the surface of the retina. And so if we go in with an electrode array and naively stimulate each different pixel, each different electrode as if it were a pixel in the image, what we see is we're not gonna create these different patterns of activity in these different cell types if we're just non-selectively activating all of the different cells. So this is potentially very important. And this is I would say the key issue for trying to understand both the retinal implants and quite likely other kinds of implants into the brain and argument I'll make at the end of the talk. So what are we gonna do with this? Well, this has very striking implications for us. In particular, the strongest implication is that if we're gonna reproduce these different patterns of activity in the different cell types in order to create something more like naturalistic vision, then we can't be non-selectively activating cells and we need to achieve something like single cell resolution in order to be able to make these distinct patterns of activity. Now, that hasn't been achieved to my knowledge in any other work. And so I wanna go into that and show that it's actually possible in many cases to achieve single cell resolution. So the way we approach this is in our same laboratory environment where we now not only record from the gang themselves but we also pass current using technology, again, developed by our physics collaborators, Pablo Hatterby and Wadik Dabrowski and Alan Lipkin College. So we passed very small currents of about a microamp here for periods of about a 10th of a millisecond. So roughly a hundred Pico Coulombs of current in bi-phasic or tri-phasic current pulses through individual electrodes. And we do that while collecting up the activity of all the cells in the region. And that's important because it turns out that in order to understand what's going on, if you pass current through a single electrode, it's not good enough to just record from one cell, you have to record from all the cells to know what's going on. And the key question is, can we achieve something like cellular resolution to represent the image distinctly in different cell types? So an example of such a stimulation and recording experiment is shown here. This is a voltage trace of electrical stimulation performed at this time shown by the arrow, one of those very brief current pulses. And voltage is shown as a function of time recorded on a nearby electrode. And there are multiple traces where a small amount of current is passed at this point in time. What we see in the voltage trace is that every single time we pass the current, there's this bi-phasic voltage deflection shown here. This turns out to be electrical artifact that's inevitable when you pass current and record at the same time. However, shortly after the current artifact or partway through the artifact, what we see is that the traces recorded diverge into two distinct traces that are distinct from one another, these in some trials and these in other trials. And we refer to these as successes and failures respectively because if we average these and average these and subtract the two, what we get is a beautiful action potential waveform that looks exactly like a classical action potential often recorded in the retina. And in fact, it looks exactly like the spontaneous activity recorded from the same electrode at times that are unrelated to the electrical stimulation. So what we can see then is that we're able to deliver this pulse and sometimes elicit a spike in the cell and sometimes elicit a spike in the cell and sometimes not. Indeed, we can also see that the spike that's elicited is very precisely time locked to the occurrence of the stimulus. You can see the time scale bar here is one millisecond. So the fluctuation in the timing is tiny. So we're able to elicit a tiny spike with a single spike with a tiny amount of current with a very precise, very high time resolution and not going into the details the charge densities required to do this are well within established safety limits. So can we do this in the multiple distinct cell types in order to represent the visual image more completely? And to make a long story short, the answer is yes. At the top here are shown examples of the five major cell types in the macaque monkey retina that we've discussed before. I'm showing the now artifacts subtracted traces of trials in which a spike was elicited and trials in which it was not for each of these different cell types. We see that as we increase the amount of current that we pass in the range of 100 picocoulons, the probability of eliciting a spike rises from zero to one all with similar electrical stimulation thresholds. The timing of the spikes that are elicited, as I mentioned before is extremely precise with a time variability of less than 0.1 milliseconds, which is more precise than natural visual signals. And perhaps most importantly, we can often activate a single ganglion cell to fire a single spike with very high probability without activating any of its neighbors of the same type. Examples are shown here for the on parasol, off parasol, on midget, off midget, and small by stratified cells of this kind of very precise selectivity in the peripheral macaque monkey retina. Now you may be wondering, okay, so well, first of all, this is a good news that we can actually activate a single cell, but you may be wondering, wait a second, if I activate this on parasol cell, I know that there's some other cell types right around this cell, do I activate them? And so that experiment is shown in the next slide. So here's an example we have in a small scale electrode array, recordings from four of the five major cell types in this case. And we target this particular off midget cell for electrical stimulation by picking an electrode nearby that cell. We pass current through that cell of a certain amount, and we're able to elicit spikes in that cell with probability close to one. But we don't elicit spikes in any of the overlying on midget, off parasol and on parasol cells. These are actually all on top of one another. They've been separated out for convenience. So this is an example where we can activate one cell of one type all alone without activating any other neighboring cells in the vicinity. Now that's not always the case. If that were always the case, we would be done with this problem. An example where we fail a little bit is shown here, we pass current to activate this on midget cell in a particular electrode. And although we're able to activate that cell, we get a little bit of splatter onto this off midget cell. A worse example is shown here where we pass current and we activate this cell with moderate probability about 0.5, but at that current level required to get this cell to spike sometimes we actually activate this cell to fire spikes in all of the trials. So what you can see is that there are cases of partial selectivity, perfect selectivity and failure of selectivity here. And about half the cases in the peripheral retina were able to obtain partial or complete selectivity for an individual cell. So overall, this is very good news because we're able to frequently get single cell single spike activation with precise timing. But of course we're not just trying to activate single cells, we're trying to create patterns of electrical activity in the population of cells. And a pattern that we might wanna create is one that's illustrated here, which is the pattern of activity in a collection of on parasol cells elicited by a bar of light that crosses the receptive fields of all of those cells. What you can see in this activity, I hope you can see this on this live stream, is that there's background firing followed by a wave of activity sweeping across the collection of cells when the bar swoops across there. And this is what you would see in your peripheral retina if a moving stimulus was traversing that region of retina, this is the pattern of activity that you'd be working with. Can we reproduce patterns of activity like this by using this kind of electrical stimulation approach passing small amounts of current through these small electrodes? Well, it turns out that we can. And an example of that is shown here, again, for a small scale recording, in this case for six on parasol cells in a single retina, when we sweep a bar of light across these cells, we activate the first cell, cell number one first and then cells number two, three and four because of their locations and then cells five and six after that, as you can see from this raster. So those are the light evoked spikes. In the experiment then, we then do a huge amount of calibration by identifying the amount of current required to activate each cell through each of the different electrodes on the electrode array. Find that current, find the electrode that best activates each cell in the current level to do so and then attempt to reproduce this pattern of activity by passing current at the points in time shown with the red tick marks. And it turns out we can reproduce the activity very precisely based on exactly what I told you before activating, for example, this cell to fire very precisely when it's supposed to fire while not activating this other cell and so on for the other cell. So we're able to reproduce these patterns of activity in some cases, cell by cell, spike by spike. And indeed, the activation is very precise, as I mentioned to you before. So if we look at what happens with a visual stimulus that's presented repeatedly and limit a raster of responses, we see cell one fires earlier, cell two, three, four, five and six, so on, repeatedly across trials. And the single trial that you just saw is shown by black dots in this trace. Now, if we do our electrical stimulation repeatedly instead of the visual stimulation, what we see is that a somewhat striking pattern of activity. These vertical lines are not artifacts of the graphs. These are the times when the spikes are elicited. So you can see we can elicit spikes from these cells much more precisely than what occurs with the visual stimulus. So in a sense, we have some extra headroom in terms of our temporal precision for activating the cells that's not available with the visual stimulus. And that turns out to be very useful. We'll talk about that in a couple of minutes. Okay, so we're doing pretty well here in terms of getting patterns of activity in order to recreate the neural code and the different cell types. But we're far from done. One of the reasons we're far from done is the problem of axon stimulation. So this is an image of the peripheral macaque retina with the ganglion cell bodies. This is labeled with tubulin. Ganglion cell bodies are shown in these green circles and the overlying axons on the surface of the retina are shown in these green bundles. The problem of course is that when you introduce an electrode array onto this and you pass current, for example, at this electrode right here, what we might expect is that we'll activate not a cell, but in fact, a whole bunch of axons coming from remote cells. And indeed, that is known to be a problem with existing retinal prostheses. On the other hand, maybe we'll be fortunate and some electrodes will be close to a cell and able to activate that cell without activating axons. So can we do this? So let's look at some raw traces to see how well we can go about this. And I'm gonna show you raw traces, voltage traces, after passing current at a particular point in time, a very tiny amount of current at a particular electrode, the one shown here. And you're gonna see a movie of the voltage recorded across the entire electrode array during passing current. And over here on the top panel, which we'll see, so when we pass current, there will be a brief current artifact on many electrodes followed by activation of a spike at the cell right here that then travels down the axon toward the brain. So here we go. We pass current, artifact, traveling spike. Let's just do that again. Pass current, artifact, traveling spike going down the axon toward the brain. Okay, now if we pass a little more current, as shown in the lower movie here, what we see is a very different pattern of activity where instead the signal propagates bidirectionally, indicative of activating an axon. So here we'll see that in the movie. We pass current, and then we have bidirectional propagation of the signal. Again, we can just do that again. We pass current, bidirectional propagation. So that's bad. Now the cool thing in this situation is that what we do is we simply say, okay, well that means at 1.5 microamps we can activate a single cell and at two microamps we start activating axon. So that's a good situation because if we can calibrate this thing then we can just make sure we don't pass two microamps instead we pass 1.5. Now how do we verify that indeed the signal at the top is the activation of a single cell? Well, we do that by looking at the electrical image of the cells recorded using spontaneous or visually evoked activity. And an example of that is shown here. Electrical images is a spatial representation of what happens when a single cell fires a spike. And we do that by taking the recording of the cell without electrical stimulation, in this case with visual stimulation or spontaneous activities fine too. We identify the spikes from a particular cell by spikes sorting, figure out when the spikes happened and look at multiple electrodes now and see what happens at these different electrodes when the cell fires a spike. Well, when you look at these other cells you can't see necessarily anything much happening because the fluctuations due to activity of other cells and instrument noise and so on are high. But if we average across many of those spikes what we can see is very particular waveforms at each of these electrodes when the cell fired a spike. A classic action potential waveform recorded from the soma, what appears to be a dendritic waveform recorded at the dendrites near the soma and a traveling axon potential with its classic triphasic structure. And also at this electrode, electrode number four there's no signal at all. So this sort of indicates that we're recording from the cell at multiple different locations but the space time movie of the spike is the most clear thing. This shows what happens in about a one millisecond movie when we record the cell. We see activation of the soma right here. We see dendrites nearby and then propagation down the axon. So it turns out that we can use this recording from spontaneous or visually alert activity and compare it to electrical stimulation to figure out what's going on. So here's an electrical image of a particular cell recorded next to it is the image of the axons and where all the electrodes are and all that. This is the electrical image of the cell, somatic activation, the signal traveling down the axon with visual stimulation. When we use electrical stimulation of a particular amount in this case, 0.9 microamps we get something that looks extremely similar to that indicating we're activating that cell. On the other hand, if we pass too much current we get bi-directional propagation indicating that we're activating many cells at their axons. So this is the way that we can calibrate the system to understand when we're activating one cell and when we're activating many. And indeed we can do this to compare what are the activation thresholds at axon bundles and somas for each of three different retinas shown here. And what we see is that those activation levels are in the same range, which is obviously problematic. It would be nice if the activation thresholds for bundles were very high compared to somatic but that's not the case, they're similar. What's most interesting however is just to look at the collection of them, electrode by electrode and cell by cell. And what we see here is that for any given electrode what we see is that the activation threshold for the soma and activation for the bundle are sometimes bundle activation threshold is sometimes higher and sometimes lower than somatic. So again, if we calibrate in about half the cases what we can do is to stimulate the soma without activating the axon bundle. So basically half the electrodes we're able to do this in half or not. That's a great situation because that means as long as we can calibrate our device it just needs that half of the electrodes are not useful to us, but the other half are very useful to us because we can activate individual cells that way. Now it turns out that this problem is a little bit, well, it's only just, it's a little bit improved if we look in the central area of the retina. So I've been talking about peripheral retina we're increasingly going toward the fovea of the retina. So here's the macaque fovea, here's the optic disc and here's the area of the macaque retina known as the refae, where the axons coming from peripheral retina go out and around and avoid the refae actually to go to the optic nerve. So this is a region where there's relatively low axon density. And it turns out if we do the same experiment in the refae area, more cells have a higher, sorry, more electrodes have a higher bundle activation threshold than somatic activation threshold. In other words, 75% of the electrodes that activate a single gang cell do so below the bundle activation threshold. So again, the situation is pretty favorable it seems like in the refae region a more interesting region for visual restoration we may be able to get very high percentage of electrodes that don't have axon activation problems as long as we can calibrate. And let me just point out that this necessity to record and stimulate at the same time is critical. There are no retinal implants that record and stimulate. And that's absolutely necessary in order to understand whether you're activating axons or something else. So where does this leave us? So what I've told you then is that the goal is to create these distinct patterns of activity in multiple different cell types. We'd like to do so very precisely. We'd like to create different patterns of activity in the different cell types. And of course we don't want to activate axons which I just told you about. And what we've seen in summary is that the situation is not bad. We can't necessarily activate every single cell precisely in perfect isolation, but many times we do get perfect isolation. Other times we get pretty good isolation. So we're not talking about indiscriminately activating hundreds or thousands of cells as current devices do. Instead, we're in a regime where we may be able to reproduce the neural code with pretty good fidelity. So where does that leave us in terms of actually exploiting the fidelity and understanding it? Well, we've left one major thing on the table so far in all of this discussion. And the major thing we've left on the table is the image encoding. When we have an electrode array connected up to a retina and we have an image that we wish to represent in the pattern of activity of those cells, we have choices in our image encoding about which electrodes we activate with which in relationship to the pixels that are coming in from the image. And we wanna do that in an optimal way so we can produce the correct patterns of activity. But I just told you we can't exactly produce these patterns of activity. We can just get fairly close. So what choices do we make then when we're in a situation where a single electrode, let's say activates two cells instead of just one, maybe two cells of different types? Do we pass current to that electrode or do we not depending on that image that comes in? Well, it's hard to say if we're just trying to reproduce the spikes and we can't reproduce precisely, what do we do? So here I wanna ask you to do a sort of frame shift mentally about the problem. I've been describing this problem as a problem of reproducing spikes we wanna make these patterns of activity. That's not actually what we're trying to do but we're actually trying to do is reproduce visual perception. We're trying to get the person to see something as close as possible to the image that we have presented. So rather than think about our image encoding as turning a bunch of knobs to do the best we can to reproduce these individual spikes can we try to think productively about the problem of reproducing visual perception? Well, we wanna do this for now in an ex vivo situation where we're just working with the retina. So there is no visual perception it's just the retina and the dish. So how can we think about this problem productively and think about how to optimize visual perception? So here I wanna put forward a hypothesis that's a working hypothesis that will hopefully advance what we can do and it's surely wrong in detail. So when you ask me questions about it I'll tell you that it's surely wrong in detail but you'll see that it's a hypothesis that we can work with and we can extend and hopefully it's the basis for thinking productively as I said. Here's the hypothesis. The hypothesis is that what the brain is doing with the incoming visual signal is to use that to reconstruct the image as faithfully as possible. That is the brain is performing an optimal reconstruction or inference of the image. It's important to have some idea what the brain is doing because if we don't then we don't even know what to work with in terms of turning the knobs in our device. So we're gonna assume that this is what the brain does and we can talk in detail about that assumption in the Q and A. What this allows us to do is think about then, okay, can we somehow build a quantitative model of what the brain is doing with the visual image? And let me remind you that we don't know what the brain does with the visual image. If we had an answer to that you would have heard about it by now. So we need to construct such a model. So let's assume that the brain is doing optimal reconstruction. Can we build a quantitative model of that and use that to reason about how to perform our image encoding as effectively as possible in order to represent the image in the collection of cells? So the first question is, does it make any sense at all for the brain to be using the activity of retinal neurons to reconstruct something that resembles the visual image? And is this completely crazy? If it's not completely crazy how do we do this computationally? And the answer is this in concept. We present a lot of images to a retina and record the visual responses, the electrical responses we get from the retinal gamma cells. And then we try to perform that optimal inference or reconstruction computationally by taking all those fights and using that to reconstruct the individual visual image. We assume that this is done linearly with least square error as the objective function. And that's just because it's the easiest way to think about the problem. Again, we can discuss that in detail in the Q and A. So the computational problems we have are visual stimulus, we have the responses we record and there are some weights that are applied to the responses in order to reconstruct the image. If we assume that this is a linear reconstruction process as shown here, then we assume that we're trying to minimize the error that is the difference between the estimated image and the real image. What we find is that the optimal is given by the least squares linear reconstruction and a well-known solution to this problem, the pseudo inverse. So, okay, so how does this work in practice? Can we use this to develop, let's say, some sort of a functional model of what the brain is doing with the incoming spike trains so that we can optimize our encoding? Well, so let's look at that. So we can do that experimentally now, as I was just indicating. So we have a visual stimulus that we present. We record the activity in this case of on and off parasols, two of the major gang and cell types in the CAC retinal. And their activity is shown here. These are recorded responses in an experiment. We then compute the optimal reconstruction filters that I showed you in the last slide and exactly using this same notation that I showed you. And it turns out that the optimal reconstruction filter associated with this cell spiking is shown here. And for this cell, it's shown here. In other words, every cell contributes a little bit to the visual image in a linear way. And you add up, we take the weighted sum of the reconstruction filters for all of those different cells in order to reconstruct the image. Notice, by the way, that this image looks a lot like the cell's receptive field. So a reconstruction example is shown here where in the lower right is shown the actual reconstruction we get by taking these patterns of activity at the top, applying the appropriate filters that we learned, and then reconstructing the activity. So, and that actually closely resembles what we get if we just smooth the stimulus. So reconstruction is not a crazy way to go, is the point. All right, so we have now, let's say, a model for how the brain is dealing with the retinal image. It's the optimal reconstruction filter. How do we use that to optimize our electrical stimulation? So now let's move into that and consider what the issues are because we've created sort of a surrogate brain, if you will, one big matrix that reconstructs the image from this pattern of activity. So we have a couple more challenges that we have to deal with and I'll tell you how we deal with them all in a single approach. So among these challenges are, first of all, calibrating all stimulation patterns is difficult and impossible. So we can stimulate each electrode individually with current, but all different patterns of current passed through hundreds of electrodes are, that's a vast number of current patterns. So we can't calibrate all of them. And unfortunately, the calibration is not linear. That is the activity that's produced is not a simple linear function of the current passed through all the electrodes. So we can't do anything simple like that. So what we're gonna do is assume that we can create a dictionary that is calibrate a subset of the electrical stimulation patterns. For example, each electrode on each current level individually, and that's it without any combinations across electrodes. And we can talk about that later. Second problem is finding the best achievable stimulation pattern is complicated and slow. So if we have an incoming visual image and we've calibrated the individual electrodes and we wanna somehow find the best pattern of activity, how do we do that? Well, it turns out that we can use a greedy approach which is to sequentially pick the individual electrodes that contribute best toward moving the perceived image in the direction that we want it to. And so let me show you how that all this works in practice with this surrogate brain, if you will, and a dictionary based approach using a greedy algorithm to optimize the stimulation. So here's a recording from a collection of on and off parasol cells in the single macaque retina. This is the visual stimulus that we're gonna think about and we wanna reproduce that visual stimulus by activating these cells with our electrodes. If we had perfect control over the cells, this would be the image that we can produce. And of course, the resolution is limited by the resolution of our recordings from the cells. So that's where we stand in terms of the, if we could totally dial in the patterns of activity of cells, this is what we assume that the brain would construct from the visual scene based on everything I just told you. So now what we do is after our calibration we use a greedy approach to sequentially and rapidly stimulate many electrodes in a rapid temporal sequence, much faster than the integration time of the visual system. So we're gonna stimulate at about 20 kilohertz, which is very fast, so 20 samples per millisecond in order to recreate patterns of activity. And here's how it works, so here we go. So what you can see here is that the greedy algorithm has selected a sequence of electrodes for passing current and that as we do pass the current through all these electrodes, the image that's built up in the surrogate brain is shown at the right, which is the results of what we call a greedy temporal dithering algorithm. Let me play that again for you. We pass current in all these different electrodes rapidly over time over a period of, let's say, 50 to 500 milliseconds. And the result of this greedy temporal dithering, which is what we call it, is an image shown in the lower right, which is a good approximation to the image shown at the top. So this is what we could do if we had perfect control and this is what we do with the control that we actually have that we actually measured in the experiment. I'd point out that this is way better than what you get with simple grid stimulation of the cells. If you simply go in and encode the image pixels and use that to activate each electrode appropriately, the way that current devices work, you get a very bad image that has very little of the correct structure. And we think that's an indication of what goes wrong with current devices. Let me point out that the devices that are out there on the market now are not this bad. They do produce some visual perception and we could discuss the reasons for that. But we do think this represents sort of an extreme of what happens when you activate different cell types, particularly on cells and off cells simultaneously, that you really degrade the visual image compared to what you would do if you use a smart algorithm and the electrical stimulation calibration that I told you about. Okay, so I'm gonna skip over a quantification of that in the interest of time and move to, and just point out that this same kind of algorithm can work with eye movement. So I discussed the creation of a static image by electrical stimulation, but we know that the eye actually moves around. We envision that the way this will work is that as the eye moves around, we electrically stimulate according to the location of the eye and in the visual scene to build up the image in this greedy approach. And the assembled image that we would expect to be developed in the brain would be shown at the right. So we think that this algorithm can be used to run the device in the way that I just told you about. So let me try to summarize what I've told you so far and then end with a couple of comments. So I told you that retinal circuitry and signals, as we know from the basic science are very precise and cell type specific and that fine-grained stimulation in some cases can match this incredible precision. We saw that axon activation is a significant problem, but it can be avoided by appropriate calibration and choice of current levels to the electrodes and that a greedy temporal dithering approach can approach optimal system performance. Two other aspects I didn't tell you about are shown in gray that we can talk about, but those are the main results that I told you about today. So the exciting point is that if we build a device in a smart way, maybe we can really reproduce the visual signals that are necessary for high quality visual perception. So we actually have a project, the Stanford artificial retina project with a design aimed at doing exactly that. The overall design is shown here, but I'll just point out quickly what the key points are. This device needs to do three major things. First, we have to record spontaneous activity to figure out what are the cells, what are the cell types and make models of their light responses, which we can do. Second, we need to stimulate and record to figure out how the different electrodes activate the particular cells and the particular axons. And then finally, when an image comes in, we need to stimulate, to represent that image in the patterns of running cell activity. So this device needs to record, needs to stimulate and we need to be able to calibrate all this information in order to provide an effective visual image. Now this building this device is something we're deeply involved in right now. This is our whole neuroengineering effort, that's a whole other talk. Let me just point out that there are many aspects that are required to do this. I've told you about our algorithms, I've told you about our large scale recording and stimulation development of particular ASICs for the device is underway and two have been taped out right now. Microwire electrodes for penetrating the retina are shown here. Implantation and packaging device are necessary. This is a large neuroengineering effort that our group is involved in and I'm happy to tell you more about that in the Q&A or afterward. But I wanna end with a couple of what I think are important future implications for this technology. I've told you that the idea is vision restoration and that's our sort of North Star here. But there are other important implications that I think will come to your mind and I'd love to discuss if you want to as well. A research instrument is really a clear result of this kind of technology development. We'll be able to stimulate and record and produce patterns of activity in the optic nerve that couldn't be produced before and examine interactions of different cell types for visual perception, what the individual cell types do and so on. Visual augmentation that is providing images that are impossible to provide with natural light stimulation is it immediately becomes possible with a device like this. So very interesting experiments and experiments on visual augmentation are become immediately possible. And finally, we think that these kinds of technology developments, although not directly translatable to the brain in general will be very important for brain interfaces where the problems are essentially the same. We've got to activate many cell types and very particular patterns in order to restore natural patterns of activity. Finally, I wanna just end by thanking the people who did this work. This is the Stanford Artificial Retina Group, the faculty, postdocs and students. I wanna particularly highlight the work of Chris Secureneck, Lauren Chepson, Lauren Grossberg in past experiments in neurophysiology, Sassi Medugula, as well as Atlas Calgotino and Michelle Scha for the algorithms. The entire group is now involved heavily in developing the artificial retina that is funded by the Woodside Neuroscience Institute at Stanford as well as these funding sources. So I thank you very much for your attention and I'll look forward to answering your questions. Well, thank you, E.J. for a really wonderful talk. I really enjoyed that. I'm sure our audience did as well. And I've got a few questions here already, so I'll just begin working through them. So the first is, this came in sort of during the first half of your talk, E.J. Is the spatial selectivity of the stimulation dependent on the cell type? So for example, our parasol cells, which have larger dendritic trees, more difficult to specifically stimulate? The short answer is no. And let me just say a little bit more. The dendrites tend to be not a place where you stimulate, you're very likely to activate cells. Actually, you can activate cells primarily at their somas and on their axons. Those are the major sites of activation. So really it's about somatic activation. The axons, as we discussed, are a serious issue. But basically, if you're trying to focus on somatic activation, it seems to be similar across the cell types. The thresholds for activations are similar and the selectivity appears to be similar. Great, thanks. So I've got another question here. They say, hi, what are the limitations, if any, of the technique due to the inability to decrease firing rates? Yeah, beautiful question. So we, as you may have noticed when we reproduced the pattern of activity of the moving bar, I think it's gonna go to that image. I hope you don't mind me going back there to highlight that very important point. What we can see is that, although we're able to reproduce the patterns of activity even more precisely than their native ones, the reduction in the activity after the bar leaves or set the fields is not present. So we're missing a piece of the neural code and maybe this is what the question was about. We currently can't do this. So this is absolutely a limitation of our approach. We are thinking about that problem. So there's several things that are possible. But the one that excites me most is the potential for shooting down spikes, is what I call it. So it's entirely possible that we can interfere with the propagation of a spike coming down a cell, coming down from a cell, down the axon to the optic nerve by passing current at a particular electrode, either a current that hyperpolarizes at the time that the cell is depolarizing or current that produces sub-threshold activation of sodium channels that inactivates them shortly after thereby interfering with the propagation of a spike or creating a colliding spike. There's several ways we have in mind. We have not actually done the experiments to do this. We're still designing these ideas and haven't done it. But at the moment, this remains a limitation. The question was, I think the first question was simply what are the limitations of this approach? And I would say many. So for example, a big one is that it's very difficult to access the fovea itself. Within the fovea, the retinal ganglion cells are not actually, their cell bodies are actually displaced laterally. And in the central fovea, there's multiple layers of retinal ganglion cells. So it's much harder to activate the cells using surface electrodes, which is the way that we're doing stuff. So among the techniques that we're using to tackle this are shown here, we're moving progressively toward the fovea, which is the higher resolution vision. It's the greatest interest, particularly for AMD patients where this is the region that's lost to disease. We're able to get closer and closer to the fovea. We're currently about recording routinely about two millimeters from the fovea, as shown in this example. When we get toward the fovea, penetrating electrodes may be important. And so we have a important collaboration with the group of Nick Malosh. And Uying is doing the work on this in developing micro wires that we can attach to the device and penetrate the retinal a little bit for more selective activating cells. Another significant problem is just the size of the device, creating a huge electrode that'll cover a big chunk of the retina is very challenging. Our first devices will be about a millimeter in diameter. There are also many engineering challenges. So for example, connecting these electrodes to the ASIC, encapsulating this ASIC and packaging it, having, measuring both the incoming image and the position of the eye, so the calibrate for eye position, processing that information in real time and relaying the information to the interface and electrodes. These are all very substantial engineering challenges that we are deeply engaged in now. And I'd love to tell you about it, but there's no time for that. So what are the limitations? There's lots of limitations. I think the way I think of it is this, we've got a ton of problems to solve. All of them are research hard. None of these problems are like, oh, well, you work on it for a few months and you got it. Don't know, these are hard problems. Everything from the wireless to the sensing to the interface to encapsulation, all requires serious research to do it. But it's all kind of, you can imagine the answer to this. You can imagine how to do these things and we have ways to do it. So we're deeply involved in doing that. And the main point is that we can, in many cases, create very high resolution patterns of activity that mimic the natural signals. So we're in a regime where this becomes something that's really feasible. So I think it's really about having a team that has the extremely diverse expertise that you see here that can all work together and understand what we're working toward. And I think that's our strongest asset for solving all these problems. Great, thanks. I've got another question here. So the question you're asked, do arrest plots seem to imply that this approach is optimal for transient signaling RGCs? Is there a strategy to reproduce sustained signaling with some specificity? Sure, very nice question. So here's the way I would think about that problem. What we're doing is reproducing activity cell by cell and spike by spike. So in a transient cell or a sustained cell, there's a pattern of spikes where we can, in some cases, reproduce that pattern of spikes however we want. So there's no problem with reproducing activity in a sustained cell. It just means produce a bunch of spikes over time instead of just a couple of spikes and then go silent. So all of the issues with encoding, exactly what's the encoding with different cell types? How do they naturally work? All that is totally subsumed by this wonderful observation first made by Chris Secunak in the group that we can do single cell, single spike, sub millisecond time resolution, recreation of neural activity to recreate whatever natural code those cell types have. Thanks. So there's another question here. So it says, great talk, thanks. Is it known how stimulation specificity depends on retinal eccentricity? Yes. Well, is it known? We're working on that. So as I mentioned a couple of minutes ago, working toward the rat-fay area of the retina is sort of the key. And so many of the recordings I told you about are performed in a peripheral retina, maybe out here someplace. The fovea is where we'd like to be because that's the highest resolution area of vision but as I mentioned, it's very difficult. As we go in, the cell activity becomes harder. Interestingly though, as I mentioned the activation of axons is less of a problem in the rat-fay, so that's great. The activation of cells and the spread of activation across cells is more of a problem as we move toward the fovea and now in the rat-fay. And one student, Alec Tokogatino, in the lab is working on that right now. What I can tell you is that we do get selective cellular activation and a quantitative comparison about exactly how selectivity varies with eccentricity is absolutely essential. We're working on that right now. I think what becomes very interesting is asking the question, if you can be very selective in the peripheral retina and not so selective close to the fovea, where's the sweet spot? It's nice to be in the fovea because it's higher resolution vision but it's nice to be in the periphery because we can activate more selectively. Where is the nice intermediate sweet spot that's gonna recreate vision most acutely? That's where I think the stimulation algorithms that I've told you about will help us a lot in designing the placement of the device combined with the data that we're collecting now on stimulating in this area in the rat-fay. And so I think we've got time just for one more question. So I'll go with this question. So what contribution do you think starburst amocrine cells contribute to your mapping? Lots of great questions there about retinal circuitry and how again the cells are activated by different amocrine cell types and all that. I don't have a specific answer to that. I think what I would say is our goal is to reproduce the output of the retina and to make it sound extreme, I'll say it this way, we don't care what happens in all the amocrine cells and bipolar cells and all the other cells in the retina. As long as we can reproduce the correct pattern of output activity, that's all we actually need to do for this application because that's all that ever arrives at the brain. So it's fundamental to our approach that we say this point, the retinal ganglion cells, is the bottleneck, anything that's not happening in the ganglion cells doesn't get to the brain, so we just need to recreate that pattern of activity. Whether it's produced by amocrine cells or direct stimulation from the pullers or whatever it be, we need to reproduce the natural patterns of activity. I think one of the reasons that question is interesting is how it relates to stimulating other cell types in the retina with other technologies that are being developed by other groups. And there are groups that are developing technologies that stimulate bipolar and amocrine cells. And I think a major problem with that is that there's a lot of remodeling in the retina that occurs in degeneration. And so activating those cells has the suffers from the remodeling issues, which may be serious, but perhaps most importantly, in all the existing technologies, there's no capacity to record. So you don't know if you've correctly reproduced the signal. That's why our device is critically centered on this idea of stimulating and recording, calibrating and then using that calibration to reproduce the visual signals accurately as possible. Wonderful, thank you. So you should now be able to see that we've just posted a link on the YouTube feed. So if you would like to join EJ for an informal chat now after talk, do just click on that link. We'll keep the recording here going for a little bit longer just to give people a chance to access that link. And then after a few minutes, and we'll stop the live recording and then we'll just be chatting on Zoom. So I just like at this stage to remind you all that there are a number of upcoming talks, both for Sussex Vision, which this is a part of and also for worldwide neuroscience. So particularly I'd like to draw your attention to Dr. Catrin Frankie's talk on Tuesday the 7th of July, Professor Todd Teely and Emily Cooper's talk on Friday the 10th of July and Professor Thomas Euler's talk on Friday the 17th of July. And I do encourage you to subscribe to the channels. So just as we close this portion of the session, I'd like to say a really big thanks to EJ again for a wonderful talk and for all these questions and for being willing to stick around and chat a bit more. Thank you all for listening. Thank you to Worldwide Neuroscience and do tune into further seminars. Thank you very much. Thank you very much, Paul.