 So yeah thanks. So I'm gonna try to basically do a tour de force through all the tools we developed to reconstruct what we refer to as anatomically realistic circuits. And as I put in the title course I figured this is what this meeting is about from subcellular structures to macroscopic scales and how we can use then these circuitry constructions to do some simulations to bridge the gap not only across structural but also across temporal scales. So basically what I'm gonna show you today is a brief description of how we actually acquire all the data. So one difference to the blue brains and it's been mentioned is that most of our data basically all our data is from preparations in vivo. So the recordings are done in the living animal under different stimulus conditions behavioral conditions as well as the cell labeling is done in vivo and also the reconstruction of neurons which allows to basically reconstruct the neurons within the context of the real circuitry. You will see that second I will tell you a bit how we keep track of these huge amounts of data how we annotate it how we kind of do a quality management which then in the end allows us to integrate data from many different preparations to standardize it in a way that we can integrate them and build up neural networks from all this data and then I will give you kind of an example what you may then be able to do in terms of simulation with its network reconstructions. To make it a bit more kind of plastic or interesting I will try to explain the tools and methods we developed on one specific example you heard in the morning about the whisker system and basically the only thing you need to know if you're not familiar with the system is that the deflection of a single whisker is relate to the brainstem I don't have a pointer so to the brainstem from which is number two then to the thalamus and then to the cortex and this is like any kind of sensory modality you know it's always brainstem thalamus cortex but the nice thing about the whisker system is that they are kind of somatotopically arranged structures which are called these barrels also in the thalamus and what I try to explain here is how the thalamus activates the cortex and the nucleus in the thalamus that's associated with whisker processing is called the VPM. Okay so at the macroscopic scale what kind of structures do we need to reconstruct to be able to build kind of a network so what you see here is kind of an atlas like view of the rodent brain and you see the whisker cortex somatocentric the whisker cortex takes actually up a quite large area of the cortex so just for scale it's about three by three millimeters in size in the red and if you do a tangential kind of a top view onto the cortical surface you see the layout of the barrel cortex these circular structures are individual barrels and these barrels process primarily the information from related principle whisker and if you take a coronal cross section through these tangential view you see it on the right hand side down there would be the thalamus in the center of the slice and then you see the barrels and you can already infer from this kind of cartoonish version the difficulties the cortex is curved the thickness of the cortex changes the the columns or the barrels are tilted with respect to each other the depths of the barrels is slightly different for different barrels and so forth so what we do as always as the first step is do we histological sections along tangential cutting plane so we cut the brain starting at the peer surface all the way down to the white matter and then reconstruct the reference landmarks for everything we need so the reference landmarks in our case are the the peer surface which is traced as the circumference of the individual sections then we also trace the white matter which is shown in blue down there we also trace the barrel outlines and we trade trace the blood vessels which are hardly visible as these orange little circles in each brain section and these structures are automatically extracted in high-resolution images high resolution is here a micrometer approximately a micrometer and voxel size so that we have a real 3d reconstruction of all the large macroscopic anatomical structure in the system just how this may look like so this would be the reconstruction of the cortical surface from this time tangential view and then embedded in there the 3d reconstruction of the white matter surface then the blood vessels throughout the entire cortex here just because I did this on the plane my laptop didn't do a good job but you basically have the 3d pattern of all the blood vessels in this volume and then you can also extract automatically the three-dimensional outlines of all the barrels so you have a nice anatomical reference frame and we've done this for many different animals and what this allows you to do then is to standardize these landmarks from many different animals and figure out what's actually the variability between animals in terms of this 3d layout and quantify how accurate is actually an average barrel cortex if you want to reconstruct the system and what we found to our surprise is that there's very little variability between individual animals at the same age and so all our experiments are done at p28 so it's always the same age approximately the same weight same hemisphere and what I mean little variability means for instance the difference in cortical thickness is on the scale of 50 micrometer between animals the difference in barrel sizes and volumes and locations is on the scale of about 30 microns so it's very little variability between these structures so from this macroscopic resolution which proves to be kind of preserved across animals we do the next step at the cellular resolution so we want to know how many neurons or cells in the first place are distributed here so we again developed a high resolution imaging technique in this case kind of a high throughput confocal microscope and a set of other tracing algorithm and you see again here this would be just one sections again about four by three millimeters on the top left corner you already see parts of the pier in dark or in black here are the blood vessels and if you look kind of you know this more dense area would be the barrel cortex here so if you zoom in you see actually all the cell bodies here we developed a technique to automatically detect all the center locations of all the neurons here and since we want to do it with respect to the anatomical landmarks we need to know where the barrels are so we do a counter standing to reveal the barrel pattern so we can actually extract the number of neurons with respect to the individual barrel columns to the blood vessel pattern and to the pier surface and then you get the number for instance of cells for this particular column or for the neighboring column and so forth and then basically for all the columns and this would be it's roughly about half a million of neurons in the barrel cortex but now you can actually see already differences between individual columns so there's nothing like a stereotypic column all the columns are different not only in terms of geometry but also in terms of cell numbers for instance the columns in red which are in the so-called a row have only about eight to nine thousand neurons the columns on the left hand side in blue have already about thirty thousand neurons so there's already a huge difference in just terms of cell numbers between a column so if you exactly so not only the 3d reference frame is reproducible but also the cell number so the variability in terms of cell numbers between animals for the same column is on the order of five percent so very low but it's a 300 percent difference between different colors columns in the same animal okay but yes so it's about 97 percent correct compared to manual counting of all these cells yeah but compared to other studies that the only quantitative study at such scale was from the Kleinfeldt lab and we came up with exactly the same density of hundred and thirty six thousand whatever so it seems to be comparable also to other automated counting techniques done in other labs but this is just a cellular resolution so what we need to do now is to figure out what kind of cells do these ten thirty twenty thousand or whatever represent so we again developed an automated high throughput imaging and reconstruction technique to reconstruct the axons and dendrites from a large number of in vivo labeled cells so the complete axon I'm just showing one example here since I wanted to show you this kind of thalamocortical activation so what you see here is a reconstruction of a single thalamocortical axon projecting all the way up from the up from the VPM the dendrites are shown in blue through the white matter and then entering the cortex and again we reconstruct these cells with reference to all the anatomical landmarks so the that we have a way to to integrate them to the neuron distributions and to the other cortical reference schemes of course we cannot only reconstruct a single cell but we can this is a zoom in we can also reconstruct many cells this would be one of the phalema recipient cells in layer 4 just the dendrites of a typical layer 4 spiny stellate cell that may receive input from the phalamus so this is kind of the ground anatomical data or the basic anatomical data we have the reference structures the cell the the neuron distribution but you can also do kind of in determine different subtypes if you have a certain genetic mouse or molecular marker or whatever for instance you can determine the fraction of inhibitory interneurons among all cells of for instance somato statin positive interneurons if you want to and so forth so it's not limited to just all the number of neurons but you can do this for any particular stain you're interested in but then we need to integrate this somehow so we're talking here about hundreds of terabytes of imaging data many reconstructions all with respect to different anatomical landmarks how do we keep track of all this data in terms to build a model out of that so to do this we developed a database called the cortex DB 3d and the database what it's main major purposes is to kind of convert all the reconstructions course over time you know you keep adding new things to the stuff to the stuff as you figure out more things to convert it into a standardized format that is primarily useful for a consistency check of everything you did because this is done by different individuals over a long period of time so you want to make sure that everything is nice and correct so consistency check in this respect means all the labels correct you know are there any loops in the reconstructions is the annotation done correctly and so forth the standard format we're using to store our morphologies is the Hock format from the neuron simulation tools but it can be converted into any other kind of data format that is out there and the nice thing of course you can associate meter information to all the individual to make queries and then browse and whatever but the major feature of this or the major thing of this database is that you can do a standardized feature extraction of all the different morphologies feature extraction this sense means for instance simple parameters like dendritic lengths but you can also get with respect to anatomical landmarks let's say home what's the axon density of cell A in the superficial layers of this column versus this column and so forth and this allows us then to do kind of analysis that we are confident that it's correct because all the data has been treated the same has been registered to the same and it's I should emphasize that it's very important to register these neurons to the anatomical landmarks because slicing the brain is not an easy task and reconstructing cells from slices also not an easy task and you can get all kinds of systematic errors if you slice the brain and if the orientation is slightly different and we tried to do some cluster analysis of cells that we didn't register for instance to these landmarks and this proved to be almost impossible because you have all these kind of systematic errors so if you really want to do cell typing of cells based on anatomy you should take care to minimize all the systematic errors you're actually doing here and one example what you can do then for instance is to reconstruct many cells I'm only showing the dendrites here but we've also reconstructed the full 3D pattern of all the axons here but based on the dendritic morphology you can do a cluster analysis within the database and come up with kind of objectively determined cell types and one benefit of doing the cell typing here is that you not only know what discriminates cell type A from B but also you get an estimate of the location where you can find a particular cell type in a particular 3D reference frame which is indicated by the colored bars on the left hand side for instance the brighter green these are layer 4 spinny stellate cells for instance found kind of more within the center of the barrel while another cell type in layer 4 the so-called star pyramids in darker green here are also found closer to layer 3 and layer 5 yes hopefully by the end of the year yeah so we're working on we want to make it a nice format that's not embarrassing for non-computer scientists like us yeah so we it's on its way yeah and so this gives you kind of a way to classify cell types and locations and so forth but then we want to build networks out of this so how do we do this so we have the 3d reference framework and what we do as a first step we split up our 3d it's about as I said 4 by 4 by 2 millimeters into 50 micron voxels and then register the neuron distribution which I said are very kind of stereotypic across animals to this 3d reference framework so with every within every 50 micron voxel we know exactly how many neurons are located within these 50 micron voxels we also know at a certain location in space what kind of cell types do we find there so we replace each individual soma by a 3d reconstruction that was found with 50 micron position at this point in space and take it from the database and put it in there keeping the right orientation location and so forth with respect to these anatomical landmarks so what we end up with is kind of what we refer to as anatomically realistic reconstruction at least within 50 micrometer precision and this is just an example how it looks like this would be the side view it's up there on the top is the standardized PR surface and the barrel outlines projected onto the PR surface the white matter the individual barrels and you if you register for instance the salamocortical axons into its respect or first of all the soma distribution you first register the soma distribution which is based on neuron counts into this framework and you see then the number of neurons you can actually already kind of see the barrels based on the soma density here and the different densities of cells in the different layers and then you replace them for instance with the phalamocortical axons traced from the VPM so this would represent about 250 of the phalamocortical axons here you can paste in other cell types or just visualize other cell types for instance these layer force by nestelate cells but you're not restricted to one particular column so you basically register them to every place in the barrel cortex you find and then superimpose them and you can also do this for different columns so to speak as I said this is just an example you know all the blank spot is filled up by other cell types and we also have the intro cortical axons not only the phalamocortical axons in here this is just an example how this network assembly process is taking place so this would be kind of the transition from the macroscopic to the cellular resolution how do we make map connectivity in this so as I said even though the anatomical layout of the reversal cortex seems to be very stereotypic the numbers the morphologies and so forth we don't believe that geometric proximity of axon and dendrites is good to predict connectivity in this sense because as I said with 50 micron position we can do that but if we want to predict the contact this is at much smaller scales and so what we do is we count the boutons along the axons and the spines along the dendrites and convert our axon distribution which was shown here our axon distribution to a bouton distribution with 50 micron resolution and the spines dendrites to spine distributions and do a statistical estimate within these voxels how many contacts this particular cell may receive so we're not saying for instance this is the layer 4 cell I've shown you in the beginning what you see here would be kind of the 50 micron resolution innovation by the phalamus for this particular cell so the color code reflects a certain number of contacts and we can then place on the dendrites that are located with a respective voxel randomly synapses onto the cell and we're not claiming that this connectivity may be the correct way to wire the network but this is at least a constraint how it could be and we can then investigate based on these constraints how different anatomically anatomical wiring patterns may influence the cells function or the network function so to sum this up I've shown you how we can using high throughput imaging and reconstruction techniques reconstruct landmarks reconstruct these landmarks and also the cell distributions and then the axons and dendrites which is shown here in the bottom again just to illustrate it I've shown you how we keep track of all our data with this database and how we standardize it and how we then integrate it to do anatomically realistic network assembly and get finally even an estimate of how many contacts a particular cell may get from a particular presynaptic cell type so that's nice in terms of structure but what can we actually do with this in terms of simulations so the networks we're building up here basically try to incorporate all the pathways we find so we don't restrict ourselves to kind of a dogmatic view where we say this is the network and this has to mean something soon but we reconstruct all the axons and whatever structure we find is connected to this we have to incorporate to the network for instance we found that the connectivity for instance between different cortical columns even exceeds the connectivity within a cortical column so if you want to understand that how the whisker information is processed you cannot just crop off the rest of the cortex and so you have to incorporate this so therefore we needed a simulation framework that is that scales in terms of you know an increasing size of the network and in collaboration with the University of Heidelberg and in particular with Stefan Lang there we developed this neurodune simulation environment and just a couple of things I liked about it first of all is written in C++ which makes it easy for a non scripting expert to figure out what the source code actually means I personally like C++ a lot but another nice feature that is of course therefore easy transferable from one compiler to the next from different hardware settings and can be easily optimized to different hardware conditions it's based on finite volume elements which has certain advantages in terms of numerics for convergences of the solvers it is based on an n-dimensional adaptive grid which is probably the key feature of this neurodune environment which allows to automatically refine the mesh so to speak or the grid into the spatial grid as well as a temporal grid based on the status of the cell or the network which means in reality the cortex or at least in our simulations the activity is rather sparse it's not like that all the cells are constantly active and doing a lot of stuff most of the cells during most conditions are actually silent don't do anything so rather than computing all the cells with the same fine grid and accuracy we only compute where actually something's going on within the network and therefore get a huge speed up in terms of simulation times so yes and the advantage in this is you can set a certain kind of error margin you want to meet and then the grid is automatically refined and we've shown and I think that's also on the website that it's an improvement for at least the networks we looked at so far in terms of speed up of course if you have a network that where all the cells are active you know a constant grid maybe faster but in our case where you have sparse activity this seems to be appropriate no this is a different one it you have to look it up on the on the web page how it's called exactly no no it's specifically not related to anything from Gabriel it's if there's at all competition between the blue brain and us there's yeah and it's not nothing to do with it anyway okay but what can we do with it I am showing you the same neuron I've shown you an example before you see the VPM synapses as one realization how it could be and what we can do now with this Monte Carlo simulation approach we can just investigate how different structural and functional connectivity patterns influence the activity of an individual cell but within the context of the entire network so in green for instance this is a sub-sample of active synapses we chose based on functional recordings that we know if you deflect a whisker only about 60% of the thalamic neurons fire one action potential with this certain latency so we only activate about 60% of the thalamocortical synapses vary their locations and investigate what's actually the consequence for the spiking activity of this individual neuron and then since it's within a context of of the real 3d circuit you can immediately compare the simulation results to data that you've obtained in vivo or even better to functional imaging data so just one example at the sub-cellular scale what you see here on the left-hand side are the spiking activities of neurons measured in vivo upon whisker deflection so these are the little dots the color refers to a specific cell type and at the bottom axis you see the number of thalamocortical synapses that has been predicted for this particular neuron within the network by our network assembly process and if you sort this by certain cell types you actually find for some cell types not all for instance on the top left corner this would be the layer for spiny stellates that the spiking activity of this particular neuron correlates very well with the estimate of thalamocortical synapses we got here so simply speaking if you reconstruct the 3d morphology of this particular cell know its 3d location you can predict its spiking activity in vivo just based on this network assembly process there are other cell types where it's not possible for instance for the layer 5 cells which see a much more diverse input have a much more complex dendritic tree and more active conductances and therefore it's maybe not such a very simple linear relationship between input and output of the cells but if you don't care about synapses and in sub-cellular scales and and just have for instance two photon calcium imaging and care about what's the overall network activity at the cellular scale so what you see here on the right hand side is a top view of the barrel that we've simulated based on whisker deflection input and in green are cells that responded with an action potential in our simulations blue are cells that showed no super threshold activity and you get the impression from this one simulation trial that it seems that cells that are located closer to the center of the barrel seem to be more active and if you do this for many simulations with varying configurations of connectivity and synchrony and so forth you actually find that this seems to be the case that the closer the cell is located to the center of the barrel the more active it seems to be and there's a certain decay towards the barrel center and fortunately for us jason kursk group a few years ago did exactly the experiment to photon imaging after whisker deflection and they actually found this very nice kind of location specific a decay of spiking probability kind of making a proof of principle here that you can do simulations without any parameter tuning just Monte Carlo simulations that you constrain as good as you can anatomically and functionally by in vivo data and make a prediction that you may recover in vivo just no recurrent no it's so of course if you want to get the spike timing for instance right so this is just spike rate here in this example but it seems to be since this is the input seems to be the underlying theme in processing throughout a cortical column that the input shape or the shape of the input that seems to decay is kind of propagated through the column at least for whisker deflection there are other stimuli that could look completely different from this picture but if you don't even care about the cellular resolution but at a more macroscopic resolution for instance using voltage sensitive die-imaging here shown on the left hand side from Damian Wallace where you see kind of these overall summed network activity projected onto the cortical surface again with the same input whisker deflection you can transform the simulation results projected onto the cortical surface and try to figure out what particular cell type to what point in times actually contributing to this kind of macroscopic imaging data okay so simulation part I've shown you one example I mean keep in mind I'm just showing you an example here for the network we have the blood vessels in there which we don't use course we are not really interested in in blood flow but if you're interested in this you have the anatomical reference frame and if you're good equation the simulator will certainly be able to solve this and but our major interest is in how different connectivity patterns may influence the responses of of the circuit okay dog I think I'm done so I want to acknowledge people first of all Bert Sackmann my long-term mentor and who kind of pushes us to develop all these tools from scratch and not use what people have already done probably better than we but yeah we have to do it from scratch then Robert Egger a very bright graduate student in my lab who has been involved in doing the registration framework Andrew Johnson postdoc in our lab doing the data basing and feature analysis and clustering Hanno Meyer postdoc in our lab who contributed essentially to the counting procedures and developing the protocols to make nice histological sections my long-term friends and collaborators Christian the cock and Randy Bruno at in Amsterdam and Columbia who do all the in vivo recordings during different behavioral states and also the labeling of the cells and ship a constant supply of filled neurons to Florida Vincent Dixon at the University in Berlin an expert on visualization and and reconstruction methods in Amira and and I mentioned Stefan long already at the kind of master mind behind the neurodune simulator thanks