 So, good afternoon. Exascaling nuclear innovation is, I don't know, I feel like it's a lot of buzzwords, but there are a lot of buzzwords in nuclear innovation right now, and in exascale, so it seems appropriate. And I spend a lot of time thinking about this from a variety of different perspectives. And the one I'm going to talk about today is mostly about exascale computing. But nuclear innovation in general is a thing I'm really excited about. And why is that? I picked nuclear engineering mostly because I was concerned about this first bullet point. And however you feel about nuclear energy versus other forms of energy is fine with me. But I personally picked nuclear energy for global climate and environmental goals. But to meet those goals, nuclear innovation really is needed. We're sort of at the transition point from an old technology to a newer version of that technology, and there are some big challenges in that. So this is an example of an existing reactor and a model of computation of existing reactors. And we need to develop new methods to make sure we can still model those appropriately and figure out how well they're working. But we also have new reactor designs that we haven't built before or maybe we've built one as an experiment a long time ago, but we haven't built any lately and we haven't built them commercially. And this thing, I'm not going to talk very much about the details, but this one's extra cool. The fuel is a molten salt. So it's just this flowing vat of radioactive stuff that makes power. And it has all these really useful features. But the fact that the thing that is generating the heat is melted salt is, I don't know, it's exciting. But modeling it is hard. I'm also really interested in global and domestic security goals. So making sure that we generate energy in a way that is really globally responsible and all the other applications of nuclear technology is also done in a responsible way. There's an enormous amount of radioactivity used in medical applications, industrial applications, all over the place. So doing that in a secure and responsible way is pretty important to me. And economic prosperity, energy poverty on Earth is a big problem. There are a lot of people without electricity. And so I'm really concerned about how do we get affordable, clean energy to more people, to bring more people out of poverty. So these are like why I care about nuclear innovation. And this is an example of enhanced security, where if you have fuel that has been used, making sure that we can monitor it and take care of it appropriately. So these are the kinds of applications that I think about. And my little piece of nuclear innovation, because it's a pretty big bucket, is numerical methods. So from my seat, to facilitate innovation, we need predictive simulation. So we need to be able to not only model what we already know how to build, but to model things that we would like to build in a way that we have confidence that those answers are right, or at least close to right. Because doing experiments is slow and time consuming and expensive. Slow and time consuming are the same thing. But they're expensive and they can be hard to do. So bootstrapping experiments by having predictive simulation is a pretty important thing. So what I do is build tools, which I call translating applied math into code, that are used to design and analyze these kinds of nuclear systems. And I focus most of that work on high performance computing, though not all. And the algorithm development is really informed by physics of the problems of interest. And so I'm going to get a little bit into what is the problem that we're solving and what's the physics. But an example that I think will, it's a useful picture to go into this discussion with is this is an example of a nuclear cross-section. So we're interested in how neutrons interact and matter to figure out what's going on in nuclear systems. And neutrons interact and matter in kind of wildly varying ways. And one of the things we're interested in is fission. So this is energy, and this is the probability of a neutron causing fission for different materials. And so you can see how this behaves is a bit variable as a function of energy. And this is like 10 or 12 orders of magnitude. So we care about wide ranges of energy. And we have data sets that are difficult to capture and express. So this is part of what is underpinning nuclear that makes things a little bit difficult, is that the physics inside are not trivial to model. So that's sort of where we're coming from. So I'm going to give you a little bit of background. So what is the problem that I'm solving? Why is it hard? What is the current state? And then I'm going to spend most of the talk in algorithms for advanced architectures, so both big machines and big and heterogeneous machines, and how physics provide opportunities when you consider it in the context of the hardware. I'll talk a little bit about nuclear data and some of the problems with nuclear data and some of the things we're working on to try to improve nuclear data. That's those cross sections that go into things. And I might run out of time, but I might talk about this new thing I've been thinking about a little bit, thanks, Katie, which is processing data while computing. And this is only tangentially related to the talk, but it's my favorite reactor. This is the reactor at Penn State. It's a research reactor. I was an operator there, and it's really pretty, so it's my favorite one. So what are we doing? We are finding all of the neutrons in a nuclear system. And we care about that because that lets us figure out things like reactor power or shielding needs or whatever it is, starts with where all the neutrons are. Also, there is no good talk without the Boltzmann transport equation in it, so that box is now checked. On this side of the equation, we have the losses. The thing that we're solving for is this guy, the angular neutron flux. So this is where all the neutrons are at a particular point in space in a direction of motion, angle, and at an energy state. So we want to know what this thing is, and we can have them be removed from that point in phase space by either having an interaction. So this thing, capital sigma, is a cross-section, and that's a probability of interaction. And this is the probability of total interaction. And my mouse keeps going away. The plot that I showed you before that was all squiggly, that was the probability of interaction for fission. So this is for any interaction, our neutron will leave the system. We can also have neutrons physically stream out of our system, so that's this term where they're either changing direction or location or both. We also can have sources of neutrons. This term, which is one of the ones that makes things extra fun to solve, is scattering. So neutrons can scatter off of other atoms, and they will change energy and direction. So this integral is neutrons scattering from any other energy into our energy of interest and any other angle into our angle of interest. So neutrons coming from other places to where we are. And we can have neutrons that show up as a result of fission. So this is the fission cross-section. How many neutrons come out per fission? So fundamentally, fission is caused by a neutron hits an atom that is capable of fissioning. It splits into two pieces. It releases some more neutrons, 2.43 on average, if we're talking about uranium. And those neutrons are born in an energy spectrum, some of which are in our energy of interest. And if we don't have fission, this term goes away and it's a fixed source problem. But this is what we're solving. And solving that accurately is a little bit hard. So it's 6D. It has three spatial two-direction and one energy dimension when we're in steady state. And the systems that we're interested in solving can be very large and or physically complicated. So this is a picture of Eter. Eter has a 6.2-meter major radius. Meshing that and getting the solution with appropriate fidelity everywhere is kind of, this is like a rough flux simulation of this. And it's big. And the data, as you saw, is complicated. And especially the scattering data, getting all of the functions of mapping, the energies, and the angles together is complicated. We also are interested not only in where all the neutrons are, but in coupling the feedback of other kinds of physics to how the neutrons behave. Because you saw there's energy dependence. So heat transfer, fluid flow, structural mechanics, all of that deeply affects where the neutrons are and where the neutrons affect those other things. So we're trying to figure out how to couple the physics together in a way that makes sense, which also requires a lot of computational effort. And I'm going to talk about these solutions from two perspectives. One is deterministic solutions. And that's where we take our space and we discretize it. So we break the energy up into energy bins. We call that multi-group. We use a collocation method in angle and use an angular quadrature. And we do a spatial mesh and use something like finite difference. The other way that we can solve this, which I'll talk a little bit about, is Monte Carlo. And we express our physics continuously and use random numbers to sample the physics. And this is a common thing in many fields. And some of the challenges with Monte Carlo is that to have sufficient accuracy, you need to have a sufficient number of samples in each part of phase space. So that can be really time consuming for certain problems. And with deterministic calculations, it's about having sufficient resolution to actually capture the behaviors that you need. So those are kind of the driving factors of what is a challenge in each space. And so where are we right now? Or in the at least very recent history. So right now, existing tools can basically predict what we already know, but not really new things. And so we have a lot of codes that have been tuned with 40 years of operating experience of stuff we already have. So we can get the answer really well when it's something we already know how to model. We're not quite up to modeling things we don't know very well. So we need to continue to increase accuracy of our methods, increase the resolution of what we can calculate, get coupled multi-physics right, and have significant verification and validation efforts so that we can have confidence in our solutions. We also need to improve our nuclear data. So in sort of the past, when our resolution was not very high, if our data that went in to represent the physics wasn't accurate, it didn't really matter because our answers weren't finely enough resolved to matter. That is no longer the case. We are now starting to be able to have accurate enough answers where the problems in the underlying data mean that we have wrong solutions that are wrong enough that we can tell. So we're having this, I like to call it the garbage in problem. If you ask the wrong question, you will not get the right answer. And we need faster analysis strategies. So as these simulations get bigger and bigger and we have more data and we have coupled data, we have all this stuff to process and it's taking a lot more time to process things and deal with the amount of data that we generate. So that's kind of where we are. And so the question is exascale computing to the rescue? What does that mean? What does that look like? What are we gonna do? So the way I like to think of this with exascale computing is physics plus architecture. And I know most of you are not nuclear engineers. So I'm gonna try to talk a little bit generically about how we think about changes in architecture combined with our physics to think about new algorithms. So hopefully something will translate into something you care about. This is a flux map of just neutrons in a reactor. This is the Titan supercomputer. So it's a GPU accelerated CPU machine. We're also really interested in heterogeneous architecture. So we have traditional CPU platforms, but increasingly because we're concerned about power consumption, we're using GPUs and mics, so multi-integrated cores or Intel Xeon fees to combine with CPUs to improve our calculations at a non-enormous energy cost. So a big challenge is figuring out not only how do we use big machines, but how do we use big heterogeneous machines? And that is not always very easy in nuclear. So the first method set I'm gonna talk about is more generically oriented towards big machines and not specifically GPUs or mics. And this is a set of solvers that is a block pre-loved solver and really quotient iteration and multi-grid in energy preconditioner, and I'll walk through each of those separately, but here is sort of what our matrix, what some of our matrices look like and how we've solved it so far. So with deterministic, when we discretize things into groups and angle sets and so on, this is our energy space angle scattering matrix. So this is how neutrons change energy and angle for an iron heavy water graphite, which we use in nuclear a lot. And a thing that you'll notice about this, so this goes from high energy group to low energy group, so most of the entries are below the diagonal, so that means in general, high energy neutrons have an interaction and they go to a lower energy, we call that downscattering. But down here, we have a bunch of stuff in the upper diagonal or above the diagonal, and that is neutrons that are a low energy can have interactions that actually they get energy from the system and gain energy, we call that upscattering, which is fine, but it's a lot easier to solve lower triangular matrices. So the more upscattering we have in our system, the more challenging it can be to solve this problem. And as we use more energy groups, we have more upscattering entries. But historically, we would break this problem up as we would solve each energy group by itself. So within a given energy group, we would solve the space angle component, and then we've moved to the next energy group and solve the space angle component, because now we know the neutrons from the first thing have scattered down into the second thing and so on. And the outer iterations over energy, we would use typically Gauss Seidel. And then when we have fission, I forgot to mention this on the first slide, the fission is associated with an eigenvalue that tells you the multiplication level of the system. So is your nuclear reactor steady state? Is it increasing? Is it decreasing? That's the dominant eigenvalue, and the dominant eigenmode is the flux shape in the reactor, so we're usually interested if we have fission in the dominant eigenpair, and we use power iteration historically to solve that. But there are some problems. Gauss Seidel is fundamentally serial in energy. So we have methods that are pretty standard at this point where we parallelize our problem in space and in angle, but if we're using Gauss Seidel, we cannot parallelize an energy. And when we have a lot of, that's not super well worded, so when we have a lot of upscattering, it gets slow. And when we have a high dominance ratio in our matrix, power iteration is also really slow. So two of our fundamental solvers can be very slow. Oh, and I didn't mention we have, when we historically in one energy group, we usually use a Krilov solver these days. So the first method is called the block Krilov solver. So it turns out machines are bigger than they used to be, and they can store more than one energy group size of iteration vectors. So the reason that we had to do Krilov in only one energy group is that we couldn't deal with having an iteration vector in our Krilov method that was bigger, but machines are bigger now. So we can have multi-group sized iteration vectors. And that's like really all that changed is we got a big enough machine to store a bigger vector. And that made this method possible. So now, instead of doing Krilov individually and Gauss-Seidel over the energy groups, we just use a Krilov vector that is the size of all of our energy groups. And then we can do space and energy and angle all at one time. What that allows us to do is, A, Krilov methods typically converge better than Gauss-Seidel. So we get better convergence properties by itself. And we can farm out each part of the matrix vector multiply in the Krilov vector to different processors. So we can parallelize an energy now. So we gained an extra dimension of parallelization. So before, if we had this is our spatial domain and we broke it up into four blocks and then each processor would get one block, now we can also divide it up into what we're calling energy sets where each collection of processors will get the entire spatial domain plus some of the energy groups. And they'll each do their spatial angle domain plus the energy groups, and we call those energy sets. So we have now increased the number of cores we can use by the number of energy sets. So that took us from when we could do tens of thousands of cores without too much trouble, we would sort of top out in reasonably sized systems in sort of 20 to 30,000 cores. Now, when we're doing anywhere from 10 to thousands of energy groups, we can use a lot more cores and we've successfully tested these methods out in the hundreds of thousands of cores range because of this energy parallelization addition. And the only thing to change was now we can store bigger vectors. So what this made available was a better Eigenvalue solver. So quick Eigenvalue history, I try to interleave pictures and equations so it's not all of too much of one or the other. So power iteration converges like the second Eigenvalue over the first Eigenvalue, where the spectrum is defined in the usual way. And so in some nuclear systems in particular, those numbers can be very close together. And then our convergence is very slow. Like sometimes we call that the dominance ratio. Sometimes the dominance ratio is like 0.9 or 0.95 and that's bad. So we can do what's called a shifted inverse iteration which many of you may be familiar with but in case you're not, we can apply some shift. So we just take some scalar and we subtract the scalar times the identity matrix from A. And if you do that, you still get the same Eigenvectors. So the shape of our solution doesn't change, great. And we can relate the Eigenvalues of the shifted matrix to the Eigenvalues of the original matrix. So then if we find the new Eigenset, we can get our original Eigenset out of it. And what that buys us is a much better convergent situation. So here our error now decreases as the dominant Eigenvalue minus the shift over the second Eigenvalue minus the shift. And as the shift becomes closer and closer to the dominant Eigenvalue, our error rate improves dramatically. And so Rayleigh quotient iteration is when you choose the Rayleigh quotient as your shift. And the Rayleigh quotient is an optimal guess in a least squares sense. However, when we do this and apply it in the matrices we're interested in, functionally what that looks like is that every single group has upscattering. So when you were solving in Gauss-Seidel and upscattering made things take longer, having upscattering in every group was a very bad idea. So we couldn't use this method because it didn't make any sense when you were using Gauss-Seidel. However, when we're using this block Crelove method and we don't really care if there's upscattering or not because that's not slowing us down or making things serial or whatever, now it actually makes sense to use this method. So this method was enabled by the previous method being enabled by bigger memory. However, you may have thought about if your guess is really good then the condition number of your matrix gets really bad. So you need a preconditioner. So we wrote a preconditioner too. So Rayleigh quotient iteration typically requires a preconditioner. And fortunately we have this new energy parallelization thing so we wrote a multigrid and energy preconditioner. And why is that fortunate? Doing multigrid in one dimension is super simple because in multigrid you take advantage of the fact that there are many iterative methods that remove oscillatory error modes very simply but slow error modes very poorly. So what you do is you take an error mode that looks not very oscillatory on a fine grid and you map it to a coarse grid where it looks relatively much more oscillatory. You apply your numerical method which removes the now oscillatory error mode. You map it back to the fine grid and the error that you couldn't get rid of on the fine grid has now been removed. So you sort of take advantage of tricking the solver into making the error modes look relatively more oscillatory so you can remove them more easily. And the reason it's easy to do in one dimension is mapping between grids in one dimension is pretty straightforward. So we did this in energy because it was easy to implement and the energy thing was also very easily parallelized. So we now have a easy to implement, easy to parallelize pre-conditioner. Yeah, so that's what we did. And we did a lot of tests but the sort of big high impact test was a full pressurized water reactor and I'm not gonna go through all these details but it's this highly heterogeneous system with energy groups and scattering but it was 1.73 trillion unknowns. So it was a big problem. And this was the first problem I ever ran when it was Jaguar on Jaguar. And it was one of those I called my mom and I was like, mom, my computer's code's running on the fastest computer in the world. She's like, what are you talking about? Like, never mind. But it's exciting. And so for the results I'm gonna show next, we used about 140,000 cores. And we found that the new methods use less time and fewer iterations. So that was pretty big win for my PhD thesis. So when we use power iteration without any preconditioning, it took 600 minutes on 140,000 cores. And when we added preconditioning to power iteration, that was a bad idea. Don't do that. That preconditioner did not help out power iteration at all. But when we did really quotient iteration with our preconditioner, it was much faster. And took many fewer iterations. So I was really happy. For problems that aren't very hard, the new solver set wasn't really helpful. Like the time investment in dealing with the preconditioner and the shift and all that isn't really worth it. So it's only for really big hard problems that it is helpful. So if your problem is easy, use the other solvers. Okay, I'm gonna take a pause. I've been, any questions about background plus round one? Yeah, it depends. It's a normal graph. Yeah, so it depends. That's a good question. So the question was how much do the values in that matrix vary? And the answer is it depends on structure and that. Yeah, not in a reliable way because what those values are depend wildly on the material and what energy group structure you choose. So there's no general characterization because the physics vary hugely between physical materials and also what energy group structure you use. So there's no, like it always is this way or it always is that way that we can take advantage of. It is typically diagonally dominant. Good question, yeah. If I'm just correct, the integral on the right depends on those processes which will be really nasty, nasty, very irregular curves. How much does your ability to get those integrals right impact the structure of the low energy species? Because those integrals will really, really quickly hit your integrant in the initial operation. Yes, yeah, yeah, yeah. So getting those integrals right is not easy. And as we now have increasing resolution, our ability to get those integrals right as it turns out is getting worse. So that's part of it. And that does deeply affect what the structure is. And in fact, the scattering I didn't talk about but we usually expand in a polynomial expansion and how many expansion coefficients you have for a given material, berries, I mean it's kind of a, because it's, yeah, so we don't have the expansion coefficients measured or calculated for all to the same extent for all materials. So we use a Legendre polynomials or spherical harmonics. It depends on what you're doing. Yeah, like the physical material, the fissionable material. Yeah, yeah, so the, right, so like the uranium plus, you know, there's cladding made out of zircaloi plus water for coolant. Or if you're in one of these molten salt reactors, there's like a lithium fluoride eutectic. It's whatever stuff you have. So the question is about having material in homogeneities and variation. So that can be enormous and upscattering, it really affects how much fission happens. But the, like how much material homogeneity, it's just up to you how well you model that or not. Like, you know, a complex example is fuel pellets that are used in a reactor course sometimes have a very, very thin coating of a highly absorbent material on the outside. And modeling that accurately is very important. And very hard. So some people like homogenize the whole thing together. Some people try to, so the details of how well you model that is one of the challenges is kind of making it so that you can make really detailed models is some of the goals here. Like making it so that we have good enough methods that you can model that correctly. Did I answer your question? Okay, all right. Yeah, this is the garbage in problem. So we try to develop methods using whatever you give us. Oh, I forgot to end this slide. It scales well too. So this is linear scaling compared to measured scaling out to a little over 250,000 cores. So scales well, strong scales well, weak scales less well. Okay, so totally shifting gears. Monte Carlo methods on GPUs. So this is the heterogeneous architecture flavor of this. So we have a code that we wrote here at Berkeley called warp which kind of stands for weaving all the random particles. And it was developed by Ryan Bergman who's now a postdoc at PSI and Kelly Rowland who many of you know and love. And it's a 3D continuous energy Monte Carlo neutron transport code. And it was written from scratch to be on GPUs. And why is that? So compared to CPUs, GPUs have higher aggregate memory bandwidth, much higher floating point operations per second, and lower energy consumption per flop. However, they're structured very differently. So many times CPU optimized parallel algorithms are not directly portable to GPUs. And that is very true for Monte Carlo, for neutron transport. So in fact, most people who work on like existing large Monte Carlo codes are just like, we're just not using GPUs. Because rewriting their whole code is too hard and they're trying to figure out, are there small pieces of their code they can like farm out to GPUs? But overall they're like, we're just not doing that. We're just gonna wait until that hardware goes away. Is there hope? So it's up to the universities to investigate new algorithms. So that's what we're doing. So warp uses an event-based algorithm. So I know this is a bit hard to read, maybe you can see some of it. But on this side is what is historically used or conventionally used in Monte Carlo, which is a history-based algorithm. And with that idea, you follow one neutron from birth through all of the things that happen to it until it dies. And you just give a certain number of particles to each CPU to go follow and it's embarrassingly parallel. Cause you can just follow it until it dies and follows it until it dies. And every core has all of the data and all of the geometry and everything it needs. On GPUs, you can't do that cause the data is big and you can't just have it have all of the stuff in memory. And following this algorithm and having it go and get every single piece of data it needs every time would take forever. So we can't do that. Instead, we use an event-based algorithm where we have all of the particles take one step and then another step and another step together and we keep all of the particles doing the same thing together in memory so that they need similar kinds of data at similar times. And I'll go through two more slides that explain that a little bit. But if you want to know the details of this algorithm, we have a paper and another paper coming out soon that talks more about performance. But it was really, you have to totally return on its head how you track things to avoid having big data lookups of very disparate data to do this in GPUs was the fundamental challenge in warp. So we use data parallelism instead of task parallelism. So in the historical way where we're doing history-based, you parallelize what tasks are happening and you don't worry about what data they need. They'll find the data. And with GPUs, we parallelize what data they need and we bend them together by what tasks they're doing associated with that data. And we use, so this is written in Cudice and we use a lot of NVIDIA libraries because they work well. So we do a radix sort to map where the data. So this is an example of how we thought through how to take the physics and translate it into the data mapping. So if this is our index of all the particles of where they are and what they're doing next. So we've figured out what just happened to our particle and we have a variety of different reactions they can have. So all the blue ones are elastic scattering and all the green ones are inelastic scattering and all the yellow ones are a different kind of inelastic scattering. Purple means I crossed a boundary, figure out what happens next. Orange means I had a fission, put me in the fission bank and red means I died somehow. So instead of, so at the end of whatever interaction our particle has, a whole bunch of different stuff happens and the data's all over the place. So what we do is we remap it so that everything that needs elastic scattering is in the same place in memory. Everything that needs inelastic scattering is in the same place in memory. So we just do this remapping to put everything that needs the same thing next to each other. And radix fortunately helps us do that and that minimizes thread divergence quite a bit. And this just in, Kelly's laughing, results match standard solvers. So warp as it turns out is in fact pretty fast. It's still a very developmental code and has a long way to go but we compared it to serpent which is a much more standard accepted traditional Monte Carlo code. So this is the flux map as a function of energy of warp compared to serpent and they match very well. And this is, the difference between the two is the squiggly blue line and the green is the bound of one sigma error. So they mismatch pretty much within statistics is what this means. And I don't have timing information because I only got this slide yesterday because until yesterday there had been a big bug in the code and the answers were wrong. So the answers are right now. Okay, so that's Monte Carlo and GPUs and the third example I'm gonna talk about, on my own time, okay, is angle informed methods. So this is a third chunk of how we think of what, of a third version of the things that we do. So hybrid methods use deterministic solutions to create variance reduction maps for Monte Carlo. So we can speed Monte Carlo up by playing games with it to make the statistics go better and to only keep track of important particles and to use weights so that particles that are more important have a higher weight and particles that are less important have a lower weight and this whole thing. And so there are a lot of variance reduction games but one of the ways to do it really well is to use a very coarse deterministic solution to get a map telling Monte Carlo what to do. And we're particularly interested in that in shielding problems. Shielding is very hard for Monte Carlo because the answer in Monte Carlo depends on having enough particles to get an answer. The point of a shield is to not have any particles. So those don't work well together. So we really need good variance reduction methods to get answers when we have shielding. And we have actually hybrid methods that work really well most of the time but those methods that we typically use only include space and energy information. So when we have problems that have a strong angle dependence it doesn't work very well because we don't include that information. And this is an example of, this is a pressurized water reactor and this is a pump room and this is the error map. And so it's a bit hard to see but this is a pipe going through a concrete wall. So you have this like very, only one or two directions matters in that case. And dark blue is low error and light blue and green and red and stuff is high error. So trying to figure out if it's safe to stand in here it's difficult to tell because you have this big error beam of doom coming through this angle. So figuring out a way to include angular information is important in a lot of key problems of interest that have to do largely with monitoring and strategic detection. But including angle explicitly is really expensive because A, figuring out how to sample from an angle map and map back into continuous angle is more complicated than like space or energy. And it makes the variance reduction maps really big and the lookups become expensive and it just isn't a big win. So what we decided to do is take an existing method that works really well and modify it just a little bit in a way that plugs into the old method so you get the information you want without a big overhead cost. So this is the existing method and I'm gonna go through this faster than I'd like because it's just a lot of back explanation but the existing method uses what's called the adjoint transport equation and the adjoint transport equation is like solving the transport equation backwards. So for the transport equation, you start with a source and you figure out where all the particles go and you get an answer. With the adjoint equation, you started the answer and you figure out where all the particles came from to get you the answer. So to sort of understand what that means, we can look at the equation and I'll point out the differences. This is the angular flux with the little dagger means it's the adjoint. So it's the adjoint flux and before we had removal with streaming in a forward direction and now we have a minus sign instead of our streaming term. So it's streaming sort of into where we are instead of out of where we are. And before our scattering was from other energies from other energies into this energy of interest and from other angles into this angle of interest and now we're saying how does our energy scatter into other energies and how does our angle scatter into other angles? So instead of being like what's contributing to me, it's how am I contributing other places? So it's like the reverse solution and what that functionally means if you do some math and proofs, the adjoint flux is an analog for an importance map of the problem. So you can figure out how all the parts of phase space contribute to the solution that you're looking for. So if you have a map of how important all phase space is, you can do variance reduction with that. So there are a lot of methods based on that and this one works particularly well for reasons you can look up. But we can define a bias source distribution, birth weights and a weight map that we use in the variance reduction in this way. And so what we are doing, and I'll show this on the next slide, is changing how we compute this thing. And all we're doing is changing that so that the rest of the mechanics that uses this doesn't have to change. So all the methods that use the bias source distribution and the weights and the weight map, you don't have to change any of that. We're just giving you an adjoint flux that contains more information. How are we doing that? So normally this thing is called the scalar flux. The scalar flux is just that integral over angle of the adjoint flux. So what we are doing instead is taking the adjoint flux, multiplying it by the forward flux and integrating and dividing it by, so we're just normalizing it with the forward flux. Why does that matter? Well, the forward flux has a shape to it that indicates where the neutrons are going in angle. So we're including angular information that you don't have otherwise. And so we're sort of embedding that into the scalar flux and we're just doing everything before as the same way as before, other than this. Why didn't anyone do this before? Well, the angular flux is really big. And the thing that we almost, most codes only write the scalar flux. They don't even store the angular flux internally. They do the integration on the fly, so they never even have the angular flux at all because the thing that we usually use is this. So we had to modify the code to store and write the angular flux and that's only possible now again because of this larger computer situation. So because computers are bigger, we could now store more information and use it when we couldn't do that before. And here's an example of what the difference ends up looking like. So this is a labyrinth problem made out of concrete and here's a source and here's a detector. This is kind of a standard like, I have some medical facility where there's a source in one room and then there's a hallway and I wanna make sure people can stand over in this other room, easy. So you can, if you look at this and you figure out where neutrons might come from the detector, neutrons like to scatter off of things. So the neutrons coming out of here are gonna bounce off this wall and come affect the detector. So they're gonna come from over here but they're also gonna come from over here. And if you look at the original method, it doesn't really notice that anything is coming from over here in a way that is different from coming from this side and the new method that includes angular information now actually includes that effect of all the stuff scattering off the walls. So this is also a really new solution and we weren't really sure what we're gonna find out. But it turns out it does actually matter to include angular information. So that was exciting. And I'm running low on time but we found out we get an answer that is more accurate than the variance reduction method without angle and we get a lower error than not using a variance reduction method and overall we're doing pretty well. So it was good. And we're excited about more tests. Yeah, but I said there would be data. Okay, so this is a very rapid swing through the garbage in problem and then I think I'm gonna skip the last bullet point. But there's all kinds of nuclear data that we need to be correct. So here's one that we think we know well but we might not actually know it that well. So when an atom fissions, when in particular, well when any atom fissions, but this is for uranium-235. When uranium-235 fissions, it splits into two pieces. It can split into a whole bunch of different things and what it splits into varies by energy. So this is mass number and this is fission yield. So this is like the range of what the uranium atom will split into, what other atoms and it changes as a function of energy. So if you don't have this right, the fundamental thing that is happening in your reactor is not modeled correctly. This is an example of a nuclear data file. This is the official format. I know it's a little bit hard to see but it looks like gibberish. And it mostly is gibberish. There are people who can look at this and know what it means. They've been studying this for a very long time but one of the problems in nuclear is that we have like fixed format ASCII input files with gibberish. I'm gonna talk a little bit more about this and what's going on. And oh, this slide takes too long to explain at this point in the talk but how we deal with the data being not quite right is we basically play with it until it gives us an answer that we know. So inside of like fission, there are a whole bunch of different components that make up the fission cross-section and we know for really simple geometry what answer we need to get. So we just fiddle around with the different parts of it until it gives us the answer that we know is right. And there are different data libraries with different information and they've all fiddled with them in different ways. So if you break the fission cross-section up, the values will vary quite a bit between the libraries so they don't have the right answer for the individual physics but they have the integral of the physics right because they know what that's supposed to be. So that underlying premise is deeply disconcerting and the details on this map are comparing the different numbers between two different libraries and show like you add and subtract and when you get all the way around the circle, they all add to one but in totally different ways between the libraries. So, and you can't, and one of the problems is if you get one of those numbers better then which other one do you change to make the integral right? So it's not a trivial problem. So this is some of what's going on in here. So we have a project to try to help make that better. So James is writing an optimization code where you take an incoming neutron source and a spectrum that you wanna get out to the end and automatically design a what we're calling an energy tuning assembly. So neutrons interact with materials, it changes their energy spectrum. So you can layer up different materials to get the neutrons to go from this energy shape to a different energy shape. And this is kind of predictive computing. So right now in our draft, in our sort of initial cut design, this solid smoother black line is the spectrum we're going for and this somewhat jaggedy line is the thing that we say we can produce right now. So we're going to field this, this is a little busy but I'll walk through it. So we're gonna field this on two different sources, measure what we get out and then use the discrepancies between measurement and prediction to find holes in the data and by taking two different sources onto the same shaping device it will help us pinpoint where the problems are. So we have an MCMP model of what we expect. We're gonna use that to design our assembly. We are hoping to field on NIF. There's some reasonable chance that that's gonna happen. And we're also going to field this on the 88 inch cyclotron and then we will compare and then we'll do all this timing measurement and looking at irradiated foils and gamma spectroscopy and nuclear chemistry and we will get out what we think happened and we can use all of that information to try to help improve nuclear data. So there are strategic experiments that you can do to try to unravel all of this and James is doing some of that. So that's one piece is doing experiments to help get data better. Another thing that is starting to happen is community communication is what I call it. So the people who make the data and the people who use the data have historically not talked to one another which sounds ridiculous but this past summer Lee Bernstein organized a nuclear data conference that brought together people who make data and people from a couple of different communities to use data so that they could be like we need to know this stuff better and they could say we can measure this stuff and you could find between everyone what things you could improve most easily and actually do something that made sense. So that idea is new as it turns out. So we're excited about that. Data formats and processing tools. So right now there is no single person who can take from the measured nuclear data and get a data set that you use in software. There are like 15 pieces of software in between that and nobody knows how to run them all and they're arcane and they're horrible and it's a painful disaster. We would like to make that better and we're like slowly, slowly working on it. So some of the data formats are really bad like that ASCII fixed format. It's very error prone. So somebody at LBL is actually developing Aaron Hurst a new data format to get us out of. So we're slowly, slowly getting better about how we manage and process our data because it's just like, it's literally the black box that you just don't deal with. It's kind of, in fact, one of the data format processing tools they turned off the bug tracker because it was too full. It's just, so we're getting there. And we also have a whole bunch of data from experiments that we already did that needs to be processed. So another step is just slowly going through that data and getting it processed and into databases and benchmarks and things like that. So these are kind of the other solutions. And in my, I'm over by one minute, I'm gonna say we have a lot of output. We're trying to analyze and visualize it and transfer it and trying to get data out of experiments and compare it to calculations and model really complicated things. And it takes a long time storing, transferring, visualizing, analyzing, processing memory challenges. So thinking about, and I think Katie pointed me too tight as talking about this for a different field. And I think it's really interesting is how do you get, how do you design algorithms to do computing more strategically so that maybe you do analysis while you're doing computing. Maybe you are more careful about what you ask for. Maybe you're more careful about what you report and figuring out maybe your calculation takes longer but the data processing is now so much simpler, it's worth it. So we're kind of at this place where exascale computing is forcing us to really start to rethink how do we process output? Because just dumping all the data to file and looking at it later is not gonna work anymore. So, you know, there are lots of tools to try to make good software and I get involved with those on campus. So we need nuclear innovation to help solve some of the world's biggest challenges and predictive simulation is required. How do we get there? So using new hardware, whether we like it or not and using our physical problems, using the physics of our problems to inform algorithms in combination can make useful and new tools. We need to continue to improve nuclear data and I'm really excited that there is finally sort of a new generation of people who are thinking about this in a more excited way. And we need to think about new data processing strategies. So thank you. There were a lot of projects buried in here. Many people and funding agencies helped. Thanks.