 Okay, thanks. It's always a pleasure to visit. I'm enjoying it very much. So I'm going to survey this area of mostly our computers helping us do mathematics and trying to do mathematics. And so we'll start with a very quick history of the field. So we have to know, you know, really where we've been, you know, we've been the historical developments to be able to extrapolate. And so, let's see if, right. So that will be the goal to try to really survey what's there now, what are the areas that are making progress that we can really expect to see important progress. So, okay. So to start the brief history of computer science, of course, computers, the idea is somewhat old and was finally realized with electronic computers right during and after World War II. And the original applications were things like cryptanalysis, you know, to break the enigma code and calculating tables for artillery and this sort of thing. And then scientific computations trying to predict the weather and so forth. But in the fifties, people, various visionaries realized that computers were much broader concept, much broader capabilities that you could really manipulate any sort of information. They could help us organize libraries and, you know, someday, you know, search through all the documents in a library, which of course was visionary. This idea of computers as a control system was very important to term the cybernetics of Robert Wiener as, you know, kind of describes that, that branch of the field that has gone off into robotics in many different directions. Artificial intelligence is a concept that goes back to that time. And in fact, as I'll review some of the very first attempts at artificial intelligence were to get computers to prove theorems. This catch-all term of natural computing is less used now, but it was to combine these various ideas of computation that didn't necessarily follow the hardware available at the time, but were more modeled on, you know, existing, you know, physical and natural systems, such as neural networks, such as cellular automata, such as evolutionary computing. And of course, you know, computers have already had a large impact on the mathematical sciences, just this ability to do computations in a straightforward way. And I'll review a bit of that through the talk. And then AI in this period, you know, again, was very emphatic. People were very optimistic at the beginning and, you know, soon realized that the ideas they had did not scale, you know, that solving a problem just by searching through the possibilities quickly runs into the exponential growth of the number of possibilities. But a lot of the foundational technologies, such as just the ability to manipulate symbols in simple ways, were developed then. Okay. So then to skip ahead, a very major leap was the Internet, the web, this type of communication between many, you know, computers of many people that didn't necessarily know each other. I'd like to give an example from my own field of theoretical physics and string theory. So I was a grad student at Caltech, and I worked for Josh Schwartz, who was one of the founders of Superstring Theory. But this was the mid-80s, and it was actually a very frustrating time to be a student at Caltech because these incredible discoveries of the heterotic string and the globial compactification were being made. And we would hear about them through the preprints that would be sent to us in the mail. You know, we would of course hear about it, but then the details would all be in these preprints. And so every couple of weeks we would get this box and we would eagerly, you know, read all the preprints of the latest ideas, mostly coming from Princeton and the Institute, and start thinking about these things ourselves and start the project. And by the time we'd, you know, settled on a good project and gotten started, the next box of preprints would come and they would answer most of all of the questions that we had been formulating. And so back in the 80s and through much of history of physics, there were a few dominant centers of research. You know, you can think of Gooding and the birth of quantum mechanics and various centers of research that just dominated the most advanced topics. Now, in the mid-90s is this period that we call the Second Superstition Revolution when duality and related ideas were developed. And the original see the original discovery to convince people that this was possible to work on at all was made by Ashok Sen, now famous string theorist, and he was working almost alone at the Atata Institute in India. And on the other hand, this time was different because the discoveries were recognized, people all over, you know, Princeton and the Institute were with many people started working on this. But Ashok Sen continued to contribute at the very high level. And the difference was that he could follow the developments immediately because people would put them on the archive, which had been introduced about 1990, 1991 by Paul Ginspar. This was, I was even at the meetings in Aspen where it was realized that this was this was a possible thing to distribute all the preprints to save them on a computer, archive and distribute them. And so what I see is that the way we do research can be changed dramatically by such technological developments. And of course, you can find good and bad sides in this, you know, the world is a much more connected place. So there are perhaps fewer points of view, fewer schools, but it changes and one needs to adapt and try to deal with whatever problems there are. So this question of collaborative work in academia or, you know, just your knowledge production and, you know, using the computer, I think the greatest success story that also surprised people that was enabled by all this is Wikipedia. And of course, it's not the first encyclopedia by any means. I mean, that's, I suppose it took some centuries for printing to lead to the first encyclopedia here. Worthy of the name in France. But this one took only decades. And so the encyclopedia traditionally required a large group of people, editors in charge, editors, appointing editors in charge of the various topics and so forth. And this idea that people could just type in information on the topics they felt that they were expert in, you know, was most people had heard about this in the mid 2000s, didn't think this would work. And it works much better than anybody thought it would. It's by no means the solution to everybody's problems for many obvious reasons, but it's an important example of what's possible. Okay, so now we're in this third comparable leap of machine learning that has led to all these development stuff on the law and others spoke about. And, you know, we clearly are still in the middle of this. It's not quite clear what the scope of these developments is. It's got both academic and, you know, huge commercial value, many, many groups, companies, institutions around the world try to use them. And so the question of this talk beyond this one or within this one of the foresee the next, you know, 10 years is, you know, what does this mean for researchers in the mathematical sciences? What could machine learning and comparable techniques that are really, you know, new? And again, we don't know their limits and capabilities. What could this mean for people that do research that have to teach and that work as mathematical scientists? Okay, so let me quickly survey some of the things that are out there. And this page is of the category of media. So there's just, you know, not so much trying to work particularly with mathematics or science, but just to distribute information in more efficient ways. And I'm not going to describe these in any detail. People are probably familiar with most of them. And if there are terms here that you're not familiar with, you can easily search and find out more. These are things worth finding out about. So, you know, of course, we have a tech and we have the web hypertext and the archives that we've talked about various tools for us, for example, Media Wiki, which underlies Wikipedia for collaborative editing, this idea of a notebook that, you know, is most famously in Mathematica to, you know, organize a series of calculations. Social media for questions and answers like Stack Overflow is actually, you know, especially trying to get into a topic or answer some very technical questions that other people have run into, you know, can be a very helpful resource. Okay, so now this is moving on to computational tools where we're actually, you know, computer is actually helping us work with mathematics, mathematical science. And of course, symbolic algebra would be what probably comes to first the people's minds in that. So Mathematica is a famous example. Sage math is probably the leading open source such system developed by William Stein and many collaborators. There's been many specialized systems for problems in group theory and algebraic geometry and so forth developed over the years. And many of these, the open source ones are in some sense integrated into this sage math. SimPy is an interesting example to actually do symbolic manipulation within a Python program. You can call routines and, you know, there's just again a few, there are maybe 20 or 30 of these systems that are worth worthy of note. The next category is tools that are more used by programmers, but, you know, again, to the extent one has a programming project in the mathematical sciences one might use them. And there's quite a bit to learn from what programmers have already developed. So for example, the question of how do you share the files that make up a program in a way that allows, you know, version control, keeping track of successive versions allows many people to work with the files without getting confused about who did what. So there's a system called GitHub that many people use for that. If you're, you know, again, grad students or postdoc, you probably have used GitHub if you do any computational stuff. And if you're older than that, it's worth, again, learning the basics of that. So again, things that programmers have developed which can be useful for our purposes. Mathematical databases. So maybe the most famous example of that is the online encyclopedia of integer sequences that Neil Sloan and others keep online. You can type in a series of numbers and it fits a known sequence out of a list of, you know, millions of possibilities that will tell you that can be very helpful. And databases associated to other mathematical objects and classifications. The Atlas of finite groups. Database of globium manifolds at string theorists. This is about 25 years old. These are older. And then there's a curated list of about 100 of these at this website. Okay, now interactive theorem verification. Now, this is less well known. I'll go into quite a bit more detail about this later in the talk. So proof assistance that helped to verify a theorem on the computer. So Coq Lean is updated but similar to Coq system. Isabel Hall, Mizar, and the various libraries of work that people have defined many basic mathematical concepts for these things. Again, I'll talk more detail about this and distinguish them from these other related but they use simpler logics, but they're more efficient examples of verified proofs such as the four color theorem and Thomas Hale's proof of the Kepler conjecture. Finally, in this sort of survey, we'll talk, I'll list a few of the kind of more experimental things that people have done in recent years trying to push the envelope. So collaborative textbook, an example that many of you will have seen is called the stacks project for that topic in algebraic geometry. I'll talk shortly and quite a bit more detail about the idea of mathematical search, you know, web-based mathematics collaborations, this topic of AI and theorem proving, I'll go into some detail. Software project to do computer grading of problems in a high school level math class, for example. So a variety of interesting works here. Okay, so let's get now into a bit more detail and I'll start with mathematical search. So many of these topics, your applications, programs, you know, will be familiar, you know, because many of us use them quite often, you know, latex, Wikipedia, Python, Mathematica. Okay, but now the more experimental things, I'm going to start with this topic of mathematical search, which has been emphasized by a number of people, for example, Tim Gower. So okay, so what is a mathematical search edge at well? I mean, in general terms, the idea is very simple. We describe a mathematical concept and we get back whatever is related to that concept that the computer can find online. It might be, you know, documents that refer to the concept. It might be proofs. It might be an algorithm if that's the type of question or even the code for that algorithm. And okay, so that seems like a useful thing. Okay, so now the easy case of this is if we're looking up a theorem or a definition which has a name and we know the name, okay, and then we can use a Google or another search edge. So for example, if we want to know more about a solve a lot of spaces, so we maybe learn this in, you know, in the university, but we don't remember all the details and we want to refresh our memory. We can type solve a lot of space. And, you know, Google is very good because there is a fair amount of human readable information, both textbooks, web pages, Wikipedia pages available with, again, the basics of concepts on this level clearly explained. So that's a good case. And it's very useful. Okay, but of course, it's easy to come up with questions that this would be harder to use. And the basic example would be if you know that there's such a thing out there, but you didn't, you don't know the name. You know, perhaps you forgot the name or perhaps you never learned it. You know, you'd like to know if there's something like that concept, okay? Or even better, suppose you invent a new concept. You know, suppose you are doing, you know, PDE or something, and you have a new version, something like Subtle F space, you know, but with additional conditions or whatever, you know, but then you develop this. But of course, what are the chances that this is actually new? You know, is it also very possible that somebody else has thought about this before? And as a good scholar, you need to do a literature search to find out, you know, did somebody already discover this? Did somebody perhaps even, you know, write programs or make proofs that are useful? So how do you search if you don't know the name? You know, and if there's something new, of course, how could you possibly know the name that those other people gave to it? Okay, so that would be, you know, a next step. And in mathematics, this would seem more possible than in most fields, you know, if you're going to ask for something new in most fields, it's just a totally open-ended question. And of course, while mathematics is open-ended, at least we can state our claims in simple and universal ways. And so we can imagine coming up with a formula or a logical expression that captures or at least describes the concept that we have in mind, and give that as our search term. Okay, so that's, again, something that, you know, you could imagine typing formulas into a computer. There are a couple of examples of web pages that offer that type of service. So in preparing the talk, I looked around and, for example, discovered this wwwsearchonmath.com, which was being maintained by some people in Brazil. So I suppose we wanted to find the concept of prime number just to test it out. Okay, so I hope you can read this all. Anyways, this is close to the definition of prime numbers. You type it in a latex-like form, and it gives the corresponding output. So p is a natural number. There is just no n greater than 1 that divides p. And, well, that's the core of the idea of a prime number anyway. And so we'll type it into that site. And the first thing that comes up is, by definition, how is a prime number represented? And there's the definition, and it seemed to work very well. Okay, so it worked better than I thought it would. So I went on to try the several aspects. So a basic example there would be a function that's both in L2 and its derivative is in L2. And so I typed that in latex. And the first thing that comes up is not quite as good. The situation of Nuremberg-Solobov embedding. So it's not quite the definition, but it's clearly very... It is a question very related to the definition. It's given me names, which were very useful in this subsequent search. And so I think we can count that as a success. Okay, so this is just, again, some relatively simple searching based on the terms and the formulas, which is better than you would think, but it's kind of limited. And, in fact, even not as good as what it looked like. So here's another search for a prime number. And in one way, I've improved it because I've actually corrected the defect. You have to insist that the end of divides q be less than q to actually get the other prime numbers. But on the other hand, the letter p is as very commonly used to denote prime numbers. And so to avoid giving this hint, I've changed that to the letter q in this query. And so it was logically the same query, but now it doesn't work at all. It doesn't find out. So it was kind of relying on this human preference for the letter p. And that you could fix. And in fact, there's another one of these search engines. If you go to, what is it, zb, central dot math, and their web page, they have another search engine divided by a Michael Claus, who was a researcher in this, where you could type like question mark p in that formula. And it would mean do any variable and match it with question mark p. And then another appearance in question mark p in the same formula would have to be the same variable. So that problem can be fixed, but it still doesn't do extremely well. So that's both, it's a limitation of one of those first letters, but you might also regard it as saying, well, formula-based searches is actually better than you might have thought. It is worth getting to work. Okay, now there are deeper problems. Again, the next problem that you would come up with in a good system to do this, of course, is that mathematics builds up its concepts. You make a definition, then making that definition, it relies on that definition and so forth. So even things that seem simple often depend on a large tower of definitions down to the foundations. And then you either have to type in several things, in which case your query becomes much less canonical. The things may differ, the things may come in a different order, or you have to remember the names and the calling conventions. So to illustrate that, I've defined a finite simple group here. So the finite simple group has no non-trivial normal subgroup. So that's easy enough. But if you're supposed to type normal h comma g, to say h is a normal subgroup of g or g comma h, well, if you look at the real books on group theory, they tend to use this little triangle notation to refer to normal subgroups. I've forgotten that when I typed these queries, but there it is. So you have to know quite a bit to build up concepts. Of course, the computer could help you, but that's an example of the sort of thing that you would have to deal with. Okay, now I suppose that we have a good search engine and that works. So what would be the next step? Well, you could take its outputs in various ways. And there's a spectrum where the one limit would be to say, this is just inspiration. This is refreshing our memory. Now we're going to redraw everything and understand it and not necessarily trust anything. And then, of course, the other extreme is the student that perhaps has a project doing an hour and just manages to get a search and just copies it straight into his work and hope that it works for the best. And although it's easy to say one is better than two, and this is sort of what we were told in grad school, of course, that's not a realistic answer. There are many, many very well-established, well-understood results that you'd like to be able to trust it and you'd like to be able to just use it without reworking it every time. Okay, but then, of course, to the extent that you've done a search over the whole web, you don't know that you can trust everything that you find on the web. Okay, so let's, again, just in the interest of concreteness, I can illustrate this by imagining, I have a problem, I'm writing a computer program and some ingredient of the program, I've given a positive integer and I need to write it as a sum of four squares. Okay, you know, something I might need to do. So I type a program to decompose integers of four squares into a Google and the first thing that comes out is sum of four squares, Alpertron, and then the four square theorem. So that's, again, a promising thing where there's a standard name, which in this case is even easy to remember. And so that's good. And here's the web page that that points to. It is a little computer program that you type the number into a window and it puts out a decomposition into four squares. And so then, do we trust this? Well, you can look at the source code. It's there on the web page, but you have to look at the source code. And then what's more kind of in practice, again, it's quite reasonable to think that this is a serious author who has tested his program. But what about the cases that he didn't originally anticipate? Suppose you need to put very large numbers into it. Suppose you believe the algorithm will work in some extension field of Z. Do you like to try it for that? And then, of course, in practice, software often depends on previous software, which works for a certain period of time. And then you have to change, because somebody changed something, you have to change something in your software. So there are all these problems that come up in practice. And one answer to this, again, is that people do write libraries of software for the most used computational methods, such as numerical, linear, algebra. So we could try to take that approach. And although you might say, well, for four squares, what is the software library? Again, there's this system SageMath that's incorporated a great deal of standard or implemented computational mathematics. And if you type SageMath four squares into Google, you've told Google that, well, let's look for queries that involve that as well. And indeed, there is an implementation of write the integer n as a sum of four squares in SageMath, which is on a 50 page long web page. But you can find it by searching within that web page. Okay, so again, a problem that you can be helped quite a bit by the prior literature in finding it through the existing search tools. Now, we still haven't solved the fundamental problem. Again, a 200-year-old theorem is almost certainly correct, but suppose it were some five-year-old theorem and algorithm, are we sure that it's well implemented? And then there are many questions that this could lead to. So this is trying to get us into the kind of motive thought of if we're trying to get the computer to help us do mathematics, what are the questions that actually come up? Okay, so let me now switch a bit to review computational mathematics as an ongoing concern, which is, of course, not a new subject. Mathematicians of the top rank, of course, were renowned for their calculation abilities, and they used these to make important mathematical discoveries. And now, of course, we can point at important discoveries that the computer played an important role in providing evidence. So the whole theory of dynamical systems and chaos has been much influenced by the ability to just simulate ODE's on the computer. The finite group, the classification of the finite simple groups used a great deal of computer calculation. Another interesting one, the Bertz-Swinner-Dyer conjecture, you're very famous, part of the millennial clay problem list was actually based on calculations done by Bertz-Swinner-Dyer on a very early computer in the 50s. And that tradition continues. There's a whole collaboration founded by the Simon Foundation on that. So that's sort of computational mathematics of working through interesting examples, classification databases, very much continues. So that's a point where we can safely predict progress by the year 2030. And meaningful progress, which is the most basic example, would be by taking some sort of naive version of Moore's law. So it's not the original Moore's law because, as you've probably heard, chips have pretty much reached their density limits. But in the sense of computers getting cheaper, more available, and so forth, that progress continues. And so if we roughly say that every year the available power doubles, a problem that we could have solved with a thousand times more computer power than we could afford now probably will be solved by 2030. So that's not an exciting prediction, but at least a conceptually exciting one, but it's only an important one for the practical and from our point of view as mathematical scientists. Okay, so now here's another example where there's definite progress that helps people solve real problems. So a SAT solver is the simplest, if you will, example of a computer theorem prover, because now we're just giving it a list of propositions in Boolean logic. The variables can either be true or false and we can combine them with and or not. And so it's such a constrained problem and has many applications and industry and so forth. So people have developed very optimized solvers. They can deal with billions of clauses. And an example of a math problem that was solved this way a few years ago is what's called the Boolean Pythagorean-Triple problem. And so we take the set of integers from one to n and we have to split it into two subsets such that neither subset contains a Pythagorean-Triple, a squared plus b squared equals c squared. So we've colored this chart of integers with two colors and then this group has found a assignment for 7A24 and proven using a SAT solver to introduce no such assignment for 7A25. And so the basic idea is just that each of these things leads to a constraint. So you would have the constraint that, you know, a squared, b squared, c squared cannot be in the same set. And for every a, b, c that leads to a propositional clause, the variables would be, you know, the number one is in set A or set b. The number two is in set A or set b and so forth. And so you get this long list of clauses. And then the proof, the certificate that this set of clauses has no solution is 200 terabytes long, but it was found by the computer and verified and trusted. So that problem was solved. So it's an intricate sort of combinatorial problem. And then an improvement, something which is putting more problems into range. This is a problem where it's just very easy to list all of these constraints and turn them into clauses. But there are many comparable problems where you have a list of constraints like this, but it's actually costly to work them all out. You don't want to work them all out in advance. And you only want to work out the subset that you need in order to address the problem. If you could show that you don't need one because of other ones, you don't want to work it out. And so there's a paradigm SAT plus cast solver and other paradigms, SMT, that allow doing this sort of thing. And various people, in particular a group at Waterloo, the math check project has been systematically exploiting this to solve problems such as the Williamson conjecture. So this is actually a problem involving four squares, but here the A, B, C, D are symmetric N by N matrices. All of these entries are plus or minus ones. And so the question is, can you solve this sort of vaguely quaternionic condition, writing the identity matrix as a sum of four squares of matrices? This actually comes up naturally in experiment design and statistics. And strangely enough, this original conjecture was that this always exists, but they showed that it does not exist for N equals 35. But it does exist for every even N up to 70. So there's no standing conjecture as to what's the solution to that problem. So another problem which is famous in computer sciences, what's the minimum number of scalar multiplications you can use to multiply N by N matrices? So you've perhaps seen in a computer science course that you can multiply two by two matrices. Naively, you would think you would need eight multiplications, but you can actually do it with seven. And that's been proven to be the minimum for two by two matrices. And the minimum is not known for three by three already, but many ways, this group found many ways of doing it using 23 multiplications and no ways that use 22 multiplications. So there's a conjecture waiting for proof. Okay, so neural networks is something which I think must be mentioned in a talk like this, which is really stuff on and others are giving some applications of mathematical topics for this sort of thing. And they have this, you know, there are many interesting problems where modeling optimal functions and finding optimal models is an essential part of the problem. And so that will continue to have a huge impact. Okay, advances in statistics is another point where, you know, again, it's hardly a new thing. But if you actually look, and this is, you know, I think, you know, to some extent clear in physics and some other you know, applications that there's this trend towards more sophistication and using more general statistical methods. Okay, so there's, you know, again, it's a field with a long history of its own, but that successfully comes up with more general concepts to derive statistical models as opposed to just postulating a simple way of analyzing the data, which in the large amount of data limit converges. You can step back, you might have Bayesian arguments to derive models. You have more interesting ways of using information theory. This is an example of the Wasserstein distance that's had, of course, many applications in mathematical physics applied math. And I mentioned this one. So as an example of solving a problem using a more general method, so in particle physics we have CERN, we have the LHC and the ATLAS and CMS detectors. So the typical collision produces thousands of particles coming out and you have to somehow identify what do they come from at high energies? Was there some new particle that produced one of these jets combinations of particles or is it all known physics? And there are handcrafted statistical analyses that were developed over the years for this. But an example of doing this in a more general way, Jesse Taylor, his group at MIT, you can just regard the distribution of energy on a cylinder surrounding this as a function on the cylinder and then define a distance between two events as the, what's called earthenware of distance. You move each piece of energy from one point in the one event to another point in the second event and you minimize over all possible ways of moving the energy distribution from the first to the second and that defines a distance. And then once you have a distance between events, you can start to again apply more general techniques of clustering and the rest. So this turns out to work as well as many of these handcrafted methods. Okay, and even in pure math, there's I think more of an interest in probabilistic and statistical methods and an example I can cite is the model of a Bergava et al of the arithmetic of elliptic curves. So I'll just assume people know the basics of that elliptic curves. In fact, you may well know this model better than I do that we're studying the ranks of elliptic curves over the rationals. And so is there, for example, an arbitrary high rank? And the evidence from this model, which was developed using large databases and then of course turns into a heuristic model for the distribution of various groups in their ranks associated to curves, says that the infinite series of curves have rank at bus 21. So there will must be a curve of highest rank. So again, not in itself a intrinsically computational thing, but not a new idea to make probabilistic models of in number theory, but an example of I think the growing influence of this. Okay, let's so I'll go through this one a bit quickly. So I think this is a very central question for I'm obviously teaching mathematics is written in mathematical science is a very central part of it. And we have not really gone very far I think in exploiting the the new technology, even as it is in writing textbooks and teaching, you know, it's very much taking the existing model of a textbook, which has many reasons for being the way it is, but may not be the best thing. And so just to give a list of possible, you know, ideas, you know, technologies that that actually, you know, do exist, but really don't haven't all gotten integrated into writing textbooks. Okay, the use of hyperlinks and pop up links throughout a book. Automated grading of exercises is a harder thing than these others, but somebody that, you know, everybody who teaches will be very happy to have, you know, if it works, really trying to take advantage of social learning these these sites like static exchange and core and the rest show what's possible there and students do communicate with each other and try to share information it gives. And one has to, I think, work with that. You know, the notebook form has value, you know, flow charts have value. Again, things which there are too much work as it stands for your a serious, you know, educator or mathematician to put a lot of time into using them but could be made available in a platform that would be a big benefit to all of us who write such books. And then another point more in the main line of the talk is that if you think about how to teach the computational side of these subjects, then rarely is that integrated. You know, as the topic moves towards computer science, it becomes more integrated, but something like computational physics is generally taught by assuming the students know the physics and now you're going to teach them how to write the programs and has various issues with it. An example of a book that tries to do a more integrated presentation is by this textbook on classical mechanics, structure and interpretation of classical mechanics by Jerry Sussman and Jack Wisdom. And so if you want an example of how you can use, you know, so the book, you know, gives you, you know, again as part of the associated web and other material, the, you know, ability not just to execute programs. Again, in classical mechanics, we're going to be doing simulations of Newton's equations, DoDs, but even the derivation starting from understanding the variational formulation, the relation between the Lagrange and Hamiltonian formulation. All this involves computational examples in the textbooks. It was an interesting example of what's possible. Okay, so now I'm going to move to the interactive theorem verification, which again grows out of some of the very earliest work in AI that proved simple mathematical statements. And you know, what the computer could find by itself was very simple, but showed at least that this was possible. And then over the subsequent decades, not only was this improved, but there's a field of computer science called formal methods where you take a computer program, which although it looks like a logical thing, involves typically many concepts which are not parts of standard logic, such as time sequence of the various statements that you execute. So of course, you know, traditional logic doesn't have time, but you can make a logical system that incorporates time and then give a precise semantics to a computer program and show that given some specification that says what I want my program to do, yes, I could prove this program really does the thing I want it to do. And then this is something of potential great value for computer programmers or anybody that uses the programs. So if we're programming an airplane autopilot, for example, it's worth very large amounts of money and people's time to try to show that the autopilot doesn't have any mistakes in it. There's a famous example of a recall of Intel floating point ships back in the late 90s because their floating point unit had a subtle bug and they had to spend a billion dollars. They just decided to recall all those CPUs and give out new ones because of that mistake. And ever since they logically verified the floating point at least design on the HF. And so this is a subject. This is one of the many core areas of computer science that's fairly well funded. And in fact, the systems I'm about to describe, this is the primary application. Okay, so I'm not sure how well you can read this, but I'll say the important point. So this is an example just to, again, people generally, he talks about the subject, tend not to get into the details and I want to show you what it is. And so this is an example of the statement in the Koch language of what does it mean for a program to sort a list. So we've got a list of things and we want to put them in order. And the basic point I want to make from this is that if you read it, it's actually fairly clear what the statement of a sorting algorithm is. It's something that takes a list into a list such that the second list is a permutation of the first list and the second list is sorted. And sorted, well, the elements with lower index i than j come earlier in the order. So that's clear. And then what's a permutation? Well, there's a nice recursive definition of a permutation where if I have two lists, I can append an element x to the front of each list and that's a permutation. I can append two elements in their transposition in front of the list and that's a permutation. And I can, you know, I have transitivity of the condition that it's a permutation, its equivalence relation, and then you can prove this characterizes permutations. And then given this, you can actually prove one of these moderately complicated sorting programs that you would learn about in a computer science algorithm class really does sort lists. Okay, so now what could one do with this for mathematics? Okay, so as an example, let's look at a bit at the proof of the fundamental theorem of algebra. So that's, again, very easy to state. Now, this is the lean theorem proving language. Okay, so exists root, f is a polynomial over the complexes of positive degree, then there exists some z, so if I evaluate f at z, I get zero. Okay, so that's very straightforward and one could imagine writing that without a lot of difficulty. Okay, so now just to refresh, I'm not going to go through the details, but this would be the core of a standard, you know, informal human proof of the fundamental theorem, starting with the statement that the norm of f attains its minimum at some point is e0, and then working around that z0, and then using basically properties of holomorphy showing that, you know, if it's not zero, it's actually not a minimum. Okay, so this is now the lean version of or verification of your verified proof, verifiable proof of the first statement that the norm, the absolute value of f has to attain its minimum at some finite point z0. Okay, so using basically that to, you know, from closed boundary regions. So I'm not, again, going to try to go through this, I'm going to basically comment on it. So the statement that we're trying to prove is that p is a polynomial. There exists some x so that for all y, absolute value of p evaluated x plus or equal to absolute value of p evaluated at y. So, you know, syntax is a little funny and non-mathematical, but at least the concept is very clearly stated. And now the proof is really less clear, you know. If you find it kind of hard to follow this, I agree. It's kind of hard to follow. And there are various reasons for that. And it is true. This can be verified. So this is a, you know, clearly correct proof of that claim. But one, you know, various reasons why it is hard to read. Well, we have to actually define various propositions, things which you would have just said, spelled out and said, you know. So given a function, then, you know, there exists, you know, some, well, you know, again, for all, you know, y's that are, in fact, you have to feed various, you know, helper functions into this thing to construct the statement of interest. So the ordering of the statements is nice for a, you know, logical construction, but not so intuitive for a human. We've used various previously defined named concepts, and we've had to call them with these various particular orderings. And finally, we've had to tell the computer things like, here is an explicit simplification using distributivity, using, you know, here is a actual rewrite of inequality, which I defined elsewhere. So it is a doable thing. But it is a technology which is not that easy to use or learn at present. Okay, so it's much more like programming a computer to make these verified proofs than it is like doing regular mathematics. Okay, so that's why you don't see more of it in the practice at present. It's something that you could learn. I'm trying to learn, but that would probably take more than a year to, you know, not constant, but serious work to learn. Okay, so why do it if it's tedious and difficult, except unless you have an autopilot for an airplane or something to design? Well, I think this is a central part of getting computers to help us with mathematics for reasons that I will, again, you know, come back to, hinted at some, and I'll come back to some of them. So, you know, the questions that I raised before about search and knowing is this thing that I found on the web, you know, really a correct implementation of what I'm trying to do, you know, just having the language of, you know, what do my symbols actually mean? Well, this is a way that the computer can understand the meaning of mathematics and whether something's true. So, you know, again, I feel this is one of the central parts of the story, but it is not an easy part to use. And so what do we do about that? Okay, so as I said, the definition part of, you know, what is the theorem that we're proving is actually fairly straightforward to read. So we could focus on just the definitions and less on the question of what are the proofs. It could be that strict deductive logic is just a difficult formalism. Of course, people use many forms of reasoning, and perhaps their computer should use more forms of reasoning than strict deductive logic. And then, of course, it could be that AI will just come in and make the computer much more able to come up with its own proofs. Okay, so as far as the first direction, I think the best illustration of this is a project Tom Hales at Pittsburgh has started. So I'll come back to talking about his work on the proof of the Kevlar conjecture, but motivated or inspired by that, he has a project that he calls formal abstracts, which will be a repository, kind of like the archive, where you can upload a paper or a result. But it would be uploaded a statement of the result in formal terms using, to say the lean language that I just described, that the computer could, in principle, you could have a verified proof. So he's not expecting people to upload proofs of everything, but the idea is that there would be a library of statements, many of them prove it, in the various foundational areas, and perhaps reaching up to the early graduate level courses that would then allow you to formulate research-level mathematics, the statements in a formal language. And so a rough estimate, I discussed this with Tom some months ago, and other actually other people come up with similar estimates is that you would need something like 50,000 definitions filling 10,000 pages to support this type of being able to write out theorem, the statements of theorems in most of mathematics. So another aspect that he's working on now, to try to make it easier to work with. So this is what Tom is called a controlled natural language. So this looks much more like conventional mathematics, so Silo's theorems. In this section, let G denote a fixed finite group. In this section, let G, capital X, G inverse, stand for the set of G, little X, G inverse, and so forth. The conjugate class, conjugate, normalizer, eventually we get to the Silo theorems. And so this is actually something that the computer can parse. So this is design. This is not general English. The order of the words does satisfy certain rules, but the rules chosen such that the computer can actually parse this into the lean that I described earlier. And so this looks kind of nice. And this would certainly solve the problem of being able to read the things that people have put into the computer this way. And Tom's help is that it won't be that hard to write that you would not have to spend a year to learn, but a month reading a manual and you would be able to write mathematics that looked like that and then the computer could parse it. Again, the things you need to do to make a verifiable proof are more intricate and you would have to give a lot of extra notations, but at least the definitions you can work on this level. Okay, so I would estimate that this is something with a fair amount of work, perhaps 100 people typing in these definitions over a year or two, but such a system could be developed in less than five years as an example of something that certainly could be done by 2030. Okay, so now to move to this second idea, which is a little vaguer that there are other reasoning methods besides deductive logic and even more in a way clever or efficient ways of implementing deductive logic. So there's this idea of what's called a sledgehammer and in particular the system Isobel implements this, where you translate some of the things the computer has to prove into one of these more efficient propositional or first order logic systems and then it just looks for the proof using this more efficient system. So in fact, Isobel is better if I put up that proof in Isobel, it would not be quite as opaque and difficult to write as it was in the lean language because of this automation. And then to the extent that you combine this with the previous thing that maybe the goal is not to rigorously prove everything but to prove enough to have some level of confidence and to have something that you can search for and work with in other ways, then you can start to use other forms of reasoning, such as looking for counter examples and the many other idealizations of the intuitive reasoning that people do. Okay, so then of course, the thing that got me and many people into this is believing that AI could work not just on recognizing images of cats and so forth, but these complex symbolic problems was the success of AlphaGo and AlphaZero. And I'll come back to that momentarily. Let's come back a bit to the mathematical search engine question now that we see this type of language that we would want to really enable, you know, the full power of one of these logical formalisms. And then it is true that searching, giving a logical statement is a problem that computers scientists study and know to be difficult. It's a general category of a semantic search. And so we need to ask, you know, is mathematics actually simpler or more promising for this than other domains? And I would think it is, but it's not by no means clear. But we can borrow from existing work on semantic search. And an example of a couple examples of concepts of that that, again, you know, to show that this is related to things that there's a good deal of work going on. So there's this idea, for example, of a word embedding that the modern approaches to language translation, generating language and so forth, all start with this idea of a word embedding where each word is mapped into a point of some high dimensional space. And you learn this basically from a statistical, you know, you take documents and you look at what words appear along near certain words. And that very large vector would be the initial form of a word embedding. And then you could do a compression of that to get to a relatively low dimensional space. So a way that you could then work with logical queries given that is express the logical connectives in terms of functions on this word embedding space. So if cat and dog individually get embedded into this vector space v, we might look for a function v cross v into v that would be the embedding of cat and dog so that finally we get some distance function on the space to reflect how similar our queries to each other. And then this you can learn if you have, again, a set of queries and answers, then you have an objective function, you know, how, you know, once it produces queries, you know, how similar, you know, how often do I get the right answer or how often do I make mistakes? And so you can learn functions like this through gradient descent. And that's, I would say, the main approach people are trying right now to get query answering that does logical expressions in natural language. And then there are various issues that would arise in mathematics. I make a pretty close analogy to the Gromov-Hausdorff distance and describing how you would decide what's the distance between two sets of statements. So I just described the distance between two simple logical statements. But in general, we would want both, you know, basically we want them to be aligned. We want the symbols in both statements to, if they refer to the same thing, we want them to map to the same thing in our representation, you know, even if, you know, it was P in the first query and Q in the second query as in our prime number example. So that's like the initial embedding of manifolds into a common manifold of Gromov-Hausdorff distance. And then, of course, you don't, ideally you wouldn't necessarily be the same logical statements. If you could make a deduction. So from the first set, I can make a logical deduction si and add that to the set and then get the same set as the second set. Then it would be in some sense distance, you know, one unit. Or if I can make two deductions to get them to be the same, there would be distance two units. And of course, that's the analog of the, you know, the Hausdorff distance part of the Gromov distance. And then, you know, again, a very typical strategy in this business is you've come up with some sort of precise mathematical logical definition. And then you go back to, you know, some approximate version of it using neural networks on finite dimensional spaces and see if it can learn that. Okay. How much time should I, I mean, okay. So this is a part of it that many people here have probably seen in other talks that, you know, the main line of this type of work would be to be able to both verify and work with large proofs such as, you know, the classic, you know, the example that always comes up in conversation is the classification of the finite simple groups. This is an example where in physics there was a mistake and an important calculation that entered this computation of the muon g minus two, something you measure in experiment. It turned out to be a simple mistake in interpreting the result of a computer algebra program that they gave it the wrong sign because of the sign conventions. This took five years to find. So, so physicists could clearly use this too. Okay. So finally, I'll say a bit about this question of applying AI to theorem proofing. And there are several groups working on this at Google, at Princeton, at OpenAI, and other places. And so in brief terms, the idea that all these people are working on is by analogy. If we take the analogy between making a proof using the type of language I just described. So in Lean, what are the aspects, you know, we're given a statement and now we're going to have to select out various premises from all the, perhaps large number of known true statements as the relevant ones for making the proof. And then we'll have to follow this whatever sequence of tactics of, you know, let's use algebra at this point, let's use a particular logical deduction rule at this point and so forth. And so we might liken it to a game of solitary that we're given this goal, and we have this selection of moves that we can apply to complete our proof. And once you describe the problem as a game, then of course you have the inspiration of systems like Africa that use reinforcement learning to learn how to apply and win a game. And so the basic idea that's been followed is to take the existing work of humans having proven various theorems or pieces of these proofs of the four-color theorem of the phytoncine theorem and so forth and of these, you know, software engineering applications. And you can get a database of, on the order of 50,000, 70,000 lemmas, theorems that are, perhaps, where the proof is, they're short and the proof is perhaps ten lines long. And that would be the training data. So the system would, you know, you would give it these proofs and pieces of, you know, these statements and the pieces of the proofs and it would have to either elaborate the proof or fill in missing steps. And then having learned from this, you know, most of the data you give it, then there would be a testing set that you would give it the theorems without the proofs and see how many of those proofs can it come up with. And people in the two or so years they've been doing this have gotten up to a rate of about 50 percent. So well, well, you know, much better than a kind of a random approach and better than, still comparable to the most advanced, like the sledgehammer I was talking about, but still impressive success of AI coming up with the non-trivial parts of these theorems. So another question you can ask is can you read human mathematics such as the whole archive and turn that into something that the computer could then reason with, okay, that's a challenging thing. The main difference between theorem proving as a game and of course go is it's not totally clear how you win in theorem proving. I mean, if you have a particular goal, of course, that's a win to achieve that goal, but it's very limited. How do you go past that to approving new things? So every step of a deduction in some sense is a step forward, but only a subset of those are interesting and you need some concept of interesting, which the computer may be able to learn. So that's the central part of this problem. That's what I'll do. Okay, so these are a few of the major lines and I'm just going to summarize now. So I believe there will be by 2030 textbooks that do integrate computation in these various points I put on the list. Computational mathematics will continue to advance and these theorem proving systems that I described will be as easy to use as say, mathematics and the symbolic algebra systems are now. So that will become a not universal but much used tool compared to now. This type of formal abstracts will exist and be enabling searches of the sort that I described and then will computers really be able to come up with mathematics on their own? Well, the short answer to that, this is my speculation, is that we're still in this state where computers really can't answer basic logical questions except in the most highly structured form and that's the topic of a good deal of research now to answer questions, to engage in dialogue and that the analogy with problems like AlphaGo suggests that if that basic step were taken, well, the great advantage of mathematics over other areas where you might use logic, of course, is that you can make very arbitrarily long chains of reasoning and mathematics that connect, that can actually lead to a sensible conclusion and that leveraging such an ability to do individual steps of reasoning in about 10 years you could hope to have computers doing serious theorem, you know, proving on their own. So that I'll conclude with that statement. Yes, I just have one comment, which is certainly I agree that using the computer is an extremely good point. I mean, my general impression is that if I had a young student, I would advise him to totally ignore all this and say by himself about the problem before trying to look in Wikipedia and so on. What I mean is that in what you have described, you have described the outside of the brain and to become a mathematician, one has to shape up one's own brain to be able to resolve, to invent things, to create, to understand analogies, to formulate concepts and so on. And so, I mean, in a way, this is exactly the complementary. Yes, exactly. That's right. I totally agree with that. So that's right, there's a danger of just relying too much on not just a computer, but society as a whole, and to make yourself a little piece of this big society, but we only function in proportion to our own capabilities. And so that's right. Yes. So I think, you know, there is a really important issue of also trying to find out how young people will be formed, they will be trained in 2030. Right. Complimentary to what you have described. Right. Now, I think again, there's some role for improving textbooks that I talk to people that some people learn very well in the present system, in the present textbooks, and other people, they could use more help. But definitely that's, I think, an equal focus, education. Yes. I had a question about the tactics for Hall and stuff like that. Most of the things that have been done right now to apply tactics were concentrated on the satisfiability. Basically, if you have a certain tactic for fear of improving, most of the time it's not satisfiable. So you lose a lot of time just trying to put something that doesn't work. No one seems to be working on the usability of the tactic. Basically, it's not just candidate tactic work, but will it prove something useful? Do you have any idea because Well, that comes back to this point. I had to skip ahead, you know, but what's interesting. That's right. So the computer makes no attempt to decide what's interesting at the moment. And it's a big, I would say, a very important question, to what extent one can make any precise definition of what's interesting. And I think you can. I think it's maybe a topic for private discussion. There are even interesting ideas you could pursue now on how do you formally define an interesting concept. And I think you would need that go much farther. What I was thinking is that so long you cannot say what is an interesting example, there's no point trying to train a network or any machine learning thing or something. You need a database of interesting examples or if you teach a machine with stupid examples. Yeah. Well, again, the current training is human generated examples, which have some interest by virtue of that, but it's very limited. That's right. So AlphaGo could play itself and generate millions of games. And to do anything analogous here indeed, the computer has to generate interesting examples. Yes, a very interesting topic to discuss. Which was this related, but is human working on, like, a zero generation? In the sense of just high conjectures. Billions of serums? Yeah. That was equally. In the early work, that was kind of equal. And there were famous projects like a Duglan ad, famous guy in AI from the 70s. That was how he became famous. I think that's, people try to do it, but it's definitely much less developed than the things I said. Again, I think the central thing is the computer has to have some idea of what's interesting to make that possible. Are there things we know we have tested and we believe with some certainty that computer are bad for in mathematics? Well, it's probably too early to judge that. Again, for something to be bad at this moment in time, they just say that we didn't do it the right way. I mean, they're bad at most things, obviously, at the present time. I'm not sure how that... In the sense of computational complexity theory, obviously, there are problems that you can prove, or at least under conjectures, like P is not NP. You can reduce this problem as hard to that. That point may come. I think we're still in the exploratory phase. You tend to be more positive and say, if I want to get this to work, I want to get this to work. And then you consolidate and say, no, this is why that couldn't work. We're not there yet. That was what I was... Okay. Thank you.