 Hello. My name is Emily. First, I just want to say this is a talk about the history of programming, and I'm also going to say some very hand-wavy things about cognition and neuroscience. I am not an expert on any of those topics, and a lot of the languages I'm going to talk about are not actually around anymore to play with, so I'm pulling these details from papers and blog posts. So if you really like this topic, I'm going to tweet a link to all my references. Please go check that out instead of believing all the things I'm going to tell you, because there's people out there who know a lot more about this than I do. By day, I am an engineering manager at Honeycomb. I'm also a design and data visualization nerd by night, and I studied architecture in school, not computer science, and I made lots of extremely weird visual physical things then. So many weird things like, what is this? I made this thing. I don't even know what it is. But I still had to read the same academic handbook as the normal kids, and there was this one line in the handbook that always made me really mad, and that line was, to suppose that anyone who cannot write clearly can think clearly is an illusion, words are the most basic tools of thought, to which my response is skeptical face. So think about it for a minute. Picture in your mind what you think the most basic building blocks of thought are. Like, don't describe them with words, picture them. So I don't think of words at all. I think of shapes and colors and symbols, and if you're anything like me, maybe you thought the same thing. And there's a really funny thing about this statement when you think about it, which is that the words part of someone's brain made this up, like the verbal part made this sentence up. So, you know, asking the verbal part of your brain what the most basic tools of thought are is like asking this dog, like, who is a good boy? Like, this is a wonderful dog, but the dog is not a neutral party. And there's some actual science that bears this out. So two famous researchers, Gazanaga and Sperry, studied patients who had had the connections between their brain hemispheres severed for medical reasons. So these people could still use their left brain, the more verbal side and their right brain, which is the more spatial side, but there wasn't coordination between these two halves of their brain. And they found that the different parts of the brain actually think remarkably different thoughts. So they asked one patient, like, to say his ideal career out loud, and he said, draftsman, very dependable. And then they asked him to point with his left hand, so he would use the right side of his brain and have him spell it out. And then he said he wanted to be an automobile racer, less dependable, right? So, you know, when your hemispheres are connected, there's kind of chatter back and forth between them, and it helps you stay synced up and not say strange things that make your friends and family kind of wonder about you. But these kinds of divergent thoughts are living in all of our brains all the time. And the verbal part of your brain can't do everything by itself. Like, it can write great papers, but it doesn't produce all this cool visual stuff. So, you know, fast forward many years, I have a very dignified career as a programmer. Yes, very dignified. It's great, but everything is in words, like the code is in words, pull requests are in words, documentation is in words. And all of this makes me ask, like, what opportunities we don't see because the verbal part of our brain is telling us, like, words are the best, you know? So I wanted to ask this question, like, what if we programmed in pictures where we think differently about our work? And it turns out that there's a lot of history behind people exploring that what if. So I want to specifically talk today about visual programming languages or VPLs. The most likely example that you might be familiar with is Scratch. Scratch came out of the Media Lab in 2002. It's designed to make it easy for children to learn to program and make creative projects and share their work with other people. This is a picture of what the interface looked like at one point. The piece in the middle is the actual program. So if you zoom in on that, that's Hello World and Scratch. Very neat. And the pieces kind of snap together. Like, you can drag them around, you can change values with the keyboard, press a button to run the thing. It's very intuitive. Unless you think, like, that's not really programming, it does have all the right pieces. So it has conditionals, it has loops, variables, event handlers, and someone on Hacker News even told me it's Turing complete, you know, so there's that. So the program that you're looking at up here is a very simple physics engine. And it kind of looks like a toy, but you can use this to write real programs. So that was 2002. But this idea actually goes way, way back to the 1960s. The first good example we have is Sketchpad from 1963. This ran on something called the RAN tablet and it used this light pen as input. You might think that there would be this slow evolutionary process that led to our modern tools, but you actually see some of these tools coming up with like nine different new innovations at once. So this might be the first graphical user interface ever, I think, and you would kind of draw on the screen with a pen, and you would use this series of switches that were on the left side to kind of describe how you wanted this input to be transformed. So you would draw on the screen, the computer would redraw it for you, which is what you see on the right. And this was kind of a precursor to the modern CAD tools we see today. And one of the most interesting things about this is that this is actually object oriented, like the tool understands your groupings of lines, and it would know that these were independent objects that were grouped together, and it would kind of do the same operations on them separately, which is cool. Grail is another language from this area, it also ran on the RAN tablet. So this is the first example of a flowchart based programming language. You'd actually draw those different sections of the program, and draw lines to connect them up, and it would even use an OCR like technology to turn the drawings and letters you drew into program components into the names of things, which is kind of fascinating when you think about it, like we've been working on OCR for over 50 years now, and it still only works some of the time. Another one I thought was really interesting was Pygmalion, which the creator had this thesis that part of the complexity of programming was holding all of the mental model of your program state in your head, and so he wanted to be able to show all the state on screen, which is what you see in those boxes. So the idea was every piece of state in your program, you could just sort of look at it all at once. It also uses this programming by example paradigm, so you would kind of show the program what to do with your data, show it each step in this remember mode, and then you could use that to output a program that you could take with you and run against arbitrary input later. Another favorite one of mine, this is Cube from 1995. I've been told the very first 3D visual programming language. I have no idea how this works, like galaxy brain, right? I just looked at this picture and I was like, I have to put this on a slide. So this is all very fun, but what's the catch? There are some things that are very hard to do in visual languages. Abstraction is really hard. You probably have to create a new paradigm to do that. Diffing and merging are really hard. If you made this beautiful visual thing and I made my visual thing, how did we determine what's different and merge them together? It's really hard to show that. And it turns out that your face can only absorb so many shapes. We have a much better memory for unique words than we do for unique shapes, so it's really hard to work with lots of different tokens at once. So just to dig into one of those problems in a little more detail, let's talk about abstraction. There are lots of different types of abstraction, but let's think specifically about procedural abstraction. Like, how do you take a piece of code and make some shorthand for it so you can reuse it over and over? There's a program called Agent Sheets that solves this in this neat kind of recursive way. So this is a block-based programming tool, but the interesting thing is that the blocks can be nested. So if you want code reuse, you just make one of those blocks, which is what you see on the left side, and then you just stuff it into other blocks, and then it's almost like a subroutine or something like you just make blocks that contain blocks that contain blocks. It's cool. Different languages have handled this in different ways that are kind of similar, but you know, subtly different. So this is Chimera, which is a graphical editing tool, and its solution is to have this side bar of every operation you've ever done, kind of rendered visually, and then you could clip out sets of operations and save them as macros and then like drag them back in to apply them later. Almost like a film strip. Maybe the best solution out there comes from ProGraph, which is a language from the 80s. This is a data flow-based language. To me, it looks exactly like mousetrap, which both excites me and horrifies me. The cool thing about ProGraph is that it's truly object oriented. It has classes and has inheritance. So this box that you're looking at here is a description of a method. You can assemble it by dragging together those little boxes and drawing how data flows through them, and the methods can actually then be minified into little icons. You can package your method up and take it with you. And then all of your objects are represented as these hexagons. So if you click on the left side of the hexagon, you could see all of the classes and instance variables associated with that object, but if you click on the right side, then you could see all the methods attached to it. So all those little squares there. This is really neat. It's really cool. It worked well and professional programmers actually use this language, but you also notice that by being less literal, all of a sudden new programmers really need to learn all these difficult programming concepts. So the point is like most of these languages either don't have a good solution to this kind of abstraction, or they have to come up with some totally new novel design solution, and then the complexity of that kind of works against all of the good things, the good goals that you had for your VPL to begin with. And then of course, you know, take that process, repeat that for the dozens of different problems that we already talked about. So back to that first question. I wanted to know, like, did we really deliver on this promise of changing the way that people think about programming? The answers are kind of unclear. Like a lot of people anecdotally will say, yes, like if I use CAD or if I use one of these tools, like I have this different brain space that I'm in, I come up with different kinds of ideas, but there's really not a lot of science on this. The one good paper I could find about the cognitive effects of visual programming, looked at a couple of different studies and found very mixed results. There's some evidence that shows that visual programming languages might be easier to learn, but the one thing that shows up over and over again is that programmers have a bias in favor of whichever programming mode they learn first. So programmers who use text-based languages are likely to call them easier to learn and more powerful. Programmers who use visual languages first are likely to say those are easier to learn and more powerful. And to me, the takeaway from that is just that programmers really like to disagree with each other. We already knew that. So there are really successful cases for these languages, though. They could be great for occasional programming tasks. So this is a cool example from Twilio, Twilio Studio, and you could use this to build out your company's public-facing phone system. The person who wants to do this is very likely not a professional programmer, and they probably want to change it very infrequently. So there's actually this great benefit to having the flow of the whole system displayed really clearly and not having to learn a bunch of syntax. IoT devices are another good example. They're often used by hobbyists. You see lots of flow based models in the space. Kids programming is another one. This is Kodoo. Kids make beautiful games with this. And then, finally, there are all these great examples of creative tools. So this is Alice, which is a design to appeal to populations that might be less served by traditional programming tools by encouraging narrative and storytelling. So this is great. We see these cases of where these tools are making programming more accessible, and to me that means all of a sudden I don't care so much about that question. If these tools are making it possible for people to leap the gap from being a non-programmer to being a programmer, I don't care as much if they change the way we think. So if there's one thing from all of this that I want to take back to my day-to-day life, it's noticing. I notice that my tools are primarily text. I notice that it's rare to see pictures and visualizations while I'm programming, and I notice that text biases my thinking in certain ways. And I ask, like, what am I missing by not showing this some other way? The very last example I want to leave you with is Not Data Visit All. This is a project called GitHub Selfies. It's a browser extension that reminds you to add a selfie of yourself whenever you submit a PR. So yeah, I love this. It's like, take something text-based that can feel a little impersonal, like stop a reminder of your humanity right in the middle of it, and see if it changes the way you think about contributing. So take away notice and experiment, and don't be fooled by text trying to tell you that text is always the best answer. Thank you.