 My name's Tom Black, I'm a developer, designer, product strategist, look at everything to determine who's happening. You can find me online, most places at BlackTM, get up Twitter, and I use Ruby for both work and fun, but I'm going to talk about creator. So before we get started also, I'm going to put notes up for this, and be code samples in the links, and so just so you don't have to keep track of all this stuff to be able to place afterwards. Today I'm going to talk about some of my experiences and thoughts about how we can re-imagine, rethink, 2D graphics and Game of Thrones. And so for me, this journey started here in New York City, particularly in the Flatiron District, I was spending a lot of time. I was new to the city, sort of taking it all in, meeting people, and this whole, around 2012, this Learn to Code movement was like at its peak, sort of to the mayor was about to learn to code. But it was exciting for me, despite all kind of the hype, and it was tough for me to learn, and I was really excited to share that experience with others, help them out, be a mentor, all the things that I kind of really had to get back in the way. And so we've got Meetups, and after school programs, Bootups, Pants, like this one, the Flatiron School, this is their first Pants, I think in this doc, so long ago. And we keep changing, I sort of noticed something. This is just sort of the pattern that everybody's doing, it's the paradigm, it's the terminal of people, it's pretty familiar to us, that's how we make software, but when you're learning, it's sort of kind of living, it's all computers can do, it's kind of this old paradigm, reminded me of like, computers of yesterday, or the ones that you've kind of grew up with. And it kind of seemed like not a lot has sort of changed, but in many ways things have changed a lot. We've got much better visuals, we can see millions of colors, pixels we can't even discern anymore, rich audio, speakers built in, we've got still the keyboard of yesterday which we'll probably have forever, but also speakers, real sound. And we can have all sorts of different input devices, connect things, so there's just a lot you can do. It's nice, but this is a maps route to real learning styles. This is how people actually, their visual auditory is that things people can see, hear, touch, that's how real learning takes place. So I was really interested in figuring out how do we take that experience of all of those sort of senses and bring that into something that teaches in Ruby and makes it more accessible. So in 2012 I made this demo, it's kind of a vision, I imagined you could go into the terminal kind of still like you'd usually do in Titans itself, but see results kind of happening on the screen. And maybe build up some skills and use more kind of constructs there and see things happen, maybe make some mini games or something and then plug in other device controllers and kind of jump around. I think this is a Ruby impaired character or something. So just a much more interactive learning environment. And so that was sort of the vision. It was built with real code and real Ruby libraries actually to just sort of help validate this idea. But like every good prototype, it's really a facade, a trick, it's there to validate an idea, but it's really nothing more. So I kind of tested it out, and did see how people would learn and just kind of get to feed that. And that was enough to kind of dig in and see if I can build something a little bit more. So I did some research to see just sort of what was out there. Was anybody else thinking about this stuff or people developing graphics and stuff broadly in the Ruby community? And it turns out they are. I actually started going back to like 2000 and so a pretty rich history. And there's a lot of ideas in here. There's things that are kind of used for this graphics, and there's two things. A lot of different approaches. There's a kind of a neat mix of this. So I kind of thought, well what would be my must have if I were to like choose one of these or mix it up or take ideas? What would that sort of look like? Well, we probably want to have a cross platform native in the web, make this stuff easy to understand internally. That's kind of nice to learn. You could do something and see how it might work and make real things. So it's not just like a lot of sandboxes that we would see. It could be good for a lot of different things, not just education, but like you could make a real game or you could kind of like, you know, throw your things there. And it would just be easy to get started. The quicker you get to having the idea to seeing it interacting with it is pretty critical. And just making it a good idea. So I thought, you know, this is kind of a hefty list, you know, starting something new, thinking about how we could take some existing ideas and match them together. There's a lot of positives to that. You get to build on many ideas, incorporate new tech and methods and so forth. Revisit these fundamentals. You can kind of say like, do these things need to exist or does this have an updated approach? You've got the freedom to experiment. For me, it was nice because I wanted to gain new skills as a sort of weird kind of, you know, I think a lot of stupid web stuff is probably how we got into Revisit. I wanted to get dealt into this new area. The downside is there's a lot to learn to do that's a big thing. But I persevered here and thought, well, where do I start? And, you know, I thought about the simple like starting with the end in mind what experience would you want in just this like, minimal case. And so I thought of this like, MVP minimal Bible product, a new sort of application. And then what would it take to really build that? I know there's like hardware at the bottom somewhere but like what goes sort of in the middle I imagine a lot of stuff. The first thing I did is just kind of broke it down and said, well, what if we just start with a window that's pretty much the most fundamental thing that you have. And if you go around and Google it, you'll find most operating systems provide some low level library that you can access to make those things. And so this is on the Mac you have all these NS classes. This is NS window and if you read it it just says a window that an app displays on the screen. So I thought, okay, that's pretty good. So if you go and do some study in trial and error you can actually build a window and this is really all you need to do that. This is written in Objective-C and this is not in this COPA class you just kind of say what you want and run it or even pilot first linking to the COPA libraries there and we'll just output it as a window and if you want to run it, this is what you get basically a window that does not but it's a good start and it works. From there there's a lot more to do so it's not just a window but you've got to put a graphics context in. You've got to think about how are you going to get inputs in versus manage it. Maybe you're going to talk to other system things display and system timers. You start to think about all the things you need to develop this experience thinking ahead and a lot goes into that. And I'm on a Mac but I want to include all operating systems you've got to do that everywhere. So that's what basically all the stuff you're going to have to go in. And pretty much it starts to become pretty overwhelming and you have to do this and you think about all the variations and all the rest it just gets to be where you might be stuck and you start feeling like this. There's no end in sight and this could be a dead end but what is it? Well it turns out that thankfully not, game developers have been thinking about these issues for a long time. A lot of these efforts started back in about 20 years ago when they were recording games to Linux. They were trying to think of the inconsistency between Windows and Linux and other operating systems making these things work everywhere was a pretty big effort in what their games were running for. So these are called media libraries and media layers because that's exactly what it does. It puts a smooth layer on top of all these different operating systems so you can root out all these inconsistencies. You just have a nice API. When I looked at those of the three major ones I found SDL to be a simple media layer to be the most well rounded for this project there's a lot out there and they're good for different things but for what we were trying to do this made sense and it supports a lot and there's over 500 cross platform functions so you can really and it just so happens that it's used by triple A titles and indie games and lights so my general rule is that it's good enough for Half-Life, it's probably good enough for everyone to work with. So this is what it looks like, this is written in in C and this is all it takes basically to build a window on every platform and now we don't have to think about any of these inconsistencies because it's sort of like jammed into that lower level and I have a talk link about game developer with the SDL so I think we're back on track and we have a path forward but despite we might be wondering like why we're talking about C so much this could be like a Ruby conference and isn't it maybe old and crusty and all the rest of it but it turns out that C is very contemporary the standard is always evolving people are building really amazing tools and infrastructure and it's it's very beautifully simple, efficient and precise language and while you might not want to build your next web application in C that you'd probably have a bad time it's perfect for programming systems and exchange and all these lower level things it's really good for and it just so happens that Ruby loves C it's primary implementation MRI but I guess we'll probably see Ruby now it is written in C Ruby and so Ruby has this nice connection with the native world it means that just because it's an interpreted language it's not sort of just labeled so then if we go back to our red square application here square.new where are we right now, what do we have kind of reviewing what we just talked about we have an operating system, we know that it provides essential access to hardware and windows and all the rest we know that we don't have to think about all operating systems like that, we can use SDL to kind of make that a cool experience and we can sort of move the hardware out of the way because we don't really have to think about it too explicitly so that's really nice but what else, what else would we might need to make this possible because we're not quite there yet one of the things we haven't talked about is this graphics how to actually make things visual well it turns out again we don't have to talk directly to hardware there's a lot of these cross-platform graphics APIs OpenGL is a common one thanks for the open graphics layer and it's for rendering 2D and 3D maybe sort of Direct 3D or Direct 2D or Metal is a little hot thing on Apple platforms these are all just APIs to talk to graphics and kind of take a hand over some of those cool things but the price is nice so from your perspective as a programmer it looks like writing your C program you'll make OpenGL calls usually it's a GL blog or something in orchestration with the operating systems and the graphics libraries on there it'll talk to the GPU and then finally that is all displayed so that's a massive oversimplification and we can dive into it and talk about the rendering pipeline this is how rendering works in modern graphics programming and it's important to talk about this we actually program these things sort of not the first thing you have to do is take actual points into one address this is a triangle this is the simplest 2D shape you can basically draw and all the vertex data is described and just floating the next thing we have to write a vertex function this is just a function and it sort of looks like a C language with a few extra things there that's where we can actually take these points and map them to a coordinate system put it anywhere we want we can even do any kind of manipulation but it's just about the coordinate system then also we're going to need a projection it's like how we see the world and so in the world that we live in we have perspective we will see train tracks fading into the disk for vanishing point but we don't actually do anything to it we want to see things in this orthographic way which means it's a world of no doubt that you can imagine everything you see in the world is being jammed right in front so this is kind of without looking and if we implement that in the graphics world we would need to write a orthographic matrix which is not too complicated it's described here, it's math and that's a means of representing 3D dimensions basically in this 2D space and it's a distance it's a distance from the camera you can see that in the cube there and also we can do things in this vertex arena like move the origin to the upper left corner it's nice to see the expand then after that we have this shape but it's still represented mathematically, we actually have to apply it and figure out what pixels do we shade it and that's called rasterization and a lot of that is sort of done for you which is nice but then we get to write a fragment function to fill in the colors, think about those intermediate values that interpolation that happens that's done by the hardware and the rest of it that we do for you but you can choose those points and then it'll kind of figure out what the rest of it looks like finally we have a nice triangle so in the last step you write this to a memory frame buffer which is what gets displayed right into the back and then it gets to swap buffers and it gets quicker to the floor then it happens like 60 times a second if you're lucky but the graphics program experience is pretty wild you feel all of these things at various times but even when you screw up pretty badly I don't actually know what this was supposed to be but it looks like a wall paper of PNGs or something I don't know what the rest of it is but it looks pretty cool maybe you could even do art somewhere like an exhibit so this is cool this approach works and now we've implemented it in OpenGL with the legacy versions the more modern approaches which I've described in the pipeline and then also for mobile embeddings is the pocket chip so basically we can we go back to our stack you see we've got the operating system SDL, we've learned about OpenGL that kind of sits there a little bit the average just kind of provides these things or they're kind of worth a concert application, code for SDL event loops other things, a lot of other things OpenGL calls, well that's a lot to kind of just be scattered around so we can package those things to make it a little nicely and that's kind of what we ended up doing is creating this little side project which makes that easier the simple layer written in C to do those things that looks nice in all the files if you kind of look at it everything's very explicit images and all of these things you can kind of look at each file and see what's sort of essential there and I'll just point out the OpenGL code that we were talking about that's a little bit there so you can kind of check it out and it's really simple so all you need to draw across platforms is something like this and really three functions so you create a window show it and that'll go in this render to draw a triangle just write what you want it to look like and it's everywhere which is pretty amazing so now we're at this point where we can add Ruby finally after we put our simple 2D layer on top of that we've got MRI or C Ruby and also M Ruby and of course familiar with M Ruby or have used it before not too much okay well you should be familiar with it because it's pretty awesome it's a small, lightweight implementation of Ruby and it's designed for better use cases and for us it's really important because we can compile Ruby down to machine code so there's a lot of ways to do that but if we take our squared out new application which is important to us and we run the MRV compiler we can actually generate this bytecode this might look crazy but it's basically this compiled Ruby for the M Ruby virtual machine and this is an example it's actually contained in a C array which is really good for us because that means we can embed it places and all you need to do to actually run that is open a context and basically run that so just pass it over it's a really simple way so that's great so we have then we'll need to write a native extension because if we're going to talk at the Ruby layer and we might think that we would be too but fortunately we don't actually so we can just do this together and this was somewhere oops but we figured it out twice so there are differences in the MRI native API and the M Ruby and so because of these differences we kind of have to account for them but in C I think this will get easier or maybe more consistent in the future but for now what we did is just map these things so like if we're in the M Ruby world we just kind of like that's what these defined things are this is C stuff but it makes it nice so you can just choose which one you want here so just call out that in the bottom you'll see defining a module that works for both and then when you call this quad dot render because we're going to need a square eventually so if you have a quad type you would run this function that we have to find somewhere else and it calls that simple 2D draw function so it works so going back to our stack here we can finally add our Ruby layer put the native extension in the gem and we can sort of celebrate because now we can actually we have everything we need in the stack to actually do the simple application what we did say earlier that we wanted to cross-platform so we're not quite done but that's okay we can persevere so what would we actually need to do that well at the very bottom left there we can all sort of explicitly for SDL because we have a browser and the browser sort of does a lot of those things javascript APIs you can get images you can open, you can do all these things it's perfectly provided by the browser and then WebGL is a version of OpenGL that is designed just for the browser there and we don't need simple 2D but we kind of need its port in javascript and we can then also cross-off both rubies here because in the Ruby community we also have Opal which is a Ruby for the Web here and we don't need the native extension but we do need an Opal extension and that's what it looks like sort of a simple part of there and if you haven't done any Opal before like I said Ruby in the browser, what does that mean well if we take our simple squared out new application we can compile this to javascript by using a simple Opal command this is the genome so you can then get access to this command tool and you run this and it's basically just saying don't give me the whole library but just focus on this and pilot to javascript and we'll get this thing which looks kind of ugly but it's real javascript and it works and it's actually something readable but we don't need to read it because it's we can pilot, push it in and instead of writing this native extension like I mentioned we have a a Opal extension which basically will include these things and what's really nice about this as opposed to the native world is that it's kind of like a a foreign function interface built in an FFI that's basically a way of talking to another language and so you just put backticks and instead of it sending to the terminal because it's really not a terminal in your own web they just go up to it and you're just writing kind of exclusive javascript so that simple 2DJS thing you can put in your drawquad right in there and you're kind of like reopening that that quad class and just kind of add it and what it will do so this is what it looks like we didn't have time to do this but there is this thing called WebAssembly on the horizon and we could deploy that I think pretty easily by using the native extensions using mRuby because we can compile it so WebAssembly is the thing which will take for now C and C++ and compile it to this new format which you can basically run in your browser so it's like running kind of like a native application in your browser but these are some of the key pieces that we'll have to add so if we look at the complete stack here we've got the operating system the SDL in the browser kind of filling the same thing the two layers that kind of make this even more simpler our Ruby stuff layer on top and then finally this 2D thing with native extension and all the rest of it so that is what the complete package looks like and it might seem a little complex separation, everything does what it's supposed to do and that's kind of what we want the right stuff in the right places and they're all kind of filling together so we haven't really talked about this Ruby 2D thing what's that all about well, this is going to be the Ruby experience for all of this this is a gem we're working on and we're kind of pitching it as interpreted native Web whatever you want to do you can build 2D applications in Ruby and so what's inside that well, it's got classes as you probably expect but here we have the real opportunity to build concepts from these low level interfaces so now we can build up these tangible things so what is one of these classes look like well the central one is the window we saw some window examples earlier this is what it would look like in Ruby this is just kind of abbreviated but you might get or set attributes in the window where you can do things events that you can call on show clothes and here's an example with a square this is actually the full implementation they're all so you can see squares are inheriting from rectangle, that size and color and it's updated coordinates so it's pretty simple and the most fun part for me and probably created is finding opportunities to design this programming experience this domain specific language for a new domain like in graphics it's kind of a little bit of a refer but there are some prior examples of folks that have really put effort in thinking of that look like one that was very inspiring to me was hyper talk is anybody familiar with hyper talk or hyper card or those things, yeah something cool hyper talk is a scripting language for hyper card this came out in 1987 and hyper card I'm going to explain it for you but it was like a stack based thing and you could kind of like early web browser stuff and you can read about it and you can teach me about it but I was looking at references and just kind of seeing like what these experiences were like there's actually hyper part of Ruby talk later today so that's I'll be going back to the warhouse but the kind of idea here was that say things that were very English like on and all of a sudden it would just happen it was a very conscious effort to think about what that intuitive expressive sort of experience would look like so taking some of that inspiration what could this look like for our Ruby world well if we're setting like a cursor to arrow for example maybe we could well we actually don't have cursor yet setting those attributes would kind of look the same and then this is the most of the example we just saw maybe that would look like this where maybe it's the sound so kind of a Ruby kind of event and then key events or other kind of events we could capture those sort of filter them out and then say okay we've got an event and then you could kind of crack it open so pretty nice so here's the simplest to the app you can really write in Ruby 2D and this will just create a 640 by 480 window just require it and show it and that's pretty darn simple I can't think of anything else to remove really and this is what our square dot new app looks like if you want this you'll get that the windows a little bit bigger than the previous examples but 640 with 480 you can set it to be whatever you want here's a test card which you might be familiar with I like test cards because you can quickly see if kind of everything's working you like to write unit tests but if you're going to do games and graphics there's not a whole lot of testing is interesting in games as well just leave it at that because you kind of have to play test things you have to see things work and things will fail in ways that you kind of didn't expect we do go ahead and catch them in kind of automated ways so we wrote a test card for a little Ruby 2D application we kind of see things working no more crazy shards and all this this gem can do a lot it can automatically detect virtually any controller even ones that you kind of hot plug in so they're wired, wireless, whatever and I'd love to take credit for figuring all of that out but really that's from STL they're all motivated to do this they want their controllers working and so there's a giant community of people talking about this controller let's make sure that it works all the mappings are correct the rest of it so we can take advantage of that and then write a really simple interface to do that so you can do something awesome here and there's maps too so even if you have a crazy controller and I always travel now it looks like it needs to which makes me look a little crazy but controllers all look different some have different joystick buttons this is an iBuffalo or a brand name this one's kind of used for iOS devices and so what we do is we map them to a common generic Xbox interface or at least that's what STL does really so that's what we do the other things we do are we can build for those native platforms if you write your groupie app app.rb or something you can say build it as a native app or for the web or for iOS or TVOS so that's what this looks like on iOS here build it and from the same command you can launch it as well so when that goes through you'll see this is not the full test card but it's something that you can modify at least if you stretch it but you can see it at least it works all of this gets put in a single C file where we have our native extension we have your application's bytecode that's been compiled down and then this Ruby2G to the GEMs bytecode in there all in a single file it's in C which means it's still not machine code which is what we want because then you can basically put it in an Xcode project get the right libraries and they're linked to the right things you can see here if you look at the frameworks we actually have another repo where we can build M-Ruby into a framework so there's only two ones that are not and we don't really have a good publishing plan there but you can basically open this open Xcode and publish it after the app store I mean I think the distance there is 2 years so as we've sort of seen here it can really help us rethink what's possible reimagine everything not just at the DSL level which is what it's really good at but even like the entire stack that was surprising to me just scratch the surface here but there's links to there's been a lot of talks at various places around game development there will be a lot of topics and so I've linked to all those and I think really we'll be getting I think we've talked about how we can really the education gaming, graphics, all these kind of tie together and so there's one more demo that I want to show that kind of combined with my development here and to do that we're going to have to put M-Ruby into space this is an N-body simulation physics or astronomy nerds out there so these are celestial bodies that are interacting in space and their gravity is kind of attracting one another and the color is actually mapped to your velocity here and here's another example where the cursor is actually a single point of gravity we're just sort of directing it in front I tried all these different variations and things so I think there's many possibilities for Ruby and I think there are a lot about that I'm excited about this Ruby on the fringe track because I think there are a lot of this is a small but growing supportive community who are messing around with graphics and games and all the rest of it so please join us, hang out, and I'll share some things and ways that you can do that and overall I think that the future is bright for us here thanks so much for having me the notes will be here and I think that's it