 I'm a software engineer from New York. Thank you all for coming to see me speak. This is my first time at RailsConf, so this is pretty exciting, a little nerve-wracking, I will say. Just before this, about a half an hour, I had to run back to my Airbnb to grab my labels because I just completely forgot a pretty vital piece of my demo, so hopefully it can only go up from here. So I'm here to talk about how we can integrate Rails with different types of protocols. So yeah, so let's get started. So like I said, I'm Danielle. This is my Twitter handle and my GitHub handle, so if you have any questions afterwards that maybe don't get answered, you can treat me at Adam's, Danielle, and then there's gonna be some code samples. You can find them on my GitHub, Daniel Adams. So a little bit about me, I work at a company called Blue Apron. You guys have probably seen our, or heard our podcast sponsorships. That's usually what I hear, so. And I work on the warehouse management system team, so what that means is that all of the software that's built within the warehouses for our employees, that's something that my team helps build and so that things like this don't happen. Another important fact about me is that I have a cat. His name is Dennis, and like most cat owners, I spend about 90% of my free time taking pictures of him while he sleeps. So some notable achievements about myself. The day that the internet went down, I posted a tweet that went viral. This was really exciting for me because I'm not even really a big Twitter user. Had 400 engagements and it ended up on Twitter moments. Another notable achievement is that I was part of a team that introduced automation and hardware into Blue Apron's warehouse system last summer. And so that's what I'm here to talk to you guys about, is that how we can integrate Ruby and Rails and get outside the web development scope and actually start integrating with some hardware. So first I'm gonna go back in time to spring of last year. So originally last year, Blue Apron, we decided that we wanted to introduce a lot of automation into our software. And so up until this point, everything that we were doing was web development based. So we had a lot of different web applications that we were interacting with, we were using Rails. And so we started realizing, okay, if we want to help the company scale, we're gonna have to integrate with these printers and the scanners that we're working with in the warehouses. So we had to start thinking about, okay, we had to start asking questions about what this would look like. We didn't want to have a scenario where we're not being able to handle the latency that needs to be handled or the different connections. And so we wanted to, we started looking at, okay, is Rails the right tool for the job? Is Ruby the right tool for the job? Are we gonna have to do this in Python? Are we gonna have to rewrite everything? Do we need multi-threading? There was a lot of questions that we needed to observe because this new hardware that we were introducing, it was really adding a layer of complexity to our systems. So that's, here's a high level, high level, this is what I'm gonna go over and talk about. And so first we're gonna go over the different types of protocols that we needed to be using. And then I'm gonna talk about how you can integrate Ruby with machines. And then I'm gonna talk about how you can integrate the hardware with a Rails system. So first, protocols. So at the time, like I said, we were mostly web development. We weren't doing a lot of software development in the sense of integration with hardware and automation. And so we had a lot of, we all of our software could speak HTTP, but what we needed was a system that could speak TCP, which is what the hardware spoke. So first of all, let's take a step back and look at what is an HTTP request and how does Rails handle that request? So first of all, we have our browser and the browser makes a request. The IP is resolved by DNS. In the middle, we might have a caching layer. It might reach a load balancer. And then in the scope of the Rails application, it hits our application server, like Puma or Unicorn. It goes through the rack middleware. So rack is parsing it for errors filtering out spam or there's an authentication layer. And then it hits Rails. It eventually gets to our Rails router and then eventually to our Rails controllers. So what is TCP, this different protocol that we speak of? So TCP was something that was introduced in 1981 with the intent of being used with the US Department of Defense. And so as we can see here, it's a transmission control protocol and is intended for use as a highly reliable host-to-host protocol between hosts in a packet switch computer system or computer networks and an interconnected systems of such networks. And that's part of the RFC that was introduced. And so for those of you that don't know an RFC, that stands for request for comments. So that's a fairly common way that people are introducing new ideas in computers and software so that they can draft this idea that they have and they can receive feedback from interested parties. So how is TCP different from HTTP? So they're actually part of the same network model. And so what we have is the internet protocol suite. And so the application level, which is HTTP. And then beneath that we have the lower level layer, which is the transport layer. And that is our TCP layer. So going back, like I said before, we had started looking at the different languages and maybe frameworks that we would need to move over if we needed to support this automation. So come summer we had actually decided that Ruby and Rails were the perfect jobs or what we needed to do. There's a lot of reasons for this, but mostly we weren't aware when we were making these design decisions that Ruby could support these lower level connections. So we started working and testing. We started figuring out how we can integrate the Ruby and Rails and integrate with the different types of hardware. So this is an example of what we looked like in the warehouses. We had our laptops running on local hosts just connecting with the different devices and just trying to figure out what works. And so yeah, so now I'm gonna talk about Ruby and its integration with machines and how we discovered that we could integrate with our Rails systems. So first of all, those of you that don't know, Ruby has a socket library that you can use. It's in the standard library, so when you install Ruby, it's already available for you. And so for those of you that don't know, here's another snippet from the RFC that I spoke of before, but a socket is essentially just a set of two values, an IP address and a port. And then what they do is they distinguish a unique endpoint and within any sort of computer. So Ruby's socket library has this class called TCP socket. It's built on the base socket class and so it has all the functionality that we need to open a socket to any sort of device, any sort of server. So usually how a connection, how flow of data looks is we would have a client, our browser. The client makes a request to the Rails server and then we make a post or an update to a database. And so this is a, as far as web development goes, this is a fairly common flow of data. And so what we wanted was we wanted a client that was speaking to a printer. So how would we do that if it's not speaking HTTP and it's only speaking TCP? So we need to open a socket. And so in order to communicate with this hardware, a TCP socket needs to be established. So let's look at what that might look like in Ruby. So like I said, we use the socket library, so we just import it in the file that we're, the script that we're using. Like I said before, our TCP sockets have an IP address and a port. So we have defined this new, we've created this new socket object, which is a representation of the socket and we've defined its IP and its port number. And then we're writing to the socket and then we're just closing it. So on the other side of this, if we needed to have a TCP server that supports socket connections from a TCP socket, we also have the TCP server class that's available in the Ruby library, in the socket library. So like I said before, this is a fairly common flow of data and web development through the client to a server and it makes an update to a database. And so how do we establish this connection between the client and server and TCP? So in Ruby, it might look like this. So we require the socket library again. We've established, we create the instance of the socket or not the socket, the server. So we don't need to pass in the IP because wherever the server is being started, it's on whatever machine that it's on. So in this case, it would be local host or whatever IP address that you're running on. So we choose a port. When you're choosing a port, there's a range of ports that you can use. It's like 1,000 to 50,000 or something like that. So you just pick a port to start your server on. Okay, and so then the server, once it's running, it's listening for any client connections. And so this one isn't doing a very good job of guarding against any sort of bad socket. So it's just receiving any socket, it's accepting it. It's reading, it's acknowledging that it's received a client and then it's reading the client, it's outputting it and then it closes the socket. So these are a few examples of the different types of machines that you can use with this library. So Datalogic is the scanners, the company that makes scanners. Zebra is what the Zebra printers are. And so they receive ZPL files, which I'll get into a little bit later. And then Intelligrate is the company that builds our software for our shoe sorter. And so I also talk about that a little bit later. So I'm gonna do a demo of what that client-server relationship looks like. So this is really small, so I'm gonna make it bigger. Can everyone see? Okay, so first we can look at what the server looks like. So like I said before, we're choosing a port and then we're just starting a process in which we're listening and then we are reading the message and then we're closing it. So we can start that. And we can just choose a random port. So let's say 4,000. So now we have on the other side, let's look at what the client script will look like. So it's a little bit smaller. Like we said, it's just opening a socket and then it's closing it after it writes a message. So we are... And so we're gonna pass it an IP address, which is local host, the port number that we're trying to reach and then a message. So we'll say hello. So as we can see here, we've received a client, we've output the message and then we are closing the same client. And so this Ruby object is the representation of the socket that has been established. So we can do it again. We can write another. Sorry, another message is fine. And so as we see here, it's outputting a different socket connection and closing the socket as well. So that's how you would use Ruby to communicate in TCP. Okay. So now, okay, yeah. So now, so fast forward to winter of last year. And so what, so as we started to, well, we had already started building out a lot of the software to support the different connections. And so we'd started integrating it. We had printers and scanners that we were starting to introduce into a production environments. So then we had this beast. And so what Blue Apron was doing is that we wanted to introduce this shoe sorter machine to our warehouses. And so the way a shoe sorter works is you have a, so you have a package that gets on a conveyor belt. There's an overhead scanner and it goes through, it scans something on the box and then the shoe sorter diverts it into a lane, whatever lane, depending on what the machine has, what the machine is read. So because of this, we had this huge machine and it's obviously very latency sensitive. We wanted to make sure that we were integrating with our rail systems smartly. And so now I'm gonna talk about how, what are the different options that we can do that so that we have asynchronous processes in our rail system so that we can make sure that everything is working as efficiently as possible. So we wanna make sure that the, like I said before, we wanna make sure that things are working as efficiently as possible and that nothing is being blocked by anything else. So we don't, in this way, we don't want the rails application to be doing too much. So I think that's, I think in any system that you're designing, you'd never wanna put everything in the same process that just doesn't make any sense. That's what multi-threading is for and things like sidekick. And so, so we did something, what, is that my mouse? No, oops. We have something called, that I like to call non-blocking integration. And so it's kind of a variant of non-blocking IO. So we have our rails application and we can have a process that's running simultaneously that can handle the connections to the hardware. And then on top of that, we can have another simultaneous process that handles the updates to databases and whatnot so that what's happening from the browser's perspective, so if we do have something where we are going from the browser to the printer, all of those processes are not blocking. We have asynchronous behavior so that we can move efficiently as possible. So like I said before, a user example would be, we might have a browser and the browser needs to print something out from the printer. So we can't do this because we already said before, like we learned in the first section that HTTP is browser, that's pretty standard, and then TCP is the printer and so because those don't speak the same protocol, there's no way that we can do this. So how would we implement this? So we could do something like this. We have a browser, which makes an update to, or I'm sorry, it makes a post request to our Rails controller. And then so the Rails controller can kick up a worker. In the purposes of this demo, or in this presentation, I'll be using Sidekick, that's awesome. And then so the worker can open up a TCP socket to speak to the printer. And so in the scope of Rails application, we would have something like this where just a labels controller and then we have another worker which is kicked off once the request has been made. And so this is the code. It's pretty standard Rails code. We have a create on a post. We've created this label and then after we create the label, we have a printer worker that kicks off a job and it passes in the ZPL. And so I'm still gonna go into what ZPL is a little bit later. And then this is a snippet of code that we would see from the printer worker. So we've created a queue for our devices that to handle any sort of hardware that we might need to communicate with. In this case, we are opening the socket and closing it in the same worker for the, like I said, for the purposes of this presentation. So another example that we might need is that we have a scanner and the scanner needs to make an update to a database. But the scanner is speaking TCP and the database has its own protocol. So how would we do that? So something that we might want to do is we'd have a scanner and then the scanner is, so we have a demon process that's listening to a scanner. And so for those of you that don't know, a demon is just a background process that's listening, the most popular example would be like mail or demon. If you email an incorrect address, it kicks off a callback and it returns an email and says, hey, this person doesn't exist or deleted their email or whatever. So yeah, so we have a demon that's listening to the scanner and then the callback for that, when the demon receives a scan, is that it kicks off another worker and then the worker makes an update to the database. So this is what it would look like in the scope of a Rails application. So we just have a scanner worker and then we have a separate process that runs parallel to our Rails server, which is the demon. So we have our Rails server and then in a separate instance, we have this demon, which is listening for scans. And so from a code perspective, this is what it might look like. We have just a script that I've written and it kicks off the demon process. And then we have the demon and so every time it receives a scan, it kicks off the scan worker and it performs the update to the database that we need. So another thing about integrating with Rails is that a lot of these different devices speak different protocols. So for instance, the scanner, I'm sorry, not the scanner, the printer, like I said before, speaks, it receives a specific language, but then when you scan that same label, you don't receive the same thing back. So and you might not receive and then as far as the scanners go, you might have three different models of a scanner and all of them are sending different protocols in the message. So it gets a little bit confusing. So first of all, like I said, ZPL is a Zebra programming language, it's labeled definition and printer control language. So this is an example of a Zebra string. So you could think of it sort of like HTML, you can deliver it to in a string form, it's just a simple string. And so if there's a certain element that's off or so there's a syntactical error, it won't format right or it might not even display correctly at all or not print at all. The printer might silently fail. So it's just, yeah, it just recognizes some simple patterns. This is at the top we have, can you guys see my mouse? Yeah, okay. At the top we have the start of the string and the end of the string. There's some stuff with formatting and how big the text is gonna be. We have something for a barcode, a barcode, QR code. And then foreign data, this just tells the printer, or the scanner, not the scanner, the printer, hey I have some data that I'm gonna be passing to you. And so this is an example of a message that we might receive from a printer. So it looks totally different. We could be scanning the exact same thing that the printer just printed out, but it senses a completely different message. So in the first line we have the message from the scanner and then on the second line we have an identifier of the printer and then third line we have some other data that we don't even need. So how do we handle this? Because we don't want all of this weird logic that isn't really applicable to the Rails application sprinkled all over the place. So a solution of this would be to have some shared code that acts as middleware so that from the minute that you receive the message from the TCP socket, it parses it accordingly or it sends it out accordingly and it acts as middleware. So that once the message gets to the application there's no ambiguity about what is this message, which line does it coming from? Is this even saying what we think it is? Okay, so I'm gonna do a demo. So all of those processes that I just started that I was just talking about having the device queue running, the daemon and then the Rails application. I'm gonna show that all to you today and hopefully everything's gonna work. Okay, so we have our Rails server, we have our daemon and then we're using Netcat to communicate with the scanner. Okay, so first off, okay. So let's look at our labels. We have no labels, so let's create some labels. Yeah, yeah, I can do that. So we've created three labels and we have our labels, okay. So now I need two volunteers to give me your name and your title. Okay, so now Megan gets a label. This is the only size that we have so I know name tags are supposed to be a little bit smaller. And then can I have another, yes? Craig? Oh, okay. And what's your title? So now we have two labels that we printed out. So let's look at our labels, our label dashboard. So before we had, so before we created three labels, they were just marked as created. Now we have two printed labels for Megan and for Craig. And so now I'm gonna scan one of these labels. So now let's refresh. And so I scanned Megan's label and now Megan's label has been updated in the database. And so now, so now let's scan Craig's label. Now both Craig and Megan have scanned labels. Do you guys wanna come get your labels? I'm gonna put some sort of hashtag on these but I forgot, here you go. Okay. So yeah, so that is, so that's an example of how you can use Rails and have simultaneously running processes to make sure that you have a flow, like a workflow like what we need in our in Blue Aprons warehouses. So some obstacles that we had to overcome. So first of all, not all hardware can have multiple sockets open at the same time. So that might not seem like a big deal but when you start to scale and you start distributing load, there is, that starts to be a problem because you wanna make sure that you are anticipating what is going where. And so you don't want something to be sending a message to one device but it's already connected to another instance and then you have a silent fail because these are lower levels and so when we do send a message, we're not necessarily getting a verification that it got back to us or that it got there. So keeping a persistent socket open, some frameworks don't support that. So in the example that I gave, I am creating, I'm opening a socket and then closing it. We might have an instance where we wanna keep a socket open and so it would be, we can't do that from within the Rails application because we have nothing supporting that process, that listener one while it's open. And then privacy. We need security, obviously. So if there are also some hosting services that don't allow open sockets on a private, to a private network. So for instance we have, so why this matters is we have devices that are in the warehouse, they can't be on a public network because then anyone in the other side of the country can either, if they know the IP or if worse they just guess the IP address, they can just start sending messages to our printers and we don't want that. So we need to keep them on a private network and because they need to be on a private network we need to have hosting services that support open sockets to private networks. And then testing. So testing is one of my favorite things about the development process. I love automation testing and writing tests and making tests pass. And so it's easy and web development to stay within the scope of your web application and your code but when you're starting to work with hardware, automation testing isn't everything that you need. There's a lot of factors that go into the success of something in production. So something as ridiculous as the thing was unplugged or the core doesn't long enough and it doesn't reach the outlet. The ethernet has been, is broken. There's a lot of stuff that you need to anticipate and so manual testing is extremely important when you start integrating with hardware. So that brings me to the end of my talk. I'm happy to say that the software that we created is being used in blue open warehouses across the country in production and it's being used to deliver box if there's any customers in here, I have no idea but it's being used to deliver boxes all over the country. So this is awesome that we did this and we were really excited and I think that my message to you guys is that I would really like you to as web developers just really think outside of the scope of web development of what we can do, how you can integrate things and there's a lot of really cool stuff you can do with Ruby and Braille so I encourage you to explore that. And that's it. Thank you very much guys. Yeah, so he said with the, as far as keeping the printers on a private network, could you just have the server that we're working with on a local network? The answer is yes. But just, yeah, but just maintaining software within our facilities is like a whole other realm of complexity so that's a possibility but it would just be a little bit difficult because then we have to physically maintain the software that's running on the local network. So, anyone else? So he asked what did we do to coordinate communication between devices that only take one, that only take one connection? It's pretty easy because even though the business logic makes that really easy so we have warehouses, we have three warehouses right now, each warehouse that, so let's see. So each warehouse has a sorter machine. The sorter machines are actually the ones that only can take one socket at a time so we just have a process that handles that machine per facility. So that's really, it's not always that easy but in our case, that works for us. He asked if, sorry, what did you ask? Oh yeah, if we were using ERB to render any of the ZPLs, no, we haven't done, I'm sure that might be in the future but we don't preview any of the ZPLs as far as rendering them as ZPLs. There's other ways to do that but it's not, yeah. He asked if this was on the same deployment pipeline and if we had any problems, if we had any problems maintaining those sockets, we have, so it is but since it's a different process, it's a little bit different so we use blue-green deployments to make sure that there's no downtime and so that ensures that, yeah, that there's no downtime. That's how we solve that. Does that answer that? I'm not sure if that's, if you guys have any questions, feel free to come find me afterwards. Clearly, this definitely makes me really excited and I love it so I'm happy to talk about it but thank you so much for coming. Enjoy the rest of your day. Thank you.