 So I'm Lee Edwards, the VP of Engineering at Teespring, and I get to follow up probably the cutest tech talk I've ever seen in my entire life. If you didn't see the talk on father-daughter pair programming, you should definitely go to Confreaks and check that out. So now that this is off to a really good start, I told everyone to leave. So yeah, my name's Lee, and I want to talk about how we do global manufacturing at Teespring using Ruby, and in the fashion of 2016 Ruby talks, we also use lots of software that isn't Ruby. So I'm super excited to be working with hardware again. I've been writing Ruby since about 2008, but before that I worked in robotics for a company called iRobot, and I was actually a mechanical design engineer, so I worked a lot with different manufacturing methods, sheet metal, milling, CNC, lathe, so 3D printing. So manufacturing is something I've kind of always really liked, and I had no idea my career would kind of swing back this way. So you don't always think about technology when you think about t-shirts and garments, but the truth is there's a whole lot of technology behind it, and the oldest one is screen printing, and we actually started seeing screen printing in China during the Song Dynasty in about the 10th century. And so the technology looks very, very different, but the principles are very similar, so essentially what you have is a screen that has a negative of an image on it. You fill this with ink or paint, and then you're going to squeeze or roll that ink or paint through it onto a substrate, like a cloth or a garment. So you can see a lot of really, really cool Chinese art that was created during this time using screen printing. But we didn't really start to see it spread outside of China until the 17th, 18th century, when they were using the blockchain through something the dread pirate Roberts created called the Silk Road. But actually the real Silk Road, which was a real road. And basically this was sort of the trade route from China to the rest of the world at the time, and the reason that silk and screen printing were tied up is obviously you've heard kind of the phrase silk screen printing. So the silk was kind of required because it was a very strong material that you couldn't really do screen printing using cotton for the screen. So we start to see it go into the west around that time. But then we see a lot of really cool advances in screen printing in the 1950s and 60s with Warhol and Liechtenstein. And the thing I think is really cool about this is what screen printing was essential to what Warhol was trying to do with his art. So his whole idea was he was trying to comment on, and I'm not an art history major, but he was essentially trying to comment on kind of the way that artists perceived and valued and how there's only one Mona Lisa and that's part of why it's valuable. And every painting you make there's just one. And Warhol wanted to turn that on his head and it's like, well I'm going to make a thousand of these. I'm going to make a hundred thousand of these. And they're still beautiful. They're desirable. And they're all exactly the same, they're identical and repeatable. And that's exactly what you get with screen printing is you can create large quantities of something that's identical. And so Warhol figured out how to make that art. And today screen printing looks, a factory that makes screen printing looks a lot like this. So it's come a long way in a thousand years. This is our facility in Hebron, Kentucky, which is called Mercury. So it's about 15 minutes from here. And I guess at this point I should tell you a little bit about what we do just so you can kind of get context for some of the stuff I'm talking about. So the way I usually describe Teespring is that we're essentially a marketplace that allows someone with an idea, a design, a brand to create a product and turn it into reality. So one thing that we do is we sort of take away the need for you to understand how to manufacture it. So we have this facility, we work with third party printers, which we'll talk about a little bit. And these creators will create a business online or they'll create a campaign where they're raising money for a charity. So for example, I'm wearing this shirt, which Bo Bergeron, a designer out of Oakland, California, created to raise money for Louisiana flood relief, raised I think over $50,000, maybe even quite a bit higher than that. So that's kind of what we do and how we do it. So screen printing is how we do those kind of larger campaigns where we're printing thousands and thousands. And the way this sort of works typically is first you'll have to do artwork separation. So you have a design that's got, say, 10 colors. That means you're going to need 10 screens. So the first thing you'll do is kind of separate it. So we have a room of designers that use Photoshop on IMAX and they are actually pulling the layers apart on the image. And one reason that you don't actually automate this is there is a good amount of artistry and craftsmanship involved. So there's a lot of things that they'll do during this process where they maybe remove some discrepancies or maybe two colors are close to each other so you can reduce the cost by combining the two colors. So there's a bit of a judgment call as to whether that's desirable or not. So basically we'll do artwork separation. And after that or possibly at the same time, we're sort of picking the blank inventory off of the rack. And if you work at a traditional like e-commerce type company, this is generally the entirety of what the Fulfillment Center does. Just receive product, scan it, enter it into some kind of inventory system or warehouse management system, stock it on a shelf, put a barcode on it so then you know it's there. And then every time you pull one off, right, you're scanning it. So we do that but it's just a part of what we do. And then also this could happen simultaneously or perhaps next. You actually create the screen. And so there's lots of different ways to create screens but all of the very, very high volume manufacturing facilities for screen printing do it similar to this. So essentially you have kind of a lie flat inkjet printer. And you have a screen that has this substrate on it. And the inkjet printer will print the positive of the image on top of the screen. And this can be done like very quickly and again you need one screen for each color. So the next step is you put the whole screen through a sort of heating process that will burn through the emulsion and turn the positive image into now a negative. So this looks a lot like the screen you would expect to see. That has a negative so you can actually push paint through it. So the next step is you'll usually have like a large inventory of paint. A lot of times you'll need to mix paint based on hey, a design came in and we don't have that color. And then you sort of slab the paint onto the screen. And I think one thing that's kind of cool about this is like a very thick latex based paint. And it's designed in a special way to be able to bond with the shirt so that it can actually survive washes. And this is how a lot of retail quality shirts that you see are created. So these things, I mean the paint will like outlast the garment usually. So once you have all that, so here you see like a rack of blank t-shirts, this is a screen printing press. And you can see the operator here is insanely fast. You kind of can't really tell, but he's actually having to line up very precisely the shirt against those screens that are sort of above each station. And that's to make sure that the print is well aligned. So a really talented screen press operator can do a shirt in about eight seconds. And let's see. So when you see the machine in action, you can see each screen presses down on the garment and then uses a squeegee to force the paint through the negative and onto the shirt. So then the last step is we have this curing step that kind of dries the shirt and it cures the paint so that the paint bonds to the fabric. So I mentioned multiple manufacturing processes. The second one is very, very, very new compared to screen printing. So digital printing, the company that makes the digital printers we use has been around since about 2002. And if you've seen digitally printed shirts from that early in its life, they're not the kind of shirts, well, you probably don't have it anymore. They didn't last. The originally digital printing was very low quality, very difficult to make off the rack quality shirts with digital. But today, it's amazing. And the way it works, if you're familiar at all with inkjet printers, it's very, very similar. So essentially, you have a pump that's connected to a piezoelectric crystal. So a lot of small electronics will have piezocrystals. It's any time you see a small device that has a little bit of motion or uses a pump, it's often using a piezoelectric crystal. So the idea there is the piezoelectric effect. And there's going to be a material scientist in here who's going to be face-pomming when I try to describe this. But the best that I can explain it or understand it is essentially the crystalline structure of the piezo crystal is kind of special so that if you were to squeeze it, apply mechanical pressure, you'll create an electric field as the crystalline structure or lattice shifts and reassembles. And the converse is true. If you apply a charge over it, you actually will cause motion on that crystal. So you can use it to drive pumps. It's used in speakers. And so the piezocrystal pumps a very small amount of ink. And then it drops through a charged electrode, which gives an electric charge to the ink. And then it passes between two high-voltage deflection plates that can ensure that the droplet is sort of deposited directly onto the substrate. So this is a very simplified version. I think a lot of the printers that we use and that you see in the industry are using more complicated multi-stage versions of this. But a lot of the key components are basically the same. So what does that actually look like? So this is one of our printers in action. And you can see here someone who is not a printing professional, my good friend Alex, is not quite as fast as setting up the t-shirt on the machine. But similar to screen printing, you do have to sort of align it with the seams and kind of make sure that everything is lined up and then sort of press that lid down. And so then the next thing you'll see when this runs is you'll see a little bit of wetness around that ink. And so that's the affixant. And the first thing that the printer does is spray that on the material. And they call it sort of wet on wet. So you're spraying wet ink on top of this affixant. And we'll get to later when you dry it, what this is for. But there's two heads you'll see here. One is sort of like white-black. The other is RGB or CMYK, depending on what you're doing. And this machine in particular is able to handle two shirts at once. So one thing that's really interesting about this is that there's a lot of configuration that goes into this. So we have configuration files that are for each individual garment that depend on their materials, their colors, how we sort of manage that. And it's actually one of the, I think, more interesting pieces of technology or IP that we have is around a lot of the hard experimentation and how do you make a really high-quality shirt using digital printing. So I'm halfway through, and you haven't seen any Ruby. So you might be wondering about that. And if you were hoping that I was going to show you all the cool REST APIs that are on top of these digital printing machines and screen presses, you were going to be very disappointed. These are all proprietary machines that do not expose nice, beautiful REST APIs. But I think what I wanted to talk about here, and I saw this a lot in robotics as well, is that a lot of times working with a production system, you don't always get the kind of awesome and fun APIs that you'll get when you build, say, like an Amazon IoT or like Arduino or Raspberry Pi. A lot of times you are dealing with very closed source commercial vendor software or you're working with machines that don't expose APIs. So what you're really doing as a software engineer at a company like this is you have to remember that the software is about people. It's a mantra that I think people on the commercial side, retail side of software, often repeat. But it's just as true on the operational manufacturing side. And software is also about processes, which, by the way, are also about people. So most of the software we write is really we're trying to watch what people do. We have 200 employees who are working, basically 24-7, someone is working that facility. And we're looking at how do they work and what are their problems, how do we make them more efficient. We're looking at the operational processes and how do we model these. And so I think the reason that we use Ruby is, in particular Rails, a few years ago when we started the Rails app for Teespring before my time, not knowing where your domain is going while you're kind of evolving a startup. One really good example is started with screen printing and now we're moving to digital. And a lot of the domain that we'll talk about in a minute has shifted during that time. So dealing with a monolith is a little bit easier, right? I think dealing in a microservices world has a lot of advantages. But one of the disadvantages is it can often be really difficult to iterate quickly. So we've really had a lot of benefit from sort of keeping all of our stuff in a, I guess to borrow a phrase, majestic monolith. So the basic elements of Teespring, our Rails app is kind of divided up into a few engines and I'll talk about a few here. So where I say commerce, I'm really glossing over the fact that there's multiple Rails engines involved here. But by commerce, what I really mean is the buyer experience. So people who go to teespring.com and they're searching, they're browsing, they're looking at categories, they're getting recommendations, they're buying things. And then the creator side, which is, as a designer, I'm uploading my image, I'm managing my campaign, I'm saying how long I want the campaign to go for, I'm setting prices. So kind of call that like the commerce side. And once I've created my design, there's a whole sort of domain around artwork. And some of that is in the Rails app. We also have some artwork related things that are written in Go. So we have an image mockup generator called Van Gogh because it's actually a law that if you make a repo in Go, you have to put Go in the title, so we called it Van Gogh. So a lot of that stuff's in Ruby, some of it's in Go. And it communicates with both commerce and fulfillment. So the fulfillment engine, and again, there's actually kind of more than one engine related to fulfillment, but kind of handles the handoff from everything that happens from an order to when you get this package in your hands. And the reason we've sort of built this stuff as Rails engines is I think, as I've said, the monolith works really well for us right now. We're kind of looking towards the future of, once we have more and more engineers, once our sort of domain is a little more solidified, I think it's going to make a lot of sense to potentially pull some services out of the Rails app. And I think a really natural division is kind of like right down the middle of commerce and fulfillment and would allow us to sort of define like a really formalized API and contract between the two parts of the application. So key piece of domain that's in the fulfillment engine is called the fulfillment job. So I'm going to walk a little bit through the life cycle of the fulfillment job, both in terms of how we handle it inside of Teespring and with third party printers. So when you create an order, I kind of alluded to the campaign model earlier, when you create an order on Teespring, you are not going to get it fulfilled right away necessarily. So this idea of a campaign, it really came from, if you look at Kickstarter, Indiegogo, like there's a lot of benefits of having a campaign model. So if you say, hey, this campaign ends in seven days by now, people want to jump on board. It also, if you have sort of the social proof of, hey, 100 people have bought already, it makes people feel like they're part of something, particularly if they're raising money for a charity. And if you say, hey, three more needed to tip the campaign, right, you get that sort of sense of urgency. But I think a bigger reason around why we have the campaign is if you think back to what I said about the manufacturing methods. So when you screen print something, you're spending time and labor upfront and money to basically print the screens, there's a big fixed cost upfront. And then every shirt you print, like I said, it can take as little as eight seconds. So the marginal cost is very, very cheap. And so if you think to like Econ 101, the way that you kind of amortize upfront fixed costs is you have a very high volume. So it's kind of analogous to injection molding in sort of the mechanical design world. And digital printing, on the other hand, it's sort of a fixed cost. No matter how many you print, it's the same amount of time to put that shirt in there, bring the thing down, print it. But what's nice about it is that you can actually print, say, three items. And it can still be profitable. So the campaign model really was built around the old idea of screen printing. And hey, if I run this campaign for, say, seven days, I can collect a whole bunch of orders, and then it's a lot cheaper for me. And so it was cheaper for us. So we charged the creator a cheaper amount of money, a lower amount of money. And they were kind of incentivized to run these longer campaigns. But yeah, so that's essentially why the campaign exists and why it's a central part of Teespring. So Campaign Ender is a worker that runs on Sidekick. And basically, if you read this code, it's kind of like one of those great methods that the only way you know how to refactor it is to basically take all the procedural code and extract it into methods and give each one of them a name that's a sentence long that describes what it does. Everyone's seen these methods, right? So what it's doing is it's checking that the campaign is valid and it's valid to end. And then it checks does it meet the minimum to print. So it used to be the minimum to print was really high when it was screen printing. Now it's down to three for everyone, and it's down as low as one for some people. And then it checks is it profitable. And when I say is it profitable, I don't mean is Teespring making a profit. I mean is the creator making a profit. So based on the cost curves, the manufacturing method, how many were sold, we can calculate are they actually going to make money or have they charged less than they're actually going to spend. And we actually, we don't want to put them in debt. So we only tip it if it's profitable. And then have all the orders been charged. And if so, it ends successfully. And so the Campaign Ender has a reference to this campaign object, which is pretty standard, like active record object that has a state machine attached to it. So then the next step is to essentially figure out what printer does this go to. And so there's a number of printers inside of Mercury, our facility in Kentucky. And then we also work with a bunch of third party printers. And the third party printers exist for a few reasons. There's geographical reasons, right? We can actually ship things faster if we are printing them from different locations in the US or even we have print partners in Europe. And also we use them to work with as we add new merchandise. So obviously we do t-shirts and you may have seen the tote bags floating around that we handed out yesterday. We also do mugs and stuff like that. But as we test out new merchandise, they could always make sense to work with the third party printer to test out, does this work, do our customers want it before we decide, hey, let's operationalize that and bring it in-house. But regardless, the way that the next step is the job has to essentially get assigned to a printer. And there's two ways that can happen. So one is the auto-assigner. The auto-assigner runs periodically and will basically say, okay, are there any printers that have signed up for our auto-assignment program? Like they're eligible and they have capacity and they have the capability, then we'll just automatically assign the job. But most of our assignment actually happens from a place inside of our Rails app called the printer-assigner. And so someone looks at that every morning and they're looking at here's our capacity in Mercury and here's the capacity of our partners and here's the type of job. And you know, oh, we're really slammed today so we're gonna send it over here. So basically manually assigning the jobs. And so what happens when a job gets assigned is it actually creates the fulfillment job. And the fulfillment job is, again, an active record object that has a collection of line items, of fulfillment line items. And the analogy there is if you're familiar with sort of really common e-commerce domain modeling, it's kind of like an order in a line item. It's a fulfillment job and a fulfillment line item. So each fulfillment line item could correspond to a particular product that you're actually printing and shipping. And the fulfillment job sort of exists throughout the entire life cycle of sort of the fulfillment process. And the way that it's managed, there's, well, there's two ways that it's managed. Inside of Mercury and some of our print partners use this product we have called the printer portal. And so this gives you a view on all the fulfillment jobs that you've been given. You can actually accept, reject them. So if you reject a job and you say, hey, everyone is out on vacation today, we have to reject it. So it'll go back to assignment. But basically printer portal will take you every step of the way through the processes that I described earlier. And the other way, and this is relatively new, is you can actually integrate with our API and fulfill a job that way. And so there's sort of two halves to the API. There's the pieces where there's a request from Teespring to the fulfillment partner and there's request from the fulfillment partner to Teespring. And for all of our, for this, and we have other APIs, we have APIs for creators and we have lots of internal APIs. And we have APIs for the mobile app that are all kind of subject for different talk. But for all of our APIs we're using Grape and we're using Swagger for the auto documentation. And we have a style guide gem that we put on top of this to make it look as pretty as possible. So yeah, the main pieces of domain, it's a REST API and it's I think pretty predictable. You have jobs, you can sort of get a list of the jobs assigned to you. You can get information about a specific job by passing an ID. You can update a job by saying, you can say like, hey, the status has changed from this to this. And the same with shipment. So shipment is a physical thing that's shipping out of the facility. So inside of our facility, we have a system called Apollo. And other facilities may have something similar to this, but this is the software that we run literally inside of Mercury. So this is actually built in Python and it interfaces with our Rails app through some APIs. So the idea behind Apollo, well one of the ideas behind Apollo is that you essentially can't improve what you don't measure. So Apollo exists on every station throughout the facility and you may be working one station one day, one station another, but whatever job you're doing, you're scanning your badge to get into Apollo and you're logging what you're doing. So I think one thing that I've always loved about manufacturing and software development, agile software development, there are some shared principles in Kanban if you've ever done sort of like Kanban software development. The principles exist in manufacturing and it comes from Toyota Lean Manufacturing. So one of the ideas in that is Muda, which is Japanese for waste. So I don't have any Ruby in this talk, but I do have Japanese. So basically if you're able to eliminate waste at every step of the process, that's sort of one of the Lean Manufacturing principles, right, and another is if you imagine these sort of Kanban columns on a traditional Kanban board, if you inspect sort of what's in each stage of the process, you can figure out where are the bottlenecks, where are the sources of error, what parts of the process can we actually remove? And you can't do any of that unless you're measuring it. And so other facilities might have their way of doing it. This is how we do it. And of course it's all driven by software and it's very, very closely tied to the specific operations that we're actually doing. So once, so Apollo manages the job all the way from beginning to end and the very last step, well, I guess the second to last step is actually bagging it. So we have this thing called an auto bagger and essentially, so you'll see here, she's scanning this barcode and the barcode is on the garment through the whole process. And if you were to order from Teespring, you actually get that barcode in the mail. We don't even take it off when we send it to you. So the auto bagger will bag the thing in our cool little branded thing and then we've got a label printer. So another part of our Rails application, we have, so actually either way that you handle fulfillment, you'll be printing labels. So in our Rails application, you can log in and say, hey, for this fulfillment job, I need to print all the labels. And when you work with the API, what's kind of interesting is you actually request the label like we'll send it to you. So obviously the shipping label is what it takes to get from the factory to your door. And then the last thing, the last piece of machinery or hardware that's in the facility that is possibly easy to overlook is the conveyor belt that just sort of takes the bag from the bagging station all the way to a truck. And the fun anecdote I like about the conveyor belt is every time I go to Mercury, the layout is changing, right? As we're learning new things, we're always moving things around. Every time I go, something's different. There's new tape on the floor that says don't stand here. This is the whatever lane. And there's all like computers in different places and screens all over the place. But the conveyor belt can't move because it's bolted to the concrete. So it is like the one constant. And the last thing is hopefully that all works perfectly, right? You get your shirt and you're like, this is awesome. It looks perfect. It's exactly as described. I'm so happy I got it on time. However, sometimes it doesn't happen. So we actually do have customer service in returns. And a big part of the Rails app, and you can see without even like needing to read exactly what they say, but along the left-hand side, these are all the different sort of functions in our admin. And there's so many of them. So customer service will have to deal with, hey, what's the status of this order? Or hey, I need to issue a refund or I need to do a reprint. Or someone's asking a question that will look up the answer for them. So basically this Rails-based admin allows our customer service agents to manage all that. So again, I think that the takeaway that I'd love for you to take from this, you may not remember what century screen printing was invented in, but I think, especially where I live in San Francisco, there's so much commercial software, consumer software, and people there are obsessed with the UX and the phrase software about people, my roommate actually has this shirt. But on the back end, it's also about people, right? Because it's about processes and processes are made up of people. And so to us, I think the technical challenges that we face are not, if I showed you a lot of this code, you'd be like, yeah, that's Rails code. I've written Rails before, that's Rails code. But what's interesting is actually trying to identify these problems and solve it, right? And we have hundreds of people who are doing this stuff day in and day out and how do we make their lives better? How do we make their jobs faster, more efficient? How do we reduce error? How do we model new and innovative processes as we come up with them? That's kind of what we're doing, and at the end of the day, you don't have sort of clean, beautiful, REST APIs, but what you do have is software and lines of code that you can run, and out the other end, you get a physical t-shirt, which is pretty cool. I mean, everyone wears t-shirts, right? So, yeah, thank you, and I'd love to answer questions. Yeah, so this one here. So you can kind of tell by, screen printed shirts will often have like a little bit of elevation on the top of it, so this one's screen printed. The digital shirts don't have quite that elevation, the ink is kind of like part of the garment. So there is definitely a stylistic choice. I mean, I talked a lot about the economics of the two methods, but there's also, and I think some people describe the screen printed look as retro, and I think part of that comes from like Warhol in the 50s. But yeah, so they definitely do look different. Yeah, yeah, so the way it's architected, we have, so the question was on the sort of pieces of domain that I showed in an earlier slide. So those are implemented as Rails engines, and yeah, the reason for that is essentially there is definitely some shared state between those Rails engines and shared active record models, and it's all hitting the same database, right? But what the idea in making them engines was that in the future, we sort of have some of this stuff separated out already, the tests that belong in one engine are already there, the controllers, some of the non-active record models, which we have a whole lot of like services and non-active record models are all kind of separated out into their domains. So it's not gonna be easy to move them out into services, but I think maybe easier. But yeah, I think at the moment, we do benefit a lot from the monolith, but we're just kind of trying to be a little bit future looking. Yeah, so the question was about like sort of why the APIs don't exist for these kinds of printers. So I think that's interesting. I mean, there may actually be printers that do have the APIs. The reason that we use the printers that we do, I guess I'll give them free press because they're good guys, but so Corneat is the company we use, and we pick it because it's top of the line at the highest quality product. It doesn't at the moment have these kinds of APIs. It's actually really interesting, so I've been to their factory, which is in Rosh Haim, Israel, and they actually, I would hand made is maybe too strong, but they definitely don't have like a large scale manufacturing process for these machines. There's actually a very limited number of them. So even though they're like a publicly traded NASDAQ company, they're still, they're doing a lot, a lot more manual construction of these machines than you'd expect. So I think they're sort of speaking for them, I guess, but I think they're, I think they're sort of not in the business of allowing that level of automation. It could change in the future. So the question was if we had ever hacked the corneate interface, and this is being recorded, so hi, corneate guys, we've definitely never hacked your interface. No, I mean that sincerely. I mean, I think the question is at a limitation. I mean, I think one thing that is really interesting is I think the people that are involved in these processes, I don't see them necessarily going away. As I said, there is a good amount of skill involved in, in sort of screen printing and digital printing. So I don't think it can be fully automated at this point. I think there is a fair amount of craftsmanship and artistry that really does require people. So yeah, so it's, to me that's actually one of the really rewarding things about this job is this is an industry where a lot of that manual labor is overseas and we sort of, we employ people here in the US and sort of give them fair wages and healthcare and lots of really great benefits. So it's a point of personal pride for me. I don't think I would necessarily want to automate it away. Someone who works at Teespring asked, is Teespring hiring? So the answer is yes, I'd love to talk to you about it. Great, well thanks everyone.