 Yeah Can I go all right, so welcome to James's talk on IPv6 James is in here Okay, right. So this is the talk that I delivered last night at the AWS user group meeting It's on how to build a scalable email processor using AWS The technologies that I specifically used for this from left to right I used EBS which is elastic beanstalk not elastic block storage because I keep like mixing the two up SES the simple email service which as of just over a month or two ago now instead of just allowing outbound commercial emails normally bulk email sending Also now as an inward processor so you can receive emails as well with this year's a school SNS the simple notification service which will send JSON notifications to HTTP or email endpoints All configurable, so it's sort of a publish subscribe system in the back end that Events get published either triggered by the system itself and then can get published out to Your own server on a web endpoint for example an RDS which is a relational data store I'm not using that at the moment, but I've already built in the capabilities into the application and of course S3 Which everybody knows Scalable simple scalable storage. That's why three S's For those of you who don't know who I am My name is Stefan February. I work at this little company called Neil And we are hiring in case you you know anybody For the camera the reason they're laughing is because we are at neo It is actually not a joke that we are hiring at neo. We are hiring All right, yeah, I've got my blog at hash bank bin dot sh and the slides are up at GitHub.com Stefan February and I also put them up on slide share if you want to slide share URL. Just ping me Now what am I trying to do this thing won't fit into my iOS wallet wallet for those Who you don't know used to be called passbook and the idea behind it is that you have these digital passes or coupons which go into a which sort of secure kind of digital technology that goes into the wallet and The idea is that I'm supposed to look like that Why is my phone showing one thing and the screen showing another? No problem So this is an example of what An iOS pass looks like this is actually a boarding pass type The the iOS wallet has several different types of passes that it supports. It'll support both a things like a coupon store credit card Boarding passes for example They're only like four or five predefined types and the temple for the templates the layouts and then you can go around and you Can mess around with sort of the colors and the logos And the but the layouts tend to be fixed for the different types So a boarding pass will always look like this. You'll always have like the Airport your leaving's Shortcode a little airplane or icon that you define and then the airport that you're arriving In and then at the top you'll have additional information for the airport's the terminals you're leaving and the rest of your flight Data is on there now as I mentioned The objective is this email that I get when I do a web check-in. This is not the itinerary that you get after you have booked your flight. This is the web checking Confirmation email you get if you do an online check-in That then you get an email like this Now obviously I want this to actually be turned into a boarding pass because along with this email They give you a link if you click this link here Where that link will take you is actually the printable boarding pass Which is a PDF document that you can or a PDF formatted document that you can print out and present this at your For you as your boarding pass instead of queuing up for example, you know You just want to go straight through you don't have any check-in luggage makes it very quick and easy But there is a Digital version of this thing Air Asia for example has their own application or their own mobile app Which presents a digital boarding pass Which the airlines may or may not not the airlines but the immigration officers may or may not expect except Depending on where you are in which country you're in It's lost the connection. I'm gonna have to use my Keyboard or would I It's re-establishing the connection Okay, that's what keyboards are for All right, so what did I do to build this thing? The application is already running. I built it in Scala. I use this framework called spray. It's really cool Alternate framework to the one that we're currently using in our current project. So I can't actually say that can I? Okay, we're using play we're using the play framework at the moment, but that is fine We won't say where the client is We're using the the play framework at the moment Spray is an alternative to that it's similar except that spray is built from the ground up to be Aca actor based so it uses actors as the HTTP endpoints instead of a instead of servlets That would normally be used in a in a Java EE framework Your HTTP endpoints are actually a car is an actor actor that would have to spin off other actors To actually get the work done. So it's asynchronous message passing based really fast like super fast Question Yes, so when you start up spray it implicitly boots up an actor system for you So, yeah, it's it's spray. You can't you pretty much can't use spray without using Aca actor sorry without getting Aca in there as part of the deal and then of course I'm I'm using shippable and the reason I mentioned shippable here is because I've actually from the very get from the very start set up my development environment in such a way that Deploying is a git push. So I do a git push and Shippable is a CI continuous integration, but it also does deployment It will build run my tests and immediately push to elastic beanstalk where my deployment is now elastic beanstalk is of course Just the management interface in front of a cluster of easy to servers But the reason it's interesting is because the AWS SDK command line tools. They give you A really quick and easy way you just do use the command line tools EBS space deploy after you have configured your your project and it will deploy automatically It'll do rolling updates on your on your on your on your Cluster so that you don't actually have any downtime all sorts of really cool and interesting things happen So I'm using shippable for that Next this thing keeps losing connection. Okay, I'll just turn you off right, so How how is all this email processing actually happening? You've got an incoming email which hits the the simple email service endpoint the ACS endpoint The downstroke there that is the Air Asia email going or being The notification of the email arriving being published onto an SNS topic So remember I said SNS simple notification service publish subscribe pops up model So it publishes the notification of the email to SNS SNS then notifies my web server that email has arrived Concurrently to publishing the the the notification it sticks an actual copy of the email into an S3 bucket So the the email itself Goes into the S3 bucket Wrapped up and represented inside of a JSON file I get a notification on SNS on my HTTP endpoint on my server to tell me a new email has arrived I can then start processing that notification Fetch the email from the bucket and and get going and that's basically the how the magic happens I fetch the email. I do the processing. I extract all the information I do some web scraping and then I build the pass which involves Some cryptography some digital signing with the Apple developer search search so that the iPhone will actually recognize The pass is a valid pass because you can't just stick any anything into the eye into the iPhone's wallet You actually need a developer account. You need developer certification In order to be able to generate certified passes that go into the wallet and Then I get a confirmation email back with an attachment which says okay your wallet Your your your wallet version of your pass is ready. You click on the ad button and it goes into your wallet Yeah, that's how it works you forward you forward your the email that I showed you before This one you forward that to the email endpoint that I will share later Because I'm also going to solicit some help from you guys at the end which is not now So what am I gonna do next? Well, the next steps would be to build and test an auto scaling environment because right now I'm running a T1 micro instance, which is just a single instance inside my Inside my elastic beanstalk environment. I need to hook up RDS because I'm not actually storing any persistence state at the moment All the information although I've got all the information stored It's all stored in JSON format in emails and places But I would like to have sort of a the the final product in the intermediate artifacts that I'm building also stored in RDS But the servers themselves are completely stateless They don't actually maintain any state on the server and of course if I have RDS hooked up then my my service layer is Completely stateless, which would allow me to scale out nicely Gotchas AWS permissions hard Things everything well, they've kind of locked things down So I guess it's more secure But it also means that you might inadvertently make yourself insecure because you don't understand how to how to flip and toggle all the knobs with the permissions and it can also be a bit of a nightmare debugging why my Why I came from a regular easy-to-instance access my S3 bucket, but from the AWS Service rather the easy-to-server running inside elastic beanstalk all of a sudden I can't right Or you thinking it should work, but there are very strict permissions on what can access what at what times and you know Fortunately, there are policy tools to build policies which don't necessarily work the way that they should because the Documentation is not actually 100% which is my next point the documentation for the notification service messages Do not actually line up with what you're actually going to receive at some point They've changed some of the API has changed slightly some of the keys and the key value stores have changed a bit And the documentation hasn't been updated So I had to actually dump the messages to actually figure out what I was supposed to be getting and how I was supposed to parse that Yes, so It that that that is that is that is frustrating Fortunately, they Amazon does not do that, but something they do do which they probably should do differently is that they should Actually declare the content type or the mime type of the content as part of the HTTP request They currently do but they say they're sending you a text plane Content type which of course buggered up my my my my server side on my rest on my restful service endpoint Because I told it you're gonna get application JSON and then it inspects the headers and says nope This is not application JSON. It's actually text plane So now I'm now I'm accepting text plane and then assuming that is JSON on my endpoint Okay, there are lots of knobs in AWS lots of shiny bits to fiddle with I've scratched the service the surface of what is in there And you can really end up spending a lot of money running up bills on your on your credit card by fiddling with all the knobs, so you know, it's it's kind of a Goddess you have to weigh one up up against the other Do I really need to fiddle with this knob right now or can I get my work done without it? That is the end I will now take I guess I will now take questions I will leave this slide up, but or maybe I should do one thing on this on this final slide Instead of adding in my Let me just put in here This is not my email address. This is the email address where I would like for you if you have an AirAsia email of Some sort not some sort but specifically a web check-in email that you have generated Please send it to me at that email address Not to send it to me send it there and see what happens I'll also see what happens if you manage to break the system now. We know also really helped me break the system yesterday so I managed to Fix at least one use case which wasn't covered because email parsing is surprisingly hard and easy to mess up the especially when what you're actually getting is a mime encoded email that is HTML body So the original email that actually comes out of AirAsia when they send it to me or to you is base 64 encoded mine which the email that which most male clients nowadays know to Decode get the HTML document out of it and then present that to you If any get anybody who knows my friend Kai Kai is a really big bug being his bonnet about people who don't have text plain Content in their in their emails as well for people like him will still use mutt. So he's he's not very happy with with AirAsia But anyway, so if you guys do have any emails that you would like to send to me Even if it's for the other airlines, you know If you've ever done a web check-in to get an electronic boarding pass send it on to me Stefan dot February at gmail.com or if you guys have my new email you can send it there Let me say it again. Stefan dot February at gmail.com Just send me your electronic boarding passes the emails you can forward it to that email Because I want to build out the service so that it supports other Airlines as well apart from just AirAsia. She's the only one it supports at the moment. Let me do a quick demo I'm gonna try and run the demo through my mobile so that it gets picked up by the recorder Nope, I do not want to see that Let's close all of this stuff down Hopefully it's nothing embarrassing on the desktop What am I gonna run? I want to run quick time and Open up my email client here. It's quick time running Okay, all right, why? Okay, there we go Of course because it had to open up Good time is gone. I Had this yesterday as well for some reason when you access the phone using and it keeps opening photos New movie recording all right No Don't open photos All right So I actually have a check-in that I performed a few days ago if I can find it There was on Saturday Where I was coming back from Kuala Lumpur Ah self-check-in This is the one So this is the email that I received this is similar to the one that you saw me show you guys early So the the way that it would would work is I would forward this to the email address erasure at into pass comm and then send and Fingers crossed, you know this live demo these normally don't go the way you expect them to If we wait tick-tock, tick-tock, tick-tock Like just wait a few minutes seconds. It should be seconds. It should not be that long. I should receive a Any minute now Today ah There we go So before I do that I'm going to just quickly just for the purpose of being honest show you that my wallet is actually empty right now, right? And I'm gonna come back here and I'm gonna open up the email There's an attachment in there and this is the dot pk pass file, which is the actual wallet Pass once it ends up downloading the email client is actually intelligent enough to recognize what it is and Will show you the little icon which is attached to the wallet or to the pass It'll present the past to you when you click on it and then I can click on the add button and that would actually add it into my into my Into my wallet. So just to confirm that it is actually in the wallet I'm gonna click on wallet open it up. My pass is now in my wallet And as you can see it is all the flight details for me And if you actually were to this is a PDF 417 format the 2d barcode If you actually were to scan this you would get a really nice string. Actually, I do have a What do you call this thing a PDF 417 barcode scanner But how am I gonna scan it if I can't see it from my mobile that is not gonna Okay, that's not a bad idea Okay, so I'll open up the email just on my on my computer Where is the mail? All right, let's see This would be the Email that I just received. Okay. I'm gonna have to download it first So Mac OS can actually also there is another way actually get this into if I open this on my desktop on the Mac It will actually show The the the pass and I can actually from here Import it into my mobile via iCloud if I've got iCloud connected here, it will actually move it from the desktop as well So now that I have it up Have you got it up? Yeah, it's up. So I will now attempt to scan it with my actually let me do this I will Minimize that I will use this scanner The PDF 417 oh, it just has to be connected right so I don't know if it will okay. I'll try to scan it on the phone On the on the desktop Come on. You can do it You can do it Scan There we go so that that is actually my My flight information and so this is also the scary part right that your actual printed boarding pass If you ever think I just throw this piece of paper away. I'm gonna throw it in the airport or somewhere It actually contains some personal information about you. This is me. Iata the International Association of Travel something It's got everything in it M1 is sort of a short Flight code for the type of information that is in there and then of course my surname and my name then the next string is My booking number followed by Kuala Lumpur to Singapore on a K Which is the the flight code for the airline my flight number is seven to three and then the next series of numbers relate to things like my booking type the type of fare that I'm on my seat number is in there, which is 5e if you can see in the middle of that string that is 5e and the rest of the information also means some interesting things So, yeah, you probably do not want to leave your boarding passes lying around in case some nefarious hacker Actually with a PDF 417 scanner decides to just harvest your information Yeah, so that is it. Are there any questions from anybody? It's not great The black and blue is not great. So the black and blue Interestingly enough. So is so yes, you can the styling can be changed. What can we change the styling icon can be changed Background color can be changed. You can give a background image if you want to the text color can be changed You can fiddle with pretty much all of the parameters except for the layout in the positioning of the of the elements on the form I have no idea why apples default boarding pass template is this apples Default is like black on blue. It's completely, you know, you can't see what is going on on it But yes, I completely I agree with you. That's not the best and the moment I have I have time I will actually make some time to to fix that. Oh Speaking of speaking of colors, right? interesting security feature that I bumped up against With Air Asia and the immigration offices for Air Asia in Kuala Lumpur They will actually allow you to present an electronic boring pass But only if the electronic boring pass has the flight code, which is the KOL or the SIN flashing red-black Because that is what the air Asia's built-in app does to show that the flight is about to leave We are actually boarding soon This is a valid pass. We are flashing the light. So now everybody can know And they looked at the pass and go why isn't it flashing red-white? Sorry. Do you have a paper? That so yeah, now I'm gonna have to figure out probably can't be done But I'm gonna have to try and figure out if it is it all possible. Maybe I can can show it to them like quickly Or have two passes one with the red-black But yeah, so I'm hoping that they will actually allow me tomorrow I'll try to change the styling a little bit and see how far I get with it. I'll keep Testing my wallet pass until I'm allowed to go through through through with it, but yeah Yeah, yeah, that's that's how I'm testing my app So this is three weeks worth of work Literally hacking in the backs of cabs on the way to work and in the way home from work In airport lounges and pretty much anywhere in between Like over lunchtime or in the morning before we start or in the or after work in the evenings before I go home While I wait for the surge to drop on Uber So it's been a part-time effort for for the last three weeks. Yeah It's on a t1 micro instance it costs about $50 per month In Amazon because you don't pay for Elastic beanstalk the service you don't pay for you really only paying for the time of your Easy-to-instances and since I'm running on a t1 micro, you know, it doesn't cost that much. It's actually quite cheap Yes, yeah, if the email format changes then I'll have to also change how I parse the email Because right now I'm actually taking the rather oops, I'm taking the rather rigid approach of unpacking the HTML from the email body and then parsing that into a DOM and then using Scala's expo spot expo Style expressions to search through the DOM Which I think is more efficient way of parsing it than doing a grip in terms of computation But it's not necessarily the most flexible way of extracting the information Grip is more unique Maybe a little bit more bashy as well. Yeah, you should appreciate my domain name for my blog Yeah, Kai would have done this in five lines of bash He wouldn't have yes, sorry Kai, sorry Because I know Kai is gonna watch All right, any other questions nothing all right applause All right, thanks guys