 Good evening everyone. My name is Gary. So a little bit about myself if I start talking about transactional emails. So I'm working at Trey Gecko as a software engineer. I've been working here for almost five years and actually started learning Ruby when I joined Trey Gecko. And even though I've been here for like five years, I've been to this meetups for multiple times I guess. And then this is actually the first time that I decided to talk in the meetup. So please be gentle. So we're not hiring. Just to make sure we're clear on that. Okay, we're actually hiring. If you're interested in joining us, development team or non-technical team in Trey Gecko, you can check out Trey Gecko slash jobs. You have all the information there. So let's start. So this talk will be more about tips, about transactional emails. And most of these tips comes from when we were actually fighting against these spammers. So as an inventory system, we send lots of transactions in emails. So for example, when someone creates an order, we send them an email and then it's usually on behalf of the users, our customers and I find that there are lots of articles in the internet about transactional emails but it's about how to set up transactional emails but never about like these are the best practices and this is how you protect yourself against the spammers. So like the tips that I give you is not very advanced. It's like simple but I think sometimes when in terms of emails, what we do is actually we set it up and then we forget about it. Even though like emails is probably very important for our customers, we don't think about it that much compared to for example how we do our controllers or how we do our background jobs and stuff. So that's why that's why I wanted to give this five tips. So if I can help you with this tips then that's basically what I wanted to get from this talk. I think this is full screen and let me check. Control command F. Control command F. Yeah? Okay, cool. Thank you. I think you should be the host and not me. I should figure that one out. Sorry. Okay. So just to start off, what is transactional email? Just one simple example is that someone registered to your app and then they want to invite somebody in the app. So just add the member and then your app will send invitation email. So basically transactional emails are emails that is being triggered by the user interaction in the app. What are not transactional emails? Things like marketing emails or like if you're sending like summary of your sales every month and those are not transactional emails. But transactional emails is something that the user can trigger by interaction in your app. So this is like very simple like in your Rails app. Usually you know there's a user controller and then when you create new user and then you send the user mailer. So this is I think you've seen this in your Rails app as well. And like when the app is still small like you're usually just like setting up like your Gmail address and then using those as your SMTP and then like at that point like the email deliverability or like the reputation of the email that is not really important for you. So like you trust your customers or your users that they will send things that will not be accounted as spam and then if like in this case like it will work if you're only sending from like one single email address like you have I don't know we have like notification at tradegecko.com or but if you want to send to using different email address then this will be a problem. So when deliverability becomes more important that you look at this kind of services you look at things like SNS AWS SNS I'll send great postmark mail gun so like the reason why you go to this services is because like if you want to send 10,000 emails a month like this like their service is really good at doing that and like if you're doing it on your own then you probably have problems doing it and then also what they do really well is try to optimize your deliverability and also your email reputation. So what they do usually is if they know that the email address that they're sending to will has high probability that will mark your email address as spam then they will actually pre-block that email. So yeah they do all that kind of smart stuff and like in this is in send grid they will have like suppressions list so things like if there's spam reports and blogs and valid they will list out those lists for you and then basically if another email address that is in the suppressions list then they will just not send it and that means that your email reputation will not go back. So but like these services are always charged by a month of emails. So like in here like the most basic send grid plan is gives you 400,000 emails a month and then if you go through that threshold then basically for every 1,000 emails you will like pay $1. So it's the same this is I think mail gun. It's half the price but the same concept. So if you don't keep track of your usage then like you know the cost of sending email can blow up. So this is something that we also struggle with is what happens if there's a spy can use this? What happens if suddenly your app just starts sending 10,000 emails in a day where usually on normal days like only 100? Then like depends on what kind of email service provider you're using like they will have a certain threshold and they will suspend your account. And that means that when your account is suspended then you cannot send any emails. So in that case this is my first tip for transition email which is like don't rely on once email service provider. So this is especially for us developers in Southeast Asia trying to get your account reactivated it will take a long time. So the process of reactivating your account is usually you need to email their support account to support team and then they try to figure out what's wrong with your email and then why are you sending so many emails in a day and then if they need to verify that the issue is not there anymore. But if most of these email service providers are in either US or Europe so that means that and their compliance team usually they usually have support team all over the world but their compliance team is in US or Europe. So that means that they can only look at your case when their work day starts. So that means in our case at one time where our customers could not send any emails for 12 hours and that is for merchants or e-commerce business that is a showstopper for them. So which is why I think for us it is beneficial to have like two email service providers that we can switch to when one of the account is suspended. So some of the suggestions on implementing multiple ESP is that so all of these ESP has APIs where you can tap into like you can send those emails to APIs but like the APIs are always different and then so if you need to implement like two ESPs in your Rails app for example then you probably need to create an adapter and then you need to create like something a layer where you can actually generalize all of this sending of the emails. So what I suggest is that instead of using API you go to the lowest common denominator which is like using SMTP. So just you can just in your mailer you can just set the SMTP there. Just set the username and the authentication credentials and then the address of the SMTP server. So using SMTP makes it easier for you to switch when this happens without any like creating lots of codes with adapters using multiple ESP API. So preferably use light gems or no gems at all. So like then we're using like these small gems. So we have like gems for mail jet, send grid and postmark. So what we do is we only set the credentials in our initializers and then based on which one we want to use at that point which is like on the configuration which is set the delivery method to one of them send grid or postmark or mail jet. So yeah, or like if you're not using any gems you can basically just have like a class there where you can set all of the credentials for that SMTP server and just use it as the delivery method options in your e-mailer for example. Okay, so tips number two is that you need to identify your legitimate users and then try to restrict their e-mail usage. So this is what's happening with us I think in July so this is how much how many e-mails we're sending but at one point we start sending lots of e-mails and so what we found out is actually despite what's caused by trial customers actually try to invite thousand and thousand new users and then like we immediately identified that these are spam accounts and like one thing that we did straight away there is actually just start restricting all trial customers from abusing all the resource. So for example if only for trial customers we restrict them to like in finding I think like 50 users or something like that instead of like when you're paid account then you don't have that restriction. So that's one thing that we did. So like for us the line is between the trial and paid accounts. But like for your app it might be something else so whether your user is registered or not or whether you have like e-mail confirmation or not whether they confirm or not. So like in DeFi's that's just pretty easy like you have like DeFi's registerable or confirmable and then you should basically the idea here you should not treat your customer the same if you cannot verify them. So like if they are because if you have so many registrations in a day for your app and sometimes some of them might not be legitimate users we just need to make sure that those legitimate non-legitimate users are not like abusing your resources. So okay the third tip is that you back your e-mail with a model and what I mean with that is actually before I show you this like you create a user and then it sends an e-mail but what we do often in our app is that you create a user and then instead of creating a user creating the e-mail straight away we create a user information model and it saves in database and that will send an e-mail. The reason for that is that having a model there allows us to have like very advanced validations in the model or like if you like, if you save it into for some post guys database you can have validation on the database itself just probably to make it more robust. So like what we usually do is like for example one trial customer is allowed to send only 10 other confirmations and or there are no link in the subject. So like this kind of validations gets harder. It's very hard if you if you're actually using like this part where we actually just send the e-mail straight away and without having a track, without tracking like how many e-mails you actually generate in a day and then like probably this case still like creation of a user might block it if there's any invalid params in there but what you should avoid is that having a controller where your front end can send an e-mail straight away without tracking on your back and like how many times does that happen and then also like what I like about having a model for every e-mails is that they become the source of truth like from your code like how e-mails are generated. So most of the time you will have cases where I think I sent thousand e-mails but why does SendGrid says or why MailGun says like I sent ten thousand e-mails where this currency is coming from so having that makes it easier for you to audit and then just to see where the leak is so tips number four and this is something that we actually experienced recently which like length validate your recipient and CC field to give like the similar story so this is our order how many order e-mails a day so like this is basically the this is the weekend and this is the weekdays this happens recently I don't know why we need to look at that actually but like take like 19th of September 19th of September is just the same as any other days basically so if you look at this you probably not notice like there's nothing wrong in 19th of September but on 19th of September our SendGrid account got deactivated the reason for that is that there's a spike on 19th of September so at that day we're sending around 1,000 e-mails more than that actually so yeah due to that like our account is suspended and then as you see like our batch is saying like we haven't sent any like where's this 300,000 coming from but like the SendGrid batch says like we're actually sending 300,000 e-mails so what is happening why so that is what that is me actually so that's what I did after looking at our metrics but the culprit is this so we have this e-mailing form in our app and basically the customer can put to there can put cc and put the subject and all the message so what happens is that one of the trial accounts which is most likely is a spammer account they just put like 500 e-mail address with comma in here and then also in cc and then send that to SendGrid so that went through so with one e-mail address we're sending 1000 e-mails and they're doing it over and over and over again so that's what happened so what we did is after I think after 12 hours trying to figure out why we're sending 300,000 e-mails basically the quick fix is we validate the ccpn and cc and then we're just doing like using the length you can do like try to chop the e-mail address and then just count how many e-mail address are there but I think the thing is we don't want to use like regex and e-mails also have formats where you can send the name first and then there's the parent thesis and then the e-mail address so it gets really complicated just using the easiest validation just like string validation cannot go larger than that like I'm probably like the worst case is if someone sends a at dot a then that's four letters and then that's 100 something 125 e-mail address okay so but yeah like with this we actually stop those spammer accounts what's the stop them to just spam them I'm sorry because you're limited the length right but it still plays sense so basically what we do we basically detect try to detect like all these spammer accounts and then this we actually shut it down for because like we can see straight away from their account activity that they are actually not going to use the product or not going to pay later so yeah so we can basically it's still kind of manual right now it's like we have e-mails of all those graphs that you see earlier and our engineers get those and can see like oh that's certain strange activity we need to look at it so yeah I think like in our case it's recipient and cc but like potentially in your app all the fields all the attributes in your model where you potentially can send e-mail to you need to validate those so it's not sending so like it will not cause you to send like 1000 e-mails at once and then I think sandgrid has like a limit of 1000 e-mails in recipient field so they actually allow that but like I think depends on the e-mail service provider they might limit the amount of recipients in e-mail and then yeah so this is something that you need to be very careful about because this can blow up your e-mailing costs like if someone can send 1000 e-mails at once that means like if you're using sandgrid that's one dollar and then if it goes on and on and on again in a month it can go to like a lot of money I guess unless sandgrid shuts you down so that's not alternative which is also not pleasant so the last thing I have is just like track daily usage and then setup alerts be vigilant about it most e-mailing service providers they have like basic stats and alerts so this is in sandgrid you can they have stats like how many invitation letters you're like if you set up the categories you can basically get the stats for every category category e-mail so what we are doing is actually we're tracking all the e-mail that we send so using like the model that is backing the e-mail and then we're also tracking all the responses of the e-mail so when someone processes the e-mail or open or when it bounce or mark a span we have those data using so like with sandgrid you can get them through api as well but we're using the webhooks so we subscribe to their webhooks and every time those e-mails are being open or bounce or mark a span then we get notification on our end as well so this is just a sample of code so like using the sandgrid micro you can basically just on your config you can just set up the code back there and then this is where you just process all your webhook data and then save it so what we do is actually using this if we had this data before like we validate the cc or recipient field then we would have noticed that it was the recipient field or ccc causing the issues just because that means that for one e-mail then there will be like a thousand processed webhook so yeah okay so the conclusion is that you know transaction e-mails are usually very easy to set up but it is always I feel like it's always afterthought and you like you just set up and then like just because it's so easy to set up you just leave it there but like if you don't keep an eye on your transaction e-mails then you know like your usage your cost can blow up in terms of like e-mailing and then yeah these are the tips like you can go through them again it's like one don't rely on one e-mail service provider and then second just segment your users and then try to restrict them so they don't abuse your resources and then the third one is like back your e-mail with a model so it's easier for you to track and easier to validate and then fourth one which I think the most important one is like validate your recipient or cc field if you're sending e-mails and then the fifth one is like just track your daily usage and then set up alerts okay that's it any questions? I think this start happening this year like we have we actually have this like spam accounts that actually registers to our site and then they are able to create an order but then probably they're using some kind of like script on their browser where they just keep like filling in random stuff and then click send and then I don't know refresh and then click send so yeah this is happening I think since this year I think before that we had e-mails with problems with e-mails as well because I think the content are sometimes we have customers who are selling like phapes and the kind of stuff and then they try to send an order e-mail and then it says like oh there's phapes and then they think oh it's spam so we used to have that but like the spammers it starts like this year I guess we haven't quite figured out why though in honesty we have this huge problem where people sign up in your accounts to claim the first user e-mails so what we do is that we actually have a probability score based on your delivery address, based on the card that you use we have an updated probability score that oh it's very probable that you're doing fraud and then we just block your order so in your case it might be a good idea to do the same thing as well where you track like oh like the length of the two recipient like all these like different factors and then you can update your probability score and then you can block certain accounts yeah that's definitely a good idea I think most of the time you actually in terms of business usually having like false positive is a lot worse than having true negative I think that's for the longest time that's what we're trying to our mindset is like we can just handle this manually but like as you scale that kind of solution is appropriate also with the ESP how do you switch so you mentioned earlier that to use multiple ESPs how do you switch between the two is the second like a backup or do you distribute between the two just when there's our account gets deactivated for whatever reason and then it's just still manual at the moment for our customers it's very bad for them to not be able to send emails this is actually just temporary solution for them while we're actually trying to figure out why it's being sent and then especially with the support of these ESPs it takes a long time just because of the communication basically just like emailing them and then trying to figure out and then usually they try to give you this form where you have to fill in and then you send it back and then they get back like two hours after that that's just temporary solution for us can you explain a bit more about how the middle way work because the previous slide it seemed to me that you didn't do any communication from the common QA to standard 2 which one the middle way yeah so this is basically just well just sample code but usually you usually add the call back address with like the long hash right and then basically that long hash is the one that you put in and some I think some of the ESP supports like authentication so you can basically set like when you send this report but yeah like usually if you just put that very long hash in here just to make sure that we know that it's from standard yeah because I think it's important to make sure that's from standard otherwise other people can search for that extension in for reference in general I have a question, do you detect when the email the customer doesn't receive the email and if you detect for example because the server because of the poor reputation the server rejects your email what do you do then? we have similar problem and because it has a lot of pain because some servers just reject our IP because we were added to some black list yeah I think this is so I feel like it's an afterthought like email reputation is very hard to build and then it is very easy to ruin especially if this with these spammers so usually what happens in those cases is that like your reputation goes down and then like oh you realize some of the customers are not receiving the emails but like for those cases I think we're using like subdomains for the emails that we're sending so it's not like trichico.com but like s.trichico.com for example so like although that might not matter but like we can always switch the subdomains if you want do you have a pool of IP addresses that you're dedicated to? at the moment just one IP address for send grid I think that depends on the ESP as well send grid only allows one or like I think you can get more but for example postmark they handle it on their own so like you basically don't need to buy IPs basically warm up the IPs as well so they handle it on their own which is why they're also one of the most expensive ESP out there any other questions? Back in the day when I was working at a social network we had a specific email validation rule for g-mails and gavu-mails where you put a prefix on the back and you switch around the dots. There was a spamming case where there was a long email address and many multiple email accounts but just moving from the first to the last so if you put multiple emails, the user accounts just took that little hack and we had a special validation just block that kind of behavior. Have you guys experienced that? That's a possible attack factor so take note of that. One more question. Do you have models for each email that you send? For example order translation or refund etc. I guess that you send a lot of different transactional emails and do you have each of them backed by some model or is it just for the most push or pull? I think yeah we're using just for the most ones. Order is actually the most important one because if it breaks, our customer suffers but there are smaller ones. For example we've got order creation but someone cancels the order so is order cancellation still backed by the order model or is it a separate order cancellation model? It's an order so how we implement this is actually we pull out email templates so we have email templates classes where actually the content can change and so the parent model is still order email and then order email will have email templates and then we don't need to create one model for cancellation and for invoice or for shipments. Yeah so I think there are other emails that we don't send quite often so for example if someone wants an ECSV we send emails as well. That doesn't happen quite often so we haven't backed up the order model for us but for those ones that can cost lots of emails we do have one of those. No more questions. Thank you Gary. Thank you.