 Hi, I'm Giovanni Becchis, I'm an OpenBSD developer and mail server administrator for my own company. And this is a ten years story about how we move our setup from bare metal servers running in postfix to VM in high availability running both postfix and open SMTPD. with shared data. The story of the setup was on my SP some OpenBSD mail servers and they're running the classical postfix, Spamassin, Namavis and crew remap. And then during the years we found something should be changed to improve. One of the things was definitely postfix because complex setups are being too complex for many reasons. Due also because of Namavis that xlux proxy as its own configuration and so on and we wanted something simpler. Some of the things we didn't change yet is crew remap. We decided not to go to DoveCut for in the past it was mainly because we do not need some of the cool features DoveCut has. Now it's probably something we have to do but I know there's a script that should change, convert all mail deers to DoveCut formats but at the moment we have something like half terabytes of emails. So we need a very strict reason to switch to something else before trying to run a script on such mail. There was initially there was no absolutely no load balances, no share storage, just bare metal. It was more than 10 years ago. Some fixed pieces we want to not to move is OpenBSD because we think it's rock stable and being one of developers is something that I'm very familiar with. One thing is I'm not going to move from is Apache spam as seen because I want one of developer there as well so I do not have any reason to move to RaspamD or something, some other software. The first step towards moving to OpenSMTPD was now when OpenSMTPD has been committed to OpenBSD Source 3 mainly because we want to differentiate from the main customers, mail servers to dedicated machines to let customers send newsletters, marketing stuff and so on. We decided to try the software. The software initially was very simple, was just plain virtual setup like pass-through-the-style lines and it goes very well. So we decided to move to a better setup and to try to integrate on our infrastructure OpenSMTPD in a better way and to use it as a primary MX as well. The main problem at that time was that there was no filter interface so it was complicated to integrate anti-span software so we had to wait sometime till last year where filters were finally integrated in a very fascinating way and there are definitely production ready. On our setup we use in a web panel that's SP Configure. We modify the depth to support OpenBSD and the biggest part is just let the web panel create blowfish passwords. Otherwise there was some path hard-coded to fix and nothing complicated. Using the panel let us save data in a way that we can use the same data from PostFix for OpenSMTPD in a clean way and our user can create by on their own filters and whatever they want. New aliases emanate their web server. First things towards a real high availability setup was absolutely lock balancer so we start moving with PF and RELADE integration as lock balancer and to separate, to have more machines and then a shard and FS storage to have measures. The web panel uses MySQL as backend so we decided to go with MySQL master-master replica so all our users are replicated on more machines. The MXR couple will master and slave and if that's reached master-master replica continue to replicate and so we can both change user names and users can use their address books and calendars. We use Sogo as calendar server to have this functionality. The simple RELADE setup is just having an MX primary and a fallback and does a TCP check at the moment that just checks if the power is live or not and if it's not a fallback to the secondary and for the secondary the configuration is inverted so we can have one of the two nodes down and the whole traffic is moved to the primary or the other one. There should be in the future some improvements there because we would like to have not only check the TCP connection but also check the average queue of emails and some other things. We're looking into some SNP integration to have more information from RIMX servers to detect which is the server to move the traffic to. One important thing it was the replica in MySQL. This setup is possible also with Postgres. The web panel doesn't work with Postgres but you can do whatever you want if you want. In master-master replica you can set the connection between the master and the slave and it does work just with the first line of configuration file when you say start replicating from this log journal file. The second one is a new type of replica that's more stable than the previous. Master-master replications is first table on OpenBSD. The only problems we had was during some upgrade because there already be certificates. Master-master replica only on the same version of MySQL. They certificate for mixed version but there are a lot of conditions like officially supported operating system, full moon and a lot of other things. And so you have to stay to maybe it's wildly to stop replica before upgrading both servers and reinstruct later. For the main mail server we wanted a setup that's in which we can switch on from Postfix to SMTPD on a single server with a single command. So we decided to go to a configuration that could be easier on the SMTPD part. Maybe for queries also are some queries a bit tricky but what we wanted is that if doesn't need to switch from Postfix to OpenSNTPD and back for whatever reasons like security issues or like there are some configurations that are better handled on the Postfix part and other that are better handled on SMTPD part. So there has been cases in which for a specific customer it was better to move to also OpenSNTPD or Postfix and we decided to go this bit part for that reason. First lines are the certificates. As a start certificate you can use whatever you can use with your own Let's Encrypt or certificate you both somewhere. Then there are the definition of the table where your virtual user are, your virtual domain, virtual users and the credentials are the users that you use for start authentication to relay a bound email. The trickiest part is the MySQL queries because in SMTPD the user cannot be any mail address but on many situations this is not true. So we decided to go for a replace query and we decided that the underscore was not allowed or not stopping on for email addresses. This has to be done both in the user info query that finds where the mail address part is and in the query alias that is using MailV alias which is a view that does more or less the same things than the query user info. So this way it's the aliases are mapped into a fake user and then it's reconstructed later at the MDA level. Then does the entity spawn setup that does a lot of filters available, some filters we're using, some not. At the moment we filters on some regasps for the hello name and we check the reverse DNS. There are many other filters like for DNSBL, if you fully trust in DNSBL you can disconnect if it's coming from one of these blacklists. So instead of disconnecting the users there's another keyword that is junk so if the connection matches this particular filter the email is moved on the junk folder of the user and it doesn't check all the subsequent filters. These are very useful if you trust for example a particular filter and if you're not using pop free because otherwise the mail will not be downloaded by the users. So filters are not only to anti spawn setup related, they're related also to some other things like Deakin signing or reporting. The protocol is very easy, it's an ASCII protocol, it's not binary so you can write a filter in whatever language you want and it's very easy to write and make it work. The Deakin sign filter has a peculiarity because with the POSFIX setup we use Damovis and Damovis can sign but you have to have a key for every domain. In the Deakin sign filter you have one key for all your domains so it's a bit different in the setup from one to the other. Then there's a filter spam assassin, there's also a filter respondee if you're using the software that just connects to the spam assassin servers around the rules and copy back the results and the mail is rejected or accepted based on the spam assassin configuration. Then filters are applied so you can decide based on whatever you want which filters you do want. In this setup you have an address connection so the public interface where some filters are applied but it's possible also to change some things. So you can for example in some conditions apply some filters and in some other conditions apply different filters. One of the cool things is that for submission for example TLS authentication is required and there's no way to do a submission or SMTP relay plain text like it's possible on POSFIX or another software. Clam the integration is a bit special because there's more than a way to do it. One way is to use filter clamor but at the moment it's not production ready because it has some bugs and there are some races so it's possible that emails are mixed up. One to connect, concatenate it to another and not all viruses are detected because in the filter there's something that we still don't know what that changes the email content when sent to clamor maybe some end lines or not all viruses are detected this way yet so it's not something to put in production. Another possibility is to use a clamor called byspermassin. There are cool things to use there but something that's not perfect. The thing that's not perfect is that if a virus is detected this way the email will be rejected as junk as not as virus so the message that will be put to the upstream mail server will be not correct. The cool thing is that with this plugin that is not officially integrated yet but we think we will integrate maybe in 4.0 or 4.1 version. It's possible to give different scores to different signatures from clamor. So for example in clamor there are a lot of unofficial signatures that leads to phishing contents and macro viruses or some other stuff, bad URLs etc. and some of these are a high possibility to have some false positives. With this plugin we have the possibility to add different scores to different clamor signatures. So the official scores, the official signatures we are certain that are correct there will be no false positives and this way we can mark as with a high score. In the unofficial ones we can change the score or use other rules so we can check if it's listed in an official signature in clamor and some other conditions are met then we'll mark as palm as other way it will continue and will be accepted. As far as we know for S1D users there's not such feature available but I'm not sure. So this is something that you have to think about because integration with an antivirus is something that needs to be on a mail server. This is an interesting feature that it doesn't fully work with POSFIX for my experience. That's SOS, it's a standard rise scheme that is when you relay towards another external server with dot for word with an antivirus that's the problem that SPF is not respected so SRS rewrites the smith sender and this way SPF is respected and the mail goes. Gmail, Facebook and others have set up such that this kind of relays without SRS doesn't work. With POSFIX there are external mailters for SRS but in some cases when vacation is enabled it doesn't work and it triggers some loops so it's not something we've found a real way to make it work in a fine way. This way is definitely easier and is one of the cool things that make us move some of the infrastructure into open-send TPD. It just has to define a crypto key that's used internally and define that when you relay you have to apply SRS and that's all. One of the trickiest part is the use of mail drop. Mail drop is the sieve for Dovacot. It does the same things that sieve does for Dovacot so it filters mail based on the mail drop script language and it checks for quotas and so on. This is the part where we have to use a custom MDA because mail drop wants the user in the setup to be an email address but we have a user that's not an email address because it doesn't work. So the custom MDA just replace the underscore that we set up first with a SQL query and put the head sign back. It checks for quota and quota is available on a global manner so you have to specify 90-90% so that's warning when 90% of the quota is full. The email will be sent to the user saying you're quite full. The email is the user that we use for virtual delivery so the user that will exact the mail drop script and a lot of other parameters that I use inside mail drop to detect the sender, the dialysis and so on. To go with passing iscripts and doing actions. Once we define the actions we can define when those actions are applied. Like with match from any to one of our domains put it on their mail folders through the custom MDA or deliver locale if the mail is locale or if mail is authorized you can relay outbound with SRS in this case. Log files in SMTPD are a bit I think strange for respect to what other mail servers are. There are a lot of informations but it's not as easy to correlate those informations. I think this is something that could be improved and maybe a wrapper filter could up this way to create a custom log file to parts to use for example in the Kibana or other analysis tools. This is one of the things that's missing is that when for example Spun B detects the desert spam in the line there is no information about the users this spam is sent to. This is because the user that is passed to spam assassin is unknown in this case. This is something that is discussing on spam assassin. You can send it to the server a user that usually is the email address that you are sending to. This is used to have custom rules per user, custom scope per user and some other informations. As I said before there are different user by user on user cases not on alias cases or even on domain cases in some way. This cannot be done at the moment in the filter because the user where the email will be delivered is unknown at that stage yet. We have just the informations of the mail cells for the RSCPT2 but that information is not correct for spam assassin because for example the RSCPT2 could be an alias, could be a message from to a mail list so the information is not fully there. So some other filter like Mailter and Postfix for example try harder to check what the user name could be by looking into RSCPT2 headers and doing some tricks to try to detect what the correct user will be and passes to spam assassin. This could be done but there are some parts in the Mailter code that could be used to mimic the logic. There are a bit strange like it try to connect with some mail to itself to detect if the user exists or not. There are some strange things that are not that correct to do so we are trying to find a better way to find the correct user or something that is at least similar to the correct user and put it to spam assassin and this way we will have also in that line some informations about the user that the mail will be logged to. So at the moment log files are, there are quite all informations so you can look into but they are not created in a way that is easy for a log analyzer to check and to store into a web GUI like Kibana and so on. To do that we use the log stack file bit so we have some filters that analyze the log files and correlates with some other informations to produce those information and save them into Elasticsearch. This is done with the same filter for both POSFIX servers and SNTPD servers so we can have a global informations about who is sending, who is relay with which server, how many people are relaying for example with SNTPD or with POSFIX. We use that also to analyze which are the most hit rules in spam assassin. So we can decide for example if there is a particular new spam campaign and a lot of rules of the same rule that has a very low score is it that we can think about improving this score to block this campaign proactively. And it's, there are in log files there are informations to use geolocations so to detect even if a user for example is trying to send emails from different nations at the same time which is something that will probably be an alarm because it could very very probably be an attack of impersonation of that user. So the part that's missing at the moment is the spam assimper user setup. This is one thing that will I think will be fixed very soon and this is something that prevent us to move some users to this setup because it's something that some user wants and it can be done for some domains but not for all. We would like to get rid of the mail drop wrapper and use directly mail drop because it's the mail drop wrapper just fixes the parameters with some simple checks and then calls mail drop via exactly. So it's something we would like to get rid of and the only way will be probably to modify the web interface to let it save directly the user. So it's something that will probably touch more the web interface than mail drop so we have to decide which is the path to go with maintaining a matter of wrapper and let it do for example additional logging or something like that or change our web setup. One other thing that could be done is gruelisting. At the moment there's no official officially committed gruelisting filter but there is a gruelist filter that's been written by Gila that is a bit different from the typical gruelisting setup because this filter is SPF aware so he knows and records if a mail comes from a domain that has SPF setup so if a second connection comes from different IP addresses he can map these IP addresses in the SPF address space and let it go in without waiting, waiting and waiting like all classical gruelisting setups. This is interesting and it's something that I think could be implemented and should be committed to the official source tree. Then definitely a real AD setup based on more data this would need implementing scripts or SNMP or something like that to be able to decide in a more fine way, a more granular way which is the server that's currently alive. This is mainly because TCP connection works most of the time even if for example there's a high loader average and in this case TCP connection works so emails continue to flow to the wrong server and it's been wise to switch the server on another, switch the main queue on another server so the load average could go down and it could be analyzed if needed to. So this is more or less finished. We are going to move some more domains to this kind of setup and to improve that to be able to not get rid of postfix definitely but to integrate postfix with something that is mainly BSD based and more easier to configure and more easier to manage. Yeah? Do you have any other plans to do pre-queue filtering? This is pre-queue filtering. In the past we used the Amelies with open SNMPD like a proxy in post-queue so it's possible to yeah. Please don't forget to repeat the question. Okay. Thank you.