 Hi, how do you see me here? I'm so, I jumped right ahead. My name is Ovaliel. I'm a French guy, so it's pretty hard to pronounce. Don't worry about it. I'm used to people mispronouncing my name. It's fine. I don't care. And I guess I've been on the Federal Project for a long time as a contributor. And I've been in the Federal Engineering team since 2012. It's been three or four years. It's been fun. I'm working on Hapakiri. You've probably heard of it. Otherwise, you're just standing in this room, which is fine. It's unlikely. How many of you are using it today? Like on a daily basis? Maybe nobody. On a not daily basis, but sometimes. Like when you want to, okay. Cool. So, you probably have a lot of questions. And I have a lot of room for that. So, we'll have time to talk and discuss where we want it to be going. So, you probably know that maybe it's a problem that we have today. We have a lot of developers who are really liking the many mispronouncing it. All the filters like they want them to be. And who are used to switching between the male clients and their channel and working that way. And the workforce of most users is quite different. So, we have two populations of people who are not talking to each other. And that's too bad. It should be fixed. That's what we're trying to do with Hapakiri. I'm trying. So, this is what we're going to talk about. So, that's the status. And then I'll show you where we're going. What you have to do if you want to install it. Mainman 3, as opposed to Mainman 2, is a much, much more model platform. It's a very, it's a completely rewrite. There is almost no code in common between Mainman 2 and Mainman 3. It's a 100% Python 3 application. It's one of the first that was out. One of the first big application that is a, it's not Python 2 compatible. Just Python 3. It's a very modular design. Really interesting from an engineering point of view. From a software engineering point of view. There's a lot of good stuff about it. You don't get those monthly reminders with your password in pure text. Every month. That's, yeah, that was a problem. It's gone. So, this is good. It has a West API that is very expandable and very easy to use. There is also a Python library that connects to this West API to help you deal with objects instead of dealing with the same points and all that. It has virtual domains. So, you can have several different domains handled by the same Mainman servers. For example, in Fedora, we have Fedora project that lists that Fedora project.org and lists that Fedora hosted.org, which are handled by the same Mainman servers before we have to have two instances. And you have this concept of user that was not really present in Mainman 2. And a user can have several addresses, different email addresses. So, for example, if you are subscribed to a list with one address and subscribed to a different list with another address, you can have a global view of all your subscriptions and do stuff like I want my general setting for this value to be as such and have an override this value for no different subscriptions if needed. So, that's a really interesting concept for many lists. Mainman itself is just the main handling system. It only deals with LMTP and SMTP. It doesn't have a web interface as Mainman 2 used to be. Mainman 2 used to have everything in the same package. Mainman 3 doesn't do that. Actually, the Mainman guys, that's for the coding themselves, they realize that doing a web interface requires a different skill set. So, they say, well, I know how to handle a web, and I know how to deal with SMTP and all that, but I don't know how to make a web interface. I leave that out for the people to do. The web interface, the web admin UI is for Posterious. It's based in Django, so it's Python also. And it's a much more modern design than what we used to have with Mainman 2, obviously. But it is not the admin UI. It is the admin UI, it's not the archiving UI. Posterior doesn't do the archiving. I mean, it's the role of Hyperkilli. So Hyperkilli is a very small plugin that goes into Mainman and connects to a new email arriving for archiving and sends those to a web application, which is the main Hyperkilli codebase. It's based in Python and Django also. And it stores the email that is stored at the base, so it can be queried afterwards. It runs some analytics and statistics about them. And from the design standpoint, from the user perspective, it kind of looks like a web forum where you have threads and people replying to those threads. It doesn't look like the static HTML thing that was produced by Mainman 2 at the time. That's one of the things we wanted to avoid. You can do a full-text search inside the lists. The good thing about this is you can do a search across different mailing lists that wasn't very possible with Mainman 2 before, where you could always search Google, but if your list is private, then you're stuck. So you can also search in private lists. If you are connected, it will recognize which list you are subscribed to and will be able to search inside the lists. So it's really a deeply integrated search system. It has a lot of modern world features that you can see on forums. You can like posts, you can add tags, you can favorite some threads and keep them in your home page to get back to them later. And it is mobile, finally. The UI resizes depending on the size of your screen and puts the different widgets differently if you have a smaller screen. So it's not only getting smaller, it's actually really easy to make it easier. So this is the architecture. Mainman 3 is up there. It talks to SQL Database. We are using PostgreSQL in the federal infrastructure. But it can also use a very lightweight database. It can also use SQLite as a database. It is going to be able to support MySQL. The pull request is being worked on right now. So I think it will support MySQL in a short while. So it has this rest API that every interface communicates with. Of course it receives mail from your mail server. Today I think there is another mail server that is supported besides PostgreSQL and I can't remember which one it is. I think it's Xen, but I'm not certain. So the ID UI is PostgreSQL as I said before. It only talks to MySQL and the rest API. So everything that PostgreSQL does on the web interface, you can do it on the common line or using any kind of web client. That's for example a script that you would write that would directly cobalt with the rest API. If you want, for example, to an inquiry mailman for mailing lists or memberships, users, or create mailing lists dynamically from something else, you can do it using the rest API. But PostgreSQL doesn't have a database in itself. That's not entirely true, but it's almost true. It has just very sick SQL requirements that Django has. It doesn't have very specific models. Happily on the other hand it has a very specific database because it stores the database. It also has a rest API, but it's kind of lacking right now. Happily is one of the things that you want to improve on, having a better rest API for Happily. So this is for list admins. It can also be users if you want to subscribe to the list. Most of the UI is made for list admin traders who want to change the settings on their mailing lists. Happily is more for users who want to browse the API from the web, subscribe to see what the general trends are on their mailing lists, have an interesting meta-level to have otherwise. In the federal infrastructure right now, we have four servers. There's development server, that's in the federal client. I'm almost the only one using it at the moment. There's a staging server where I put the changes and there's one under two production servers. Currently we don't have a setup that will allow high availability between those two servers, but that's one of the things that we plan to do. It's the point using APIs. It's a web server in boxes. So I have APIs for a server that are really easy to install. When they are glued together with Ansible rules and templates, the thing is, since mailman can function without Posterous or HyperKill, it can work by itself, it doesn't need the web interface, and Posterous can be installed without HyperKill. HyperKill can be installed without Posterous. It's very independent, so I wanted to reflect that in the APIs. The APIs are very independent, but if you want to have them all together, it's better to write the configuration file as such and make sure that they are all the same Django instances instead of different training instances. So I have a couple of files in the Ansible rule that you can look at. All the things that we want is in the Federal Infrastructure and Git repo, so you can just check it out and see what it consists of. More of my web for mailman 2.0, 2.1, of course. It's currently supporting 2.0 domains. As I said before, there's Federal List.FederalProject.org and List.FederalPost.org. The migration has been very progressive. We started last summer, actually, with automating mailing lists, like the commits mailing list and something that aren't really user-talking to each other. It was less dangerous and also we could try and test them out very easily because those lists are very high-volume. But it's now been progressive and we've had actually setbacks and we had to rollbacks a few things. There were features that were not present in Mailman 3 at that time and weren't really used in Mailman 2.0 setup except for one list. For example, the topic system in Mailman 2.0, because it's not in Mailman 3.0 anymore, was a system where you could define filters in Mailman itself instead of having everybody set up the filters on their main clients. For example, you could say, this is the package allowance list and I said I have a pattern matching F23 on the subject or F24 on the subject and F25 on the subject. And people could subscribe to those filters instead of to the whole mailing list so they would only receive the email that they were originally interested in. This hasn't been reported to Mailman 3.0 as of now so we had to wait and actually switch the system for the migration and do something different. It's not supported anymore. If people want to get updates about new packages for different architectures they can just look at the RSS feed-in body and they are much easier to pass, of course, than having anybody send to you. HelloFilters was something also that was kind of not really ready in Mailman 3. You couldn't really set filters and have different actions. HelloFilters is something that was thought of for spam filtering. So this means to say, on top of the main spam filtering system that looks at the ex-spam header, I want to consider spam patterns like something in this subject or something else. In fact, because we have a lot of different usage and a lot of things that want to do specific things it was kind of abused into doing a lot of different things like rejecting email that came from an automated commit system if it didn't contain the proper header we received from or if it didn't pass through the right servers during the processing of the emails. It was very complicated and it was not. The people who work with Mailman didn't think about this kind of use case so I had to actually continue to make something much more general. So this is done, what it has sent us back a couple more. It was mostly done by December last year and migration is done in this frame, so it's very recent. I also have fast integration in Mailman in hyper-kiddy actually and also inside the moment. So it uses the open-editing logic that we have in past in Excel actually. Your first email addresses are retrieved and added to Mailman as a secondary address as before you can have several different addresses in Mailman. This is something that has been working at some point and didn't work for some user, I modified it and the people who logged in at first didn't get this and it's only on the first login that you get it so if you don't have, if you have to manually enter another email address it's not because you want one of the other people trying it. Which is good and too bad at the same time. So that's where we are right now. We want to upgrade to future version of Mailman but I don't know if the version 3.1 is being worked on. It's going to be out probably by September but we're not quite sure because it's summer and it's mostly there are very few people who are too full-time on Mailman. What's going to happen in Mailman for the next version is mainly a reliable migration process that I've been using Fedora to test. As usual, Fedora is helping some communities with approving them and helping them with their features. So I think it's a good thing that we pilot this in Mailman. It probably would have taken a lot more time if Fedora was in here to help this migration. There's also currently not an unsubscribed workflow so if you just send an email to the unsubscribed list it won't check that you're actually the owner of this email. Don't try this at home but you can currently unsubscribe anybody from the... I should turn off this squad cam. You can still if you unsubscribe anybody just by sending an email to the right address and spoofing their squad address but that's bad. It's going to be fixed. You will be able to edit the messages that are sent to your users when they log in and when they... not when they log in but when they subscribe to us. There's a lot of automated templates like welcome to the list. It's currently in the file system that you can only edit if you have a login access to the file system. It's going to be available on the web interface for the next one. And translations. It's currently not translated except it's not available in anything else but English at the moment because it's been moving about this past month and years. So, yeah, it's going to happen. I don't think we'll be ready for free. It's going to take a bit more time. The web UIs will also get improved lots in the coming weeks and months. We're relying on personal authentication. I don't know if you know about it. It's a service that was provided by Mozilla that will let you authenticate and check the validity of your email and use it eventually over services. For example, if you log into Personnel with a federal project.org email, it will send you to epsilon and fast for authentication. It will connect to Google for Gmail addresses. Do a lot of things except Mozilla decided to retire it. The service is going to end by October, I think. So we'll have to find a different try on something else. We're going to have local applications. It's something that we wanted to avoid because very certainly you have to have a way for people to be reminded of their password when they're using it. You have to make sure that it's an actual email who will trade the account. Do a lot of stuff that goes with local applications that we wanted to avoid but we're going to have to do it anyway. So it will be available by the end of this month, right? We want to have some time before personalized actually shut down. So translation is also a big thing that we want to improve on. So, quickly, it's not specifically for attachments currently. If you reply from the web, you can't attach anything. It would be nice to be able to do so. So that's something that should be worked on. It's not as easy as it sounds because it has to be done mostly in JavaScript and that needs to be imposed on... Oh well, it's not instantaneous. The REST API also did some work. We currently don't have many people using the REST API in HyperK. That's why it's not so fully featured. But with the coming of Hubs, it's going to be used a lot more. Hubs is going to have a HyperK widget where you can do a lot of what you're doing and you're really missed and maybe reply to people. The specifications are fairly defined yet, but it will be there. And that's going to talk to the REST APIs. There is also a need for system design available in mobile and single architecture. I think main end should be already working with a load balancing system, but it hasn't been ready tested yet, so it should be done before we lose that production, obviously. The components are really well separated. It should work. And as always, when you have new projects or very big new releases of existing projects, something that must be announced and mainly communities across the world need to be moving to main end 3. It's going to take a while. Red Hat is also very interesting in moving their internal main end system to HyperKID and main end 3. They currently have several private main end lists that need to be moved. They have more main end lists, but less traffic than we do in federal. So it's kind of similar, but not exactly the same. Drone matches would be nice too. Some people apparently like Drone. I've heard that somewhere. So if you want to have HyperKID on the system, there's a couple of things that you can do. You can use a small system called main end bundle, which is just a build out, a very small build out script that will bring all the dependencies together. It will provide them with a complete file that binds it all together and brings them all together, it binds them all together and in darkness... No, that's not the right word. It's not for development setups though, because it lives in a single directory. It doesn't trade less users, so if you want security, SNAP is not great. So it's not for development setup but if you want to try it out for me. I do encourage you to try it out for me. It's good. And for in-production deployments what we are currently using in Fedora is RKMS 4.7, or CentOS 7, of course. It does have a scenic support, so that's a good thing. It is protected just like main end 2.0s, and it works well with the scenics enabled. The migrations are a bit simplified than they were before. You have a command line that will help you migrate through this system. You can have main end 2 and main end 3 at the same time on the same machine. They don't use the same directories on them spaces, so it works. And you can use it the way we've been using. There's no reason why you shouldn't be able to do that. It is a bit specific to the Fedora infrastructure, but not as much. We have a rule that is only main end and hyper-killer-specific and doesn't depend on anything that we have in Fedora. In Fedora info. There are a couple of things that you need to watch out for, though, when you do your migration. You have to warn users that there has been changing in the way headers are done on the email send to people. There used to be this X being the header that was there, that has been there for a long time, but it actually has been deprecated since like 2010 or maybe nine, but since main end haven't had a large new release in that time, no one saw it. So other main end systems stopped using it and main end has finally stopped using it, but a lot of people are still using X being their header for their filtering. So now the standard is this ID. It's not exactly the same. It has brackets. This ID has a dot instead of the add symbol, but it's really usable. It doesn't have these topics. As I said before, this kind of server-side filtering is gone. So if you rely a lot on it, you'll do something else. Migrating data can be complex because the data structures that were there in main end 2.1 are really different. There are different ways to be subscribed to a main end list. There was a concept of non-main end list that was really a common case in main end 2. So migration of data is not one-on-one. So when you forward to migrate to existing main end 2 system, make sure that it's still working as you want to afterwards. Like check that the permissions are correct and the one on the list is allowed to send an email to the world before. I don't think there are problems, but since we've tested a lot the federal lists, but we don't have that many differences in our main end setups. We have mostly public lists, very few list-high-private, and we don't have very complex setups on the end-side. If you have something more complex, then the data is migrated properly. And if it's not, call me. I really need your help. Call me.com.com. There could also be performance issues. We have had some in the federal and federal courts. One of the main problems is that the main end-rest server that postal lists always communicates with is single-failed. So it can only answer one request at a time. Request should be slow, but when you have a lot of them, it can be a bottleneck. So we are thinking of putting some kind of proxy in between to really allocate that. There's also some caching done on HyperKey that is not as good as it could be because we don't... HyperKey cannot subscribe to main end signals at the moment. It's really separated components. For example, we don't get notifications. HyperKey doesn't get notifications when someone subscribes to a list. And since it's a check that it has to do almost all the time when you connect to the web interface, it is cached. But caching validation isn't really correct because we don't get the signal when someone changes the membership. So here is caching validation. Always an issue. It can lead to performance issues if you set the cache to low. It can be... Make sure you have MMCacheD server on your machine if you want to relate with more like 10 lists, for example. That will help. I don't know if any of you guys have straight mail servers at the moment. Nope. So if you want to work with all that, there's another thing that can be done. Mostly trying to approve and testing it or testing the federal system that we have. Well, don't send useless emails to the release code. People won't like you. There's a lot of Django optimization that could be done. Mostly on PosterEus. The guy who worked with PosterEus didn't have much experience on Django projects before that. So there are still things that were done. It was written at the time of Django what that was for. So there are still some things that could be upgraded to newer infrastructure newer requirements. We currently require Django 1.8 because it's a long-term supported version. And there are a lot of things that could be improved that I think on the code now that we have this higher requirement than before. One of the goals of HyperKD is to give you a quick idea of how your mailing lists are doing. What is the traffic on the mailing list and where it's going, if it's dormant or not. And that's something that we're always trying to get better metrics. The portfolio of the mailing list could be analyzed by looking at the number of people who work with it. It could also be a very non-relevant metric when you have a lot of messages and a few people who are actually sending them. There's a lot of thinking to do to find a relevant metric. That doesn't imply writing code, just thinking about it. We hope this idea at some point in HyperKD to try to detect freight patterns and do something about it. For example, when you have the thread where two people are replying to each other in public and it goes on for ages. It's actually very easily detectable in HyperKD. And we could do funny stuff with that. We could keep them in their own bubble, so we wouldn't disturb everybody else. We could do a lot of things. There are a lot of fun things that can be done with many software. Some people ask for very simple questions and everybody will answer the same thing at a few minutes in a few minutes interval. There are interesting patterns that could be detected and acted out at all. That could be fun. For example, it could add tags to this discussion. Yeah, there's a lot of funny things that we could do. It has been one of the ideas of the project for a long time but we won't actually be able to write something about this. So why not? The department, of course, needs to be improved. Currently we have RPMs for R7 as I said, but there's a lot of things that could be... Well, not everybody uses R7, R7 to R7. I know it's bad for that, but still. And there's also a lot of technical things that could be done writing tutorials or having ideas on design to help people understand my image better. People aren't used to my image today like 20 years ago and it should be made more accessible, that's normal, and it should be made more accessible. So our production services at this department are connected all and that's what you can solve. The first line, a mailman builder, is the dots for the builder installer that install everything all together which is also very interesting. First read if you want to deploy mailman in your infrastructure. And there's the tutorial on HyperKey but this one only talks about HyperKey. There is nothing about mailman. It's the installation of mailman itself in this tutorial, in this documentation. And the code is installed in the cloud like the rest of the mailman project. If you have mailman, you have all the mailman projects. Mailman is a new project so it can be installed in GitHub because it's not open source. But GitLab is fine. We have mailman, server, HyperKey, all the dependencies, not the dependencies, but the libraries that will help you access the server and all that. That's also where you can deploy or ask for future mailman. So I think that's the answer. Yes, it is. If you have questions, or if you have questions, I'm ready to answer them or IGs or anything. Yes. I was interested in monitoring the statistics and the standardization. So how does it work? There isn't really a proper plug-in infrastructure in HyperKey at the moment, but when there is an email arriving in HyperKey, it goes through this basic function that creates the object and everything and there is a signal sent, it's a Django signal, sent to the Django system in general. So what could be done is have a look at the models in HyperKey and so it's very straightforward that you have an email class with a content attribute where you have all your content and all that. And yeah, if you're doing a Django or a platform, okay. So it's not very hard. The only thing that you will probably need is to understand the ORM for what connects you to the database and the structure, how to make requests in the database. But other than that, there's currently some function that rebuilds the threads using the headers and reply to headers and it's a very separate function that actually uses... I think it uses NetworkX, so it's a powerful library to do networks and representations. This could be a good start to look at if you want to know how to handle like, make requests in the database and so on and so forth. But like, do under that HyperKey the API that you just made will be able to... It's... At the moment, it doesn't have really a program system with an API but you can get the signal and then do work on the email, on the object that you are there, like make requests. And then you can find the thread that is connected to this email and add tags to it. It's... I don't know if you've worked with a HyperKey or ORM before. You can say it can look the same. You have a tags property that you can add things to. But yeah, you should talk to me. I'll do it for you. That's interesting. So you think like, you should work on this now because right now I think we are developing lots of... Yeah. Yeah. I think it's quite independent from the hardware just. The thing is we're trying to have something that only is very remote from HyperKey to the REST API. And I don't... For example, you will get a notification on the REST API when the email comes in. You can only get this by connected to the signal in general in Python. So it will be different, I think. I can start with a... With a widget, with that. But it's going to be quite different for basis. It's not common, but I think it's way, way better. Thanks. Thanks. Thanks. It's actually the most... Mostly, yeah, I just think before, it's mostly designed as well, that they are really good. Making a website in jambles in the world that hard. Even in the... There are common cases because it's actually connecting to REST API downstream and making sure that... Well, it's actually a whole stack, but I think that's really, really interesting in HyperKitty is the design and the ideas behind it. And that's... It's a good thing that there's not a lot of code, means more people can contribute to it. But... My... Most of my work of these last months have been on the REST of the stack, actually, on MainMan, on the service, because that's where it was needed. It was more... HyperKitty was... It's been pretty stable these last months. I could base wise, I mean. So... Yeah, but thanks. Thanks. I think it's really good. That's what it is. Okay, so I'm going to stop the thing.