 Hello, Richard Bodden. All right. So, yeah, we have roomful people. First question. So I'm going to talk about hyper-killing. First question is, who has already seen one of my talks about hyper-killing? You do? You do? I know you do. Okay. Yeah, you do. All right. Cool. So a lot of new people. I will give a brief presentation of what the state of the mailing list is. It will be quick because you all know what it looks like. And then I'll show you where we're going to. So first one. My name is Aurelien. You can call me Aurelien or anything that is easy to pronounce for a newly speaking person. I'm in Paris. I've been contributing to federal for a long time. And I've been in the engineering team since 2012. It's been three years. I'm the lead developer of HyperKitty. I'm trying to help fix the mailing list problem that we all know about or guess at with the help of a lot of people here, like this person here, and her numerous interns that have been helping me over the years. For example, Karen. She was there last night. She's a very interesting person. Work on the design. So that mainly consists of development of HyperKitty, writing the code, deployment, mentoring people, communication, which is not my best part. And all the rest as usual for a software project. So this is what we're going to talk about. I will try to give a demo of what we have right now if the network does not fail, which is always the case at a conference. So we'll see how it works. So MadeMan 2.1 is the current version that we are all using all over the world. It is a MadeMan branch. It has seen one release a year, about one release a year since 2006. So it's kind of outdated. It's a very monolithic package that was designed in the last century. So it has a mailing list engine. There is a web archive that produces static HTML. It has a web admin interface that is CGI based. It's a lot of 90s era technologies. The problem with that, yeah, I've just said it. CGI is not very nice for admins. They don't like it a lot, especially in the AC Linux days. They hate it. HTML, well, static HTML, we've done better over the years. And it's a lot of different environments and skill sets that are necessary, a lot of different technologies. So the people who write that and maintain that don't know all the ends of everything that they are dealing with. So it's kind of hard to do. MadeMan 3.1 tries to fix all that. It's a much modern and modular design. There are a lot of improvements that you will see when we switch to it, which is soon. First, you will not get an email every month. Yeah, no more happy mailman days. Yeah, exactly. You will not receive your plaintext password every month. So that's even better. For the server-interested people, there's the REST API, which is very extendable. You can do a lot of things via the REST API. Actually, the web admin interface only uses this REST API. So everything you can do in the web interface, you can do via the REST API. It's very easy to work with. It handles virtual domains. So you can have one server that handles fedora.fredorproject.org, or Fedora InfraCloud, or Fedora Hosted, or Fedora Anything, and all the domains. Pager.io wants to go into that, too. So we can have virtual domains very easily. It handles the concept of having one single user with multiple addresses. So you can be subscribed to different mailing lists with different addresses, and it will reconcile all that and let you handle your preferences globally. It's been released. It has been released this year, in April, just after Picon, which was in Montreal. It has been released in a form that is considered good for new installations. The mailman team right now does not feel like the three.org release is easy to upgrade from the mailman to the one. But I've been working on a lot of that upgrade procedure, and I think for Fedora we can do it. And we've been doing it recently. So hi, Pager.io. What is it? It's an archive for mailman3. So the idea is that in mailman3 you have this mailman core that is dealing with the routing of emails and the distribution and the handling of error and all that stuff. And you have two web interfaces that are different. The admin interface is connecting to mailman via the REST API, as we said before. And you can have any number of archivers that are all independent. And the main one is hyperkilling. So it's just a plugin, but it's a big one. And you can have other archivers. Or you can write plugins that work like archivers, but actually do other things. Ralph here wrote a plugin that gets the email from... Yeah, that's his guy in green. That gave the email from mailman and routes them as Fed messages. So you can do that too. It's very easy to extend. Well, you are the god. Hyperkilling is a web app. It's based on Python and Django. The emails are stored in a SQL database. It runs some analysis and statistics over them. And it looks like some sort of web forum, but not only a web forum. It has a full-text search engine that can search across the list. So you don't have to... If you don't know the exact mailing list where an email was sent, you can just search globally and you will find an email. That is very interesting for private lists, which are not archived by global search engines such as Google. It has a lot of modern web features, like favoriting posts and liking, disliking, and all that stuff. And it's more by friendly. You will see that. So I have this nice drawing of what happens when you receive an email in Mailman 3. So you get the email from POSFIX, for example. I think that Xim is supported, but not as well as POSFIX right now. Well, we use POSFIX anyway. It received an email via LTP to Mailman 3 that has its own database. And the emails are stored into Hyperkilli via the archival API. So Hyperkilli stores the email itself into its database. And then you have the friendly users who go into Hyperkilli and request the status of the mailing list on the archives. Hyperkilli has its own REST API that you can interact with with trips. If you want to get the emails or get information such as what email was, which was the top posted email this week or this month, who favorited, well, what was liked a lot this month, this kind of thing. Then you have POSFIX VM in UI, which is an UI for other people for REST admins. And then both of them interact with Mailman 3 via the REST API. And of course, you can have your friendly users admins who write scripts to interact with the REST API. So that's the main center of action with Hyperkilli. Yes? The REST API, is that just for administrators or do the end users have access to the REST API? This one here? This one here is only for administrators or for the local administrators. Okay, how about the one on Hyperkilli? The one on Hyperkilli is authenticated in the same way as Hyperkilli so you get the same permissions. You can use that. That's solid. But you can do different things. But I can query it and say, hey, what was the most hated email this month? Yeah, you can do that. What was the most liked email this month? Which email might have been tagged? Right. It hasn't been a real priority, I would say, in my development plan these days. But the possibility exists. Yeah, yeah. I can prove this. Right. No, no, good, good. If you have questions, interact with me. That's fine. I will get back to my talk. Alright. I'd rather you interact with me than you store your question for later and never ask it. But they're playing it now. For new instances of Main Main 3, well, I've written a small vendor script that will install Main Main 3 and Hyperkinian Postories at the same time in the same environment, virtual environment. It's all very well integrated and ready to go. So you can do your proof of concept and install with that. It's working and it's what the Main Main team recommends to install a new stuff. Of course, for real production deployment, you will probably want to have something more isolated and more well, that you will integrate with your own infrastructure. So you will probably want the separate projects into separate environments. I have RPMs for L7 that are working. They are also working with the Selenix Enable, so you can use that if you want. I have RPMs for Fedora, but those are a bit bold. I need to update them. The migrations are really simplified for real production deployment. I have a few scripts that are available. I have written some prebooks for the Fedora infrastructure that you can reuse. Most of the Fedora-info specific parts are exported to variables, like the southern names and the database names and all that stuff, so you can use this. I'm not sure you can just take them and apply them, but it will be really easy to adapt. So that's what it looks like currently. It's just a screenshot. I'll show you the real version. That's the main page. It's not the most... Well, it's still a very big improvement of what we had before. At first glance, what you can see is that you have an idea of which lists are the most interesting and what activity you have in them. Well, this one here had a big spike of activity recently. This dev-all list has a lot of activities, but even me just reviewed it, and you have all those sorted sorted by what is the most popular. You can sort by different criteria and you can filter them. By default, you will hide the inactive list, so you don't have all those inactive lists cluttering your screen. And you will hide the private list by default, if you're not logged in. So you don't... Yeah, it's protected, but you... It will not mess up your view. This is the front page, and this is what it looks like when you go into... when you go and visit it with a mobile view of your phone, for example, or a table. So this collapses nicely into something that you can use. And I'll show you more in-depth demo, but this is interesting to see if you want to use it. So you will be able to browse your mailing list with your phone, yay. Currently in the federal infrastructure, we have four server deployments. There's a development server that I'm using. There's a staging server, and we have two production servers. We are deploying VRRPMs because that's our policy in the federal infrastructure. And we're using Ansible Playbooks. As I said, it's mainly using roles, so you could extract those roles and use them. Oh, no, I don't have to process them. If you want to try Hypokitty and check it out, you can use the staging server. This address here. Well, that's until your favorite mailing list is migrating, which is going to happen soon. It mirrors the federal mailing list. It's a manual process right now because I don't want to have something automatic SSH-ing into a production server from a staging server. That will not be very nice. So it's manual. So sometimes you will see emails. Well, you will see that the latest emails were like a week ago because I didn't do them. I didn't run the script, but at least it's a good way to test it again. The production servers are real functional mailman 3 instances. There is a test mailman 3 list that you can use if you want to just try it out with no consequences. And I have written a few scripts to transition from the old servers to the new ones. So when your mailing lists are migrated to the new server, I will first notice you and then you will see that the transition is actually you will get a lot of redirections that will avoid you being lost between the old server and the new server. So it should be pretty transparent for you. And there are a few lists that are migrated already. Most of them are automatic lists like SCM commits and commit lists. But we have the infrastructure list that has been migrated very very recently, like last week. It's working and people are actually communicating with it and it's working. I've been migrating a few lists over the last weeks and the only roadblocks that will fit is a feature that is not yet into mailman 3. It's a feature that is called Topic and it's very we don't use it a lot. We use it only for one list to my knowledge, which is package announced list. It's a feature that will let the list admin define sub topics based on the subject of the emails and people can subscribe to those topics instead of subscribing to the whole list and they will only get emails that match the topics that they have selected. That's the work of mailman 2.1 on the sisters list, right? Yes. Yes. Sorry to miss this one It's a closed lunch So if a group wanted to migrate to mailman 3 now or you're starting up a new list, what would be the process for getting on mailman 3 instead of mailman 3? Well first I would recommend using the bundler script which is called mailman bundler. You will find it. For a list? I'm excited. We are all in love list. What's the procedure for moving? You talk to me. I have a very small procedure to migrate lists and I just do it but yeah, you just ask me. I'll be very happy to have new users. Yes, yes, yes. So yes, this topic features is not very widely used in the structure but when I migrated this package list a lot of people were complaining that they were getting all the emails for all the released version of Fedora and they were not very happy about that so I rolled it back and I'm waiting this feature will be implemented as well differently as a result of a Google smart project this year so it's going to wait a bit but we can migrate all the other lists and wait for this What's done right now is a lot of comment lists as I said I think all of them and the infrastructure list There's a few things to watch out for when you migrate lists First, the X being their header that was added to emails by Mayman 2.1 is not added by Mayman 3 anymore because it's been deprecated for years but no one knew it since there was no release so this has been replaced by ListID so if you were doing your filters on X being their you would have to edit your filters and use ListID instead It's a fairly simple thing a lot of people from the infrastructure have done it already Most webmails use ListID anyway so if you were using them or I think Gmail uses ListID too it's just like a lot of people who are using prog mail and while manually filtering and X being there things change over 15 years Yes and there is no list topics yet but it's going to happen probably in September when the Google Summer of God finishes So what we're going to do next we have a few technical improvements to do we were doing our CSS with less and it's getting a bit a bit it's not outdated but SAS is really getting traction and less needs requires us to have a note on the server which is not very nice not JS so it's some administrators are not very happy about the list of dependencies that comes with it so we'll probably change that we're all using bootstrap as a framework for CSS so it should be not too hard to change because bootstrap supports SAS there is a new attachment system that I'm going to write currently you can add an attachment where you reply from the web interface did I mention that you can reply from the web interface I probably didn't mention that so that's one of the nice improvements of HyperKitty over the previously static HTML that you can actually reply from the web you'll see you have a reply button that you can use and post your emails from the web but you cannot attach anything right now so that's something that I'm going to add next that's a feature that I've been requested a lot of time but it's more complicated but it looks the previous main main web interface allowed people to download the inbox file for the list which is just an inbox file so just a mailbox file with all the emails in the list this has not been implemented yet into HyperKitty because since all the emails are into the database it has to be built on the fly and can be a pretty hard load on the server so I want to make something a bit asynchronous that will not cause a lot of load for the server so it's taking a bit more time than expected and then since it's all out I should help people, well we should help people install it and so there are a lot of main main communities across the world that will be interested in that I've been contacted by the Apechi Software Foundation by a very large group of people who use main main in France too some organisations are really eager to move to main main and Red Hat of course is interested too No, it's still What? Gloom is? Okay, cool, perfect More people, yeah and some people will also be happy to see images virtual images that set up main main and HyperKitty and Pistorius in a nice container that could be used I've had someone propose me to write a Docker image for main main and HyperKitty but it doesn't have any results yet and I think the guy didn't reply back to me so if someone wants to do that it's over so on the technical side you can help by testing the thing of course trying the installation and the strange thing that I haven't thought of which is probably a lot of things if you know Django very well you probably know of a lot of musicians that I haven't yet put in to use there is a lot of statistical analysis that is very interesting to do on mailing lists because a lot of people use it a lot of people use mailing lists for very strange personal and social things very interesting to analyze people talking to each other on public lists it's always very interesting to analyze for example that if you have an idea about if you know virtualization and Docker very well who would be interested in that too but it's not only about technical issues you can if you have design ideas to improve the main use feasibility you will be very interested you still have a lot of ideas on your website but what we can do but if other people have other ideas and tutorials for installation it has evolved a bit the installation has been made easier and easier but it's interesting to have tutorials and it will probably make people more comfortable so now to the demo yay! if the staging is written that's the address if you want to try it we have documentation here and that's the new that's the place where we have the code with all the rest of the mailman project so that's where you'll find the mailman bundler project too in the GitLab mailman group so I'll show you that move okay oh that is kind of our idea oh yeah that's old I'll show you the real one it will be even more interesting I'll show you the infrastructure one so that is the infrastructure list that I've been migrated recently I don't need you so at the glance you can see if there were big spikes of posts recently all this is data over the last 30 days you can see who is the most active posters and how many posts they have sent what can I say about this I'm logged in with my email address so I can go directly to the discussion that I have posted too which is very useful all this gray envelope here shows the topics that I haven't yet seen so if I go to this one here for example I can read this topic and all the new emails and all that stuff here and I have, if there are new emails in this thread I can go to them directly using this key here I can reply to this discussion but I'll go to the other one move to mailman 3 so I can say that this has been inactive for a day and it's a discussion that is 7 days old that's also useful and for example if I want to add something to that I can reply to this email here say hey guys I'm currently you move oops say hi so I could I can reply directly in line if I like to use my email fan barrier I can just click here and it will generate the proper link with the proper headers that will actually put the email back into the thread I can create a new thread if I think that's more relevant and change the subject if necessary I can send that, yay new reply has been sent and since that was so extremely useful and I will like my own posts and now that I've liked a post I can I can go to my profile for example see which thread I have read over time see my votes this post that I've just liked or cancelled see my subscriptions see my favorites, oh yeah I didn't show you that I can favorite this thread let's go back to it I can favorite the thread and now I will get it back I will get it in my favorite list here so this is very, yeah it's not inactive yet it's not active anymore because I've just sent an email into it oh yeah yeah what else can I show for example when I hit the reply fast has my email address that is not what I use after a project that I run so it's subscribed I don't know if this is good or not but it's subscribed by fake email address which is just I never give it out I just use the project as a forward and I have that at the same time which is nice because you don't have to go to the subscription part but you'll probably have to go back to your subscription list and edit but it knows that I'm me so I just have to change the address and you can go to subscribe to this list but I'm here and change the which address you're subscribed with there's so much to just be able to post because I can never remember so yeah when you reply to a list that you are not subscribed to it will subscribe you with non-delivery option so you will not get them into your email it supposes that you want to check since you've been into the web interface you want to just use the web to browse this list so that's easier when you just want to interact quickly with the mailing list and you don't want to get the flow of emails if it's a private list of course it will just say something like your email must be or your subscription must be approved or your email must be approved because you can't be automatically subscribed to a private list of course so there are a few corner cases but it usually works yeah you can add text to a list say oh very interesting and everybody can add text to it can make it easier what else can I oh yeah searching so on the mailing list we've been talking about Flock search for Flock and get a list of mail that are relevant to Flock it's sorted by relevance it's of the pen and text search engine but you can search by date you can order by date if that's easier let's go back to the list of all of you oh yeah I'll show you the mobile view you can see you can see it collapse when the screen size changes into something that you can actually use but that will be you will have the same categories here and you will have tabs for the statistics and this will collapse into a menu that you can use too so currently it works the only problem that we've had is the X bin there header that some people use there's also a minor bug that sometimes blocks mailman free but I've found a way to reproduce it this morning so I'll fix it otherwise it's all good and if you want to try this new interface just ask me come to me and I'll move your list yeah that's about it we still have a lot of time so if you want to see things and try things by yourself yeah you can say that I don't think we have we will send an email to the list first and explain what's going on and see but we don't have well the idea is not to get a big date where everybody's switching at the same time we're just moving the list one at a time we can do it and I think it's better than having a big day switch where everybody is concerned and surprised and more people I think it's better to get one list at a time and see what that goes and then we can this way we can see which lists aren't very active and aren't leaving behind yes that's a good question I think we should yes there isn't something about that at the moment I think we should send an email to the whole list you can file a ticket if you want to move your list infrastructure infrastructure I think yes yeah we can do that I can extract the list of lists that have been migrated and put that into a wiki and keep that to date yeah that's easy to do yes I haven't had a UI idea so we were just talking about which will be really nice because right now we have a lot of discussions and then somebody we don't tend to put too many attachments because our graphics tend to be very large so we'll put a link to image or something so we have inline images so we'll post a link to an image it displays in the conversation yeah why not it would be amazing there are a few things to make sure of make sure it's not a huge image there they just try the layout but why not that could be something at the bottom of the emails like inline images can you send me an email about that okay on this you can put something like markdown in the program so that if email happens to be written in markdown you can write it down that's that's a kind of worm that I'm not sure I should open the question of rich formatting is very touchy for a lot of people in many lists some people have told me you'd better not make an html editor into that thing because otherwise I will never use it so I don't know maybe I don't know then trying to detect markdown maybe I don't know that could be fun though but I don't know maybe something like those JavaScript plugins that inspect areas and turn them into hashtags if they find it if they find something interesting you know what they do with the code I think it's a minefield but we can try maybe later maybe later you have a question? however you would how can you solve this I know we have two problems piracy designers don't understand either if they can just go to the website and just talk it's going to solve some of their problems inlining images that should be not too hard can I take a look at it and then using zoos and plugins for pasting our pictures it's always just have a look at it you want to use a base one you don't like they'll give you a directory and then there's like 10 things which one do you want me to talk about and if it's because there's I'm going to say this and people are going to think I'm awful I used a Mac once 10 years ago and I had this IRC client called Colopy and the one, this is like the only thing I've ever liked about anything Mac is this one client in an IRC channel there's channels you don't want this to happen but if somebody posted a link to an image it would just load it inline in the channel there was so much easier to look at all the conversation and it was just dead simple it was just if it ended in .vng.gif or .jpeg so it doesn't need to be anything complex right it shouldn't be hard if you want to store it on the server it's not really I can't store something I don't know I'll see maybe it's not necessary to store it on the server maybe it is but for our purposes the link is still there if you walk and you throw a 23 so you paste the URL you cannot see in the picture I get to see I always have downloaded which version was it from the dark the last one we're always just drawing out ideas so we don't really need a historical record of what was the 20th version of that thing we did so if we go back to the archives and try to look at it other people might want it but it's just a web link so if the external host goes down the image gets just deleted I don't have to cache it cool yes French I just wanted to say there's a security issue with that where it's not going to make you do an image it's not really an image and it causes a problem with the track so you have to get around it by copying the image to servers and make sure on the servers the image under the tracks you wouldn't get it so you don't want friends to do that they would have you could make it pull the image and host it I guess my concern with images is somebody people are viewing through the interface somebody might start going like this and then go and see but we have moderation stuff so we can manage that it down can you delete an image or something an attachment to an email so it's not displayed for this for this list we turn on the the design list wants that but then somebody subscribes and trolls it's their way to go through where are we at with moderation stuff right now there is no high-pitched specific moderation so it's just payments we wouldn't need moderation do of course we have the design list one person high-moderate each may have that so there's moderation that's imposterous yes, that's for those so if he has this guy on moderation it wouldn't show up and have your kidney until he proves it's imposterous you wouldn't need to approve the emails anyway so would you have bad people people would be hate battling you put them on moderation it's not going to stop written standards but if you have good girls set up in place by default it should be okay for your future future moderation it can become a bit harry I've seen this happen before with our mailing list they'll send something to a mailing list and it's innocuous there's nothing wrong with it they'll get very upset because then it gets archived and they want us to pull it down so there's a way to delete a post I don't think there's currently a way to delete a post once there's other servers we cannot there's no way there's currently no way to delete a post in high-paquity there's always an exception traditionally there has been our base down here is there anything you can show you maybe there's a regular month-based view of what is happening to your list this thread I have favorited so that's okay there are new emails really oh yeah, hello thanks I want to see everything that has both I don't think that's I don't think I can do that yet there is no advanced search yet where you could have multiple criteria but the tags here are sort of we can click on them and you would wait oh, really huh, well this one I can delete because I have said it well, that should okay, strange alright, the tags here you should be able to click them and get a list of threads that have this tag I'm not sure you can do both you can do multiple tags at the same time good question, I don't know maybe, I don't know but it's easy to do if necessary yep yeah I'm kind of what are your dependencies on that what do you need on the server to have this long-running right right yeah okay I agree with it would be interesting I have currently not added it because of the dependencies and I wanted to make it easy to install first and it should be optional, I agree with that yeah yeah, that would be nice that would be nice to have everything show up and update dynamically yay, cool anytime, do you have time to cut that go ahead so yeah, that's about it oh yeah, I can just answer your question by searching this new interesting tag do I get it that would be by date, of course nah, I don't think it doesn't look like I get it but it should be in the search criteria that's very easy to add I don't trust my memory at all yes that's just our little ids are good too, you don't have to implement everything but yeah, working ids it's even better yes, something else, when I go to my it's kind of hard to do that right now but if I go to subscribe to this list here I will get into I will get a nice error message no, I will get into this I get into Posterious, which is a web admin UI and this here is, well, that's the view of the list that you will not see because that's for admins only but you can go to your account my setting here and you can add different email addresses to your account so if I'm logged in with this this one here I can add my other email addresses and when I see my subscription when I look at my subscriptions it will collect all the subscription with the different email addresses that I use I have this one here but I also have this one here that I use to subscribe to different lists and I have an historic address that is not showed here you will get three emails you will get three emails in that case I don't know oh yeah, I know, the one you're logged in with but but I think if you subscribe with two addresses I think you will get two emails because, well why would you want to subscribe with two different addresses otherwise yes, you can still do that you can still go to your subscription settings and disable mail delivery yeah right yeah, so you can still do that let me get back to the headers have changed is it the xbinther header? I think these are because we're using a matching to relay the emails oh, that's possible no, we're not using mailman 2 anymore I don't think that is well, I'll check it out okay, I'll check it out okay, so if you have no more questions well, you can always come talk to me about this even if you want your list to move just ask I'll be very happy to try to do my new stuff on your lists sorry? yes yes, absolutely absolutely and you can set hyperkitty to only show on the front page the lists that are specific to that domain if you want to so yes, you can have multiple domains, that's one of the main improvements well, thanks for your attention yeah, right that's enough