 Okay, up next we have Peter Tzonic with what is new in Pseudo and Syslog NG. Hi, I'm Peter and let me say a few words about myself. I work at One Identity, which is upstream for Syslog NG and in a way also for Pseudo. Well, Todd Miller, the maintainer of Pseudo is my colleague, but Pseudo itself is not One Identity software. I help with RPM and FreeBSD packaging of Syslog NG and I'm blogging about this software and speak at open source events. So let me give you a quick overview of what I'm talking about today. I try to define what is Pseudo, what is Syslog NG and how they are related to each other. Then I will mention a few version numbers, which versions you should use if you want to test what I'm talking about here today. Then I will show you a couple of new Pseudo features released in the past one and a half, two years. And at the end, and I will also show you a couple of Syslog NG configurations related to these Pseudo features. And at the end, I will introduce you to Syslog NG for news, which is expected to be released in the coming month. So what is Pseudo? I heard quite a few explanations from users as I asked it a thousand times at least at various conferences and events. And many people told me that, well, Pseudo is just a prefix for administrative commands. So here is a definition from the Pseudo website showing that it's a bit more. Pseudo is a system administrator to delegate authority by giving certain users the ability to run some commands as root or another user by providing an order trail of the commands and their arguments. So even this one shows that it's a lot more. But you will see that there is a lot more to do with Pseudo. Here is a very basic Pseudo. This is what comes installed by default on most systems. Here, members of the real group have practically all right on the given host. Various fields here means who, on which machines, which users, and which commands are allowed. And you see that practically anything is allowed for members of the real group. But even this very simple configuration is pretty useful as if you have more than one administrator, then you will see who did what on a system. So unlike when you shared the root password and all you see in your logs that root user did that or that, but nothing more who was actually running those commands. And so what does Pseudo does? It can control and log access. You have seen the previous slide that you can control who does what. And it is nicely logged on your system. But not just that, Pseudo can record everything what is happening on your terminal and even what you are typing in a Pseudo session and play it back just like a movie. And Pseudo itself is modular, which means that most of its features are implemented as plugins and you can replace those or extend with your own code. Recently even using Python. And how is Cystogengy is created? Well, it's developed at the same company, but also Cystogengy is parsing Pseudo logs automatically and creates name value pairs from the different values in the Pseudo log messages, which makes alerting from Pseudo logs much more easier and also storing the Pseudo logs into various SQL databases. So you can easily search what has happened in Pseudo sessions and create reports and so on, which is not so easy if you do not parse the log messages. I mentioned that you can extend Pseudo. It has various APIs. My favorite one is the IOX API, which gives you access to input and output from user sessions. I created a couple of Python examples. One is a simple data rig prevention that Cystogengy, Pseudo, is checking what is wrote to the terminal buffer and terminates the Pseudo session before it could appear on the screen. Or you can also implement command line analysis using Python and terminate the session before something nasty happens. Here is a very simple example code. It's just five lines or six. And what it does is that it's checking the terminal buffer and if a given text, in this case my secret, appears in the terminal buffer, then it writes an error message and terminates the Pseudo session before it could be displayed on the screen. And it's really just six lines of code. So what is Cystogengy? It's an enhanced logging demo with a strong focus on portability and high performance central collection. It was originally developed in C, but now it can also be extended using Java and Python. So practically any features of Cystogengy can be extended using Python. Here are the four major roles of Cystogengy. Using Cystogengy you can collect log messages. It has various drivers for platform-specific sources, so it can collect log messages on all of the BSD variants. Of course, Linux also the journal. And you can implement your own source drivers using Python. You can process log messages like parsing JSON messages or free-form messages using Python, DB and many other parsers to create name-value pairs from log messages. Those enable you easier filtering of log messages. You can also rewrite log messages, and you don't have to think about falsifying log messages when it comes to rewriting, but for example, an anonymization of log messages like removing user names or IP addresses or credit card numbers as required by PCI DSS and so on. You can filter log messages, which means throwing away surplus log messages and routing log messages to the right destinations. And finally, you need to store log messages somewhere either locally or forwarding to another Cystogengy server or right into your database, Elasticsearch, MongoDB, and so on. So there are many possibilities, and if you do not find the driver for a specific destination, then you can either use one of the generic drivers like the HTTP destination, which is used by Elasticsearch, Telegram, Slack, and other cloud-based destinations, or you can write your own destination as well easily in C, Java, or Python. What I show you in my talk requires pretty recent versions of SUDU and Cystogengy. On the SUDU side, 198 is needed if you want to use sub-common logging, which I will show you. And on the Cystogengy side, at least 3.31. That's the version which can work with JSON-formatted SUDU logs. If you want to test Cystogengy 4 features, then you need at least 3.37. We also have already 38 released. The good news is that the various BSD ports are pretty up-to-date. From SUDU, all of them have the very latest version. And for Cystogengy, mostly as well, free BSD is always up-to-date, and the others are usually just one or two versions behind, so they're also mostly quite up-to-date. A bit history of the Cystogengy port in free BSD. It's in the free BSD ports since the year 2000, so just when the Cystogengy project was two years old, but it was not really well maintained. In 2011, I helped to update the Cystogengy port from version 1 to the version 3 series, and I worked with the port maintainer quite close and ever since, which means that usually free BSD is the first one to receive a new Cystogengy version, often a lot more earlier than any of the Linux distributions. In 2012, I was at Fostam at a talk in the BSD dev room. I do not remember any more which appliance was giving a talk there. Probable freeness or BF sense, I'm not sure anymore, but that was a talk about extending free BSD-based appliance with various features from ports, and after the talk I went there that, well, using the description you gave in your talk, installing Cystogengy is not possible, and then why? Is Cystogengy something interesting? Yes, it's a nice logging software. I'm part of the team, and I would be happy to see Cystogengy on the appliance discussion stopped. I have never heard back from the guy again. On the other hand, half a year later, I realized that it appeared in the first free BSD-based appliance, and soon practically all free BSD-based appliances had Cystogengy inside. It was all just a discussion after a Fostam talk. In year 2020, I had an experiment that, well, I'm helping with free BSD, but there are quite a few other BSD variants as well, so I took a look at various BSDs and found that quite a few of them are not really up to date when it comes to the Cystogengy port. So I helped update the various ports to the latest Cystogengy version, and also helped to merge Cystogengy fixes from ports, which means that right now, as far as I know, you can compile Cystogengy on any of the BSD variants without needing any patches, which makes maintaining Cystogengy a lot more easy in the various BSDs. Here is a not really politically correct cartoon from XKCD, which describes how most of the users imagine Sudo is working. Make me a sandwich. What? Make it yourself. Sudo, make me a sandwich. Okay. So if you are not really offended by this little cartoon, I have a couple of stickers with this and also the Sudo logo which is based on this little cartoon. I have a couple of stickers with me. So let's go to the latest features. As I'm coming from the Cystogengy world, my favorite new feature is about logging. Starting with Sudo194, you can turn on JSON-formatted logging in Sudo. And now comes a kind of zigzag, as I will also show you a couple of Cystogengy configurations, how to configure Cystogengy for Sudo log messages. This is our starting configuration for Cystogengy. Well, it's not the whole configuration, but the part for Sudo logging. As you can see, the Cystogengy configuration is built from building blocks and these blocks are connected together. So here we have a filter block which is selecting Sudo log messages. Then we have a file destination to store log messages. And then finally we have a log path which connects all of these building blocks together. Well, the source is not shown here on the slide, but we have a source from where we are reading the log messages. We have a filter which we defined here to select Sudo log messages and then we have a destination where we store the log messages. And when we use this configuration, here is how the traditional Sudo log messages look like. These are really short and do not contain much information due to the constraints of the original Cystog specification. So these were plain text, difficult to parse log messages and contain just minimal information. If we turn on JSON formatted logging with defaults logformat equals JSON on the Sudo side, then we will have JSON formatted logs which have a lot more information and in an easy to parse structured format which means that many log management applications can parse easily these log messages and create field tags or store just the necessary fields from the log messages and so on. Of course, JSON formatted messages are difficult to read by humans, so there are many utilities to help you to read like JQ on the terminal but most of the time you will forward these messages to Elasticsearch or other applications where they can be displayed in a human readable form. Now let's have a slightly different Cystog and J configuration. What we change here is the file destination. We have the same filter and the same log path but in the file destination instead of having the simple text file we change to JSON formatting where we include all of the Cystog specific fields and also name value pairs are recorded to the log message. What you do not see on the screen is that Cystog NG is parsing Sudo log messages automatically out of the box so you do not have to enable any parsers yourself but it's done automatically. So when you store the parsed log messages then you see a nice long JSON formatted log message which has all of the fields from the Sudo log and from the Cystog header. What you can see here is that all of the values here have quotes around them which means that in Cystog NG 3 all of the name value pairs are created as text out of the box which is understandable from the Cystog point of view as your log messages are text but the various numbers are also treated as text which is not really good if you want to create a report in Elasticsearch or MongoDB or any other destination which is type of air. So what is coming up in Cystog NG.4 it's not really easy to spot as a human but for a JSON parser it will be quite a difference that when Cystog NG is parsing a JSON message or when using some of the parsers then it's type of air and for example here is a number and there are no quotes around it which means that it's treated as a number and forwarded as a number to the destination and you do not have to do any tricks on the receiving side to treat it as a number but it's automatically used as a number. Then there's lists in there too? Yes, and the list is also and that's another improvement. Back to Sudu in 193 change route and change working directory support was added to Sudu and previously there were quite a few situations where you had to give the user full route shell access for example if one needed to start an application from a user in accessible directory. That's not a problem anymore as you can use the change working directory option and you do not have to give route access or it was very easy to get full route access if you gave a user change route access on a system. The change route command that needs route privileges and even if it was run through Sudu it was easy to give change route the route directory and then you got full route access on the host which is mostly not what you try to achieve. Change route access needs to be enabled explicitly in the Suduers file just like change working directory as there are two possible ways how to give the permissions to the user. One is giving a wide guide access so change route equals star and the user can configure where to change route or where to change working directory but it has the side effect that the route directory can be used and that gives full access again but the important change is that at least it's logged nicely so there is a new field change route equals route and you can easily create an alert on it on Syslog NG. The other possibility is fixing the directory in the Suduers file but it has the side effect that all of the commands a user tries to start is started from this directory. I'm not sure how much you can see from this I hope you can see this. Here what I changed is the log path the filter and the file destination state the same to the log path I added an if statement that's a filter practically and I mentioned that Syslog NG is parsing Sudu log messages automatically and creating name value pairs this one is a name value pair created by Syslog NG and this is the directory name where Sudu is change routing so what I do here is checking if this name value pair equals to the root directory and in this case I send an alert here in my example I store just to another file with a special formatting but in a real-world situation you can create an email alert or send a Slack message and receive it on your phone in real-time it's up to you what you put here. The last Sudu features I want to talk about is logging and intercepting sub-comments you could check what your users are doing even before Sudu 198 added support for logging and intercepting sub-comments but it was quite boring and time consuming I mean you had to watch session recordings if you wanted to see what your users did when they accessed the shell through Sudu and I know that some of the users had three-day long Sudu sessions which are let's say quite boring to watch With logging you can check your log messages if there is something interesting in the session recording and then watch the recording based on the log messages it works in most cases but of course not everything can be cached by Sudu as for example built-in commands from shells are not detected by this method it's practically run by the shell directly you can enable sub-comment logging using default lock sub-comments and it's worth mentioning that you get a lot more information when you also enable JSON-formatted logging here is a screenshot from my favorite text editor and most of the text editors have a feature to run external commands or start a shell within the text editor so here I started a shell and then run a few commands like ID and LSD these are nothing harmful so no harm was done to my host but as you can see all that is logged is that I started my text editor nothing more this is how Sudu worked for many years but if you enable sub-comment logging then you see everything what was done even the various commands started by the profile of the shell so here the first line is the very same as you could see on the previous slide the text editor starting but then all of the various long list of what started by the shell automatically and the last two lines are the actual command lines I executed from the shell so nothing left without logging and if you add JSON-formatted logging to the mix then it is much more easy to analyze the log messages from these Sudu sessions there is a UUID which is the same for all of the commands from the same Sudu session and also there are quite a few useful information about the commands executed another possibility is intercepting sub-comments which means that you can prevent applications from running enabling this is a two-step process first of all you have to enable intercepting in the Sudu RS file defaults intercept and then give the actual rules to the Sudu RS file in this case my user is not allowed to execute the very harmful who command here you can see that even if I have full root shell access on the given system when I try to run who then I get a permission denied message so you can prevent this way commands from running even if you need to give users full root shell access you can also disable shells this way but it has a couple of side effects here you can see how to do this defaults intercept then here I created an area where I list a couple of shells you can list here many more but then it doesn't fit the screen and then I disable running shells for my user but this also means that I cannot use shells anymore this is somehow expected on the other hand I was a bit surprised when it came to playing with vi I wanted to start a command from within vi text editing session but as the editor is using the shell to run external commands I could not run any external commands at all finally a few words about what is coming in Cystogen G4 I already mentioned typing support why is it important? because you can store to Elasticsearch, MongoDB, SQL or anywhere where you receive for example in JSON formative log messages the right type so you do not have to struggle on the receiving and to configure what data is what type but it's automatically sent with the right type to use this you need at least Cystogen G version 3.37 but already 3.38 is in freebase reports and hopefully coming to others as well so you can already test these features the trick is that Cystogen G configuration starts with a version number it is used to help to if anything is changing how a value within Cystogen G is handled various defaults it gives a kind of compatibility mode and here we change this version number to a future version to version 4.0 and this way you can test Cystogen G4 features in the current release so type information from PartenDB and JSON parsing are preserved and for other parsers or other data you can rewrite type information manually it's not so nice but at least it works so let me give you a quick summary of my talk recent versions of sudo let you control c and control a lot more activities than previously possible you can get a lot more detailed and easier to use log messages there is a lot less need for giving users full root shell access and even if you give them full root shell access you can track what is happening inside and even prevent some of the commands from running and using Cystogen G you have built-in support for sudo log messages which means that it's much easier to create alerts based on these messages or store them to nsql or other type of destinations thank you for your attention and let me know if you have any questions I had a question for you so in the python world the pydantic is all the hot stuff pydantic basically does all the typing conversions for you did you guys think about using something like that or did you just write what did you do to do the type conversion this is right now under development this is the python support for type of destination is close to merging but it's still something being worked on currently everything is treated as text okay yeah because I was just thinking the solution is you could just throw like an ODM between the syslog and the Mongo with something like the supports pydantic like a beanie and it'll do all the typing conversions as it stores it in the database even if you throw a string with numbers in it and turn it into a number because it's like oh this is a number so this is still under development so I will know more about it probably in two weeks from now we had some discussions what I couldn't follow as I'm not a developer and they were discussing it a bit but hopefully I will know more about it in over two weeks just mention pydantic and see what they say because it's really cool the part where you were talking about the pattern matching don't look at my credentials it's also good to connect it to an external data store like have a database which can dynamically be updated or pattern matching have a user name or a pattern list which you need to update yes we have a technology called pattern baby that's an ugly external database but at least it works it implements pretty much what others implement using regular expressions but it's using something deep protocol I don't know from the top of my head so it's much faster than using regular expressions you can describe a log message that what is your username, what is the IP address what is everything and then it will pick out the values for you from the log message and with the right typing and everything so this can be used to store to alert on various user names like when root logs in using SSH or whatever and it can also be type a verse so if you parse numbers from it then from version 4 it will be handled properly I can try to find you an example of that at all and I already mentioned I have a couple of stickers any other questions thank you one more question thank you all thank you all thank you really great talk we've played around with this log engine in Kafka already yes we have a Kafka destination oh yeah I need to look into that when you think it's not too old it's pretty old but many bestows enabled it in the disk version of syslog engine only recently but it was developed 5 or 6 years ago by a Swiss company they used it I don't xx scale so it's a hosting company and they use it for logging everything on their network yeah we also have a page about if you want to mark Kafka and stuff right so that was up to date there was someone who was competing we moved these spots and talks last week or something so if you have a lot of printouts keep an eye on the everyone's schedule well you can't click on anything and see it abstract that was meant to be done but we all looked at WordPress and he demanded sorry that we didn't like it yeah I have a solution for you what use WordPress? yes to use pre-talks it does everything for you pre-talks one minute I'll pull it up events pre-talks let's take the config management one okay if you need a hosted one just bring me we are welcome to do the ticket sales and the call paper pre-takes for the ticket sales pre-talks for the event so if you look at ours here well look at this one this is the one this is the one this is the one this is the one this is the one this one has it so you can have this as a as it is now but you can export it to a static site and then just host it there so something that we can type and mark it down at and have it remodeled into HTML I started looking and then I remembered why I had to yeah but here you can go to this and you can link it to YouTube female and a whole bunch of other stuff in previous years we didn't manage to get the abstracts and speech files up but this year we've actually created the Elementor or something in WordPress which is supposed to make it so much easier to get your data in and my experience of this has become basically impossible please do remember your masks again, if you need help, just bring me thank you very much but I have solved the problem for me in a bit more than 24 hours Uh, this is your, oh, this is the, this is the open power version. Yep. Tomorrow we- The Linux one is tomorrow. Let me just simply check this annual that this is, well, it's the second talk I attempt to watch. I don't pay later. Thank you so much. Yeah, cut. There it is. Wait a minute now, we're going to be in one, we're going to be in one minute. So, uh, like dog. Uh, would you like to speak in an introduction or because I personally don't like speaking introductions because I will mispronounce your name. Get your background wrong and then you will, you will reintroduce yourself anyway. Okay. I think we should leave speaker introductions to subject matter experts. Well, I have a slide about who I am. Well, that is exactly my point about why I don't like speaker introductions. Oh, well, you pay along your background. Did this, that is a roller. Does this mic work? Yep. Okay. Yep. Definitely. You're getting close to the end expects. Yeah. Five minutes. Yep. Okay. Go ahead. We've got a long. You've been to load days. No, Chris. Yeah. Well, we hand out beers and we throw beers around that load days. So that's nicer. We had a speaker at nine in the morning say, Oh, I could use a beer. And one of the organizers goes and puts a beer on the table. I don't think it will go over, but we'll see. Otherwise. Okay. I'm getting started. I'm not wearing my mask so that you can actually hear me better. So, but you're supposed to keep your masks on. So yeah, this is the talk about free BSD on open power. I will be going over some history. So open power as in the open power foundation, I'll be going over some history of power in general. I'll be talking about what free BSD is doing and what we are trying to do from the foundation to help free BSD. The idea is to replicate this for open BSD and BSD and any other BSD in the world. But this talk just shows efforts that we're doing now with free BSD. So yeah, who am I? My name is Toshan Barvani. I have my company. We are involved in open source. I'm mostly a Linux user. So I've been a Linux user for about 25 years. I've been a BSD user for about 10 years. So not that long. I'm not a BSD developer. So I'm just a user. I will now get more acquainted with development because of the efforts I'm trying to do, but I'm not at this point. And yeah, you have a whole list of stuff I do. I run some conferences or config management camp, which will run after FOSDEM if you're interested, and load days, which is a smaller one. And then yeah, I have a blog and a Twitter handle. I am also now the TSC, so that's the Technical Steering Committee chair at the Open Power Foundation. So that's the foundation that tries to lead the efforts around open power. So like I said, a short introduction, an overview of what we are doing with the foundation. So if you see OPF, that's Open Power Foundation, but sometimes I'm lazy and I don't write it fully. And then the road to open power, so some history, some other efforts that have been done and that we are doing, what free BSD has done on their own already and what we are trying to achieve. And then some developer resources that are currently available that we are making available and that will become available soonish. So yeah, the Open Power Foundation was founded in 2013. We have more than 300 members. We typically have working groups, so we are divided in working groups and then everybody can focus on that specific area. In 2019, we had some new leadership. They wanted to be more software-driven and keep the hardware-driven part in there so that there's a more tight integration between software and hardware. We want to also have newer interactions with the communities, which is why people like me come and talk here to get interaction from people. I list the board members not because you need to see it, but just to understand that it's not just IBM anymore. It's typically a misconception that all power is IBM. Yes, IBM invented power. Yes, they are still one of the main companies behind it, but they are no more the only company behind it. So in many of these efforts, you will see that there are also other companies, smaller companies that are involved in these efforts. So yeah, some of the goals, so create an Open Power architecture. So typically previously you had hardware manufacturers that just build stuff and then they try to sell it. So now we're trying to create more of an ecosystem where people can get value from. We also want to have interactions with the academic world so that we can have more cross-discipline incentives. We are building what we call a landscape. So if you know the CNCF landscape, then we're building something similar to that, which is basically an overview of which software runs on power. So just trying to find ways to get all the information on one portal. And for instance, for the distributions, BSD is going to be one of the areas that we want to support. But also for other projects around BSD, we want them to be listed and to be maintained and give resources to the ecosystem so that they can actually maintain it in the future themselves. One of the other interesting things is that now the ISA, so the Instruction Set Architecture, was contributed to the foundation, which means that people can actually see what is happening, see the evaluation, can give feedback. There are specific rules about who can vote and who can do stuff, but you can still come to the meetings, you can still interact with people, you can still give your opinion. You can get what you want, but that's typical when you have so many members trying to pull each way. We have connections with open copy, so for acceleration, I know we've tried looking at ZFS and doing acceleration on compression, on encryption, so that's where the CXL open copy is now looking at. Previously, those were two separate consortiums, now they're being merged into one. For the software enablement, we have created what we're calling the Open Power Foundation Hub, so that's basically several providers that will give free access to power machines, because the machinery that is currently available isn't cheap. It's not something you as a developer typically would buy and put under your desk. If you put it under your desk, you might get confused that you have an airplane below your desk, because when it starts, it's so noisy you can't hear yourself think. We are having some efforts around this. One of the things I did when I became the TSC is that we are now running the foundation on open power technology and all open source technology, so the website has been migrated, our infrastructure for the members has been migrated, our chat system runs actually on power. For the moment, the majority runs on Linux. I am looking to change that also to some BSD stuff so that we can actually prove that it works on anything. The idea there was like, we have proprietary tools which build open source technologies. Why can't we use open source technologies to build open source technologies? In that effort, we've come a long way, we haven't finalized everything. There is still some proprietary stuff in there which I'm trying to move, which will happen over time. One thing to remember, it's not just enterprise focused. Yes, IBM is IBM and will focus on enterprise, but there are, like I said, many other members like Inspire, Winston, and some new ones like RedSemi which are focusing on smaller devices and going back to the networking stack, like Google, the embedded world. So the things that were known on the PowerPC are coming back now again, but not by IBM because they don't have a focus on that. But they will help enable the ecosystem. So here's an overview of all the workgroups. You find this also on the website. The interesting one is maybe the PowerISA. If you're writing low-level coding and you want to be able to get all the up-code optimization, that's a workgroup you can join. You need to be a member to join that one. Now you can become an individual member at no cost. So only if your company wants to become a member, then there's a whole costing. You can find that on the website. I'm not going to go through that. But as individuals, you can become a member. If you're part of the FreeBSD Foundation, as the FreeBSD Foundation is a member, you can get access through them also. I am hoping to talk to OpenBSD and at BSD as they also have foundations and let them join in. Because for the foundation, it doesn't cost any money. It's just some paperwork. But then you can, through the foundation, get access on that. Yeah, the system software workgroup is the other one, which is very interesting. That's the one where we create the documentation and the specifications for the architecture. So primarily in the past, it was Linux-focused. Now that I've come in, I'm trying to open it up and get people from BSD also to be there. IBM has already people at some of the BSD and they're trying to get more open in that way and not only focused on Linux. Because when the platform was opened up to the world, it was Linux-focused. Now we need to open it up to everyone. The hub is the one that would actually document like how to port and how to optimize. It's also the one where you can get access to these type of resources. I'll come back on that later. The Libre BMC is another new one. That's actually taking the BMC, which today has an open source software stack. But we're also trying to open source the hardware. So it runs on an FPGA. You get a soft car. You can program it all. You can compile it yourself and you'll be able to use it yourself. I have some pictures later on on how that looks and which machines actually can get on that. Then the ones with the star there. So the PowerPie is our initiative to get developer boards. I'll come back on that later in detail. But it's basically more affordable dev boards. Because when I come to the development resources, you will see it's not that cheap for the moment. And we will be starting the ambassadors also. So I think Peter would be very interested in that. But the ambassadors was actually on hold because of COVID because there's not a lot of activity. So now that we are having more in-person events, we will start that also. So yeah, the road to open power. So power as the architecture was actually a research project at IBM. Power stands for performance with enhanced risk. So it then trickled down to the RTPC, which was a workstation desktop that they made. Then there was the Deadpool and the ThinkPad. So there were several editions of PowerPC in a laptop. And I bring that up because later on we will see initiative to do something similar. So the first word there, now they don't exist anymore, then the RS6000, which is the most known one, and which is what the current architecture was derived from. We then have Power or PowerPC, which is the AIM. So Apple, IBM, and Motorola joined forces to make this. It had a lot of embedded use. So a lot of the switches in the past were actually running PowerPC CPUs. That one also went to Mars. So if you go to Mars ever, you can still find a PowerPC machine there. And then it went more towards Power.org, which was the more server in the enterprise. And at that point IBM pushed their server and enterprise view on the power architecture. And a lot of the embedded stuff basically died out. And that is where you also see that a lot of the development has stopped at that point. I mean, if I ask the people in the hall here, everybody will know PowerPC as in the Apple Mac, the G4s, G5s, people will hardly know that it is in some of the Cisco routers or in some of the Cisco switches or even Nordel switches. So that's when the focus actually was lost. Now with what we are calling the Power ISA, which is the foundation actually taking over and becoming more interactive, we are trying to get a focus for those type of use cases again. So there are efforts to make switches again with power. There are efforts on making more privacy aware machines so that machinery is fully open. People can actually examine what it is. And there is actually currently already a vendor who is selling machines in that way. So they give you the full documentation on everything. They even give you the specification of the motherboard, the layouts, and where all the blobs are. You can actually rebuild that all yourself. The only thing, of course, is that you need to also rebuild some of the learning data and that can be a little bit more intense work for somebody who isn't used to that. Another thing is that in this effort, so a year after the foundation was started with the release of Power 8, IBM open sourced the firmware. And so what is typically called OPAL, it's the open power abstraction layer. It's basically composed of OCC, which brings up your machine. Then Hoseboot initializes all the interfaces and the IO. And then Skiboot actually bootstraps the kernel. And at this moment, obviously, that is a Linux kernel, but we should be able to swap that out to a BSD kernel. And then Petitboot, which is basically just a 2E, so a text-based interface so that people can easily select which boot they want to do. So similar to what you go to your UEFA menu or your BIOS menu. We do have Coreboot that is being actively developed. They are actually busy. They have some alpha releases. You can actually buy a Talos 2 and get Coreboot running on it. The BMC software was opened up a little bit later. That was only the software stack, so it's still an ASP on that, which is a closed proprietary system. And so that software stack is open on all the dev machines that you can get your hands on. You can actually compile your own BMC software and put it on it. So now we've started an effort called Libre BMC, not to be confused with the open BMC. That's why we chose a different name. And that was actually to have an open-source BMC hardware. Now, because of the way that that's working and because of some other influencers within the foundation, that's an FPGA-based implementation. And that makes it that you can actually run a soft core like MicroBot or LibreSock, which is a fully open-source software core, which typically could then be made into an actual CPU. But you can actually program them. You can inspect what it's doing. You can see everything that's happening on that board. Yeah, sorry. There's a spelling mistake, but it's several machines have open-source, so Raptor is one of those who have done it. IBM has two machines that is also fully open-sourced. And some of the Taiwanese manufacturers like Tian and Winston also open-source their machines. So if you look at some of the technologies, so this is, of course, an older version, the Power 8, which was the first one that was released within the foundation. We have, and the reason why I write the V2.07 is because unlike Linux, we don't want to make the same mistake of only saying Power 8, which is an IBM processor. So the ISA standard is the V2.07. And so even if you have other manufacturers making cars, they might not implement all the op-codes the same way. Some of the op-codes are optional. Some of them are mandatory. There are actually different levels in the ISA. So you can then have different sets of type of processors. So yeah, IBM made them, and you see Google also had their own machines. Rackspace had them. Raptor is the one that is the most well-known one because it's mostly promoted for their openness and their individual accessibility of the hardware. But if you look, there are also some Chinese vendors. And so those Chinese vendors at the time made their own CPUs. So they're not running the IBM CPUs. They're on their own that were manufactured by them. If we go to the next step, it's the Power 9 or the V3.0. We have the Raptor one, which is today available. So you have the Talos 2 or the Talos 2 Lite, which are the most common ones, and the most affordable ones for the moment. But you see other companies are still making power machines. Obviously this is from the enterprise perspective. And it's still all rack-based server machinery. But if you go to the next one, we'll actually see some other ones. So we have Libresoc, which is for the moment a software car, which basically means if you have any FPGA today, you can already run it. You have RedSemi, which is actually taking the Libresoc designs and putting that in silicon and actually going to build the CPUs. So we hope to have those ready by middle of next year, where you will actually have non-IBM power CPUs. One of them is going to be targeting networking. So they have a gigabit router design that they are building with help of the EU, for instance. And then we have the new... Well, I'm not new, it's an old effort. PowerPC laptop project. Currently only available with the E6500 car, but we are hoping to update that to a newer car. But these are the efforts that are going on. So you even see like Microsoft and Google, they offer this in their cloud. Sometimes you might not know that because it's optimized or specialized hardware. So if you run certain code like you want to do some high encryption, it might be running on a power in place of an X86 machine or even an ARM machine. So there are a lot of misconceptions. Technically, there are no open power machines. There's open power technology and then vendors make their machines, actually. Power isn't for enterprises only. Yes, IBM is of course focusing only on that market, but we do have other new vendors that are focusing on the other side. Power isn't really open source. It is actually the entire stack is open. And you can find the specifications today on GitHub, on the foundation website. And one of the problems we are facing is that this is so scattered everywhere. So we're trying to consolidate all that information in one place. And in that way, it will be easier for developers to find that. Yeah, IBM is in power. So I mean, IBM is obviously the inventor of it and still the main party involved, but it's no more the only party involved. And there will be other people building CPU cores. Well, there are other people building CPU cores that will be available next year. Yeah, so it isn't ancient yet and it isn't dying yet. So what did FreeBSD do? FreeBSD actually ported to Power8. So they have their PowerPC64LE, which you can actually use to bootstrap and to add support. So it's for the moment tier two only. There are a lot of ports that aren't fully supported. That's where we are trying to put the effort into getting more developers involved on one hand, but also being able to see how much feedback we can get from actual users using these machines. So we need to work on that. The FreeBSD Foundation has been very helpful in that. They actually have people who are doing that now already. And we have worked out a loaner that should be soon at the foundation. And then they will actually have full access. They already have several machines, but with this additional machine, it should make the parts easier to work on. So there are some people within the FreeBSD community who have done that. So those are the three main people involved in the project. So I would at least like to thank them that they already did this before I was involved. But with our involvement, we hope to get more people. If you have specific projects or sub-projects, you want to promote or to get posted, let us know. The Open Power Foundation is also a foundation. It's not like we have money like other companies or like Intel that spend or throw money at developers. But we can work on certain type of solutions in that area. So yeah, I came to this conference to talk to the other BSDs. So I think yesterday I spoke to NetBSD. So they were showing some interest. I'm trying to talk to the people of OpenBSD. They also already have some functionality with Power, but fairly limited for the moment. So we want to expand that also. So the basic idea is just to have all BSD supported on this and to have it into the stack so that in future when we write documentation, it's not just focused on Linux, but it's focused on any type of open source operating system that is out there. And with that effort, we will hopefully get also more users, people who are more aware of what they need and want. So the developer resources. Currently, if you want to buy one now, you only have two choices. If you want something slightly affordable. So you have the XC922 developer machine, which is a machine if you buy from IBM at 50,000 euros list price, which they are promoting at 5,000 for developer purposes. So we got them down to one-tenth of the price, but it's still a lot of money for the development. I mean, we are not kidding ourselves. So yeah, if you want, you can buy them at IBM in the US or in one of the resellers in the EU. The other one, Raptor, which is touted as the most open source one. Again, it comes to a similar price tag. If you can buy it from Raptor in the US, there's a reseller now Viking store. I think a German company, which is reselling it, which makes it at least easier for our developers to get access because previously you would have to buy that Raptor imported yourself and do all the paperwork. So it makes life easier. But again, this isn't something you can buy and put under your desk. Even though the Talos is considered a workstation, it's going to be loud and noisy. And I think Viking is busy working with Raptor on making water cooling. So it's actually going to be a little bit quieter. But yeah, it's still very expensive. Let's not joke about that. So what do we have that is maybe another current solution? Are these FPGA based devices? So an FPGA you can typically get around 200 to 50. There are even ones at 50 euros. So you have, I think it's a butter stick or something like that, which sells at like 75 euros. MicroWatt should work on that. If you're active in that space, I would suggest joining the Libre BMC because that's where most of the effort and the discussions are going on. But we have two soft cores, so MicroWatt and Libre Sock. And we will have by middle of next year a third soft core that will also implement the full power ISA stack. So here are two examples. So this is an RTX A7. So that's around 200 euros. So it's still okay. And that's an ECP5 DCS-EM board. Also around 200 euros. But again, here it means that you need to compile your own software core and then you can only start doing the work on top of that. It's not exactly the same like what you get in physical hardware. There are some differences and those will always be different. But if you want to start playing with stuff, this might be the cheapest and the quickest solution than the previous options. Then like I mentioned a little bit earlier, the OpenPower Hub initiative. So this is where we give resources for free to developers. It's cloud-based. There are for the moment five providers. You can get bare metal access with Serial Console. So if you're doing low-level stuff, you can get access to that. You can even get access to several FPGAs. And I'm not talking about the 200 euro FPGA, but like a 15,000 or 30,000 euro FPGA, which is like a very heavy one with storage acceleration and encryption acceleration or compression accelerations. So accelerator access is something we're also trying to push for things like crypto or compression. For the moment, the compression algorithms aren't always fully optimized. If you have any ideas or projects around that, let us know and we'll see how we can help you. So we have the different providers there. Each one offers slightly different options, but we have two in the US, one in Brazil and then two commercial slash free ones. So Raptor has the integrity cloud. They obviously offer Talos-based machines at that. And then my company also has a few machines that we use in our RAC and in the lab that you can get access to. Some of them like the mini-cloud is a limited access, so you only get it for limited time. If you want long-term, Oregon State University will be the one that will offer long-term open source and they have the most hardware. So they have at this point, I think around a dozen systems whereby my company has maybe about five of them. And Raptor has obviously much more because they produce the machines. So you can go to that website. There's a form there. You can fill it in. If the form doesn't suit your needs, you can use that email address to email and then somebody from the hub will pick that up and discuss with you what you can do. If all that fails, you can mail me. Because like I said, we really want to do an effort on getting more open source tooling and more open source projects on this. And the only way is if you come up with an idea and we help you in some way. Like I said, we have resources available. We might not be able to throw money at you. But there are resources that we can at least give you. And then the infamous power pie. I've been speaking about this for a while. We haven't got to production yet. So we do have a plan. We have a design that is more or less ready. It's not yet ready to put in silicon yet, but we are working on that. So it's basically a single-board computer similar to a Raspberry Pi. We will start with a dual-core power V3.0. So it's similar to a power 9 to the depthboards you saw. We are trying to get a BMC integrated. So it will have a small FPGA that you can then flash and program yourself. Or you can get the standard image. Or my acceleration was planned, but I think we will have to scratch that out if we want to be ready by middle of next year. But this is going to be a platform that will be mainly focused on developers being able to put it under their desk. Or on their desk and say, okay, this is something I can play with. The price tag we are targeting with this is 500 euros. It isn't cheap yet as a Raspberry Pi, but it is at least, again, a level down from the current depth machines that we are at. And the reason why I say there will be several versions and generations is because hopefully in the next one we can go to a lower price range. And then if we can sell enough of them or give enough of them away, we can convince some other players to put in money and to actually be able to give some of fully freely away to developers. So yeah, my conclusion is we need to empower BSD and make sure that BSD isn't left out on the platform. A lot of efforts are being put in Linux, but I personally think we also need BSD support. Do you have any questions? Okay. I have a very mean question. Okay. Don't you say that our difference number is five? Well, that's not a mean question. It's typically a question I avoid because of personal involvement. So my opinions are of course because I'm a power user, a power developer. But the main difference between risk five and power is in power you have stable ISA releases. You have a guarantee that opcodes will be the same in the future because there's only one body that can push that forward. The other difference is in risk five, you have a hidden problem which many people do not know of and that is the whole IP that risk five is built on is flawed by definition. So if one of the patent trolls actually attacks risk five, risk five would have to close the entire foundation. And their members are also not covered. So technically, let's say Red Semi is one of these examples. They are an offspring of Libresock which is an open source project. They became member of the foundation but by becoming member of the foundation they automatically get patent guarantee and patent enablement from the foundation and indirectly from IBM because IBM is still one of the main players there in that space. So if they tomorrow produce something that is compliant with the power ISA stack, they know that if there is a patent war going on, IBM and the foundation will help them. In risk five, that doesn't exist. Think of that as NATO for patents for the ISA. Sorry? I could think of that as NATO for patents for the ISA. Yes. Usually we would have a treaty. Yes. And presumably if you join the foundation you commit to not using your patents to deny anyone else the ability to use power. No, you have to. So let's say like Red Semi is now coming in with a special proposal. I can't tell you which proposal yet is still being discussed. But if that proposal comes in, what happens at that point is that technology that they have developed will become an inbound patent to the foundation and then anybody within the foundation can use that patent. So you contribute your patent to the foundation. Another problem with risk five and maybe I know a little bit too little to say that fully is that risk five does not have the same ability in producing CPUs which are compatible with each other. And so you can have a CPU of risk five that has specific upcodes and you can have a different CPU that has those same upcodes doing different things. In power that's not possible. That's illegal by definition and you actually need to create illegal traps for that. So in the long term the divergence between risk five players is going to be very big while the divergence within the power is going to be limited because you have the ISA stack. So you have fixed, sorry, fixed points. You have floating point. You have little endian and you have big endian. Those are the four compliance stacks you come to. Now it means that if you want to build an embedded processor you typically will do SFFS. So a scalar fixed floating point. But it also means that you can take some of the upcodes from the little endian one. Now you cannot take just any upcode from anywhere. So do you have these four levels? You have also a bunch of optional upcodes like MMA which is in power 10 is an optional one. So you could get a CPU that doesn't have MMA. Now if we built the ecosystem in such a way that you can define that which is what we are trying to do with the Linux world where there have been many mistakes is where you can then say that if the CPU has MMA support execute those upcodes. But in the long term the divergence between risk five is going to be a hell. And to be fairly honest looking at their way of working and their foundation and the way that they allow members to do things it is going to be at some point a problem. Thank you. Anyone else? Yes. Now that I attended the main question you didn't guess well let's see for power 10 and I remember that they announced that there probably won't be power 10 due to issues. They didn't explain why? Well. Or why? Has there been some development? Or has this been instilled by now? Okay. So first thing first Raptor makes power nine machines. Yes. Raptor does not make power 10 machines. Raptors treat and block I don't think they made a block post but they treated about it is that they're not making power 10 machines because there's a binary blob there that cannot be rebuilt. That is not true. That binary blob that they claim is actually documented on GitHub. So I don't have the link with me now instantly but you can send me a mail and I can send you the link where you can find how you build that binary blob. Yes. It needs to be a binary blob when you put it into the machine but there is a way for you to rebuild it and to do a binary verification. So it isn't fully open and that is not because IBM wants it that's because another company called Microchip hasn't provided the RTL to that but they have provided how to build the RTL to it and that's where Raptor is creating a fuss. Now in my opinion if you know how to build it and it's not publicly available but the description is still good in my opinion. IBM has been trying to convince Microchip to fully open source it as in put the code actually out there. I hope that at some point they can do that. I know the person involved in that she is very adamant in getting this done so that we can get this misconception out of the road and the actual code is there but it's not IBM's decision it's Microchip's decision. So that's the reason why there's no power 10 from Raptor. Another reason why I presume Raptor is not going for power 10 is they have stocked up on so many power nines they haven't sold the quantities that they expected and at that price range you can imagine they're not I mean this isn't the system just for anyone even if it would be at half price it would still be affordable for a workstation but at this price it isn't super affordable and that's the reason why I think they're not going to power 10 and they're trying to find some excuse why they're not doing it in place of just being honest and saying we're not going to do it we'll skip to power 11 because we just don't have the resources I mean it's not only about resources of humans but also money. Tim Pearson the owner of the company has invested a lot of money a lot of time he needs to make money so that he can do the next stuff and he has done a lot of nice things I mean he has a new FPGA which will actually replace the A-speed FPGA I can't tell you the exact system yet because I don't think they released it yet but it's basically a card you can put in and then you have an FPGA so like Libre BMC his version is called Run BMC it's a different name but it does exactly the same thing so yeah I don't think there will be one of the players for power 10 I hope that changes Any other questions? Remarks? Suggestions? No, not yet Okay, thank you I guess my question is like what's the what would you say the minimum developer experience is part of it like what would you be doing as a developer do you just try to import the software to the power architecture? We want it to be easy for people to port to power and we want it to be so easy that if you have a CICD pipeline you can just add it in there and it should just build one of the reasons that IBM chose for Little Indian and is pushing Little Indian in the Linux world is because compatibility with the Intel world and the ARM world is so much easier Personally Sorry? Because our feature is too low to say that Sorry? I didn't hear it That summarizing argument probably means that software developer won't work I was just going to say in IBM's infinite wisdom they thought that software developers are idiots in place of letting the software developer be smart Now obviously they are targeting actually the more Intel based ones which think that there's nothing outside of Intel and the problem there again is a lot of the good developers just don't have time to do stuff they're so busy with their work that you need to sometimes have other people be involved in this and personally I'm against what they're doing on that level but I can't change that on my own as in my company still has Big Indian running on multiple places and we will continue to try to support that and I actually have spoken to the foundation that we should do that and from the foundation we are doing that it's IBM who's pushing Little Indian and the reason why I think they're also pushing it is because in many cases it makes stupid developers' lives easier and they just want to be quick and dirty about stuff they don't care about every little nitty-gritty detail because they are a commercial company they need to make money obviously we also need to make money so that we can pay our bills and do our stuff but we like to be correct they're not always that correct with everything Any more questions? Thank you very much for your talk