 all right okay so shall we dig in yeah let's okay so hello everyone welcome to our talk on Fedora on Raspberry Pi development so my name is Joel Savits I am a kernel engineer at Red Hat I went to UMass Lowell graduated last December with the computer science math major adult major hi my name is Thomas Gouma also a senior at UMass Lowell I'm currently doing a co-op and also worked on this project last semester in the spring hi I'm Charlie I'm a senior at UML studying computer science I had an internship at Red Hat this summer and I've been extended to continue doing work on this project I decided to get into it Joel is going to talk about well the origins of the project go ahead Joel yeah thanks so in the beginning we began this project back in in full 2019 basically I was a co-op at Red Hat and you know I was doing various things I ran into the Metro Boston research interest group and through them I found that there was this manager in the sef storage group Jeff Brown and he had been running in an IOT class at UMass Lowell where he had he'd been having people do various things with raspberry pies you know turn on turn on lights you know button stuff motors you know various a lot of key things and raspberry pies by default use use raspberry or use use a different OS than than fedora and Jeff Brown was interested in using fedora on his raspberry pies or it's pretty good OS it's got got open-source software it's linked into the whole Red Hat network you know he works at Red Hat so he tried it out and it didn't work very well it was pretty janky and so he wanted to improve it and so we wanted to get some students to work on improving fedora and you know baking tweaks so we could use it in this class so we put together a group of students and sent Peter Robinson an email Peter Robinson is the maintainer for the arm architecture on fedora and we got a couple of different responses so we formed a group of students at UMass Lowell who are working on this for for mostly for course credit uh and anyway so we emailed this guy Peter Robinson he gave us some suggestions on what to do so first thing is if RPI GPIO is so good why didn't they make an RPI GPIO2 and well that's exactly what we did I mean what well RPI GPIO2 is is a re implementation of of a popular python library for uh for Raspberry Pi for interacting with the GPIO pins uh basically the way that it worked previously no longer works on fedora uh uses kernel interfaces that are that are no longer supported uh oh this link is actually this is out of date but anyways it's uh it just relents the API and uh so that was the first thing we did over a couple semesters and um yeah so that was the first the first part of this the second thing that that Peter Robinson asked us was you know where the upstream drivers for the sense hat so well what is the sense hat the sense hat is this attach it for the Raspberry Pi 4 that uh it's a hat stands for hardware attached on top it's their like kind of clever name where they're like ha you know it's you put it on like a hat and then it's also an acronym you know people people in tech do that all the time uh and that basically has this LED grid a little joystick as you can see that you can use for controlling things it's got an accelerometer temperature stuff you know various various sensors uh that work over over i squared c so here's the here's the equipment that are that our group has got this uh this past semester um yeah over the course of of working on this um you know people people have uh come in and out of the project as most current iteration is uh mostly a group of students who joined at the beginning of this year and worked on it as a as a directed study for credit and then worked on it um most of them joined red hat as interns to continue working it over the summer and this is this is just the kind of batch of equipment we used to set up uh uefi booting of fedora and there's a just an example of the the pi itself so in general what we're doing here what this project is about is just trying to extend fedora usability stability and performance on the raspberry pi specifically the raspberry pi four four bs that's the latest one uh and so it's partly just a development project you know a bunch of engineers learning you know about about the raspberry pi improving the raspberry pi uh and and fedora support for the raspberry pi but it's also about uh introducing students to linus kernel development and low-level development which is a notoriously esoteric and um somewhat difficult area to break into of of the uh software engineering space uh it's just seen as kind of the you know weird kind of magic thing that makes the machine work and and most people don't worry about it and it can be kind of an intimidating place to get into so a big part of what we're trying to do is is continue to get students involved um and you know teach them about this space and so we want to we're continuing to do that this semester as well and we're going to bring some new students in into the project so and of course we would like to make fedora a default choice on the raspberry pi obviously not the default choice because you know raspbian is is kind of the official os but we want to make something fedora move fedora closer towards being an operating system for the raspberry pi that people turn to um you know one of the first couple that people turn to maybe maybe even their preferred os so quick aside how do i get started with fedora if you might be interested you know it's it's a it's a great os it's in the brad ecosystem you know it's it's sponsored by red hat and it's uh you know but it's independent it's their own thing it's uh you know completely for an open source software at least in the official repositories and i'd recommend dual booting it on your pc if you want if you want to check it out so uh now i'll i'll hand it off to thomas to discuss the independent slash directed study in more detail which uh i guess we'll just mention you know for those that aren't that aren't familiar that's just a class you know an academic class that uh that is kind of self self-directed so get course credit so thomas uh off to you do you want to do you want to switch to sharing yourself uh yeah i can do it okay um can everyone see my screen yep yep um so the main project of yes the main thing that we did during the directed study last spring semester was to read this book the linux device drivers book a third edition to gain some knowledge about device driver development um something to know before i go forward our carnal module and a device driver a carnal module is a piece of code that extends the carnal's functionality and can be added and removed from the carnal upon demand and the device driver is a file that enables one or more hardware devices to communicate with the operating system and this can be implemented as a carnal module um here's an example of a very simple carnal module it only has two functions the hello init and hello exit functions the printout messages to the system log the last two lines at the bottom of this program are macros in the carnal the module in the macro and module exit macro that define the hello init function and the hello exit function and make sure what they're executed a module insertion time and module removal time depending on whether the module was removed or inserted um the following commands are used to insert a module in the running carnal and remove the module and display a system log uh respectively in this mod inserts a module in the carnal our mod removes the module from the carnal and d message displays the system log and um you can see the following messages hello world and goodbye cruel world when you run d message um the main goal reading the book was to understand the character driver development a character driver is a driver that communicates with hardware by sending or receiving single characters and it has four main operations which include open read write and release the open function initializes an allocation memory that might be used for later operations like read and write read simply reads data from the hardware peripheral and sends it to user space and write does the opposite it sends data from the user space to the hardware the release function the allocation memory that might have been allocated and open and uh basically shuts down the driver these are the four main operations so i'm going to talk important things that we covered include debugging memory allocation and communicating with hardware um debugging i'll be going over printing and watching debugging by printing is where a driver is written to send messages or write messages to the system log when certain operations fail for example memory allocation if memory allocation fails then you can write a message to the system log saying something like the memory memory allocation failed or something like that and the developer would know where to go when they they ran d message um to check out the system log debugging by watching uh can be done by basically testing the driver for certain operations like read you can call read in user space and study its results if they're they're appropriate then you know it's working if they're not then you know something is going wrong um moving on to memory allocation chem malloc is a very common function used in the in the kernel for memory allocation which is the equivalent of user space malloc i think um chem malloc they work differently um chem malloc does certain things differently for example it only allocates continuous physical memory while malloc allocates virtual memory that's fragmented another difference between the two is the fact that chem malloc can be used to allocate memory for certain specific operations like dma operations or um just regular kernel ram but malloc is used to allocate a memory that's just you know like bite-sized memory specified by user communication with hardware is done over i squared c for our seneside peripheral um i squared c is a serial communications protocol that that where data is sent bit by bit along a wire that's connected to the processor these are some of the important things that were covered in the book i thought the book was a good introduction to linux and driver development especially for a student like me who was uh who was looking to gain some experience on a project outside of class i think this book was real interesting some practical exercises we did include a kernel compilation this actually took me a few attempts to successfully do it we also studied the seneside drivers and made annotations to understand how a driver works we tested the drivers and a few examples from the linux device drivers book and uh overall i thought this this was a good um this was a good learning experience and i'm glad that i joined this project um i'll i'll pass it on to charlie now i just realized my microphone is muted apologies about that all right so can i can everyone see my screen yeah we can cool perfect all righty so yeah i'm gonna be speaking mostly about what we did during our summer internship so both me and thomas uh we're working at redhead this summer you know at an internship we're both continuing to work on this project now during the semester and yeah this summer internship was uh was an opportunity to really dive in and focus uh you know almost exclusively on this project for a summer which was an exciting opportunity and the main thing that we were working on um was the driver for the seneside we wanted to try and get that accepted into upstream linux um so you know to do that we had to learn obviously how to interact with the linux kernel uh community so we had to learn how to create patches um uh patch sets and git uh and learn how to use git sent email to to share them with other people um and you know that i was involved posting on the linux kernel mailing list which could definitely be a little bit intimidating but i think that um you know my experience was that it was it was really not a big deal uh i mean obviously um it was uh you know it's it is a lot of helpful people there that are trying to help you make your code better so for the most part i had a good experience with that and you know obviously yeah we're receiving feedback uh based on how things were um you know uh really we were looking for feedback uh on how the code was so we started in the very beginning with the code directly from uh the raspberry pi source tree so we get an initial request for comment um where joel literally just took uh the code basically exactly as it was from the raspberry pi tree and just made sure that he could get it compiling um on the kernel and so um we got some feedback we posted this rfc and we got some feedback uh essentially there were there were some pretty substantial issues just because you know we really hadn't done much work on it um the um the you know there were a couple things that were platform specific um you know some fallbacks and some um error handling uh situations were specific to the raspberry pi and so we had to remove those for um for upstreaming the driver um but the biggest issue was that the uh driver for the display um for the eight by eight uh matrix of leds was written as a frame buffer uh device and uh that sub system is being deprecated at the moment and so we weren't going to be able to add a new frame buffer device we're going to be able to get our driver existed upstream uh if it was a frame buffer device so really um that set our big goal which was like okay so we have a little bit of you know fixing up to do but the big thing was like okay can we get rid of this frame buffer device so that we can um so that we can get this you know like that that needs to go um so we worked on those issues and then we created a second rfc um that was about a month ago maybe a little bit more um and having um having uh having obviously made the code a little bit closer if we were able to receive more detailed uh feedback uh you know if the feedback you get from one of the one of the modules is just like well this is a frame buffer driver whole thing has to go uh you're not getting a lot of uh detail so um having made it a little bit closer you were able to be able to receive more more uh specific guidance um so you know things like uh using we should have used the string you know raspberry pi instead of rpi because raspberry pi is more common upstream whereas rpi is more common maybe within the um the raspberry pi specific stuff you know or things like saying if not pointer instead of comparing the pointer to null or or little things like that where um you know just you know find details um we also decided to do a big overhaul of the code based stylistically because the code as it as it came to us uh referred to you know all of the file names and all the functions and everything was referred to as rpi since um rpi since this rpi since that and we decided to switch it to cents hat uh just because we thought it uh thought it was nicer um so having done that we submitted a third rfc and that was published two weeks ago uh today um and you know we're still collecting um comments uh feel free to check it out there's going to be a link on the last slide um yeah so we're we're hoping to get more feedback about uh this driver we're we're getting pretty close to to having it submitted well at least i hope so you know knock on wood um and uh yeah um i guess the the real question is um what are our future plans so obviously we want to um we want to get this driver upstreamed um and then we're hoping to get more students involved so um we're planning a small direct to study class again at um l this fall and we're trying to pull in you know some more students maybe junior or senior level students who have taken the class and operating systems but want to go deeper um and we're hoping to work with these students and we're going to continue working on the sense that driver and hopefully get it upstreamed um and when that clears up or maybe in parallel we may look into other features from raspberry pi os or other drivers to try and yeah to try and get it um really really polish the details and make make fedora really a first class option on raspberry pi uh yeah thank you for coming we've got about 10 minutes for q and a now um and feel free to email any of us with more questions we have our emails there and also our google group which you could email or join we have a link to our github where you can see all of the um you know all of the repositories where we where we develop our code and there's also a link to our most recent rfc and a link to our website so uh yeah does anyone have any questions so um yes our group is uh open to anyone i mean we have a lot of people who come from red hat um we have people uh who are students who are just interested i know we had someone actually who was a um who was a high school student um and uh they were interested in our project and they helped us with um uh yeah that's cool um so the idea uh i guess when you get your mic working um maybe you can try uh you can just interrupt me yeah so we have we have lots of people who get involved in uh this this project uh it's open to anyone uh we have a lot of people from red hat um and we're actually looking you know even in the vein of universities yeah so in the vein of usury we're also looking into um some other universities uh bobby ashera who works on this you know he's been involved in this project is looking into potentially doing a similar thing to his direct to study at a university that he has connections with um so Heidi also asked how do you advertise for students aside from devconf um i don't know actually thomas if you want you're welcome to answer simon i i realize that i've been talking for a while i don't know if anyone else uh wants to take a shot at answering some of the questions um i don't know we basically we've been uh we mostly focus on uml students because we already know a professor that's willing to uh oversee the director's study but um my mentor bobby ashera has also been uh looking to another university some are trying to get students but this semester i think it was too late for something like that um so try to get more students from other schools um no yeah probably yes hopefully oh there's no okay i'm gonna probably stop sharing actually maybe so that um yeah that's nice oh so oh yeah so i have a uh a demo short demo so here is the uh the device itself so i'll plug them and everything at the hat so uh here's the led thing so um i'm gonna run a program at charlie right so when i rebooted my computer earlier i was trying to reboot the raspberry pi that's what happened so there's this program where uh that charlie wrote called frame buffer test it's in a repo that just allows you to control the um one square arrow keys here and just puts a random color in so that's just something you can use as a basic test you know the joystick over here i mean this doesn't do anything interesting because it's i'm doing it over ssh if i was doing this over running this on the on the tgy i think this would actually control the um you know the led but it doesn't so that's one thing and then there's one other there's one other program that's uh kind of interesting that's called uh color test all right i'll just show a bunch of colors that's that they work so and then the other the other feature is the um i guess charlie charlie uh the other features uh in like the accelerometer temperature sensor various things those are all um those all have other drivers so those are those are already in there yeah joel um id was also asking how do how do we record what um id was asking how do project how do we recruit yeah uh well anyone who's interested can can start contributing i mean a number of people at redhat uh ran into place who are who are involved and then i mean you don't have to work for redhat to be involved i mean all this stuff's all open source um i mean any students can get in contact with us like if i i mean any student in a university they could if they can arrange something with their with the professor they could potentially arrange something for for credit um you know on a kind of individual basis you know i apologize i cut out there for a second yeah we're all we're all great with our uh our setup here i think there was another question about how um how well fedora works with the raspberry pi in the chat there somewhere uh i'm i think i'm looking at a completely different chat right now okay fair enough i can scroll up and find it um yeah so um i believe joel you you know like the question is where is where the gaps you know basic uh less compatibility with raspberry pi or is it mainly getting peripherals to work like the hats and the camera and everything i think joel you had some experience uh where like even things like a keyboard didn't work or something i mean i don't want to speak to that and then yeah so there are some issues getting the keyboard to work or the i don't know it could have been my keyboard as well but at times it would boot when i was booting without the uefi i think when i was booting like fedora 32 the keyboard didn't work uh and the you know there was a time when you had to uh i don't know like a year ago when if you loaded fedora you either had to use raw hide fedora or you had to at least one of the ways the way i think robert grim did it was like he just took the in this blog post we found which is widely known uh he just took the rpms for a newer kernel and put them inside of the fedora image and then booted a fedora image that didn't have ethernet support upgraded the kernel from the rpms he had pre installed or pre downloaded and then that enabled ethernet but now waiter versions just you know pretty much work out of the box at least i mean i've only been using it uh with the cli so this is pretty smooth a lot better there has to be pie three it was pretty difficult to use just because of how slow it was to do basic things like uh you know run any dnf command would take a few minutes but yeah i mean there are also some gaps in you know in internal support for various things that are listed there's a couple of different github pages that list it um one of them is just a random github issue on like an unrelated repo that has just like upstream stuff so that's that's there and then we've also been looking into some different things you know we had to find a couple of areas like uh where there's there's some gaps you know different utilities like the raspy config thing isn't available uh you know it's difficult to you can't virtualize the raspberry pi 4 i mean that's kind of that's kind of a general thing or like kimu doesn't have support for the raspberry pi 4 um well there's a there's a question about using another hat uh we haven't used it for this project but i don't know joe have you tried uh any other hats with fedora well i know i haven't personally tried but um but there are um there are there as listed on that that link that i posted there are a bunch of people who are who are there's a bunch of people working on different support for hats i believe and i know there's one for um actually there might be no there might be hardly anyone working on it but the the power over ethernet hat has i believe has a driver in upstream linux that was worked out by uh nicole sands julian or something i think he was upstreaming that and that would be listed at that link you know i think i just realized i posted that link in the just the main event chat that i would explain why and show up that that's that's that link that i was referring to i don't know if i can delete it from the other one but that's too late whatever yeah thanks everyone for coming sounds good uh i would recommend you guys uh post all your links uh slides and everything uh in your chat profile uh so if people search for you they should be able to find folks like right thank you so much that was a great presentation and down yeah thank you thank you bye bye everyone bye all thanks for coming