 I think I'll introduce himself as well. My name is Paul Urofti and today I want to talk about the work I did on the Oction machines. I want to take you through the drivers and the obstacles that I faced during the last year or maybe year and a half now and I'm gonna wrap it up with what's coming to the OpenBSD port. Who am I? I'm a reverse engineer. I work for six years now in the antivirus industry. I designed an emulator for antiviruses and I also wrote different static and dynamic engines. I'm also an OpenBSD hacker since 2008 I think and I didn't really find a single spot on which to focus on OpenBSD. So I started with really easy stuff like ACPI and power management. Coming from a reverse engineering background it looked appealing. Little did I know though and then I moved to Lungson which is a MIPS 64 machine that the Chinese produce and it's supposed to be entirely open sourced and as far as hardware goes but of course it's not because there's one piece there which is called the embedded controller which does just that embeds all the control in a single chip of the entire machine and it's not documented so that was supposed to be fun ended up not being so much fun but at the end of the road you say yeah I did that it was nice so it was fulfilling in the end and because of that I got to see how a MIPS 64 machine works. I moved on to Oction later on but in the meantime I also do compact Linux support for OpenBSD because well when I first started working at Bitdefender I wanted to use OpenBSD and IDA which is a reverse engineering tool was working on OpenBSD so I used they have they have a port for Linux so I used compact Linux for that and it used to work I haven't I haven't used it for a few years now I don't know if recent versions work but they should and also I port miscellaneous applications to OpenBSD which is the most fun thing to do because it's a lot easier than the rest and you actually get to see things working not just panics and blue messages popping at you and if that wasn't enough I'm also following a PhD in a very well connected subject of parallel computing on the GPUs which has nothing to do with anything so far but it's very interesting as well I do signal processing and I'm about to finish my PhD I have some courses on parallel programming at the faculty of computer science in Bucharest so that's about it once a year I go out and have fun how did I get in touch with Oction it was almost random as I said I played with other MIPS 64 ports I tried HGI for a bit but I couldn't get it working the model I had and Miod finally did that for me so it wasn't interesting anymore but then I had a first contact with Oction during the Toronto hackathon which is which was a very good hackathon for me but had nothing to do with Oction besides the fact that I sit next to Jasper who was hacking on this small box always mumbling stuff in Dutch yes thank you I was Netherlands these no Dutch thanks and I like when I see people angry because I can I can understand that and I can relate to that so I kept asking him what's going on and he was working on a bootloader for this Oction machine and it was really interesting I I was mostly looking at his console output and following my work for a few days but that got me interested and I wrote small phrase in the undeadly report which is the OpenBSD Journal about this experience and Diana which was kind enough to write me an email right away so do you want one or what and I was like oh okay I didn't even think about that I was just telling a story about the hackathon and soon enough I got a DSR500 machine which is a D-Link router switch that's based on Oction and that's how the nightmare started because I was looking forward to this and the first thing to do when you get a box is open it and then try to boot whatever is in there but of course it was running U-boot which is very popular with ARM people and you can see that because U-boot has a thousand variations just like ARMS so there's no coherency you can't read a single document and see how you're able to program that so I was lucky enough to learn how to boot to kernel through other people's work and hair loss since I have enough hair now but Miod was not so lucky and he wrote most of the instructions that don't really make sense typing boot Oction linux-psd is kind of schizophrenic but anyway once I got through that I managed to go all the way through the copyright and then the kernel crashed which was awesome actually I'm really excited when I see that because I get to fix things and I get to I get to talk about things and visit nice places like Bulgaria and meet you guys so I knew this was coming when I saw that crash so I'm looking at getting past the copyright which is not so interesting I had to look at how the Oction memory is organized and it's split into chunks you don't really expect to have a lot of memory on an Oction machine so you have different chunks of buffers that memory gets mapped onto so as you can see you got the first range which is the first 256 meg the second range and then what whatever else is above 512 gets mapped in the last section but my DSR 500 had 128 meg and nobody expected that because we're in 2010 plus and it was a small system memory and the code the kernel assumed that each the memory should at least have 256 meg and it was doing this substitution this this subtraction here which of course ended up with me having terabytes or petabytes of RAM because it overflowed so fixing that I got a big reward but I was a bit scared because I got all the way to user land so there was nothing to fix so we're free slides in I couldn't present something with only three slides so but I was kind of happy and I started looking at the d-message for more problems and there were plenty even though the d-message is really short if you're used to AMD 64 d-messages which take like 10 pages this was like a less than a screen which is very nice because you can see it all at once so first of all oct cf is short for oction compact flash which is not configured so no internal memory awesome this memory address conflict coming from a the PCI address space which I still haven't fixed today um and then there's the fi to which the network was connecting to the network uh adapter was connecting to and uh that that means unknown fi so there were some problems there so I was kind of starting to uh get a bit excited about the machine again and also there was no symbol table what that means is that when you get a crash you only get hexadecimal outputs on your screen no symbols that means no no no trace no parsable trace once you start reading in hexadecimal um Cyrillic seems fun um so where to from here I I wanted to add more drivers which I did and I'm going to talk later on I wanted storage support and networking of course these things seem important for port and I especially wanted to help Jasper to improve the second stage bootloader in order to get the symbols and make my life easier which didn't happen until today uh but I we worked on the bootloader this year in summer and the hackathon in Ljubljana in Slovenia and it's getting there so the end goal being we should have a port that's able to stand on its own and the users which I hope to be amongst uh should be able to just create a cheap router or switch out of the Oction and get the sweet OpenBSD message from it so then I when looking at drivers and implementing them I got my first major disappointment which is the SDK license that comes from the KVM people the guys that produce Octions and they were very nice to open source all their SDK for the drivers so you only need to call a few functions and hook them up to your kernel but the problem is that they have this it's almost like a bsd license except for the us export uh clause which uh we don't use in the project I don't want to get into politics because I don't understand them really but since the project decided that and I kind of agree with it we need to be coherent we have no such license in the tree I couldn't use it so I mailed the guys I sent an email I got a lot of replies towards zero I the the weird thing is that the SDK covers USB random number generator I don't know network card everything every driver and they also have a crypto accelerator which I'm guessing it's why that clause is there and I asked them if they can just resume that clause to the crypto accelerator that they have on the chip but I got no answer because I think that's why that got there and then it was just copied over and over so I wanted to start with a simple driver because I wanted to see how the registers look and how how I could use the octane without the SDK so I wanted to start with a random number generator which should have been pretty easy because I just need to start it get the output and fit it to our random number subsystem so I started looking at the magical figures that the SDK is using for the control register for the random numbers and it's as easy as I expected you just read the control register make sure you set up the output you set the output at the entropy flag and just after you write it back you should get randomness from there on so you just do this once during boot so the numbers are written in the entropy register which is different from the control one you fit it to our API for randomness and then you keep reading every 10 milliseconds easy the obstacles I faced this was actually the driver that took most of my time I think was because of engineers so the register addresses are written in a different engine in the SDK than what you're supposed to fit there to fit to the control register so if you want to do an actual read and access register you're supposed to flip all the bits because that's why I don't know why so besides that there was a read after write required whenever you set the control register just to make sure it was actually written there which is not uncommon but still for something as simple as that I didn't expect and another when I finally got the driver I was feeding eight bytes of data that I was told that only four are needed to be fed to the random sub system which you can ask Fio why tomorrow during his talk moving on I I got a message every time I did NFS boot which is really annoying because it's in caps and you know IRC culture you get bleeding eyes when you see caps so I want to get rid of that so because we didn't have a clock we only use the file system to get the time of day clock and the octane boards have some really good clocks I mean the Swiss must be proud of this technology of clocks of a one second resolution I mean they're awesome and they're the ones that they're used to get time and set time for the time of day clock if that wasn't scary enough in order to get to the clock you need to use a two-wire serial interface which is a twisted way of getting across to different devices so you get stuff like an internal address of that serial interface and you get nice things like a read bit which needs to be checked if the serial interface actually read what you told it to read and all kinds the valid bit it's very stable and robust and you get like all this 32-bit metadata just to write 32 bits which in the case of the RTC clock is even less but sure it's supposed to be like this so you set the address field for the real time clock device and afterwards you just use internal addresses you don't use the entire address to access the time clock and you set the operation of reading or writing in the metadata that they showed and you just do reads and writes from the data field that should be easy I don't know why I'm complaining so much well maybe because you do it in two steps the first being that you need to set the read bit in order to read something from the RTC clock and then you need to set the valid bit this is done in sequence of course and you need to write to the register which is like controlling the entire interface and telling you what to do but the second step you need to read back and keep reading back until the valid bit is set and if that is cleared then the operation was done and you can get a byte of data awesome which is one second resolution so you have enough time enough time to loop so yeah and the writes are done in a similar fashion in that you set bits in order to write and you finally you finally configure that register for write and then the second step reads back what if the operation was completed and loops all over except with the extra bit of having to read back even after you got the valid though okay I wrote there you still have to do an extra read so afterwards you get useful information everything is bcd coded which yes it's good I guess and you get to feed the set time and get time routines the bcd formatted data so that got me to a real time clock but of course the issues if it wasn't obvious so far is that it's very fragile you need to read there there are a lot if you're looking at the code if you have nothing better to do there are a lot of checks integrity checks there because of the loops and it can actually even time out I think it's it's complicated and some models don't even have a real time clock so when I committed it I expected everyone to have a real time clock but some just panicked because there was nothing there when you tried to read or write to that real time clock like the ubiquity edge router so buy that that moving on I was able to nfs boot without caps but you know in order to nfs boot first I had to tftp boot all of these involve a network which was working until I reached user land when I couldn't even ping I couldn't do anything I can get media and the status was no carrier so it was like no cable connected but that far it could reach the network so I wanted to fix that of course looking at the missing phi for the Broadcom device there remember the unknown phi message from the dmessage that I talked in the beginning I started looking at other people's work which is what I mostly do because I don't really like to get other people's work and adapt it and call it mine so openwrt had a wonderful driver for it which I adapted and trimmed down for different reasons I basically just got the values that I need to read from the media information because the openwrt driver was a lot more complex and was doing a lot more things like switching which we don't have support for in open bsd for you know doing multiple port switch and like the zrouter project from free bsd and stuff like that so I wrote a minimal driver which does done mode switching so whatever you plug in there sends out so I need to get to to the status of the phi to see if the link state is okay what kind of mode we're in and what the speed is so this was done through registers from the red from the status page of the phi so in order to get that I need to make sure that I'm on the right page and of course you need an extra readback operation even after you after you set the page that you want to be in and then once you're in the right page you get the data from there in two bytes of data that tell you what the cpu port is doing so what media it has the link state and everything and the next for the next ports you get the inflow in the second third and fourth data registers my ideas are for 500 has four ports so that's why so after that I got media right here and I could ping so network was working which is really nice to have on a router so for the switch support I was planning on integrating one of the two frameworks the z router one which I am more familiar with because I was in contact a lot of for a lot of time with Alexander and there is also the ij solution the ij people from Japan they have a commercial support for the oction based on open bsd and they open sourced solution for the switch framework which I tried to integrate into open bsd during the jubiana hackathon this summer but the guy developing that was very busy and didn't have we couldn't sync in order to discuss integrating it moving on the hardest part as the usb support because the compact flash even if you even even if you have a driver for it it's like 32 meg so the z router people are stripping down free bsd in order to fit in that compact flash and then they have usb support for the kvm sdk so you can add extra storage so there's that driver and then there's the ij driver which they also open sourced but they only work for the 3000 models of the kvm machines and it's also very specific to their to the machine that they sell and it's not really working on the machines that we support and then there's my driver which I finally got to a point where it no longer fries ticks I managed to fry like four or five memory sticks with it with a small spark coming out of it when I plug them in that was lovely so I'm going to talk about that wonderful implementation I did I started by doing the machine specific the host controller specific initial initialization routines which is a clock setup core setup dma and setting up a proper interrupt and this was pretty much stealing magical values from the sdk or looking at what the ij people did it's always a mix so I always had like four or five virtual screens with externs to their code and once I did that I managed to get an attachment so the oct oction host controller was attaching to usb zero which I know doesn't sound like much but to me it was a glorious day and I even got some error messages so more work awesome so I wanted to fix those and the first thing was to add an interrupt routine which is it's it's really easy to start with you just add bits of pieces there and but the memory sticks keep frying whenever you think you're making progress so the best thing I did was look at both implementations and make a and I was able to make a coherent interrupt routine that looks if you open other host controller drivers in open bsd they're going to look the same and you're going to spot the machine specific bits when looking at the two in parallel so I think that's nice even if I have an accident or whatever mpi can come over and finish the driver for me so oh by the way I never looked at usb before this so this was also a very nice experience to start learning about a different layer and mpi which is a usb master nowadays in open bsd was very kind to answer my ridiculous questions so after I've I've I've finished the interrupt routine or so I thought I keep adding code to it I got a proper demessage without those errors that doesn't mean that it was working it just the messages went away so I moved on to adding root hub support which is like like the media support for the fi you you start notifying people about what's happening in your hub which is the status you get the scripture of the request that are sent to the hub and also fixing the usb2 descriptors because you have to register the host controller as a usb2 controller fixing those and acknowledging the fact that I was getting commands from outside helped fix the frying problem so I was able to plug in things without sparks and I was able to use the devices afterwards so this was I'm not sure if it was progress though because some people offered money for the driver before just to be able to make pranks on first of April to people and but in my point of view it was progress because now the usb port was doing the same thing as before nothing you plug in a device and you get nothing back you pull it out and it still works so back to base back to the basis and then in order to actually get something out of the usb host controller I started adding more support for usb ports and the endpoints of course they call and endpoints channels that took me a long time to figure out I was like channels what's that I was asking an MPI do you know a channel definition in the usb specification no what's a channel so that was fun changing that name thank you so these allowed me to keep track of when you connect to a port or when you do a port reset and I was able to send notifications upstream upstream about what's was going on and this actually moved things further as nowadays I need to implement device transfer and control methods which in terms of the actual devices means that I need to start sending data through so the control is almost there because to give a large picture I the green things are things that are done and the blue things is what I'm working on and the red is what's missing so you might think it's half and half but you're being overly optimistic it's just so that I can show that it did something it's this will take a lot of time and it's very sensible work here but at least it's not destroying devices and it's you're able to actually see how things are moving in a usb stack and not getting panics just not moving data or not detecting the proper the proper events so this is the point where I where where I am today because I was saying the boss the boss methods are done and the hub routines should be okay now so I'm moving on to devices so the major showstoppers because I'm working on this for a while now is the SDK which would have the SDK is like two function calls and you're done you have a host controller nice right and the fact that the SDK and the ij code is very different and they actually set registers in a different way and in a different order and it's a lot of guesswork and poking and going back to reverse engineering and USB is really hard I mean like this pack is hard enough and doing it without the proper documentation isn't harder and the time requirements are really high and I can't I'm one of the people that need to focus on a task for at least four hours in order to get something done I can't just work for an hour in the evening after I'm done with regular work on the USB because I need to get in there and remember where I left off and start thinking about the magical values in this part of the world so progress is slow because of that moving on to the flash memory I started working on that as well as time off from the USB layer which is it supposed to be really easy to implement it's a complex flash after all after all the FreeBSD has a driver for it that's going through CFI and I wanted to get the code from there and I probably will end up doing that the problem is that they have a lot of layers that they go through and I just want to get the core of the driver I'll show in a later slide I think the layer that it goes through or maybe we can see it here yeah the CFI it's a CFI D connected to a CFI zero it's it's they support a lot of compact flashes because they respect this specification that Intel AMD and other people agree upon and that's different because of course Intel or AMD decided to violate that and do things opposite so a lot of if-teps and I haven't I have never done a disk driver so I discussed with DLG David about it because his the authority in this domain and the idea is to write a small ATA driver for it but the problem is that this device is so dumb that it's not using most of the ATA things so it's it doesn't do concurrent commands it just there are no port multipliers it's just so easy that the ATA might be even more complicated than what the FreeBSD people are doing so the the idea now is to avoid doing something like PCID which is very it does a lot of layer violations and to to do stand-alone driver maybe because the reads and the writes are done through registers you don't do DMA it's so I I'm still changing the implementation every time I'm going back to it so I'm I haven't settled on a direction so this is I as I was saying I don't know if I should support the entire CFI specification this is an extension of it I don't know if I should add that maybe I should start small I don't know so progress on this is really slow as well so to conclude is a lot of work was put into it even though it's not so obvious the lack of documentation as always is slowing things and the SDK copyright was is making things even harder and I would enjoy to have to get some help in these open problems there and the I I will continue working on the port and I want to complete it as soon as possible to make it stand alone the the memory storage being the main problem with it nowadays so thank you for listening to me and questions this thing where you had to loop over the real-time clock just to to be able to write one byte and wait for the valid bit to to change value how many times do you have to loop over that before it it actually accepts what you're trying to tell it I happened I I measured it a few times and it wasn't that bad actually I mean it seems that it's rarely occurring the problem there you always get it the first time most of the times but sometimes you can get like three or four iterations before there was no that's rare amounts like 10,000 no no no it's not that bad but it's yeah it's tricky yeah so I know it's not good but did you try playing with your SDK to see if it would be easier actually I tried to play with the sd meaning to integrate it and see how the sd works I tried that with free bsd and it works really good and with the zrouter but not with the actual open bsd tree because it it's it's not so easy to integrate the SDK I mean it needs a bit of work so I prefer to play with other projects that already did that work and try to yeah I'm yep well it can give you some pointers so that you don't have to reverse engineer everything yeah no I'm I'm looking at the SDK and I'm I'm helping myself there from there with different values and stuff but the logic it's really like it's the abstraction they use is very deep so you have to in order to read the register you go through like 10 or 12 functions until you actually do the read operation because they call it from different places and different drivers and that makes sense for them so integrating that into open bsd would be would take a bit of time and it's not impossible and I know that it won't get committed because of that license of course yeah thanks how finicky is the hardware that you are working with debug does it have a convenient serial console or yes it has a serial console which is like serial not jtag or whatever so did you no soldering necessary no you just buy it and you plug it in and it was the there was a d-link that you were working with yes d-link dsr 500 but I think the others are the same I'm I don't remember anyone having to solder a serial but I might be wrong yep and it actually comes up as a phi as well the serial because you get a like a network cable on one end and on the other the actual serial cable it's yeah works on the fly that might be a very generic question but what I mean are those devices popular the d-link are there many devices using octane or is it yes d-link has a lot of models using that they even have wireless mine only has wired network some have wireless a lot of people are using it as because it comes with a product I mean the d-link I have I could use it right off because it's they sell those it's not like I have to write an email and they send me prototypes so yeah a lot of people seem to be using that I I'm not sure if the dsr 500 so to say but there are a lot of octane models produced by the d-link and not only them but yeah okay yeah I think claudio so can we use these devices for like firewalls and stuff already or is it not really good that's what I want to do I have a server that's running since 1996 or 1997 and I wanted to once I got this I wanted to do the pf thing on it and I I didn't test with open bsd but free bsd works yeah sure but they're using the SDK so I don't know if the fi needs further work or maybe some but yeah looking at the open wrt driver we have most of the things there so I I haven't tested with open bsd for like actual performance but then I only run a home server so I don't I'm not the speed junkie yep so you don't know how fast it is no I don't know that but you should ask jmf you or dlg I know they experimented stuff on them and they're because they they're building the infrastructure at the university they probably looked at the networking figures so yeah for the questions we still have time as far as I can tell so again so we still we still some of their break time earlier so I have to know my questions thank you