 Siam salty is Andy Simkins talking about developing products in the open. Andy. First of all please forgive me if I'm a little nervous, I'm not good at this. I'm usually that end of the camera, so okay, today's talk is not going to go – there we go is not going to be technical. I'm not even going to technical on hardware side of things either. Mae'n gwrth gyd o'r holl o'ch lwinion lle'r holl gwrth am yr un aeth yn eithaf o'r holl yn y leidio'r holl. Mae'n gwoedd erbyn yn ddim yn ddisgrifet o'r holl, a'r holl mae'n gwaith o'r holl. Ysgolau ym Mhredin Cuirnau, mae'n cyfweld ar gyfer y bach o ath gweithwyr. Mae'n gweithwyr lle adael yr holl a chyfoddogol i'r holl. what I wanted to come across. So if we look at software today, the statement there is fairly true, you know. There's nothing to be actually gained by reinventing a wheel. If it's already been done, why do it again? If it's already been done in the open, help to support it. Oh, on my way. I learnt to drive. Likewise, if I asked any of you starting a new project, a product in this context is an embedded product, this is something containing a CPU. If I asked any of you to create a new product where the end product itself must contain some sort of operating system because it's sufficiently complicated. Maybe you need an IP stack. Maybe it needs some display drivers and bits and pieces. You're not going to develop that product today by looking at chipset vendors for a display, looking at a CPU vendor, and then writing your own operating system and microkernel to support it. You're probably, in fact, I think everybody in this room would start with, hopefully, a Debian-based system and work from there. So essentially we're saying probably the only buzzword I'm going to use here for your value add is the thing that your product is. It's not the CPU inside. It may be the case, the enclosure, the display, the nice little user interface, but it's the thing that your customer buys from you, not the internals of it. They don't care what's inside the box. It's the thing that does what they want it to do. So your customer is buying your product, ideally because you're the person selling that product. You're the company selling that product. It's at the right price for them. It does what they want. They're not going to the competition because it's you. You've sold them your ideal. You've sold the customer your service, your support, what the actual product itself does to an extent as long as it meets a need. That's almost commoditised these days. So the very first battle history lesson was moving to floss. From a company point of view, not necessarily from your own points of views, the no licence fee is a huge, huge win. If you're having to pay $40, $50, $100, $200 per licence per seat, per instance of a product that you're going to sell for maybe $100, you're dead in the water straight away. So key things that drive embedded, certainly in a commodity sector, is total bill of material cost and not necessarily but usually non-returnal investment. So that's the cost of how much something is to build and design and develop. By moving to open source, you can share in the development costs. Furthermore, you're not tied to one vendor. You're not tied to a particular software stack. You might be tied to a particular software stack. You're not tied to a particular vendor's view of a software stack or a product. If your product development is anything like any of the companies I've ever worked in, not only is it this product that you're developing, but the next product's based on that, and the next product's based on that, and the next product's based on that, and so on, and so on. Turtles all the way down. We see in software people are still running with, well, the Y2K people are still fishing out cobalt from the 1960s, and I suspect come 2038 people will still be fishing out cobalt from the 1960s. Hardware is, I'm afraid, exactly the same. You get tied in because it's so much easier to just tweak a little bit, modify a little bit from the last one, and then obsolescent hits you like a brick. Some kind of soul drops your micro process. Some kind of soul drops a bit of RAM. The way, certainly the companies I've been working for go, that the hardware engineers, the software engineers have been warning about this for the last five years that you've got a problem, and then they come, well, this component's gone obsolete. What can we do about it? And the simple answer is start from the beginning again. So from a software point of view, you pretty much achieved all of this. So from a better point of view of software, yeah, you've pretty much won. Nobody these days would take an embedded system without really thinking about why they wouldn't use an open system to start off with. Okay, there's a few bits that we've still got to think about. There's still the bean counters who don't really get it yet. So how can we support something that doesn't have royalties? How do we pay for that support structure? There's, and I'm struggling to see my notes on here because they're not showing up. There we go. You're still tied to board support packages, which are tied to forks in old kernels. Yeah, still exists. Changing from a closed, a closed shop to floss meant that you had to go through a whole series of retraining. You also need new skills. Your software engineers up until now in an embedded arena haven't needed to do the kernel building, the root file systems, the bootloaders. They've not really needed to do device drivers. And of course you've moved into this open environment where actually you now need somebody on board who can do that. So systems engineering, there's still quite a few job opportunities if only you can persuade the companies that they need you. So where are we now? You've gone through the pain of that transition. Hopefully you've reduced your license fees. Theoretically, your time to market has been greatly reduced because you can use software that already exists. But you're also probably still tied to a platform. You're probably still buying your platform from a third party vendor. And by platform in this case, I'm not talking about an operating system. I'm talking about a target board where you're basing your design on or even just using that target board with a daughter board around the outside to make it fit your form, maybe add your power supplies, maybe add specific interfaces, all of which are brought back into a target board typically along the target board's own interface channels. So you might be bringing it in over I2C, SPI, maybe internally you're running it over Ethernet, but you're bringing it in on a way that a third party vendor has told you to bring that your data into your product. But by moving to open source, I'm going to say you're no longer tied to that platform. Why is that? Where do we go? So we've got RRS, we've got that on, we're quite happy with that. Where do we go next? Well, you can move up the stack. We've already identified that your value add is the thing that makes your product unique. If your product contains some sort of data storage, I'm fairly convinced we can find the stack to deal with that. If your product contains networking, that's already part of the stack, you don't have to do it, it's already been done for you. You can also improve your standing within the community by becoming a good open source as a company. We'll look at that in a minute of how that can be a bit of an issue within company structures. We can also start to move back down the stack. That's below the OS itself, the drivers, the operate, and down into the hardware. That's realistically what I want to be talking about today. Moving up the stack, you all know this. You've all done the benefits of open source, I don't need to sell to anybody in this room. Moving further up the stack with all of those products available to you, why would you write any of those today if you're developing a product? Unless, of course, that particular, I hate to use the word middleware, but that particular application is your product. That's the thing you're writing, your value add, people using these things. Let's use them, they're already out there. We need to tell our companies, the people we work for, that there's an advantage to being a good corporate citizen. I'm assuming this is where a lot of us are today in our workplaces. Either you've all ead completely embraced open source, or if you're in a manufacturing company, you're probably still fighting that battle. They may have just done one or two products, and are just beginning to see the benefit. Hardware industry is very slow to move. It's a dinosaur. It takes a while. Try to explain to your being counters that participating in your, in the projects that you are using for your middleware is a good thing. It's like, why should we do that? There's a perfectly good product here. We take it, it does what we want it to do, and we get the old bug, and we fix it, and they don't push it upstream, because they don't see a benefit from it. Because you're buying benefit in kudos or karma. You're buying benefit by participating so your voice is heard. How do you put a monetary value on that if the previous concept within your company is you pay for a support contract to do somebody else's problem? Who do you pay your support contract to? Don't get me wrong, there are plenty of very, very good open source support companies out there, open source development companies out there that you can use, and ultimately, if you've got a very nervous company you're working for, putting them in touch with the likes of CodeTink, Collabra, amongst others is a good, it is a good option, because your R&D team internally, or your R&D management team, are just not ready yet to take the leap and employ people to do something, because they've been sold so heavily on the support model of, I want this, I buy support for it, they don't yet realise that by bringing it in house, they can influence the direction and they can make it do exactly what they want it to do, as opposed to just be tied with something. Again, this is stuff you guys all know about, so I won't dwell on it for too long, so we can come down the stack, and this is where the nitty gritty of this talk is really about. Up until now, most development companies have been buying in hardware, typically if you're in the volumes of sales of, I don't know, somewhere between 500,000 units a year and 40,000 units a year, it's almost certain you've been buying in compute module, or you've been buying in a specific design from a third party, or you've paid a lot of money to a third party to give you a board support package, and that cost a lot, that cost a lot of time, effort, development, and you're now stuck with it, you wedded to it, because your proprietary operating system that you've now freed yourself from, only has a particular way of doing things, you're now free of that, so you can change your platform, but you no longer just tied to the configuration, so staying with the target board you've got, you're not necessarily tied to the ways that the software vendor you bought that board from, or the hardware vendor you bought that for from, has the SOC configured, most modern SOCs have GPIO matrix, so they can do considerably more than one function on each pin, TI's devices for example, the OMAP 4 stutter on the panda board, these things have got an IO matrix of seven or eight different functions for every pin, so you've got multiple different interfaces, but the board support package assumes it's configured in one way, by taking control of this, you can now reconfigure that board to do what you want it to do, you've not changed buying your hardware vendor from anywhere else, but you can now reconfigure your platform, so look I needed an extra multi-channel SPI interface to talk to something, or I needed another I2C channel, or I didn't, I wanted some GPIO here instead, I'm no longer tied to the way it was originally configured, furthermore, I'm no longer tied to the specific compute module, I can take a basis of a compute module from somebody, and I can now create my own board support packages, I mean this is what Wookie does all the time, this is nothing new to within ourselves, but actually industry is just beginning to wake up to this, and I'm not talking big multi-billion dollar industry, but I'm talking the medium and small enterprise studies, up to around about a million product shipping a year, they're not there yet, they don't know about this, so you're no longer tied to a particular target board, so you can develop your own, but you may choose to not to develop a target board, but to embed everything onto one PCB in its entirety, it's quite radical thinking for a lot of hardware vendors, hardware designers, because why would I buy a compute board in, that's my piece, but I can make savings by not having connectors, I can make savings by not having wiring, but that also gives us a lot of reliability improvements as well, because I guarantee you all know that the biggest thing that goes wrong is a connector, or the wire between the connector, or somebody pulls it out, I mean you've already got to walk into the Amsterdam room and look at the video setup, the wiring is the nightmare, the rest of it works, so if you're putting a new design together, and I suspect anybody that's done a bit of hardware work in here would agree with me this, we're now at the point of wise component selection, we've got this C-thos that we can use something in the open, so we can buy a chip on that VLSI, could be all sorts of functionality, but we can pick a chip which has got an existing open support structure, something that's already in the kernel, and yes that chip might cost me ten cents more, but I've got to sell that to my company, that ten cents saving on my bill of materials may be offset because it's already in kernel org, with one, two, three, four man years work of the development work to write those drivers, get it tested properly, if that's already there for me, huge, huge savings, the battle you're going to have is unfortunately development costs are the domain of the project manager, manufacturing costs are the domain of your manufacturing team, and they've both got a budget, and they're both fighting to save every last penny, I mean I've been worked on projects where saving half of one cent was a target goal for the project per product, and I hear me coming in saying well actually we want to save, we want to spend another cent more on this product because it means we can offset it against our non-returnable investment, the battles you have to fight there, so it's not necessarily a design battle you're fighting, it's a mindset battle you're fighting, much as I don't know, 15, 20 years ago flash, no 10 years ago flash was a real problem, JEDEC stepped in and they as an industry body and they standardised a set of calls for flash, so you've got a series of open calls for reading and writing flash from software, you've got a physical footprint standard for flash, and you've got some things similar for RAM and bits and pieces, the world hasn't quite done that with other peripherals, but the differences it made when everybody started talking a common interface for a common peripheral, the differences that made were so big that it's happening on most interfaces, ethernet for decades has run on MII of various speeds, there are still proprietary interconnects out there and I would advocate not looking at those, pick something that is well understood where there's a lot of support for it, be aware though just because your hardware you've picked is in kernel log and there's drivers for it, you may be struggling, you may still struggle because you don't know how that hardware has to be strapped, how it has to be configured for that particular option in the device drivers, much fun on how laverty can be had by using the existing drivers and suddenly realising that you've got the wrong clock or you've got the wrong clock source, your clock frequency is absolutely fine but it's coming from the wrong end of a tree, so still be aware of that and there's often a lot of work that still needs to be done and of course please contribute it and push it back upstream, so step in open hardware. As designers we've had the ability to do everything ourselves, modular electronics, buying individual modules, people have done this sort of thing with electronics for years, they've taken a date sheet and they've pretty much copied the reference designer, who owns that design, who owns the licence for that design, do you actually have the right to use it with the hardware vendors themselves, don't really care because every time you use it it's a sale of one of their chips or two of their chips or the entire chipset but who actually holds the licence for that, so perhaps we should be looking at modular and open designs where we can pick a lego block or jigsaw piece of an entire circuit, not just a chip but a chip, it's supporting strapping, so it's how it's configured, how it's clocked, how it's powered and some method of identifying what's on this reference design, so the whole thing becomes a simple jigsaw piece, we can develop so much quicker because we're just picking up modules of a different preset design, now obviously for industry we're not going to be necessarily taking individual modules and plugging them together bit by bit, we've already said connectors are not the way forward but if you've got a truly open design, fine I'm just going to lift that bit of the design, import the card and drop that onto my schematic, we're not there yet, we're quite a wave from that, where we are at the moment is we've got a maker community, I say a maker community, it's a bit chaotic, it's growing rapidly, people are beginning to take notice of it, I want you to think back, does anybody remember, I don't know, shareware, right, how long ago was that, 2015-20 years ago, this is where open hardware is today, so you think maybe 15-20 years back of where open software was, hardware is about that state, you know it's been around before software but you know we're slow, we're only just beginning to recognise that instead of cramming everything onto a single chip, instead of having big companies force us down to have everything on one particular device, actually we're not interested in this new wonder chip, what we're interested in is standard functionality in building blocks, so open software pretty much challenged the landscape of the software world, I mean does anybody remember companies that didn't adapt and have fallen by the wayside, no even the big name companies, big blue, IBM, how long ago did they change to open source predominantly and become a consultancy, they used to be a hardware vendor until they realised that actually they weren't selling hardware at all, what they were selling was actually they were selling a software solution and they pretty much realised they're not actually selling software these days, they're just selling solutions, so they've changed dramatically, my warning here is if you're a hardware company pay attention, if you're developing a product and you're not embracing open you're going to go the way of the dinosaur, there are small groups of people even individuals out there and they're already finding ways to bypass large production methods, Kickstarter has become the default method of putting together a product for the first time for a group of individuals that don't want to do something with big industry and the likes of I don't know Adafruit, Spark, Fun and Bits and Pieces are small companies that are catering for the maker community, individuals wanting to tinker in electronics that you couldn't really do for the last 10 years because everything's gone to surface mount, everything's become harder and harder to build by hand now yes you can place surface mount components by hand, yes you can place BGA by hand and I've seen people do it but it's hard and I certainly know my wife gets upset if I start using the oven to bake a circuit board something about lead poisoning and stuff, I mean she's an electronic engineer as well she does know that I'm actually using lead free solder but doesn't like it nevertheless so you know we're finding ways to bypass sort of big fab plants by putting a series of modules together having open modules we're slowly slowly getting there so how does this happen in industry you know industry really is a dinosaur it's very slightly changed it's typically we're still divided into hardware and software teams the good companies out there have a water cooler or a kitchen or a coffee room and the engineers go and talk to each other and find out what the hell's going on despite the fact that management are saying no no no no you guys are separate you can't develop in your own independent silos you've got to get together and talk we've got to change the structures of researcher development teams and we need to change management expectations and I'm saying management that's a bit unfair what we've got to do is change structure and management hate changing things because well it takes time it's slow and it's hard to do right engineers hate doing it as well we hate changing things but it's not our problem it's theirs so I've kind of suggested that there'll be functional blocks in open hardware where you can you can buy individual parts to build your product up and I've also suggested that we're going to take individual bits and put it all on one to one circuit board but actually nobody knows the exact way it's going to go I suspect it's going to be a bit bit of both if we take the the other project for example anybody not heard of project or wow okay it's a mobile phone it's a mobile phone with quite a lot of industry names participating in it google are pretty much fronting it but the form of motor road of R&D were part of it, Leonardo's part of it, Toshiba, Foxcom who else do I see on here that I recognise no there's quite a lot of smaller R&D and think tank type companies involved in this but this is a open mobile phone platform which is modular so you've got your phone block you've got a screen block and lots and lots of individual parts so you can customise your phone by plugging in bits of boards together to do what you want it to do so it's I think their patch line is it's tailor made for five billion users and all of their individual needs and it's a great system and it's you know jigsaw pieces but instead of just being a single chip it's the entire ecosystem for a chip but those boards have got to talk to each other somehow you've got to identify what's on those boards somehow so how do we help as a an open source community that's been here before let's embrace it talked about these boards being modular they need to be able to talk to each other and identify themselves um beyond ecosystem within Linux is famous for for a few rants about everybody doing everything differently and the primary thing primary solution out of that I think was device tree how do you identify things that are not identifiable if you've got something attached to an io line on hardware it could be on or it could be off or it could be a one or a naught if you're reading it how do I know what it is I'm talking to how do I know what the power requirements are for this module how do I know what this module actually does and what it provides so alongside the actual functional block I also need some self-describing interface now that's fine for the likes of PCI where that was put into a specification open hardware hasn't got that specification together yet it's just beginning to come together so now's the time we can actually get in there and make those changes and actually say look let's do it right let's try and make sure that on every module every board every design we have a common at least a subset of a few common standard ways of doing something where we can have discoverable hardware where we can say what's on there but not just discoverable hardware hey I'm a toaster or I'm a fan or whatever but I'm a fan of this hardware revision using this particular design at this particular date because I guarantee the second you put software into the open people will make changes if you put hardware designed into the open people will make changes so we need a method of identifying exactly what hardware is in there so we can configure the software the right way how are we going to get around that well we do have some red tape to worry about the usual one is the bash from regulation you know it must be c approved for sale in Europe it must meet FCCID for sale elsewhere yes and no you're not actually required to pass any specific test to sell something within Europe you're required to comply with european directives the the best practice is to comply with a series of standards but you're not forced to you so long as you can warrant and certify something and people will happily accept it so you can self certify for CE now okay a lot of the large customers will not buy your product because you are not following a set of standards and standard tests to comply with you to prove your project and that's okay we're having fun we're building a product we're getting something and if somebody wants to then take this overall system and put it through a series of tests fantastic you know it's just the same as you guys get a big kick out of seeing a bit of software that you worked on all the huge pieces of software that you worked on out in the world and being used by somebody who doesn't know who the hell you are we're going to get a kick out of people seeing a product hang on i've just had the 10 minute board from down here it can't be 10 minutes now up there i reckon i've got eight minutes the other thing we're going to have to fight against is consortia this can be a good thing and it can be a bad thing there's various trade bodies business bodies out there that will suddenly spring up to support a given bus usb has its own steering committee ftdi has its own steering committee every single bus you can think of has its own steering committees and you're not allowed to use the logo you're not allowed to claim it is that bus unless you pay them dollar lots per year to be a member or you pay them dollar lots to certify your product so we're going to need somebody to step up to the plate maybe the open source hardware alliance or open source hardware station sorry i can't remember the name to start putting together some specifications that are open that we can all follow put to solve the the interconnect problems yeah these problems are not not impossible to solve they're not impossible to solve i have pretty much where i am at the moment it's pretty much where the world is and we're just beginning to to realise that there's more we can do as individuals there's small groups to get coming together rather than lots of large corporates doing them individually we can now take value add from somebody else's smaller project and add it to be something bigger and much more complicated but start thinking of things again as small subsystems now this has been a ramble i've bounced around quite a few places why have i done that mainly because my thought processes are all over the place at the moment i'm trying very hard to make this concise and straightforward of what's going on but also because that's actually where the industry is that's where the wild is at the moment we don't really know because no there's no one solution yet there's no general idea that we should be pulling in a particular direction a few years ago i used to work for a fairly large marine electronics manufacturer and it nearly put me in hospital trying to drag a company kicking and screaming into an open source environment we were using a closed source ls and the problem we had was the 3d gpu that we were using had gone obsolete and even if it wasn't obsolete it wasn't powerful enough for our new record for our next generation product we had probably the best part of 12 months to develop a new product from scratch and so around that time the product was something called the ti had a product coming out the omap 4 and we had ti come in and talk to us and ti came in and we had a session and they tried to sell us an omap 3 or the satara product line because it's now rebadged and commercially available and it's like no no no we we want this and they explained to us that we wouldn't be able to buy that product because that's for a select few mobile phone companies it's a chip you're making it sell it sell it us in the end we succeeded we were promised a lot of things and didn't get all of them but we got we got a few and i'm very proud to say i was part of a team that got the product to market inside nine months and we were the very first people to get that particular chip to market we we beat the tablets we beat the mobile phone companies and talking to ti afterwards they were very surprised that a small company was able to do this because they thought it needed to be a large team who've done this sort of thing before that they've worked with before actually you need the small teams to be agile you need a small team of dedicated people wanting to do something and make that change if only you could persuade the big players to let you have access to them nice shiny toys in essence what i'm saying is for the last few years i've stopped being a hardware engineer i like to think of myself as a hardware engineer i'm not a software engineer by anyway but most of the things i've been doing is company engineering trying to change the mindset from within and i install the virtues and beliefs of open source not just in software it can be done in hardware and it can be done pretty much everywhere else by pulling together sharing information and not hiding it by working as teams by helping each other we can do an awful lot more and i'd like you to come away from this today perhaps having a think about is there something open source software can do to help your neighbours in another area can you help us learn and to show us the right path any questions it really was that boring wasn't it really nobody Daniel you say that you've been primarily company engineering trying to to instill open virtues yeah in the companies that you've been working with have you also been doing any hardware engineering that you want to talk about um last year at debcom i turned up with a prototype board uh for the company i'm working for um it looks remarkably like a beagle bone neil half eluda to it in his bits from the dpl thing it's a small communication aid for people who have no voice now this is probably the fifth generation of this product it was originally started in 1973 so it's as old as i am um neil wookie uh who else have we had on working on this over the years noodles yes um vinsel though we all deny it um yeah the product's been around for for a while and yes that's uh that's now actually going to be for companies first fully open product the hardware is it is entirely open um i've been doing something with neil uh uh for the lanaro something called open tack which is just about at the point of of gaining some traction for software because we've just about got the hardware into a workable state and by open hardware i don't just mean here are the gerba files here is a bill of materials and here is a pdf of the schematic i really do mean open hardware so we're providing the CAD files in raw all the reports to run against the CAD all the design rules all the library files are also available as well as clearly the gerba files if you can't be bothered to to open it um the schematics are available in pdf and you can search them but yes they're available as part of the CAD um and then obviously as you i suspect eluding va um we would desperately desperately like to make an open source laptop um but reality is at the moment we're getting absolutely nowhere because we don't have access to the silicon i'm getting a silicon vendor to talk to us about that is hard because we are this tiny little team a few people wanting to change the world anyway i believe my time is over please do come and have a chat with me um if there's anything you think you can add the world needs the world really does need people to come out and say yeah we can we can make a change here we can make this better um and it is going to be from small numbers of people it's not going to come from big industry this time thank you very much Andy for your enlightening talk developing products in the open