 over here at the embedded world, Nuremberg. Hello, what's up? Hey, how's it going? So two years ago, you were talking about Android is the way to do embedded, right? That's right. So is it being going on? It has, I mean, it's been going on for a while now. And I was just doing a talk about that today. We were doing a four-hour session on embedded Android, and we also did a talk on Android Things. So Android Things is pretty cool, you know, an IoT, essentially, spin on Android. And, you know, it's been something that I've been following for a few years to see how we can take Android and do, you know, headless things with it. And now Google's actually investing effort into that. So that's fairly interesting. Since when are they doing this Android thing? So Android Things actually is the new name for Brillo. And so Brillo was announced at the Google I.O. in 2015, and they had that running for about a year or so in a developer preview. And then they decided to rebrand that into Android Things. And so the main difference essentially is that whereas Brillo had taken away all of the Java stuff, Android Things brought it back. And so they now have an extended API that allows you to talk to traditional IoT-centric buses and things like that. So I, too, see GPIO, SPI, that kind of stuff. Why would you want to have the Java Things stuff back? Why is it good for embedded? Why is the Java stuff good for embedded? I think the relevance really is the fully integrated Android stack that app developers are accustomed to. So it's not so much the Java side as much as it is that you are back to the same development environment that you're familiar with if you've been doing some Android work. And now it's been extended to actually provide those IoT APIs. So Android is a thing, is the right thing to do? What would you say? I think that if you look at obviously the fact that we've increasingly got a number of different devices that are connected to the internet, and that Android is an operating system that has been engineered over the years to encompass increasingly levels of sophistication for all sorts of purposes, there's an actual juncture there where essentially you can look at, essentially, Android as your go-to operating system for all things that are connected to the internet. And so one thing that Android Things allows us to do tap into everything that Android already does and do it in a fashion that is geared towards IoT. So I think there's still some mileage to be covered in terms of, say, making it more than just as it is now a set of demos around IoT boards. So they have support for the Edison board, they have support for Raspberry Pi 3 and a couple of other things. And the present kind of things they've got out there are mostly, I don't want to say this in a negative fashion, but they're toy circuits. They're not mass production things. There's some stuff that you would deploy on a massive scale. But I think that's where you have to start. You have to start where you say, hey, look, these simple things we can do and look, by the way, where we can go with this. And so I'm really looking forward to see where that goes. So do you have to make Android thinner, lighter, make it run on Cortex-M or still Cortex-A stuff? Oh, no, no. Okay, so this is the full Android. And really those boards that we're talking about are fairly high level in terms of capabilities. I mean, they're really powerful stuff. In fact, one of the aspects I was covering in my talks is an interesting, say, thought experiment where we could mix, say, Android things with Graybus, which is the bus that was created for project era. And so in the context of Graybus, we were tunneling those things like I2C, GPIO, and Spy, between the phone and the modules that were connected to them. But then you can take that and bring it into a more widespread use with Android Things. And you can say, for example, imagine that your Android Things board becomes a IoT Hub. And then your IoT Hub is controlling all these gizmos around them over this Graybus protocol. Because there's a lot of IoT ways to do IoT. Arm is talking about embed, there's Zephyr. But all these things are kind of smaller IoT stuff, right? They are. And so you would, for example, run an M0 on those MCUs that would be on the other side of Graybus or you'd run Zephyr on the other side of that, whatever you want. Unfortunately, it is hard to predict what catches on. And despite all good intentions that organizations and companies put into specific technologies, the market has a weird way of designing on its own. What actually takes off. And so that's going to be interesting to watch because there are a lot of people trying a lot of different things. And sometimes the thing that sticks comes out of dark corners that nobody's thought about. So since you did books and training on Android and stuff, right, what's the latest you've been doing? So as I mentioned, I've been taking a lot of interest in Android things, given that Google has over the years been interested into this aspect. So they had done, a long time ago, they had added a flag called RO Config Headless which was for the Nexus Q, I believe. Then they came out with Grillo, now they've got Android things. So there seems to be a long-term interest in that aspect. So I'm definitely tracking that. With the rest of Android, embedded Android, that has continued to gather market share and gather thought share and so on. So I mean, we're increasingly seeing products that are based on that for sure. And all the stuff Leonardo guys have been doing, basically it's been merged into the full Android that everybody's using right? Yeah, that's awesome stuff. I mean, Leonardo guys upstreamed the high-key support into AOSP. So I look forward to more boards actually going in that direction. I mean, I think that, so a long time ago, there was mainstream support in AOSP for the panda board. Obviously that, whatever, didn't work out. But so seeing the high-key there means there's a space for that kind of stuff to be found in AOSP. All right, so we're here at Embedded World and you have a friend right here? Absolutely, so meet Chris. Hi, Chris and I, we've been doing some work together for a long time and Chris has a lot of expertise in embedded Linux and he actually had a book come out last year, right? I did. What happened with you? So who are you? Just in case I was interviewed by Charbox, I actually bought my book along with me. That's awesome. So this is Mastering Embedded Linux Programming by Chris Simmons. It is probably the best book on Embedded Linux that you can find right now. Wait a minute, wait a minute. No, this is good. This is a very good book. So Karim wrote one some little while back. That's right. He hasn't updated it recently. I haven't updated it in a while. And no, no, no, this is a, I definitely recommend Chris's book for sure. This is updated as of last year. So it's only one year out of date. Yeah. So how do people program on Embedded Linux? It's changed a lot over the years. So we used to do what we used to call, roll your own, R-Y-O. So you used to sit down with your computer and you get a copy of the kernel and you can cross compile that and oh, we need cross compile U-boot and then you cross compile every single thing bit by bit. And then eventually after two or three weeks of frustration, you'd have a system which actually would boot and give you a command prompt. That has changed a lot. On the one hand, we have things like Raspberry Pi, BeagleBones and such like, you can buy something off the shelf and embedded board off the shelf and power it up and it will just work. But for professional use, people these days mostly use a system called Yocto Project. So Yocto Project is a framework, a set of tools that will allow you to create an embedded operating system, embedded Linux operating system with comparatively little overhead. The whole thing has changed a lot. The whole thing has moved up into higher layers now. Doesn't mean to say it's any easier but it's just that the work you have to do is moved into a higher layer rather than sitting down there cross compiling every single component. So that enables more differentiation. People can put their efforts into something more interesting. Exactly. It means that the engineers doing products today, they can focus on the differentiating factors of their product and not just doing the basic compilation and system management. So what do you think about Android? Is it the best Linux? So Android is the most popular Linux distribution ever. At 1.5 billion installations, there's more Linux in Android devices than anything else. That makes it the best, right? It makes it the best in certain categories. No, no, no, you're quite right. No, so the thing is, the one thing I should add is, so I love Android and obviously I'm doing a lot of it but the thing is, Linux is in a vast amount of products that are not necessarily accounted for, right? Because, so Android is pushed by a company that is keeping track of its progress and that's awesome, you know, that's great. But you buy a ton of products on the market. They don't necessarily advertise they're running Linux, right? But they are. In fact, you know, it's an open question as to which is running more devices, plain vanilla Linux or as an embedded Linux versus Android, you know? I'm not exactly sure. Does my camera run Linux? I don't know, I mean, we wouldn't, you could actually- It boots up in like a second. I think it does. I think Kerry and I know the guy who does the software for so many cameras. Yeah. Now, so actually the best bet to know if you are running a Linux product, at least if from a major company, is if you open the manuals that come with it, right? They will usually tell you that the licensing of the software is such and such and it includes this package and that package. So if you want to find out if your camera has Linux just get the manual and look at it. I went with my camera very up close to one of the booths startup over there and they had the magnetic thing that was turning around on the table and it kept rebooting my camera. Oh wow. I don't know if that proves anything. Oh no, that's just their RF compliance that's going on, that's, I'm not sure. Or the device on the table was emitting RF, it wasn't supposed to be emitting. So there's so many- You should probably go for a checkup. There's so many Linux distros and some want to focus on embedded, others don't care or how does it work? Some are like, they specialize in different areas. Is that how it works? Whoa. So an embedded is mostly not distro based most, so the classic embedded manufacturer these days is using something like Yocto Project to create their own Linux distros. So the thing about Yocto Project is you give it a list of instructions and then you type BitBake, something or other and then you walk away for a while and when you come back it's built a complete Linux distro for you. And then you put that on your product to where you go. There are some distros that are kind of familiar to people but also on Raspberry Pi you have Raspbian and if you're working on Buildable and then they have a Debian distro. There is a differentiation however, the problem with distros is they don't really fit that well into the embedded market's place. They're difficult to update, they're not so robust, they have a whole bunch of features you don't really need. So the thing about an embedded computer is that it's a computer that does a specific thing. I like to say it's a computer that you don't know is a computer because the computing part of it is hidden inside the camera or the TV or the pizza control oven or whatever it is. So you want a very specific single purpose system. If you use a regular distro you end up with a load of stuff in there that you don't really need. You end up with vulnerabilities, you end up with network ports being open that you don't really want and so on and so forth. So professional embedded tends to be based around things like Yocto project and build route and PTX dist and certain other similar build systems. So are we going to be able to get security in all these devices or still not? Security is not a technical challenge from my perspective, it's a business problem. The issue is if you are in the business of selling devices, then you need to sell more devices to stay in business. So servicing or updating older units is not in the best interest of your business and we can't really blame the manufacturers. I mean, that's how their business model are built and if you're a consumer, ask yourself, are you willing to actually continue paying a subscription service because that's what it would be for devices that you already bought so that the manufacturer would continue updating them and most consumers would probably not want that. So I mean, somewhere in there, somebody is going to have to find a solution and I think you look at some of the work that Google is doing, I think they're trying to actually put things in place to make that a bit more viable, long-term for everybody, right? Because that's a challenge, you know? The manufacturers have to continue selling and the consumers don't necessarily want to pay. So how do you solve that? I'm not sure. It's an interesting challenge. I think the subscription model is where it's going to have to end up because there are a lot of vulnerabilities in systems these days that have been advertised a lot recently. Cameras, IP cameras that expose ports which have been exploited in the mirror, exploits and tons of other things. And the problem is exactly as Karim says, the manufacturer makes a thing, a camera, they sell it and that's the end of the transaction and that's the end of the revenue stream for them. That cannot work long-term. Every device that has software and is connected to the internet has vulnerabilities so there has to be a way of uploading new software, new firmware to those devices to fix the vulnerabilities so that we don't all get inundated by continuous denial of services and spams and such like. So I think you're right, the subscription model has to be the way forward. There's many devices even from very big brands like Android devices that run on old Linux kernel. Absolutely, yeah, that's true. It's a complex problem because there are so many layers involved in getting Android to work on your device that at some point you have to standardize on some version and then you build the other piece on it and then you move on to the next project which is going to be on the next version of Android and so on and so forth. So we had some discussion about that actually at the Linux Plumbers conference as part of the Android MicroConf and people are encouraged to look some of that up where there is effort in trying to see how that can be realistically be made to survive in time where essentially the device that you got is going to get those security updates and so on and so forth. Because unfortunately, if you look at the media, it ends up being the manufacturers that get a bad reputation out of those things and that's a bit unfortunate because it is also part of the consumer, right? Is the consumer willing to pay for those things? And at the end of the day, if nobody wants to pay for it, it just doesn't happen. Because hardware is awesome but software is three times more expensive, is that true? I don't have the numbers myself because this is not something I've looked at recently but definitely if you look at what the hardware vendors have been doing in the Linux ecosystem or the open source ecosystem in general, they've been pouring a tremendous amount of resources and money into making those ecosystems successful and the reason is very simple, right? You're not going to be able to sell ARM chips if the software doesn't work on them. You can't sell Intel chips if the software doesn't work on them and the rest of it goes hand in hand. So I think, I'm not exactly sure what the ratios are but I do definitely think there is a co-dependence. You can't sell one without having the other. I think historically, the software expanse has been seen as a one-off expense, cost to a project and then there was a certain cost to produce every unit of a product but then you make a markup on that and that's how you make the profit. So you take the software development as part of the initial write-off and then after you've made so many units you're making a profit. Where that's breaking down now is that software is no longer a one-off expense. You have to keep on developing the software, you have to keep on supporting the software. So now it becomes an ongoing expense and it comes back to this idea of revenue model. How do you make money out of that? Do you just include that in the hardware price, maybe? Or do you have some kind of other revenue stream? So in the case of Google, the revenue stream is advertising. In the case of other service companies, they make money from indirectly from selling the service. So set-top box manufacturers, they kind of make money through selling the set-top boxes but there's also a revenue stream for the subscriptions to your Skybox or your Comcast box or whatever. There are various models but they all revolve around having some kind of ongoing revenue stream or some kind of ongoing relationship with the consumer. Maybe the taxes should pay for it, the government. That's a weird. I'm not sure how long that will go down. No, I don't think so. If you look at some of the massive efforts or open-source things that are being done by large companies, I think Chris has a good point there where essentially the software isn't an enabler for something else. And insofar as that something else is generating revenue, however that may be, ads, content selling, what have you, then justification is there for the software to be made at first and in fact maybe even to be made open-source. In fact, there was a good presentation at one of the open-source conferences, the reality open-source conferences by Matt Essay about the business models around open-source software where essentially was making the point that you can't actually build a business model around open-source software because there are so many companies doing great open-source software for other reasons which are not necessarily to build a business model out of the software itself but give the software away and hey, by the way, this is an enabler for another business model on the side. One thing I'm wondering maybe you have a feeling for it, is there enough talent out there to actually get all this done or is it limited resource? Like, there's enough guys who understand this stuff who can get it done. So what specifically are we, what do you think? Are there enough engineers to update all the stuff that you'd like to update or to keep everything updated if that would be an issue? I don't think so. I mean, I think it's, you know, resource allocation is resource allocation. If there is a problem that has to be solved and it is, you know, properly structured and justifiable from a business perspective, you can find the resources to do it. I don't think that's the issue. I think really it is a business model issue and it is a, just, you know, is it justifiable for, you know, brand name manufacturer, not to name anyone, but to actually hand, you know, put their hand down and say, well, okay, you bought this device for me. If you want me to update it, it's going to be $10 a month, right? Not sure how viable that is. But you have the metrics. You know how many people are buying your books and going to your classes and you know that there's a lot more people joining this industry, right? There's a huge amount of talent out there. There's a huge amount of people who know this stuff and Karim and I are working very hard to make sure that they keep up to date with all of that. So I mean, there is a skills level issue but, you know, a lot of these people are very self-motivated. They're quite able to read the various web forums and to attend conferences like this and so on. So there's a lot of talent out there. It's just a question of channeling that into the right areas. Some of that is done through the open source community. So I highly recommend people to contribute to open source projects. You know, it has many advantages, both personal and for the community. They get discovered, right? They get discovered, exactly. So the great thing about open source, working with an open source community is that people know who you are, whereas you can work in a large corporation, you can do the most brilliant software ever but nobody will ever know. Whereas you work in the community, your name is there, it's on the files, it's in forums, people recognize you. And they can check your code and know if you're good. Exactly, yes. So that, yeah. That's a double-edged sword, but yes. Sometimes it works out okay. That's right. As long as people understand what's in your book, there's easy for them to get a job, right? Yeah, definitely. This will definitely help. That's right. And Chris will be paying you after the video. I'm joking. Oh, I think you're... No, definitely, definitely... Definitely will be a bear. That's right. No, I saw...