 Hello everybody and welcome to the post-launch session. I don't see sleepy spaces as well. So the talk is extending Android with new devices and we'll show how you can support a joystick and barcode scanner with Android. Okay. Before we go ahead, a quick intro. My name is Sri Kumar. I like to work with my Sri. I work in Inomine software. We make devices which are Android and Android. And I blog at my website, SriKumar.io and I'm more of an adventurer. So if you want to see some cycling stories, you can Google for Sri Kumar and cycle, so you can find some. And Nithi. Hey guys, I'm Nithish. I'm an engineering student in computer science from ES Institute of Technology, Bangalore. So with that, let's just start off with these slides. So what are we doing? So I don't know if any of you attended the talk by Texas Instruments yesterday. So that was about how they expect Android to be used in a lot of scenarios, not just in books, tablets and mobile phones. So Android is expected to proliferate and that's beyond just the traditional Android strengths. So at least that's what people are expecting and waiting on. So if you're going to have to go to new areas of application, you will invariably end up having to support new devices. So yesterday one example was some washing machine or something like that. What does a washing machine have? Perhaps some motors. And then there will be applications running on top of that. They will need to use a motor. So there's new devices and new APIs. Next. Okay. So with that background, what are we going to do in this talk? So a quick look at it. So we'll first have a look at how devices support Linux. So this is important because Android is at heart a Linux kernel with modifications. Then we'll jump into what our layered approach is for supporting new devices. So we have a layered approach identified. And we'll go through that. Then we'll give you the two examples that we're talking about. The joystick and the industrial barcode scanner. And bear in mind that all these things are all this talk is from a device maker's perspective. So there's typically need for faster time to market. And the other thing is compatibility with the Android market. So Android devices are very popular because same applications run on multiple devices. So keeping that compatibility is very important. So we'll try to keep these two things in mind. Next slide. So how are the devices supported in Linux? So there's just a quick layer view. So you have the device. You have the Linux kernel on top of it. Linux kernel has all the drivers that are needed. And if a device is detected, device nodes corresponding to that are created. And there's typically a privileged demand of some sort for any kind of device which has access to the device node. So either the permissions of the device node are managed or the demand is running as root, one of the two. So if the permissions of the device node are managed, then that is done through UDAB. So if there are any Linux-based Android developers here, you would have, at an ETC, you would have those not defective and Android-proof, this file. This gives you access being a regular user to the NDB functionality, which is only going USB. The privileged demand just set you ID or set GID. So of that, you typically have a device API. And finally comes the user app, which interacts with the device API and gives you what you want. Next slide. So what's our Android approach? So Android is entirely Linux kernel, as I was talking earlier. But it has a very different user space. There's no UDAB, all the regular services are done in a very different fashion. But the underlying thing is Linux kernel. So we start from there and we build a layered approach all the way to the user application, which is running on Java. So we start from the kernel, from the device. So typically the kernel won't have the required device drivers for the kind of devices you want to support. So Linux kernel has a lot of drivers. But Android will typically have something which is very much stripped off. So you may want to add device drivers there. And that would give you the device node, which the upper levels need. So from a service point of view, we use a JNI interface to tap into the device node. And device provisions are managed by another demon. Not actually demon, it's something else. It's called UEVNT in Android. So if you go to older versions of Android, you will not really find UEVNT. But Gingerbread and Davao have, I think, have UEVNT. So this is basically a very stripped-down, UW kind of environment. And that has the effect equivalent to giving a particular app ownership, a particular user or group ownership of the device. So after that, you learn your services, which are on Java, which use the JNI interface to talk to the device. So the service should be running at the right permissions as the right user. So if you're an Android app developer, you know that apps run at user IDs 10,000 and higher. You do PS on Android system, you'll see app underscores and things like that. So all those are non-privileged users. And they will interact via IPC with the service which is running as a system user or some other user. And finally give you the required functionality. And one thing we have done here is that we use a platform library approach. So what's a platform library? Basically what happened is platform library is a way for device manufacturers to have device manufacturers have their own specific pieces. And so once you have a platform library, applications can be built which have a dependency on the platform library and market will actually do the filtering based on that. So people who don't have specific functionality, they will not see your device when it's searched for in the market. When they search for device in the market, if there's an app which needs a particular functionality, it will be used only on those devices. So first example that we have. Supporting a USB joystick, why did we choose this device? So we have a student working with us as an intern. We did a simple example, so we started off with that and he chose this device because he had it. So we would play games with it, so he started off with it. And our intent is to map concepts to details. So we already shown you a quick layering. And this is actually a very minimalistic requirement for this kind of device because it's a USB device. So you only need a system and a USB host capability. So with that I want to give it to Nitish. Thank you guys for explaining about the steps involved in providing support for the USB. So I kind of wonder that it would be nice if I could show you the example first, the application that I have developed for the USB. And then I could map the same concepts so that you will understand the kind of layered overview. So that's what I will show you. First show the application and then I will talk the concepts through about that. So we have Android X86 running on this. So yeah, what we have here is Android X86 running on our laptop. So we don't have an ARM-based system, it's an X86-based system. So I have installed a Flory on top of my laptop. And let me start my application. So yeah, I hope you can see. I know the last part is cut out, but that's how my keyboard drives it inside. So if you can see it's a logitech version gamepad, which is a 10 key gamepad. So the simple application that I have here is I just start and then you can see the buttons coming up. Look at that. So you have all your buttons there. So you are able to see those red dots there. So that's when I click my buttons. So that's what's happening on the UI. So this is the application which I did. You can take it to any other levels that you want to play in a game out of it. But pretty much anything that you get from it. So I'll explain you to the concept that we can go with this. So switching back to the slides. So step one. So that part on that corner will tell you what layer of the Android frame that we are modifying at individual steps. So the first part is you can see the device. You don't have to do much with the device. The device is already ready. So you can just start utilizing it. The next part I will have to work on is the kernel. So the kernel comes below with the HID device drivers. That's what you are a josek user. So you don't have to do much with the device drivers manipulation there. All you have to do is boot up and check if it comes out. That is one of your input subsystem device nodes. So that node you would have to get in the plus file structure there. Slash there, slash input, slash event X. X could be one, two, any of the numbers there. So you can do a small regular expression and find out that mine is a lost equation. It has event 15. That's how it turned out in my system over there. So this is your first step how it goes about. So now that your kernel is out to talk with your device, the next step we have to do is communicate with the device node itself. So how would you go about with Android? So you have your JNI interface where you can write your native code. So the first thing that you do is write a normal C code which can actually talk to your device itself, right? So the file, they identify that it's the last step slash input slash event 15. So the first step that you do is reopen that file and start reading events from it. So it's as simple as open up that device in read only mode and then start reading from the device. If you can see, the key part over here is the event structure. You can see that. So this structure, this is a struct input in underscore event. Now this maintains your key mapping itself. The important part in that structure is the scan code and the value that you get. The scan code represents uniquely represents your individual button in the key joystick and then the value for key down or key out. So that's what we'll have to do with it. So that's the basic step. And then you would provide a JNI wrapper around the code itself. So if you attended the talks yesterday, one of the last JNI talks, so there was a requirement of the JNI all over where you register all your native methods. So what is the simple structure which registers the method there? This for those who don't know JNI. So that's the way in which you access, which along native code from Java. So basically write native code, write wrappers around it and the JVM loads that native code and can execute it like the class method in the Java class. So in my JNI code, I just have three simple functions, right? So if you can see it's joystick open, joystick get key and joystick first. So that's the three primitive functions that you would need to communicate with the device. So what you'll be doing with the JNI device is you'll generate shared library which will be used by the Java code itself. So that's what the JNI stuff. So the JNI on-load function there is used to register the methods. So when the VM loads the shared library, then it's called JNI on-load. So there we register the methods. So that part of JNI wrapping is done. So the next thing is that now you have a JNI code that can communicate with the device. But if you see here the slash step, slash input has permissions. So you require your app to have permissions to access the device. The next thing that you would do is, like she talked about, you have UAMD that manages the permissions for your device nodes. So you would have to go and change your permissions. So what I've done here is I've changed my input device permissions to AID underscore system. That's an Android user named system which has access to most of the Android features that. Well, if you didn't talk about it, system code in a device is not seen. So this is all part of the Android source code. So here we are actually modifying the Android source code itself, specific to this device, and changing the permissions of a device node so that that node can be used with the demons that we are going to run next. So that's the part for the flow issue of using the jibbon about UAMD.RC. Now that you have permissions, that was the next step, that you had permissions to access the device. Now you know you can communicate with the device. Now you have your native code that can interact and print the messages from your device and you have permission to access it. The next thing is that you would have to make all of them available for your application program to have it. Your application should be able to access it. So that's where the services comes into picture. So what the structure that you see over here is, this is all my DNA code which actually talks about the native applications which access my device. And then here is my service. Now my service uses my native features to access my device and provide the same features to top the level of my application. So what my service uses is the joystick API, as you can see, and joystick call button. Both of them are interfaces, ARDL interfaces. So I don't know if you guys are interested in the ARDL services. So let me explain what kind of those interfaces were. It's just an interface to the service where the applications can interact with the interface. So here I was telling you all of them are my native codes. And then here is my service that uses my native codes and then makes the same thing available to my application at all. And then these are my interfaces which the application can use to talk to the service. Okay, see if you can go to the next. So the service that you create, since it's accessing the device, it needs to have permissions, special-handled permissions. So here you certify your service saying that with your local certificate platform, and then you're creating your own device and service that you will have to create a permission of your own. So that's what is over here. So when you say permission and by name of com sample, it's called joystick. So your applications when it uses the service will have to use permission of that permission. So that's the permission that's being created over here. So this is the next step where your service will take permissions. So yeah, so this is where the inter-process communication between the service and the app takes place. So I'm running a little bit fast since we wasted a little bit more time there. Your service uses this interface to provide, as I said, service to the application. So this is what happens. So your joystick API has two methods, set callback and clear callback. So whenever there's a key press, your application can set a callback. I mean what has to happen when that key is triggered? That's what takes place over there. And then this is what happens in the service place. In your service, you create a new API. Every time there's an application that's binded to the service and you put it in also over here. What has to happen during the callback? That's what's supposed to be over here. So this gives you an idea about what's happening in the service. Okay. So here we come back to the platform library. So as she told, it's kind of a feature that Google came up for the OEMs to provide support without modifying the internet framework. So we'd have to have your platform library available. So when your application uses platform library and you're downloading applications from Android and not getting only if you have the support, will that be available for you? So in order to have this platform library, there's this one thing that you'll have to do. So you'll have to have an XML file that decays your library and you have to put it up in your slash systems, slash ETC, slash functions. So that notifies the Android framework that you're having a platform library of your own. So the output of this platform library is not an APK file. So that's what here is saying. So it's a normal charge file containing all the features and the services and the interfaces that the application is going to use. So that's the output of that file. So I'll tell you where this will be used during the application program. So now that you have your, the kernel part is done, the GNA part is done and you have permissions with the UoIndy and you have your service up. Now the next thing is that, how else the application uses the service to start interacting with the device. So the first thing you do is bind to the service that we created previously. That's the same thing as what we said there. So just a service. That's the service that the first thing you do. You bind to the service then when the service is connected, right? So that's when the service says that it's been connected to your application then you create a new API. So this is where you create a new API. So this is the main communication between the app and the service. So once your service is connected you say api.setCallback of your callback function. So if the service knows that at any time there's an event, key event happens that this callback this is the basic structure of how your application looks like. The next thing is the callback itself. What happens in the callback? This is the function that has to be defined by the application. So every time the callback gets done you have to do your logic over that. So what I'm doing is just updating my UI and since the Android UI library is not thread safe you have to run your device on the on UI thread with that itself or you can use the async task to do the same thing. So every time there is a key event that happens I get the key back. So if you see on keypress in R0, R0 gives me what was the scan code that was clicked over there. So I get the scan code and I update my UI to transfer to the scan code. That's what is happening over there. So now you clear the usual steps that take place to just come up till over here, right? So that's part of the application. So I previously told you that with the platform library you have to use the jar file. Now this is when it's being used. So how would your application know what does your application have to import to access those services like joystick service and joystick interface. So this is the Java library that you'd be using. So in your Eclipse if you're programming Android you have to have your user library that is the same library that was the output there. So that when you have that library included you can start recording the packages. The service the joystick service and the joystick callback and the joystick ABI interfaces. So that's when this comes into picture. Next thing is it's all done, right? So now this is the overview that we've seen what happens in this video. So this is that. So if we go through from first so here's the application that first binds to the service and inside the service the upgrade method is called where the joystick text starts opening. So I didn't show you the sort of the service but since we're running short on time so just abstract that saying that there are the worker thread starts off and then it returns your joystick ABI back to your application. My application instantiates the service and then that's how it can talk with your service running internally. So when you set up a callback you again go back to the joystick service and then that's where your worker thread starts. So you start up pulling your device asking for is there any events that's generated from the key. So anytime there is an event generated a key, you get a key and that key is sent back to your application. Your application then updates you based on the key and that's how the complete workflow goes around with it. So that's pretty much basic about how you can interface with the joystick. You know it's kind of a little bit abstract but you need more information so you can just contact us after the session so we'll be glad to help you with that. Any questions till this morning? I'll take a question. So I think then we'll go back. So Nithya actually covered what happens with the joystick completely. Thanks. Alright so our second example is a Bible scan in our case we have chosen the Opticon MBI 2300 okay. This is a very engineering description. So the company that makes this is Opticon and they actually make a module so this is actually a development board okay which has the scanner module which is here and the image processing module which is here okay. This is mostly development board which is which are all interfacing and everything okay. So first thing you can ask me is why do you want to use a dedicated device right. Everybody knows that there's an android barcode scanning application and everybody asks me why we need a dedicated scanning device right. So a final dedicated scanning device may look like this. So there's an android X 1900. So why do you want to have that because first it supports a ton of symbologies so that means that it can decode barcodes of a lot of time. So as you can see on that right side there are lots of types of barcodes so this you can scan and convert about 46 types of barcodes and a lot of subcategories inside that. Whereas the ZXC library which you perhaps see on your android phone that can only support a few and some of them as you can see are alpha quality so people may want to use barcode scanners they want absolute reliability they don't want some somebody just walking around and decoding barcodes one at a time very slowly with the auto focus of the camera working on that. So a real barcode scanner device is typically very fast it has other features like a very long cable illumination the red light that you see. So all these are very industrial features. So it's not the same thing as you're going to a bookshop and you're scanning a barcode and trying to figure out what this thing is. So that's what ZXC is primarily meant for. And the last thing would you really drop your smartphone? Real world applications do not feature. Okay, next slide. So the first thing about the barcode scanners is that they're very different than the kind of devices that you're typically used to. So to get the barcode scanner to work you typically scan barcodes. That's one of the interesting things. So this particular Opticon barcode scanner there's a website called opticonfigure.opticon.com you can actually go there and select some menus say that, generate a PDF file scan that and the barcode scanner can be configured. So that's how they work. So they are very specific purpose-built devices. Okay. And the actual barcode scanner can vary on the device. So don't try this on your supermarket. Okay. So if I can go there I think we will have a quick demo of how this whole thing works. Right. So you can see the application is running. So we start from the beginning. So this is the Android again. Okay. And I hope it will work. So we have done some hard coding. Like on the level for certain things. Just simplify coding. So this is the application which you can go and run. And it's waiting for barcode scanning to start. Okay. These are all very realistic applications. Just realistic concepts. Barcode scanner here. Okay. And you need real barcode. So I have sample barcode sheet here. So I just scan one of these. Right. So you got a value there. Of course, at this point you just have to believe me that it's the same as that because all the work is being done by the device. So barcodes are used for a variety of purposes. For example, in our office we have ID cards with barcodes in the back which import an employee ID. So this is just a simple open switch. Okay. So now let's go and have a look at the internals. So barcodes can have a very simple device. And there's not much to do. When you shoot you get a barcode. Everything else is application logic. And we are just talking about just the system implementation. So we will do the strict analysis with that. So the first thing that you need to do the USB device that we talked about earlier. The joystick. It had drivers inside the kernel. Android kernel. Okay. So it needs a cdcacm device driver. And typically the next kernel will have it. It will be disabled in the android config build. So you have to enable that. Sometimes it will so happen that the kernel tree on which android is based might be somewhat dated. So you might have to take the latest changes. So these devices are USB based. So they typically don't have a direct dependency in the hardware. They have a dependency in the protocol. So the protocol is going to work mostly irrespective of the kind of Linux version that you have. So you just have to find the newest driver with the latest patches. If you see that something is not working. So you find a Linux kernel which supports your device. Let's say your desktop. You try out your scanner. You come to android. You enable the configuration options. Boom. It's already. So what you need to do may be dependent on the exact kernel that you have. So on the kernel that I am experimenting with I need to add a few lines to the driver so that it recognizes the device. This particular optical module. So after you make those changes you should typically boot your device with the device from London and check that it's really supported. See if it's really interactive. Okay. So how would you interface with the device? Now you have the device running. The kernel is ready. You can actually interact with the device. So again we adopt the same method. We write some C code which can actually go and look at that device node. So typically when you plug in a power port scanner it will show up as dempttyacn0. Of course these are very hard ported. So in reality what may happen is that the device node number may change. You will have to go and root through to figure out which exact device it is. You have to look at the USB, data ID and product ID to figure out that it's the right device. Okay. But for now that's enough. So bar port scanner is a very simple device. So you open the device node you will typically get one line per every bar port scan. Okay. So this again is a simplification. But you would typically for typical bar ports you would get one line per bar port scan. They are all delimited by a new line. Okay. So what's simpler, right? So we just wrap it around using a native port put some JNI on it. But again you hit the same problem. Sheesh. Mistake in the slides. I used the wrong thing. So dempttyacn0 the node it will not have permissions to it will have permissions to root. It will not have permissions to any other process. Okay. So we will not be able to use it directly. So what we really need to do is we need to fix dempttyacn a shortcut to make it working is to make the owner system. Okay. So there is a file called und.rc which controls as I was saying earlier what permissions are assigned to a device. So we can go and say .dyacn0.660 that is read write permissions to the system and root. So owner of the system and root is root. Okay. And then any process which is executing as android.urna.system So any process which is executing as android.urna.system will be able to access the device. Okay. There is an alternate method so you always talk about android compatibility. There are strict compatibility requirements which means that we cannot alter the API. But in the manifest file you have shared user ID. So one method that I have found is that you can define your own shared user ID in which you can give any permissions that you really want. So this line here actually assigns if you use com.sample.uid.acn inside your manifest file that will map to the user ID usb. So 1018 is the number for usb. Okay. So it's very simple to do. This does not break compatibility and provides you a way to run applications with any permissions you want. Okay. So again the API for bulgur scan is very simple. You just scan bulgur and you register a callback. It's exactly the same thing as that. So we will discuss this more. Of course you have to sign your apk using your zwapk with android.urna.system and you have to sign the apk with the platform key. Okay. Application interface is also very simple. But just like what I showed you. Okay. So I think we quickly come to the end of this talk. So as a matter of what we discussed till now. You probably steps in enabling a new device. Okay. We kept, we gave you two examples which are kept simple on purpose. This illustrative of samples. We skipped a few things. We didn't talk about power management. We didn't talk about deeper system integration. So we made services which are independent of everything else. That's not the way motion panel works. But this keeps the concept very separate. For the device maker you can actually tamper with system server and add the services that you want directly there. Okay. So that we come to the end of our talk. Any questions? We have time for one. Yeah. One question. One question. Are there any questions? Okay. Okay. So thank you very much for attending. I hope it was useful.