 Who here is excited about web capabilities and project? Fugu! Whoa! Great. Awesome. So my name is Tom, and I'm a developer advocate at the web team at Google. And I see the web as the universal development platform. And when I describe my role, I tend to say my job is to enable developers to do anything they want to do on the web, as they want to do on anything they do on the web, and expect them to do on native. And I'm Pete. I'm also a developer advocate on the web team. I love my job because I kind of look at it as I get to take all of the sharp edges of new web tech and make it easier for you to use. Now, I'll admit, I am a little bit biased. I kind of love the web. I think the web is the best platform out there. What I love is the scale, right? It reaches users all around the world, essentially on any device. It's easy to use and easy to share. There's nothing to install. It just works. But most importantly, it's an open ecosystem that anyone can use or build on. Almost everything I do today, I do on the web. I've got a tab open to Gmail and Calendar and Drive. I do all my code reviews in GitHub. I write all of my samples and stuff to make your lives a little bit easier on a glitch to try and just, it's all on the web. It's kind of awesome. And in the last couple of years, the capabilities of the web have grown tremendously with things like web USB, camera controls, media recording, and so much more. But let's be honest, the web does have some limitations and there are some apps that are just not possible to build on the web today. If you want to build something that interacts with local files on the user system, nope, can't do it. Want to do something that interacts with the user's contacts on their phone? Sorry, nope. Want to register as a share target so you can receive shares? Sad Panda. And as cute as it is, it's still a sad panda, right? We call this the app gap, the gap between what's possible on the web today and what's possible on native. And on the Chrome team, we want to change that gap. We think web apps should be able to do anything that native apps can. We want to remove, yeah, who was that? Who, yeah, all right. We want to remove those limitations and so that you can build and deliver apps that were never possible before on the open web without requiring some kind of framework or anything like that, just straight on the open web. We've been working to close this gap for some time now and with things like the web app manifest, which makes it easier to create an installed experience, with service workers that allow you to get that performance and reliability that users expect from an installed experience, web assembly that gets you closer to the metal and so much more. But some of these features that we're going to talk about over the next 40 minutes or so seem a little scary at first, especially when you consider how they're implemented on native. But the web is inherently safer than native. Opening a web page should never be scary, right? Adding these new features needs to be done in a way that respects user privacy, trust, security, and the core tenants of the web. Our internal code name for this project kind of alludes to that, Project Fugu. Fugu is a type of fish that when it's cut properly is considered a delicacy, and if you've ever gotten to try it, it is pretty delicious. But it's also poisonous. You cut that sucker the wrong way and you're dead. Like, that's it. We look at these features as the same way. We need to do them in a way that is absolutely right. So nothing should ever be granted access by default. Instead, they should rely on a permission model that puts the user in control, right, and is easily revocable. It needs to be crystal clear when all of these APIs are being used and how they're being used. We've got a good white paper there that dives deeper into some of our thinking around this, so you can check that out. Now, obviously, there's a ton more I could talk. I could talk this whole session about the security model for all these things, but that's not all that exciting. So every one of these APIs needs to be done in a way that balances the needs of you, the developer, with the needs of keeping our users and their data safe. Users should never be put at risk simply by going to a web page. Now, we've identified about 75 features that we think are critical to closing that app gap, and I'm sure there are probably many more. You can see the stuff that we're currently looking at by going to crbug.com and looking for bugs that are tagged projfugu. Now, guiding a feature from sort of that idea to a standardized shipped API is important to us. And I showed this quote earlier, but I want to show it again because I think it's really important. As we work on these, it's really important that we maintain the core tenets of the web. That means that we're designing these and building them as web standards. Everything I'm going to talk about is a web standard. And I'm really kind of psyched about the fact that other browser vendors are getting involved in this. Microsoft is actively working on several new features, including app icon shortcuts, and Intel implemented the generic sensors API, and they're currently working on WakeLock and web NFC. So how do we do this? Well, the first step is we identify a need, all right? Specific use case. Now, many of these are things that you've told us about, either by talking to us directly or by filing a feature request on crbug.com. As we start to think about the API, we put our initial thinking into an explainer, essentially a design doc that is meant to explain the problem and show some sample code about how we think this problem is going to get solved. Once the explainer has a reasonable level of clarity, it's time to publicize it and start to solicit feedback from you so that we can iterate on the design. We need to hear from you to make sure that the use cases are actually covered properly. Is there anything that we're missing or did we maybe just completely biff up the whole API implementation? And during this time as we get feedback from you, we keep iterating on that explainer to get it better and better. But we're also looking for public support at this point. To get public support from you can be as simple as a tweet from an engineer saying, hey, that's really cool. We'd use that in our app if it were available. Your public support, A, helps us to prioritize the APIs so we know how many people are going to use it and how important it is to you. But it also tells other browser vendors how important it is to you so they can go, ooh, lots of people want that. Maybe we should do that. So once the explainer is in a good state, we transition from the explainer to the spec. And where we continue to work with developers to get your feedback and other browser vendors to keep iterating on it. As the design starts to stabilize, we typically use an origin trial to experiment with the implementation. Origin trials allow us to try new features with real users and get feedback on that implementation. This real world feedback helps us shape and validate the design before it becomes a standard. Finally, once the origin trial is complete, the spec's been finalized, and all the other launch steps are completed, it's time to ship it. So enough background. I've talked too long. Let's look at some real stuff. Tom, you want to do the first one? Sure. All right. Thank you, Pete. So Bluetooth or energy or just BLE is a wireless technology that you can find in many gadgets like hardware sensors, beacons, lights, and also toys. But many times when you want to interact with a BLE device, they make you download yet another app. And actually, you should just be able to use a web app. Web Bluetooth can be used to control all sorts of devices after pairing them, which is painless and quick and works via a picker. For example, the Batmobile from LEGO can be controlled from a web app without the need to download an native app. We actually have the Batmobile, not the Batmobile, the Batmobile over in the web sandbox over there. So please come over and play with it. Before connecting to a web Bluetooth device, like the Batmobile, I first need to know its ID. The Bluetooth service UUID that you can see here is the unique identifier that LEGO gave to the Batmobile. I can optionally also provide a name or a name prefix parameter. You can think of these services filter as a way to not only look for certain devices, but also for certain categories of devices. For example, to just look for hardware sensors or just for devices whose name starts with Hub, which is the name that LEGO gave to the Bluetooth controller inside of the Batmobile. When I then request a device, I will see then the connection picker, where I can then pair it and start using it. Web Bluetooth is a very powerful API, so it requires a secure connection. And the connection picker only shows after user gesture. After the pairing step, I can then connect to the device's server that provides the services of the device. In a concrete case, a Batmobile remote control service. How cool is that? Web Bluetooth is an established standard that has been in Chrome for a while. It's fully shipped, and you can use it today. If you have a great use case, I have built a cool demo like Nils with the Batmobile controller app that we've showed at Sandbox. Do let us know. Awesome. Sharing is a critical feature for many apps, and it means whatever that means. Let me try that again. Sharing is really critical. Whether that means you're sharing out or you want to be able to receive shares, many native apps can plug into the system sharing APIs. And the web should be able to do this too. With the web share API, you can. It allows your site to share links and text via the system share picker. Working with the share picker is pretty straightforward. It's best to use feature detection because it's only available on some devices, right? Like your Windows or Mac OS device doesn't have a share. So you want to fall back to whatever the native sharing mechanism is. But if it is, you just provide the text, title, and the URL that you want to share, and then call navigator.share. And that's it. It's pretty easy. We just also added, I think this one is pretty cool, the ability to share files as well. So you can share not only links, but you can share files. The other side of the equation is the ability to tell the system that you are a share target, that you want to receive shares from other apps. And the web share target API allows an installed web app to register with the OS as a share target so that users can easily share apps to you, or share links and text to you. In fact, Twitter just recently implemented this. If you've installed Twitter, you can try sharing and see, hey, I want to share a link over to Twitter's installed app. To register as a share target, you need to add the share target entry to your web app manifest. And when the user chooses to share to your site, it opens a link to the action page that I've got linked there, and sends over the required information. So for Twitter, what they do is they just compose a new tweet with the information that you provide. Just one pro tip here, if you're going to do this, make sure you pre-cache your share target page so that it loads instantly and works reliably. Otherwise, if you do it and it's not loading, users are going to get a little cranky, and they're not going to use it. So web share and web share target are available today. And they provide a nice way to integrate with the system. But we've also been working on web share target V2, which allows you to receive files. You can now send files, but you can also receive files. And so I suspect that'll probably land a little bit later this year, around Chrome 76 or so. So on your computer, your laptop, or your desktop computer, oftentimes you have media keys that you can use for playing or pausing your media content or for skipping forward or backward. For web apps, these keys, typically, are not accessible. But there's actually no reason why they shouldn't be accessible. When you build a media player app, you could for quite a while now use Android's media controls in the control panel via the media session API. And not only use rich album artwork, but also, obviously, use the media controls to skip forward and backward, pause, play, and so on. What's new is that now you can finally also use media keys on physical hardware keyboards to control playing, pausing, skipping, and so on of your media. And this also works while the media tab is in the background or in full screen mode. To make this happen, I set up Action Handlers in the navigator.media session interface for the previous track action. Yeah, you can applaud for that. It's really cool. It's a great API. So you set up Action Handlers in the media session interface for the previous track action, for the next track action, for play, and for pause. The media session API is fully shipped now, and it works on Android, Mac OS, Windows, and Chrome OS. And support for Linux is under development. Sweet. All right. With gate user media, getting a video stream from the user's web cam is pretty easy. And you can use JavaScript libraries to do face detection, or you can do QR codes, and that kind of thing. In fact, a couple years ago in the Chrome experiment movie Conti Revo, that's how it worked. You navigated by moving your head to move through the experience. But doing shape detection in JavaScript is kind of expensive. And most native platforms have APIs for this built in. So the shape detection API is something that is about to land. And it's got three interfaces, a face detector, a barcode detector, and a text detector. The face detector does exactly that. It can find a face. But it's just finding your face. There's no facial recognition there. And sometimes it can find landmarks, so like it could find your eyes, your nose, and your mouth. So you could use that for maybe taking a picture when everybody is looking at the camera, or imagine trying on a pair of sunglasses or glasses before you even try them on your hand. And you guys all look better when I have my glasses on. So that's a good thing. The barcode detector does exactly that. It can scan all kinds of barcodes. And Tom's going to dive into that a little deeper in a minute. But the text detector, what do you think it does? It finds text. Now, each detector has a detect method, which takes an image or a video or a blob. It'll check that image and asynchronously return whatever it finds. We just finished an origin trial with the shape detection API. And one of the key reasons that we do an origin trial is it allows us to validate the design. In this case, we realized that there were some things that we needed to change and make some updates. So we're updating the spec a little bit and making some of the methods a little bit simpler. We also realized that text detection wasn't at a level that we were happy with. So we probably won't ship that one for a little while until we get the quality level up. So as you can see in Pete's demos, there's some really neat use cases for the shape detection API. But pulling all of that together so that it will work across process is a bit tricky. The shape detection API is coming out of an origin trial, but not all mobile browsers support it yet. But fortunately, this is a really good example of where JavaScript library can help and make lives easier for you. Using platform APIs were available and JavaScript and WebAssembly where they aren't. We're happy today to announce an early version of the Perception Toolkit, a library that allows people to navigate your site with their camera. This idea of navigating your website by pointing the camera at physical things in the world has a lot of power, actually, and it can help users complete tasks faster and easier. For example, if you want to learn more about a product, sometimes instead of typing in a search box, it can be way easier and way faster to just scan its barcode to find the page that you're looking for. Another use case is museum tours where rather than forcing your visitors to download yet another app that they will never use again after the museum visit, you can just scan the actual artworks from the web and just deliver directly a rich and rich online experience. The Perception Toolkit does planar image detection, which means if museums have digitized their artworks already, it is a really small step for them to make these artworks recognized by the toolkit. The Perception Toolkit does three things. First, it manages a camera session for you and can detect barcodes, QR codes, and 2D image targets. Using native app ads when available, and as I said earlier, web assembly when not. Next, it allows you to link real world targets, like barcodes with pages on your website. And this is done using structured data, so it's easy to manage. And finally, it makes the user interface really easy, from onboarding the user to rendering cards. You have full control over the user interface, but it takes care of the heavy lifting. The whole thing is open source, so there's a lot of flexibility to customize how every one of it. But the general idea is that you can get up and running really quickly to add camera-based navigation to your site. If you want to try this out, we have a live demo over in the Web Sandbox. You can do this on your own device, so just go to the URL that you can see on the screens. And yeah, once you are in the Web Sandbox area, try pointing a camera at physical things, signs, logos, and so on, and see what's coming up. Sweet. Installed apps can frequently alert users about new information, either through notifications or through a badge on the icon. Badging makes it really easy to subtly notify the user that something might require their attention, or it can indicate a small amount of information. Badges are less intrusive than notifications, and they don't interrupt the user, which means that they can be updated much more frequently. And because they don't interrupt the user, they don't require any special permissions to use them. There are plenty of great use cases for badges. Chat and the email could indicate progress. Games could let users know it's their turn. And plenty more. Just one small thing about them. They only work when the app is installed, right? Because if it's not installed, there's nothing for us to badge. So setting the badge is pretty easy. Call window.badge.set. It takes an optional number if you provide that number. It uses that number. If you don't provide a number, it just sets it with a dot, and then you can clear it with clear. It's available as an origin trial right now on Windows and Mac, and we're working on Chrome OS support. So you can start using it today. During the origin trial, though, you'll need to call window.experimentalbadge.set. If everything goes smoothly, this should be available and stable around Chrome 78. We're still looking for signs of public support for this, so if this is something you think you might use, please tweet us or let us know, because this is a pretty cool one, and we're getting close and want to see some use on that. Yeah, all right. Woo-hoo. So the last talk had four rounds of applause. I think we're at four, so let's keep going. This is good. Oh, crap. My screen just went off. Oh, luckily it's back. So a lot of devices can keep the screen on, while they're running. For example, a presentation app like PowerPoint can prevent a screen from falling asleep, from turning off. Luckily, we can sort of do the same pretty soon again on the web as well, because presentation apps like Google Slides have very much the same use cases, so why shouldn't they be able to request something that is called a wake log that prevents from happening what just happened? So to avoid draining the battery, most devices can quickly go to sleep and turn a screen off when they're idle. This is fine most of the time, but some applications do need to keep the screen on or prevent the device from sleeping so that they can continue doing their work. I mentioned keeping your screen awake during presentations, but there are plenty other really good use cases. For example, a run tracking app turns the screen off, but it needs to keep tracking where you are to record your route. Or maybe a game where you focus on the screen while thinking about the next turn, and you don't want the screen to turn off when you think about your next turn. The wake log API enables you to support use cases like that and new experiences that until now required a native application. The API also aims to reduce the need for hacky and potentially power hungry workarounds like looping an invisible video that people use nowadays to keep the screen on. The wake log API provides two types of wake log, screen and system. A screen wake log prevents the device from turning the screen off so that the user can see the information that's displayed on the screen. And a system wake log prevents the device's CPU from entering the standby mode so that your app can continue running. And while they are treated independently, one may imply the effects of the other. For example, a screen wake log implies that the app and therefore the system should continue running. In order to use the wake log API, I need to create and initialize a wake log object for the type of wake log I want. Here, I'm requesting a screen wake log. Once created, the promise results with a wake log object. But note, the wake log isn't active yet. Before you can use it, you need to activate it. Before activating the wake log, let me first set up an invent listener so that I can receive notifications when the wake log state is changed. To activate the wake log, I need to create a request. And now finally, the screen won't turn off ever again. Until, of course, I cancel the wake log. You probably have noticed the red ribbon in the upper right corner. This means that the spec is currently being rewritten based on early feedback. And the code that you can see here will change. But what you can see here is also what you can actually play with today. Wake log is moving along really well. The folks at Intel have been busy with rewriting the implementation. And it's available behind the flag. But again, please note that the current implementation will change. I suspect we'll start something in an origin trial, probably around Chrome 76, maybe 77. But it could change depending on how we make progress with the API and so forth. Sweet. All right. So apps that read and write local files are essentially impossible to do on the web today. You need to upload something. If you want to start working on it, you need to download it and put it back in the downloads folder, save it back. It just doesn't work very well. The native file system API aims to change that, making it possible for users to be able to choose files or directories that a web app can interact with on the native file system. Single file editors like Squoosh, which is a PWA our team built earlier this year, are an ideal use case for this. Control-O to open a file, compress it, Control-S to save it, and then just repeat. And of course, there are plenty of other examples. Notepad for editing text, Sketchup for doing drawings, Figma for doing prototypes, all that kind of thing. Beyond single file editors, we also want to enable the ability to work with collections of files. So think like photo galleries or media players. And the one that I'm most excited about personally, I dream of the day when VS Code just works as a progressive web app. And I just use it straight in my browser. That would be pretty amazing. The one key design element of the API is that its primary entry point is a file picker that will require a user gesture to invoke. This ensures that the user is in full control over what the web app has access to. And we expect that some files and folders, web apps just won't have access to at all. As I said earlier, users should never be scared about visiting a website or web app. We're still iterating on the spec for this. So this code might change, but grab the file I want to open, get the file, read it into an array buffer, make any changes I want. Then to save it, get a file writer, and save it to disk. There's an early draft of the spec in the works, but there's no code that you can try right now. So this is still one of these ones that's a little bit future looking. As with our other stuff, if you think this is cool and want to get involved, please go check out some of these links. Thank you, Pete. So many devices such as microcontrollers or 3D printers communicate through serial data. Up until now, these devices can't be integrated with web apps. But what if there were a way to bridge the physical and the web worlds by allowing them to communicate together? Even though modern PCs rarely include the classic nine-pin serial port, a lot of the devices driven by microcontrollers, such as robots and 3D printers, communicate using serial data. But nowadays, commonly over USB or maybe Bluetooth. So let's have a look at how the serial API will work in practice. First, I need to filter for known vendor IDs in order to request a port to the device in question. Here, I'm filtering on the Arduino USB vendor ID. Remember, it's all serial data, so I need to agree on a board rate when opening the port that determines the communication speed that is used when communicating over the channel. Finally, I can then read from the reader until there's no more data, which the reader signals with a Dunflag. The actual code will look similar to what is defined in the specification, but some of the details still need to be worked out. The permission model will work similar to that of web USB. We started the serial API way before the capabilities project, and before its processes came into life, so some of the details still need to be defined. There's an implementation behind a flag, but most of the work has been so far on getting the permission UI right. But I can request permission to access a serial port today. The object that your script gets back isn't actually useful yet, but we're aiming to have proper support by end of the quarter. So there are a whole bunch of peripherals that communicate with your computer using HID. The most common classes of HID devices are keyboards and mice, but there are a whole bunch of devices that are not well supported by the HID driver and are often inaccessible to web pages. Web HID would make these devices accessible and provide better support for devices that have limited functionality. So many game pads, for example, have decent support in the browser, but they frequently miss out on some of the cool, neat features that they have. With web HID, the device-specific logic can be moved into JavaScript, so providing full support. And there are plenty of other use cases. Extra buttons on the telephony headsets could easily control video conferencing gaps. You could control virtual reality experiences with spatial controllers. And just one note, though, web HID isn't intended to get high-level input, where we already may have a high-level API, like key presses and pointer events. Those are going to continue to be used with the high-level APIs. This sample opens up a HID device, grabs that first matching device, opens up a connection to it, sets up a listener to be like, ooh, hey, you moved the joystick. And since HID is bidirectional, it sends out a report to activate some of the lights on the device, letting the user know that the device is ready to go. There is a draft of the spec available now, but there's nothing to play with code-wise. So we'll see that probably around Chrome 78 or so. Thank you. This one's cool. So access to the user's contacts has been a feature of native apps since almost the dawn of time. And it's one of the most common feature requests that I hear from web developers. And this is often the key reason why they still build a native application, getting access to the contacts. There are many great use cases for a contact picker API on the web. But at the core, it is designed to help you reach your friends and family easier. For example, in a web-based email client, the picker can be used to select the recipients of an email or someone to connect with on social networks. But instead of providing a site with complete and continuous access to the user's contacts, the contacts picker API is just that, an on-demand picker. When invoked, it shows a list of a user's contacts and makes it easy for them to pick only the contacts that they want to share. And users can provide access to as few or as many of the contacts as they choose. And because access is on demand, each subsequent call to the API will show the picker again. In designing the API, we want to give developers the features that they want. But we also want to ensure that their contacts and their own information is safe and they clearly understand what they are sharing. That means that the contacts picker will only work when served from a secure host, and it can only be shown after a user gesture. Users can also explicitly choose which contacts they want to share and see exactly what is being shared before they share it. Opening the contacts picker starts with a call to navigator.contacts.select that I need to pass an options object to with information I'd like. So here, I just want one contact. And from this contact, I want the name, the email, and the telephone numbers. The call returns a promise that results with an array of the contacts that were selected by the user. So let's say the user has selected Sundar Pichai. Why not? Each contact will include an array of all the properties that were requested. For example, our address book entry for Sundar has his name, no phone numbers, but it does have an email address for him. So go email Sundar. We're in the early phases of experimentation with the contacts picker API. The explainer has been written, and a very first implementation is available behind the flag. But a lot of the details still need to be discussed. For example, whether the addresses should be accessible via the API. Like the other APIs that we've talked about, we're looking for public support. So if this is something you think that you could use in your website, do let us know. All right, well, that didn't get a round of applause. OK, there we go. Yeah. That one's cool. All right, so I'm going to cover this last one, and then we'll wrap up real quick. But designers frequently have customs fonts installed on their system that they use for creating apps. Company standard fonts, specially license fonts, that kind of thing. Browsers can use them if they know about them, but there's no way to get a list of those fonts. The Font Access API has two components. First, it allows apps to enumerate the list of locally-installed fonts so that it's easy to use them within your app. And second, it allows access to the details of the font, such as the individual glyphs and ligature tables, so that web apps can take fine-grained control of rendering using things like HarfBuzz. Getting access is pretty simple. So we just call navigator.fonts.query and give it a query parameter. In this particular case, I've asked only for local fonts. I can then loop over the list of fonts and add them to a font selector. We're still working on an early draft of the explainer, and we're looking for feedback to make sure that we've got all the use cases covered. So take a look at this and let us know, especially if this is something that you're really interested in. So as Pete said, in the last couple of years, the capabilities of the web have grown tremendously. Today, we've only covered a handful of the APIs that we've shipped. What that we are planning to ship later this year that will all help close the app gap, but there are a few other features that we're working on that I want to at least quickly call out. We're adding support for programmatically copying images to the clipboard, which is the most popular bug in Chrome Spark Tracker, actually more than 1,800 stars. Currently, the asynchronous clipboard API only supports basic reading and writing of text, but we will be rounding out the existing synchronous clipboard API with support for images, again, helping to unlock a couple of really new creative use cases. Then we have the SMS Redeemer API that gives developers the ability to be notified when specially formatted SMS messages are delivered to the user's device so that they can pass out OTP tokens, for example. This can help reduce the friction when verifying phone numbers and can be used as a mechanism to limit abuse in account creation. It can ease account recovery, or it can be used to confirm critical operations. Web developers have the ability to display notifications using the web push and notifications API. Notification trigger is a mechanism for preparing notifications that are to be shown when certain conditions are met. For example, the trigger could be time-based, or it could be location-based, or something else. This makes it possible to display notifications at a particular time without involving a server and improves the reliability and improves the reliability by removing the dependency on a network connection. This is essential for certain types of applications, like, for example, calendars. So while most of the APIs that we described today aren't available yet, some, like the web share and the web share target API, already are available. And there are a few others that are in origin trial that you can all start experimenting with today. I'll give you two, three seconds for taking pictures of that slide. All right, cool. So if you want to participate in an origin trial with your site, the list of the currently active origin trial can be found at developers.chrome.com slash origin trials. And finally, if there's any features that you think we were missing, like we mentioned, like 75, but there might be more. Who knows? We want to hear about them. Please file a feature request on bit.ly slash new dash fugu dash request. So after all this, we've just two links that you need to remember. The first is for the code lab, where you can experiment with all the ready-to-play with APIs. Not everything works on all platforms yet, and sometimes you may have to set a flag to use them. But as the admittedly biased author of the code lab, I do think it's a lot of fun. And the second link is for the Capabilities Project landing page. We will keep it up to date, and it is outlinks to all the capabilities that we work on and that are currently in flight. From there, we also link to the full list of all the upcoming capabilities that we will get to in the future. The web is an amazing place, and we're really working hard to close that app gap. So thank you very much for being part of our journey.