 Hey, everyone. Welcome back to the State of the Web. My guest is Minhas Kazi. He's a developer advocate for Google Data Studio. And today we're talking about visualizing data at scale and specifically how we use Data Studio to visualize the Chrome UX report. Let's get started. [?]. Minhas, thanks for being here. It's a pleasure to be here. So what are the options for developers who want to make sense of and visualize a data set? So when developers are visualizing data sets, I feel like there are three main problems to tackle. The first part is they need to connect to their data. The data might be in different places, on different databases, on spreadsheets. So that data needs to be brought out to be visualized. So that brings us to the second part where developers need to visually analyze the data and to create the visualizations. And the last part is once they have visualized the data and got some insights from it, they will need to take that and somehow share that information with other folks. They might share it with their team members, with their clients, with the general public. So these three problems, the connecting, visualizing, and sharing, I feel like there are different tools that solve these problems separately. For example, you might have a spreadsheet that lets you visualize the data. Or you might have a database tool, a querying tool that lets you query the data. There are some tools that will let you do all of those three things, visualization tools. And for those, I feel like there are three main counts for those. On the one end, you have spreadsheets, like applications which are not that much powerful, but are very approachable and easy to use. And then you have very customized solutions that have a high barrier to entry. So if you are a developer, you need to have expertise to get into those tools. And lastly, you have very customized solutions where you have to code it from ground up. So probably, if you're a developer and have a data set, these are the ways you can approach the problem. So what are some considerations for developers who have extra large data sets, like HTTP Archive and the Chrome UX Report, which are millions of records and they're updated on a regular basis? The issue that you face as a developer when you're working with big data sets is that they're huge. You have to have an infrastructure that supports that data, that will to query that data. So if you're trying to generate insights from that, you need to make sure that you're pulling in the right data that you want, your queries work in the way that you want them to. And once you build a solution or build some kind of dashboard around that, you probably would want to share that with other people. Then you need to make sure that it's scalable, that thousands of people will be able to view that dashboard. It should also be cost effective. If you're pulling in gigabytes or terabytes of data, you don't want to break your bank doing that. So what ends up happening is that most of the solutions that we have right now out of the box solutions won't let you do that, all those connecting, visualizing and sharing in a seamless way within one application. So developers end up creating their own solutions. You basically create your own database as a buffer layer and then you create your own web server, you write some JavaScript visualization and then you basically publish the whole thing. That takes a lot of development resources and time. And also if the same developer wants to recreate the same thing with a different data set, most of the time this pipeline doesn't work. They have to recreate everything from scratch again. So this is not really replicable. So those are some of the challenges developer face when they want to work with big data sets and visualizing them. Right, and with the Chromium X report, one of the challenges we faced was the exploration and sharing part where the data set was available on BigQuery with the raw data, but getting people to learn SQL and learn the schema and explore the data set was proving to be a challenge. So can you describe how Data Studio address this problem? Yes, so Data Studio works very well with BigQuery. An example would be a dashboard that Philippe, one of our developer advocates created based on the FIFA World Cup data set. He took the data from BigQuery connected with a Data Studio dashboard and then he took the dashboard and published it in a medium blog post. So you can go into his medium blog post see the dashboard, interact with it and see what countries scored which goals were and it works seamlessly and it's pulling in data through Data Studio from BigQuery. And if you're working with it, you wouldn't know that this has tons and tons of data behind it. So that's how Data Studio makes it easier. The way it works is that Data Studio natively can talk to BigQuery. It has a native connector. So if you have a data set, you can create a data source and pull in data. That works very well at personal level, but if you want to scale that up, if you want to create solutions around it, you can create community connectors. Community connectors are basically connectors created with Apps Script. Apps Script is Google's scripting platform that's a subset of JavaScript that lets you build solutions around G Suite applications and GCP APIs. So you can use Apps Script to create a connector that pulls in data from BigQuery and you can also attach template to this connector so your users can go in, put in some configurations and based on that configuration, they will get the data that exactly they want, put into a template that's ready to be edited and used. So all the user does is goes in, puts in some configuration and they get a completed dashboard in seconds. So they can either go in and change the dashboard, tweak it, or they can just take that dashboard, publish it, share it with other people and it becomes a seamless experience for the users. Right, so it's like, all that plumbing is built in for the developer who wants to just plug and play with the data set. Yes, so it is lowering the expertise barrier for the users. They don't have to know SQL, they don't have to know how to design a dashboard, they're just putting in some configuration and getting a already populated canvas that can be edited if they want. And can you elaborate on how we use Data Studio to implement the Chrome UX report dashboard? Sure, this was a very fun project because we had this amazing data set that gives a lot of insights about how websites are performing. You could obviously write a SQL query in BigQuery and get the results for a specific website. But if you wanted to give that power to end users, then well, users could use BigQuery with Data Studio but they would have to write their own query or they would have to have a billing account that would incur costs and that gets into a lot of friction. So what we did was we created a community connector that uses a service account to access the data on BigQuery so you don't have to go in and put in your billing information. That's the first part. Second part was that we wanted to make it very efficient. We didn't want it to spend a lot of time doing the query work and we wanted to take advantage of different levels of caching. So we introduced caching at the BigQuery end, then we used a Firebase real-time database to add a middle cache layer and we also included the Apps Scripts cache. So if you were using only the BigQuery connector, what would happen is that let's say you make a query today and seven days from now, you do the same query. Both of the queries would hit that gigabyte database and pull in the data. But the Chromium dataset is updated every month so within those seven days, the data would probably not get changed. So we're caching those results in Firebase and serving from there so it's both incurring less cost and it's very fast. So when you add a new URL, you instantly get the results. And we also attached a template with our connector. So what the users do is they go into the connector URL, they see a interface where they have to put in the URL for their website, they put that in, click Connect and immediately they get a completed dashboard that's created specifically for them. It's not a dashboard that's replicated and it's shared by multiple people. It's a separate copy of dashboard created for each user where they can edit the dashboard, change it and then share it again. So users can let's say take a dashboard that they created for a specific site, share it and then put that dashboard in a medium post or on Reddit or they could even embed it in a blog that they have. So it completes a whole pipeline from connecting, visualizing and sharing and gives users the power to create from there. So what options do developers have to integrate Data Studio with their own existing datasets? So if you're a developer and you have data set if you have your data on any of the native platforms you can definitely build data sources directly from there. So maybe have your data set on BigQuery or on Spanner or on Sheets, you can create a data source from there and share that data source. If you want to go into more at-bone solutions you can definitely build a connector and what a community connector would let you do is that it will let you access datasets that are even outside of that ecosystem. It can be behind a web API. It can be on a Postgrease or MySQL database. You can create a connector that fetches that data gives user the ability to add the configurations and you can also attach different templates to it. For example, we worked with partners like Kaggle, Data World and Socrata who built their own connectors. So if you are a user of these services what you can do is go into Data Studio and then just click on the connectors put in the configuration maybe the ID of the dataset or the URL, click Next and immediately you get a blank canvas where you can put in your visualizations based on that dataset. If you didn't have this option what you'd have to do is probably download the data upload it somewhere else and then create connections from there and that creates a lot of friction. These connectors as a developer would let you take that friction out of the workflow for your users. And as a developer advocate you've probably seen a lot of the work that developers have been doing and how the landscape has been changing. What excites you about the direction that we're going? The things that excite me most are that a lot of new user-generated content are coming into the scenario. People are building more visualizations. For example, in Data Studio thousands of people are building connectors. We have a gallery of connectors that have more than 100 connectors that connect to more than 500 data sources. We have a gallery where people are creating dashboards and submitting those as templates. So more and more people are getting exposed to this and creating content. This in turn is helping to reduce the barrier to entry. People don't need a high level of expertise to create this content. They don't need high level of expertise to create connectors or dashboards. And they can also piggyback on the work other people have already done. For example, let's say I don't have the expertise to create a connector. I could go in and fetch a connector and fetch data through that. I don't have good design skills. I can take a template, attach it with the data source that I have and create something new that didn't exist before. Without these user-generated content, I wouldn't be able to create this new thing. So this lower barrier to entry and the community content are sort of helping each other to grow. So I'm really looking forward to more user-generated content in the ecosystem in the future. Are there any resources that you can recommend for developers who want to learn more about Data Studio? Definitely. So if you haven't tried out Data Studio yet, developers should definitely go to datastudio.google.com and try out the tool. And if you want resources about doing development around Data Studio, you can go to developers.google.com slash Data Studio. That site has reference documentation, guides, code labs, code samples, anything you need to get started with your development work. Manas, thank you again for being here. You can check out all the links in the description below. Also, the Chrome UX dashboard is at g.co slash Chrome UX Dash. Thanks for watching. We'll see you next time.