 All righty good afternoon everyone my name is Thorsten and I'm a product manager at Snowflake in my talk today I want to walk you through a few new capabilities in Snowflake and explain a little bit more how they rely on foundation to be Given the limited time I had to be quite selective So I just picked my most favorite new capability in Snowflake and I'll walk you through that a quick architecture slide As a lead-in for those that are not familiar with Snowflake This is from a 30,000 foot perspective The architecture that Snowflake uses you can see the clear separation between storage at the bottom and then compute at the top and Customer query processing data processing all happens in the virtual warehouses that you see in the middle and The storage and the compute they scale completely independent of one another So you can have a very very huge data set in storage You only pay for the storage and then then you can put as much compute on top of it As you need at any given point in time that can be very small for instance during the weekends But it can be massive during a regular work a work week where you're running for instance your data science workloads at the top Of that sits let's say the brain or that that stitches all of this together This is where metadata management transaction management security and query optimization all happen most of these tasks rely heavily on FDB because the complete system state for Snowflake all of our metadata is Is stored in in FDB? so Let's zoom into one of the features and try to understand a little bit more What it means to to add a feature to Snowflake and what's the impact for FDB? The one that I picked is what we launched earlier this year at our summit conference Another umbrella of data pipelines in the scenario here is Particularly with IOT scenarios mobile applications Sensor applications you have data that's constantly born and this can be websites devices Mobile phones the data either comes to a messaging system such as Kafka or it's stored somewhere in in the public cloud And from there what customers usually do is they load that either in in batch or through some of our continuous load functionality into what we call a staging table the staging table is Essentially an append-only table that keeps track of all the changes that are happening at the source systems And then what customers want to do is they want to inspect the net new changes that came into the staging table And then figure out what are the changes that they need to apply to the actual business tables that are being used by analysts or business users And those are these transformations that you see in the middle here Those can be simple merge statements in snowflake sequel But they can also be more complex stored procedures that have really rich business logic in them The challenge here is you want to run these transformations? Automatically ideally every time when new changes are landing in your staging table You want to reliably figure out what those changes are Send them through your business logic through your transformations and then apply the resulting changes to the target tables That you see on the right-hand side in this picture here in order to support this use case We introduced two new concepts to snowflake one is what we call a table change stream it gives you the ability to run a sequel statement against the Change stream to figure out what changes what net new changes have happened to the underlying staging table Since the last time that I ran this this statement This is typically used in what we call a task a task is the ability to schedule a piece of sequel That could be that merge statement or a call to a stored procedure And the sequel code in these tasks usually then refers to a stream to figure out what are the changes? That have happened to the staging table then apply the business logic to it and then persist the results of that Into the target tables now all this scheduling all this state management for these tasks Is happening through fdp so we keep the metadata obviously for the task definitions in fdp But also the queue management for what's the next task that needs to run the scheduling all of this is done Through fdp that you see at the bottom here So I want to walk you through a quick demo of this So here we're connected to a snowflake account Let's start by creating a staging table for a customer scenario Here is my staging table for the customer customer ID name date and a zip code Now let's create the target table here Again called customer ID name date and the zip code Now let's pretend that a change is happening to the staging table and let's insert a new row to the staging table here and Let's see when that finishes Not sure why not There we go All right, so let's take a look at the staging table. Let's move this up a little bit So here we can see this is our staging table. We have a new event that snowflake moved in ad zip code 401 But if we look at the actual customer table, it's still empty So that change record has not been picked up from the staging table yet. It has not been processed yet So here comes a merge statement that implements the That the change processing and it does two things so first it checks whether there is already a customer record in the target table If so, then it will just apply an update with the new values to that table if there is no record for this customer in the target table, then it will run an insertion you can see that here in the When matched clause, we are doing the update and in the not matched clause. We are running the insert so let's run this merge statement over the The the change that we have in the staging table and now you can see there is snowflake ad zip code 401 As it is in our staging table now I ran this merge statement manually just to illustrate the processing that we want to happen now Let's automate this using streams and tasks Let me first clean up my staging table here and Then the first step typically is to create a stream over the staging table That gives me the ability to incrementally look for changes that are happening to our staging table So now let's add a change record into the staging table. Let's say that snowflake moves from zip code 401 to 402 Which we actually did And you can see that change here in the staging table But it's not yet processed for the actual target to table So it's still sitting there and remember we haven't defined any tasks to do that, but let's take a look at the stream So here is the stream object. It tells us there is a record sitting in the staging table ready to be picked up And it's due to an insertion that happened to the staging table So now let's define a task that runs every minute in a given Snowflake warehouse and the body of the task is the merge statement that we looked at earlier So this essentially means this task wakes up once every minute and is looking for Changes to the staging table through the lens of the stream and then running the merge logic on top of it So let's get this task created and Let's kick it off with a task resume statement here is the task and We can look at the task history and the task history is essentially a look into Into fdb metadata Which gives me visibility into the queue which changes or which task executions have been scheduled in the past Where they're successful or not what where the error codes that we have hid along the way And what are the scheduled tasks that are subject to Execution next and you can see here that in about half a minute the next incarnation of that task is supposed to run and until that point in time we should see the old zip code in in our customer table and We should still see the change record in our stream So let's wait for a few more seconds and look at Some of our adoption Data before I jump back So here you can see the number of task executions on a daily basis plotted over a timeline And you can see that we are now well above one and a half million task executions every day Imagine that all of those task executions trigger in the order of ten Maybe a few dozen FDB transactions each of them So there is a substantial additional load to it and you can see that one of our customers went super crazy creating tasks We obviously found that out and we were talking to to them And we found an easier way to implement their particular use case So that's why you're seeing us coming back to That that regular hockey stick trajectory that we are that we're on Just wanted to show you that real quick now Let's jump back into the demo here and let's take a look at our target table And you can see now the zip code has changed to 402. So our task has probably run Let's also take a look at our stream and here you see the stream is empty So we have picked up the change from the stream and followed it forward into the target table And let me also run the history statement here again And if we scroll down you can see our successful execution Down here, and you can see this get the next one is scheduled for About a minute Out from now with that, let me jump to my last slide here and Showing my shameless plug. So As always we are looking for great engineers. We have offices in San Mateo, California in Bellevue, Washington and in Berlin, Germany If any of these items that you see up here sound remotely interesting to you, then we'd love to chat So we're working in all sorts of interesting spaces from corded away housing to data lakes and data pipelines As you just saw and last but not least FTP with that. Thank you very much and have a great day