 Hello, my name is Dave Watkins. I work at Deutsche Bank and today we're going to be talking about Waltz. So over the course of this talk, we'll be going over what Waltz is, a high-level summary. We'll talk a little bit about its history and we'll devote quite a bit of time to going through its feature set and how we use it. Then we'll talk a little bit about the future, how to get started with Waltz in your organization and how to contribute back to the code base. So what exactly is Waltz? Waltz is an enterprise system. We have to think of it as an open approach to enterprise architecture. We're trying to help organizations deal with very large and complex technical estates. So to that end, we're primarily interested in applications as a sort of primary unit. We look at how they fit together in terms of data flows who's working on them and we look at their sort of characteristics in terms of what functions do they offer, what services do they provide, what process they participate in. We look at how the applications are entwined within regulations and compliance issues and ultimately how the organization is changing as applications are commissioned and decommissioned and functions are moved around the the organization. We've got a wide set of users, developers, project managers, analysts. We've got about 15,000 active users in the bank and we're seeing about two and a half thousand user-driven changes per month. That's people coming into Waltz and amending the data in there to correct it or to keep it up to date. Waltz is now around six years old. It started as a purely open-source project upon GitHub, so all of the commits are available going back to project inception and it was started before I joined the bank. However, Data Bank were a very early adopter and have since become by far the largest single contributor to the code base. Over that time, there's been probably around three to four developers at any one time working on Waltz plus an analyst or two which has added up to quite a bit of effort over the years and the project earlier this year transitioned to the FinNOS organizations who went through their project activation criteria, ensuring that we've got clean IP and our code meets their standards. We've been in production for four years at Deutsche Bank and at NatWest Markets have been in for a couple of years. Now I'm going to talk a little bit about the Waltz in detail. So we'll look at its feature set, the core entities and how those entities and features are used together. This will take the form of a demo, which will be covering the following topics. We're going to look at the building blocks, so applications or units, for example. We'll look at how data flows are modeled in Waltz. Some of the aggregation capabilities and some of the reporting that's built in. We'll look at surveys for capturing ad hoc data requests and looking at attestations for ensuring that data is up to date. Applications are one of the fundamental building blocks in Waltz, so let's take a quick look at one. Using my recent list to open up this sample application. So here we can see the basics about the application, its name, where it is in production, what sort of application it is, is it in-house, is it off the shelf, etc. This is who owns it, so this is a link into the organizational hierarchy. We can see, you know, with the application being invested. Aesis, we've found to be useful. Quite often applications are known by multiple names or undergo name changes throughout their life cycle. And so in the Aesis field we can track those. Complexity rating is currently kind of uses things like looking at the number of servers or the number of links an application has to other applications to try and give a ballpark complexity estimate. Alongside a bunch of other bits and pieces such as retirement dates, etc. Going into more detail, we have these sections. So we have one for say bookmarks where we look at key bookmarks and applications. So we might ask for people to fill in things like links to source code, links to documentation, links to issue trackers. We'll ask how we can bring in information about costs. So this is typically imported from other systems to give us a feel for relative sizes of applications. We have some basic stats that we capture around every application. Quite often these are things like compliance measures or stats, say number of database tables, etc. People. This is looking at, we've got usually bringing the entire organizational tree and link people to applications. This allows us then to quickly show all of the apps belong to people and pivot the data that way, which we'll see shortly. The other one I want to focus on right now is on technology. So here we consult, see, okay, these are brought in. It's a list of servers associated with the applications or databases or software. For instance, we've linked our instance of Waltz to Artifactory to bring in a detailed software breakdown for each application. And from there we can work out licenses and things like vulnerabilities in terms of risk for license risk, looking at incompatible licenses, for example. Two of the mechanisms that Waltz offers for capturing custom data around applications are assessments and ratings. Assessments are kind of like a single data point about an application. So we have assessment definitions for things like records management, legal holds, compliance with various policies, etc. They're stored here and users can either input them directly into the application or they can be fired programmatically and marked as read only. In addition to assessments, Waltz supports aligning applications to taxonomies via the ratings mechanism. So in here we've got ratings for an application. In this case it's just aligned against one of the taxonomies in Waltz, the functional taxonomy. And we can see here that three items have been chosen. So positions, risk and reporting. The icons next to them indicate things like this function is being retired from this application at a certain date. The handover icon implies that this application is taking on board this function from another application and the little pointer says that this application is handing off this function somewhere else. So I can see from here just as a glance that this application is really consolidating around the risk function. The little pie charts indicate that there are allocations associated with this application. So here we can see okay, there's a cost allocation scheme and we can see what the breakdown is for this application. These are of course either usable or can be provided via external systems and marked as read only. Another key aspect of Waltz is looking at data and data flows within an organization. To that end we have a data taxonomy which we may declare which is register all the data types. Here's a sample one that we've generated for the test data. And we can also declare which systems are authoritative for those data types. So this table here shows okay for this data type, for this part of the organization, and this is either an authoritative or a secondary source of data for that data type. We look at how that looks in practice on an application level. Let's go and take another look at the application we're looking at earlier and look at the data flow section. So here we can see a breakdown of various aspects of the data flows that we have for this application. But the main bit concerned with is this diagram at the bottom. This shows fairly clearly the upstream systems, the downstream systems, and our system in the middle, Fox Terrier. It's broken down by the data types that it consumes or produces. And we can see okay these systems are feeding it. These it's sending out to the system. These little annotations here that is showing you whether or not we have examples of the types of data that's being fed. So whether people have documented the actual flow specifications. If we take a closer look at these. So we can see okay here's the file. And in this one there should be more than one file because the icon's slightly different. Just indicating that there are multiple files there. The colors on these lines go back to those authoritative sources we talked about a few minutes ago. So American Foxhound for example has been identified as an authoritative source for transaction data. Hence this arc is in green. Whereas these applications, so for transaction data, there may be there is an authoritative source but it isn't Tiffany which is why it be read. Where the line is black that's just indicating that no authoritative statement covers this type of data for this part of the bank or part of the organization. We can focus in on these especially useful when the diagrams get more complex. And we can add new ones fairly simply. Although there's a mixture at the moment of these things being registered by hand and registered programmatically from other sources. We also have this other annotation here which shows a pending change. So this is something that's fairly new. If we look at the change sets coming into this application we can see that there's one change set associated with it and it's got in particular a bunch of flow changes which are potentially affecting this application. So here we can see one that there is a retirement of this report pending as part of this change which can optionally be linked to a change initiative declaration. If we go back to our application we can start looking at the flows in more detail taking more of a lineage view. So this is giving us very much a sort of point point of neighbors kind of view upstream and downstream neighbors. But we may want to show more of a lineage type view. To that end we have the diagram section and you can create diagrams in Waltz quite quickly using a kind of constrained editor. So this is one I just drew earlier starting at Foxteria and work backwards through the systems showing which data is being fed by which systems. So this is just showing you the upstream systems and these little things in the circle just where we have documentation for the actual flows that have been passed. The question marks when we don't and you can add basically annotations onto the diagram. Changing the diagram is quite simple. If we wanted to add a new system it then goes and queries the flow information that we have in Waltz and just shows you only the possible upstream systems. So we can say okay I want to show these three systems. And go through and again using the constraints we can say okay I want to show just those two flows. Once this is saved this diagram will be cross referenced against every system that is mentioned within it. So I think we just included one bat. So if we go and have a look at one bat we can see it is now being cross referenced on that diagram. Effectively it was starting to kind of build materials behind each diagram so that we can do that cross reference quickly and that goes down to every part of the diagram. So every flow that is mentioned on every file will be indexed and cross referenced to this diagram. So far we have been primarily looking at individual applications in Waltz and describing the characteristics of those. Waltz can also be used to aggregate applications either by sort of intrinsic aggregation groups such as org units people or via sort of user defined mechanisms such as ad hoc application groups. So we take a look at say org unit as a grouping mechanism. So let me look at one of the org units since the AI office. You can see it's got 24 applications in this group and sample data. We can see you know the applications that make this up the costs complexity etc and the indicators that are there which are the we use extensively in the bank for sort of KPIs, KRIs. This lets us sort of quickly view how well we're doing I guess across these measures but also jump off to get more detail about we know what actually makes up those numbers and also to do quick comparisons so we can compare across parts of the bank quite quickly by using this little mini navigation selector here. Going back to this list here we can see we've also got things such as attestations so in this case we've got now attestations but we can schedule attestation runs for different types of data. So far in Waltz we've got attestation support for flows, physical flows and for the taxonomy ratings. We can have bookmarks, link to change initiatives, data flows is where we start seeing the data in aggregate here so we've got a force directed graph obviously just the sample data nature it looks a bit of a mess but we can do things like filter it out according to data type or show only inbound flows or outbound flows. Obviously the full data is available as well should you wish to export it. The ratings section is quite useful the ratings that we were looking at earlier so we've aligned I think a box terrier to risk we can see now for this group of applications within the CIO office or unit this is their sort of functional makeup and we can also see here some of the other taxonomies which are in this sample data set so we've got an example process taxonomy a product taxonomy a regulatory taxonomy. Drilling in each one of these shows you the actually the data underneath so we can drill in and say okay these are the applications that make up these little bar charts which are going to aggregate it up and color coded according to the the ratings used within that taxonomy. So in this example we've just got a very simple sort of rgb sorry not rgb rag status for the ratings on those applications to the function reports was kind of new so we've added this thing called the data grid it's not an interesting one let me switch to this one this is to support some of the cloud readiness thing that we're doing in the bank at the moment so this allows us to cherry pick a set of attributes so either assessments or the measureables sorry the taxonomy ratings and define them in a grid and then the applications in this group or any group will be the rows and we can quickly see if we've got this nice data grid and we can add these to the summarizing columns so we added these to summaries we can quickly use these as sort of filter mechanisms to say show me you know all the applications which are good or adequate for bcbs and adequate for governance obviously these ratings would make more sense with realistic data but we're finding that quite useful in particular we're looking at it for cloud readiness so we have assessments based upon how how well suited we think applications are to migrate to cloud infrastructure other types of aggregation are things like people you see here we've got a person we can see that they've got a very similar set of views they've got costs and things we've got the same ratings type of view we've got the same reports which are dynamically generated just for that person and the applications that they are involved with we can do the same for the functions or any of the taxonomy items so looking at risk for the function we can see you know the same sort of thing we've got the applications the data flows we've got the the ratings and reports etc we can also define custom app groups so when there isn't a sort of intrinsic organizing mechanism such as people, audience taxonomy items we can define ad hoc groups or we can just create a new group and add applications to it there's also a app group for every person called your favorite group where if you add applications to that group either by editing the group or as you're navigating through waltz you can add them by the little bookmark icon here then it gets added to your favorite group and then any time application or any of the apps in this group change in terms of you know their records in waltz get updated so new flows new functions new bookmarks then this calendar chart will be filled in letting you know that there's been some changes to apps that you're possibly interested in and you can drill in and say you know what's actually happened so here you can drill in and see okay for this application on this day you know there's a flow added remove data from application modify the application another key usage of waltz within organizations is around its ability to issue surveys to collect data on an ad hoc basis surveys can be defined by administrators for example here's the one we created earlier where's the templates we'll get a demo survey so we've got a very simple survey which has got some questions so it's a fictitious cloud migration but we can also have things like conditional questions and conditional questions can take two forms they can either be predicated questions can be predicated on previous answers within the same survey so this is a simple this question appears if this value is checked and this one appears if this value is greater than six but there can also be ones that are more environmental predicates so this is looking at a attribute of the application being surveyed and saying does it sensitive data assessment rating equal s which is the code for sensitive in our data model that we've defined we've got to have a look at how that works in practice we'll have a look at foxtoria see i've currently set its sensitive data flag to just internal or we can set it to lower i'm gonna say it's public so now when we look at this assessment also when this look at this survey we should see okay we've got even scope yes now with the first predicate the other question was you know it was greater than six than a high estimate but we're not seeing that question about sensitive data if we go back and we change the sensitive data assessment for this application and say that it is strictly confidential save that and we go back into the survey you can see we've now got this question appearing for the sensitive data question once a once a survey is submitted it can be sent back to its owners people who issued it and they can either approve or reject the survey if it needs more work doing to it also when surveys are created there's two real ways of initiating a survey one is kind of ad hoc applications can sort of self sorry applications can sort of raise their own issues or people can raise them on a case by case basis the other way of raising surveys is you can create a survey run um where we're going to the template we can say okay I want to create a run and here we say what we're actually doing is we want to issue this survey to a set of applications or change initiatives which meet certain criteria so give it a run name some contact details but here's the key bit we can say okay send it to everyone in this app group send it to everyone in this org unit send it to applications that do this taxonomy item the exact is whether or not you want to send it to children of this org unit as well so we just sent it to the risk department who might want all of the the sub departments to also fill this in a risk IT that would include all of the other item the risk departments uh them is a question of who is going to be filling this in this is again where we start reaching into the world's data model and say okay we now know from this selector uh which applications are involved so everything to do with risk everything associated with risk IT the org unit and now what we want to do is we want it to be the data owner the architect the IT architect and the IT lead and we can then say okay who how do we want this to be filled in is it um for a group as in anyone can fill it in or is this an individual everyone has to fill it in um once we've filled in the rest of the data and we should be able to issue that survey nope and we haven't managed to catch in my sample data anyone who uh is oops is there we issue the survey and now that survey run has been generated click create an email if we wish to uh but we can also obviously we've got a new run uh where we're going to see that there are seven surveys outstanding now you may have noticed that there was a lot more people listed there is because they ended up selecting everyone and the group nature of the survey means it was only actually seven surveys issued but more people have been invited to participate um yes surveys have proven to be quite popular uh we use them a lot around sort of some of the governance metrics also the governance processes uh so funding processes uh and also some of the records management type of uh processes where we are doing uh in-depth questionnaires for applications which meet certain criteria okay that concludes the sort of demo part uh now let's talk a little bit about you know um futures you know where waltz is heading um how to get started with waltz and how you can start making contributions should you wish to so uh one of the key challenges that we're facing uh in the coming sort of months years is uh the gradual shift of the unit of sort of deployment moving from sort of applications sort of monolithic type applications towards much smaller components microservices landers etc uh trying to figure out you know how do we document those what is the correct unit of sort of measurement what policies and procedures need to be in place how can we aggregate them uh so as not to uh basically bombard the owners with lots and lots of documentation requests um another key part is opening up waltz the data set of our apis so that we've been asked to do quite a lot we've done some of that internally within the bank so be looking to how we can industrialize them with that and contribute that to the open source project um automated data capture so quite often some of the data sets are manual and we've written various feeds into other systems to bring data in it would be nice to start exploring uh some more sort of official uh integrations with other products to bring data sets in workflow support we've seen some basic workflow around things like the surveys how we're being asked increasingly to um put workflow around various change sets within waltz so that data can be tracked and assigned to reviewers etc um finally the cloud migration kind of dovetails into the first point is that we're really looking at sort of visualizations to help support the cloud migration so a lot of the recent work we've been doing has been sort of rooted in some of those efforts as the dashboarding thing you saw the grid report certainly a lot of the assessments and the ratings are sort of geared now towards looking at how we can document applications as they move from on-prem to sort of cloud systems and all of the changes that will be brought on them as part of that move i'm getting started okay there's a couple of ways to get started with waltz one the easiest is if you just want to try out waltz on on your own machines you can download a simple docker image it's a little bit out of date a moment but i'll get that uploaded in the next few days which has a waltz instance running with some sample data if you'd rather do a more of an installation process it's quite simple to set up on a tomcat container or you can run it as a simple uber jar database creation is pretty straightforward we just use liquid base and if you want sample data there are some sample data generators available in the source code it's all documented upon github look inside the docs folder reach out if you need some help we support sequel server postgres and maria db those are the main three that we support let us know if there's anything you would like us to support we can certainly investigate to see how hard it would be to add support for other databases relatively database agnostic um finally on getting started it's worth pointing out we have a blog the blog has a couple of articles on how to get started and quite a few screencasts basically aimed at both users of waltz administrators and developers um which brings us to contributing so should you wish to make a contribution then you would be delighted um reach out to us via the github issues probably best or contact us directly there's some good first issues that we've tagged in the github issues repository and again we'll be more than happy to help out there's some getting started tutorials on our github blog sorry on our finance blog and some screencasts to help you take your first steps in contributing to waltz okay that wraps everything up thanks very much for your time and we hope you enjoyed the presentation look forward to hearing from you thank you