 Good afternoon and welcome this session is entitled cloud foundry the gaps and My name is John Wetherill. I'm an architect a cloud architect at BNY Bank of New York Mellon and Little bit of background kind of what brought me to this basically what brought me to this talk so I've had a year a career in develop software development and delivery The last four years I spent it at active state building our Cloud Foundry based pads helping build it and and basically deliver it the paths called staccato and It was a really interesting role and I from the viewpoint of a cloud foundry vendor visiting a lot of large enterprises around the around the world around the country anyway and Seeing what their problems that they were facing were you know seeing the problems they were encountering trying to successfully and rapidly deliver software and And and and the barriers that they were seeing and and not just in general software delivery And then with plat with the platforms they're looking at including cloud foundry and including staccato so I had this viewpoint of seeing all these companies from the viewpoint of an external vendor and getting a lot of you know very informative descriptions of the problems that they're facing and I've since moved on from the staccato project and I joined BNY Mellon six months ago working on the Paz team and It's now I get to see the world from like the other end of the telescope So instead of from the external you know a long-distance view of a vendor I'm now seeing it from the inside and First of all, it's it's affirming a lot of what I discovered over the last four years talking to these big enterprises Yeah, they get these same exact problems And but what it also is doing is really from the external I didn't really see the depth in the How difficult some of these problems were to solve and now I'm I'm seeing it firsthand from the inside and so What this talk is about or what we're gonna focus on is our work with cloud foundry at BNY Mellon We're a gold sponsor of cloud foundry the cloud foundry foundation. We have brought it into our our shop and So the work with that and It's basically framing this with we have been building a Paz for several years for almost 15 years and So how that relates to the cloud foundry and everything that we're doing so that's kind of a background As far as BNY Mellon goes where we've been around for a long long time one of the oldest established companies in the banks in the country and We handle massive massive amounts of of you know funds and finances and currency, etc I'm slightly embarrassed to say that before I started interviewing with this company. I hadn't really heard of them and yet it's actually it's very Important financial institution institution in the world as you can see with these numbers 29.1 trillion in assets Under custody one point, you know lots of money lots of assets going around. We have 13,000 people in IT And as I mentioned earlier, we've been working hard on cloud software We we've been building a Paz for 15 years We've been building micro services long before micro services were kind of the latest trend We're a Java shop mostly for the most part so we've been working hard on all this stuff and We have built our own Paz just working. Oh, I see. It's a build. I didn't realize that. Pardon me I'll let this cycle through because I don't need it to do the building. So we have this platform called the Nexon platform which is our Platform for soft cloud software delivery including the Paz itself including analytics and Digital information. I think I just blew past this thing My apologies. Okay, let's try this I've over clicked it and this platform itself. We've been working on it for several years and We're on the third generation of it now. It has had profound impact on our ability to deliver software and I'm just very carefully stepping through the slide so I don't go too far and It's it's the entire platform for our whole software delivery system and we have incorporated Cloud Foundry as basically as a POC and an evaluation into this Paz as we'll see shortly. So I Didn't realize that they had provided me as these bills. It's okay But the point of this slide here and we'll get into the meat of this thing shortly The point of this slide here is to show how we have advanced over the years from our ability to provision and deliver software taking a long period of time to much shorter period of time so back in the old days it would take two to four weeks for set up and for same amount of four to 12 weeks for testing in QA 12 to 16 four to 16 weeks to get things out into production And that's part of the dev cycle and we've got most of that down to a single day So and that's the the speed up that you can expect when we're moving into the modern era But there's some incredible challenges We have as a bank to try to get there and that's some part of what I'd like to cover today So yeah, let's move right into that So this is this is huge and by the way when I was visiting all these companies all over You know a lot of them were banks and financial institutions, and I I swore to myself I'd never worked there because of all these restrictions that they have imposed on them and I found it working looking at BNY Mellon there they're Aware of a lot of these constraints and these these challenges, but they're actively Getting passed them and that's what we'll look at here So it's a find a huge financial industry Company in in a very highly regulated industry We have massive amounts of transactions going through our system very high volume very high amount high number of transactions and to the point where if something fails it could Potentially affect the economy of the country, right? It's that much so obviously high availability is absolutely essential for our operations We have multiple data centers across the planet. They need to be they need to stay up We need to understand disaster recovery obviously security is very important and We've been around for a long time We have a lot of old cold old codeword applications legacy apps You know vintage apps as they said this morning and more modern stuff that we have to deal with have to work with Different hardware architectures software architectures all the things you would expect Integration with vendors and partners all these things are huge challenges and just a little bit about the regulations Audits are huge everything that we do has to be audited I'm trying to just paint a contrast here between what it's like at this financial institution Versus at a you know a startup that's working with with logs or data or something like that These can these constraints have significant effect So having everything has to be auditable and trackable and Secret delivery and secret management has to be very carefully controlled and never getting in the wrong hands Every step of every process has to be reproducible Precisely as it was pre reproduced so every image that's built every artifact that's built has to be tracked and reproduced obviously fraud detection Many of these things are they have huge impact on How we are able to deliver software here's one example is separation of duties by regulation The coders must not deploy to production So if you're a coder you can't deploy if you're a deployer you can't code. There's that strict separation which you know Kind of gets in a way a little ways of the DevOps concept, right? But but but this is by regulation and that fact that one little fact has Significant impact on our whole development lifecycle including on our paths that we've built So I'm gonna skip through some of these because I don't have a lot of time I want to make sure I get into the the cloud foundry part But some of the additional challenge is just the massive amounts of data. We have running through we are global We have to be up 24 by 7 we have SLAs to deal with all over the place We just can't be down and I do want to say a little bit about DR So without a doubt Bank of New York Mellon understands disaster recovery We've been through some basically the worst possible thing you can happen have happened to a data center We have had happen to a data center about 15 years ago, and we had to deal with that So there's many people that I work with that are Experts at what DR really means and how we can recover from it So there's a lot of things that that has to be incorporated into our entire platform at all times You have to keep keep on top of that. We also have to validate this so we have to prove to the regulatory Regulatory bodies that we can recover from a disaster, and it's kind of interesting It turns out it's easier to go through a real DR event than it is to test for a DR event Because if you have to test for a DR event you actually have to keep all your existing stuff going and then gradually move it over to and you know somehow move it over to another data center whereas With a real DR event you don't have to worry about you know gradually bleeding off processes or things like that You just it's down. It's gone, right? So we have to prove that periodically and Make sure we can simulate failure failover accurately There's a lot of stuff regarding security, but again, I'm sensitive of time. So let's let's talk about some of the challenges The and as a company we're doing all the things that you hear about at a conference like this microservices And we are we are doing DevOps to the ability that we can based on the the regulations Containerization Docker, you know that kind of stuff All the things we we are heavily into open source Open source makes a lot of sense for a number of reasons and we're very sensitive about getting locked into a specific solution or a specific vendor But yeah many advantages of open open source and we're we're actively taking advantage of all the open source Or many of the open source projects that you've heard of including cloud foundry So some of the strategies we follow to try to get here Paz itself as I say we've been building a pass for all these years and I'll talk a little bit about that and how it relates to cloud foundry So cloud foundry we as I say we're as I said, we're a gold member of the cloud foundry foundation and we have Incorporated cloud foundry into our paths again We're using the open-source version We're taking advantage of a lot of its features for monitoring and log aggregation and the self-service on-demand cloud stuff You would expect to give to allow our developers to rapidly provision an application Auto-scaling all the stuff you come to expect with cloud foundry Another valuable point is it's a hybrid cloud ability to support hybrid clouds quite easily. Okay, so let's get into the meat of this That was kind of the background. I think I've got 20 minutes left so There's a lot of Gaps that we found there's several gaps We have found with cloud foundry that we've had to work around or had to meet and I'm going to cover these relatively briefly I want to set the stage though that this this is not a Defects of cloud foundry talked by any stretch there's no way that any product or application could meet any everybody's needs at all times and As I say, we've been building paths for 15 years and we have all our specific requirements And we do things the way we do them So we can't just drop in and replace all our stuff with a brand new technology and have it work And so what we've done is we spent a lot of time Analyzing and evaluating what cloud foundry does and how it fits into our existing platform And what I'm going to cover in this talk and the remainder of this talk is some of the gaps between cloud foundry and Our paths and how we've filled them up. So and there's I think there's three slides here with some of the high-level gaps and I'll Blast through those while I dive into each one and just a little bit of detail. First of all is OS certification so a lot of our products must be by by regulations they must run on certain specific operating systems and So we found bringing cloud foundry that was a little bit of a challenge because because it wasn't certified to run on some of these OSes so we don't have a choice about this We have specific apps that must run on specific OSes and then as an organization we certify a Set a fairly small set of OS as we can use internally So that was a definitely a constraint. We have to work around and I'll get around shortly how that how that happens I mentioned disaster recovery and I Realized cloud foundry is making great strides in all of these areas by the way This is our kind of how we've been in the last few months looking at things as they were But having a use case for cloud foundry Sorry a use case for disaster recovery that can handle an instant loss of a data center and Failing over to a secondary data center. We have to have a very precise way to describe that and make it happen And obviously the the whole Multi-data center operation has to be very seamless. So it's like if you've maybe heard of uber netties It's like the uber kubernetes thing You kind of need something like that with our paths and we have built that with our paths And that's what cloud foundry is sliding sliding into that So we need a single control plane so we can manage all the pieces of this whether it's the applications of the services Cross multiple availability zones across multiple data centers across multiple clouds for that matter even private versus public cloud and Another challenge of that is there's all this vendor integration So we use vendor products for monitoring and for logging and for load balancing and all these other other Facets of the paths and we have to integrate with all of this and it's no mean feat to take care of all that network and application issues, I guess So that basically there has to be ways to describe how Different app instances and services and containers if you will can communicate to each other or Cannot communicate to others. So I want to make sure that this web tier Ruby on rails thing of a jig can talk to this Java tier, but can't talk to this database tier That's just a very trivial example, but it gets much more interesting when you have you know fleets of micro services operating throughout your network How to control the access between all these different components or for that matter how to enable it? How can I get from here to here? So there's a lot of complexity involved in isolating isolating networks and then we have this whole ecosystem where I For us right containers the whole container concept. So Docker is you know, obviously I'm cloud foundry is actively working with supporting Docker Which is really cool. We were able to use that to great use a few months ago But there's some limitations that we found that we had to overcome for example I don't know if it's changed yet But when we were on it a couple of months ago is very difficult to have cloud foundry pull a Docker image You this is with Diego obviously to pull a Docker image from some random like let's say internal secure registry as opposed to having to go out to the Docker to the hub so That was a challenge that we we had trouble getting past or taking an existing container that has requirements for volumes and figure out how to get that to work inside of cloud foundry as a Docker container inside of Diego a Lot of challenges we we had trying to get get past that Check, okay And Docker has some nice features like Docker compose the ability to describe multiple Components of an application in a single descriptor and deploy the whole thing is one unit, which is which was pretty nice So we have our own version of that. We've had it for years. We have our own descriptors that describe apps I'll get into that a little bit more detail So these are some of the things that the Docker and the containerization support that we had to work around or had to deal with A little bit about build packs so very powerful concept You can take any arbitrary app and cloud foundry will understand it and provision a container for you based on on what your application is built from but We found that so very powerful But then if we needed to customize it at all we had to you know download this build pack code Which is several thousand lines of Ruby code and we're we're a Java shop So suddenly we have to have a Ruby expertise on board to understand what is inside this build pack and how to maintain it and then Issues like can we contribute that upstream our changes? Is it is it relevant things like that? So it the build pack issue is just it was I'm still struggling I'd love to talk to people about how how you get around that issue of build packs complexity and the The massive code base to deal with it what what I've done in the past is just taken a You know if I need to do something really simple like Java and Tomcat and some specific customization Like injecting some security thing I mean almost can just bypass the whole bill pack or the Java bill pack and use a bash script like 20 lines of bash to do The same thing, but I don't know if that's the recommended way Anyway, another thing so a lot of work is being done today for the you know the TCP router etc allowing non-http and on HTTPS traffic to be routed through Cloud Foundry And you know back in the staccato days when I was a staccato We've been working on that for a few years now. It's been working quite well But with with this latest venture that was a limitation where we want to make sure we can deploy any workload whether it's HTTP or not and have it correctly routed to you by the paths and you know ideally on our paths should support UDP as well and I'm not sure what the status. I haven't checked recently of the Whether the TCP router is if there's also a UDP router underway, but our paths is having to deal with that outside of Cloud Foundry Okay, I'm assemblies is pretty interesting. So I need to be able to take To describe a complex application topology with a single descriptor And again if you've used Docker so Docker composes a really good example where you can have a multi-tier app or a microservices based app with multiple components multiple services that it makes up and Describe them in a descriptor and then use Docker compose to the provision the whole thing is a single entity and These partitions for us they can include all sorts of things whether it's vendor software or Standard 12-factor applications and services or a lot of our legacy apps cron jobs services that need to be up all the time all this stuff needs to be Wired together described and wired together and and we use assemblies for that so 12 factor apps You know Cloud Foundry is really good at that good at that And but that that's kind of the the easy part 12 factor apps They're stateless and they have standard ways to log and configuration and source code manager all that stuff But we have all these other applications all these you can call them legacy apps, but Not necessarily legacy. They're just not cloud native applications. I guess All of these apps need all this stuff the automation and and versioning and Scaling up and scaling down we need to be able to do that effectively for any category of application Not just a web-based app or a 12 tier 12 factor application So what we need is to be able to manage this from a single Abstraction and we have what we call an assembly which is like a Docker compose file Which describes all the components of this application in a single place and then we can provision it This entire thing as a single entity Things like dependency management on how can I get the latest version make sure it's the latest version of a service is tied to to a specific application I'll say a little bit more about that is You know microservices has great promise obviously ability to version your all your services independently But in certain applications with with very high requirements of Availability where you know, there's penalties incurred if you if your app is down for a minute lots of money Penalties involved. It's maybe a little too risky just to allow that microservice to evolve on its own, right? And so instead you have to be able to say well this application requires this microservice version XY Zed I was gonna say that I should say Z since I'm in this country I want to be able to Tie each application to specific versions of everything and that's what an assembly can be used for it say this is my entire Complex application and I can deploy it as a unit. I can version it as a unit. I can roll it back as a unit So that's something that we we have had to build outside of the power of Cloud Foundry And we've that's what we've been doing for several years Another thing we need to look at is is like affinity or location awareness So if we have you know for for many workloads and applications You need to have the data and the processing close together and Hadoop is a good example of that But at the ability to to I don't know partition your into all your resources so that certain Workloads get targeted at those resources Hot cold storage, so I might have this you know the head of bytes of storage that has very Very long time to access it and then I have my smaller memory pool which is instant access And I need to be able to say you go over there and you go over there and we we Need to be able to use our paths to say all this stuff So specify all these all these constraints same with specialized hardware GPUs and FPGAs etc Our paths needs to be able to handle all this stuff I'm shared local file system again. I understand this is coming with Cloud Foundry if it's not already here But many of the apps that we do deploy We need to have the ability to share multiple Multiple instances of the app running across the paths need to be able to share a file system easily And we need to be able to specify this in the descriptor or the assembly descriptor Pardon me a single sign-on authentication So we have just huge Infrastructure in place to deal with this and that says here lots of customers It means there's lots of people within and outside of BNY melon It will be authenticating to our systems and the paths is what's deploying these systems So it has to have some understanding of an integration with all these different authentication mechanisms So very complex and there's some that are very challenging to do So we have to do a lot of work on that Yeah, third-party's customers Our whole stack whether it's ancient hardware or brand new software and hardware has to be able to deal with all this Okay secrets is an interesting topic as well. How's my time? Okay Secret management is regulated and I've been at companies in the past and I'm sure there's many many companies But whereas if I was a clever and malicious developer I could easily get into a database and look at things or even change things if I knew what I was doing I've I've been there and You know, there's there's ways to get around that But we have to be just very precise on how we control secrets and making sure that secrets don't get in ever Can never possibly get into the wrong hands So developers and deployers just can never have an access access to a secret that provides access to some Trusted or needing to be trusted resource Some sensitive resource, okay? Exposing secrets via the environment is is kind of risky So we're looking at sophisticated secrets management techniques to get past that and just to say a little bit about trust Um, you know, it's it might be it might seem to be a good thing to trust our developers You know trust developers in general, but in fact, that's not a good thing and it's nothing about the developers Myself as a developer. I do not want to be burdened with the trust of secrets I'd never want to have a secret because if I have that secret and something happens, you know Suddenly I'm kind of I'm in I'm in there somehow Whereas if I never had that secret never even kind came close to it that I can do all my work So the past has to be able to allow me to provision everything I need to and run everything I need to but without me ever getting my Hands on that secret any port any portion of the whole SDLC And so we have to very care carefully bake that into our paths. It's just a fundamental tenant for the past Okay, so that's a few of the gaps with cloud foundry But as I said, this isn't a cloud foundry bashing talk by any stretch There's a lot of really good things about cloud foundry And I'll just wrap up my talk by just covering these and briefly and I'm sure many of the talks here cover all of this stuff, too But it's a very strong community is is evidenced by this by this conference and by how things have been going for the last three or four years here And we are using the open-source cloud foundry, which gives us a lot of power as you can imagine Really good for 12 sat factor apps really good for microservices Really helped us with our hybrid cloud initiatives so we can take our apps and have them deployed across multiple clouds both internal and external You know as your AWS that kind of thing That's the same point here. It did allow us to do some POC's with Docker There were as I said, there were some limitations But we were able to really quickly provision Docker containers into our paths using our cloud foundry integration Many of the developers in house that use cloud used our paths that had the cloud foundry in Incorporation were we're just pleased to put it mildly of how Simple it is to provision apps when when you have cloud foundry taking care of all the heavy lifting The service broker cloud party service broker whole infrastructure. We've been using that URL provisioning is really nice. If you know the wildcard DNS while cloud foundry URL Mechanisms, it's really nice to for a developer to provision an app and that URL is instantly available That's a huge thing the logging This standard cloud foundry pitch, which I'm really good at by the way I've done it for many years, but I'm the logging the polymorphism ability to support multiple languages The security groups it has for isolating workloads and basically creating networks between between or partitioning networks between workloads The open source one we had no vendor lock-in and we had we had clouds so Yeah, no specific IS tie-ins really this far as from the from the higher level and It really helped with the separation of concerns who does what I'm out of time. So I'm rushing a little bit Yeah, so there's a lot of things that cloud foundry did very well for us But we are continuing to use our existing paths because we have built it to support all these things I discussed at the beginning of this talk Cloud Foundry is a path to components. So when you push an app to our paths It might in certain circumstances will push that we'll just pass it through to cloud foundry But you know so it'd be a single cloud foundry instance writing across these data centers as opposed to Kind of an uber thing. That's where our paths comes in But it does very well with the things it does it does cloud foundry Deals with all the things that it's good at very well obviously and we are able to leverage that Yeah, so greatly simplified app deployment on the descriptors are so much simpler as well So our app descriptors if we use cloud foundry for certain classes of apps We can simplify the descriptors and and the specification of those much faster to bring a developer on board to figure out How to provision an app to the paths to the cloud the URL provisioning. I mentioned Docker POC, you know, we can do some Docker work with with Diego and the hybrid cloud so that's kind of the The front of my talk. I don't have time for questions unfortunately I will take it just a second to say that we have a bunch of innovation centers around the globe doing some very Innovative things that we are hiring and it's a very cool place to work. So I just I wanted to put that in there I'm if you're interested and go to this URL or come talk to me. It's incredible the Initiative to innovate is coming from the top and from across the organization. We're doing some incredible things I'll plug we are there's a panel discussion at 455 today for 450 I guess in ballroom B and its Finance and cloud basically is what we're going to talk. We've got half dozen people or so to discuss finance and clouds So come to that and that's my talk. Thank you for the time