 All right, so we'll get started here. I'm happy to see, I did a quick head count, at least 30 folks, which is pretty good for after lunch session. So if you feel a little bit dozy during my presentation here, I won't take it personally. So my name is Andrew Lundy. I work for SAP in a team called the ISV support team. We're part of the partner program. So if anyone of you is partners and you're interested in working with SAP and adopting our technology or making things that work well with our things, then I'd be happy to chat with you after this session. And it's a good program. You build something, get it in our little store, and we help you sell it to our customers for lots of money. So just a quick survey of the folks in the room. This talk is a little more developer centric. So I'm hoping when I say who's developers, we got developers out there. Yeah, very good. And ops people, ops centric, got one or two. OK, well, that's cool. Not really much going on in the ops department in this presentation, but we'll just kind of model through it. I don't have a whole lot of slides to show. It's mostly me kind of poking around and demoing a few things. But I wanted to talk about this topic. It's kind of near and dear to my heart. As developers, it's easy to get frustrated. And when we look at this new world, it can be even more frustrating. So here's the agenda for the talk. Pretty much we're going to talk a little bit about what we're talking about, which is the pain of deploying and waiting and debugging things when they're not under your direct control. A little bit about what an MTA is. We'll get to that in a second. Also, why the deploy time is kind of add up. And then some of the things you can do about it, which is really the rest of the presentation. So localizing. And I'll do a little demoing here. And then kind of some things to it. So one thing I realized when I got here and started going to some of the sessions is that most folks, I would say probably nine out of 10 folks here, are not using SAP's Cloud Foundry installation, which is fine. But SAP kind of comes to the show adding a little extra sauce. And that also can cause new problems or kind of exasperate things. So let's talk about what it is. So here you are. You've been a developer for a while. Now you've gotten all into micro channel, micro service development. You've taken your big monolithic app. You've kind of broken it up. You've gone to the 12 factor website. You worship it like a religion. You've gotten all your things in a nice little microservices, containers, and all the details are abstracted away. And you're following all the right practices. It's completely stateless. And it's all good, which never happens, right? People still kind of tend to skip over things. But there's the rule or the law factor, whatever, number 8, which is concurrency, which really gets down to the heart of where you're breaking up your big monolithic application into little bits and pieces. At SAP, we've kind of added on to this concept. Because oftentimes when you're building a bigger enterprise application, it's not just one piece written in one language. More likely it's a suite of microservices, modules that all need to be coordinated. And it turns out oftentimes they need its own lifespan. It's nice to be able to consider them all kind of version controlled together, whereas typically you use manifest-based deployments. And some of this comes into effect there. SAP has this thing, this methodology called a multi-target application, which is really a collection of modules kind of bundled together. So in a lot of ways it's a deployment mechanism. So the first thing you typically run across in this sort of thought of it is a module called the app router. Now this is for your application itself. This isn't the systems router. This is the one that's just controlling your application. And we give you a stub one. This thing handles typically all the static serving of your application, so images, text files, sounds, and stuff like that, as well as it handles things like routing requests to other parts of the application. And also interacting with the authentication service. So first request comes in. If it requires authentication and it's not, then this module is responsible for kicking off the login workflow. Wherever that leads, at some point it comes back, hey, you're cool. And you can pass it along to further parts of the application. So then you typically have one or more worker sort of modules. We use the word module in application kind of interchangeably. So if I do that, don't be too thrown off. So in this illustration, I'm describing something maybe implemented in Java. Or another one, another part of the application might be implemented in Python. So for instance, you maybe you have an application that's mostly enterprise code, doing business logic. Java's probably a good solution for you there. But maybe you've got to sprinkle in some machine learning and suddenly Python works better for that kind of stuff. So let's do that part in Python. Obviously it's all microservices. So you can kind of distribute the workloads and scale and upscale and outscale these things as load requires. And this is a very simple model. But already we've got three, right? And you got to remember too that each one of these things is its own whole web server. It might be a light implementation of a web server, but it's got everything inside there that is a web server. So in the past where you had to set up something like Apache and get it all configured, each one of these guys is pretty much going to fill that function. And you can send requests to each one of these individually when you're doing the initial stages of your development. And oftentimes you want to do that just to kind of isolate your work so that you're not having to deal with maybe the authorization part of it. In which case you can directly call these things, but when it's all tied together and ready to go for production, this is more of what it looks like. Now, in SAP we also have this concept of defining the database in your code in a way such that that module can also be part of your project and be subject to all the code control and version control things that any other part of the project is. So we do that by using another module that's implemented in Node.js that actually just takes those instructions for building your data objects in the database and it starts, reads your instructions, does all the database work and then shuts down. But in terms of packaging and running things, the first time you got to do this. So now we're up to four, quote, web servers that need to be created. So this thing, when it works, it creates the database and then when it's done, it ties everything together. So it kind of goes out of existence, but that doesn't mean it hasn't taken some time to do its work as well. So if all this kind of is local on your workstation and it can be, we actually have a version of our system that will run on a laptop. Not a whole cloud foundry system, but if you're working with this sort of application model, there's a localized development scenario that will fit in a 16 plus gig laptop. Then it can be pretty quick. However, that's usually not the case. Usually things are done kind of locally and you then need to kind of move them up to the server. And just to kind of describe some of these things, when you're first bundling all your code together, and this again has to happen for each of those squares, right, if it's no JS, then oftentimes you've got to use MPM to pull dependent packages in. Now, when this happens, it's kind of like Robin Peter to pay Paul, you can do it during the bundling phase or you can just send your kind of minimal code to cloud foundry and the build pack will then try to resolve dependencies that aren't there later. But either way, you're going to have to wait for those and you're going to need a connection to the internet. So if there's anything about your internet connection that's sporadic or goes down or is slow, this can start adding up time-wise. Then on the local side, as you're pulling things together, there's certain assembly and then once you get it all assembled into what we call an mtar and mta archive file, which is just a glorious zip file, it's got to be sent up to the cloud foundry system. So after it's uploaded, so that takes time, then the system takes and creates a container, invokes a build pack. If we were doing Python for this example, it fires up the Python build pack within that container and then gets your code going in it. Again, during the build pack process, oftentimes if the dependencies haven't been resolved yet, then they have to get gotten at this point, so more time potentially, even though now you're talking the servers making the request, not your local workstation. And then once it's all ready to go, the system packages it up into a droplet, sends it into the blob store, and then immediately pulls it right back out of the blob store to create an instance, and then finally you get into the place where you're running things. So all this stuff takes time, right? And when you're doing development, you want to make a change and see what happened. So in my example, now I know with typical Cloud Foundry working, you're probably just working on one thing where you're talking about all this adds up to probably around five, three, four minutes or two minutes or something like that. But in my example, you can imagine all this thing happening four times, and that takes time. So I don't have it here in the slide deck, but I have, oh, and by the way, this particular project is available on a GitHub repo, so don't worry about trying to make notes about what you see code-wise or tool-wise. It's all listed in this repo. And I see here that I should have been in mirroring mode in my presentation because it has taken over the screen. Hang on a second. And that is not helping. Hang on. And I used to remember how to do this, right? It is. Let's just look at the repo a little bit. So when I was first deploying this project, see in my notes here, it was taking around 460 seconds. So that's getting on eight minutes, and that's a lot of time. You don't want to be in a case where you make an edit and then have to go through 460 seconds worth of deploy just to see what happened with it. So what can you do about it? So one thing you can do is you can move as much as possible local. So when you do this, so in the case of this demo, I'm going to use Python as my example. And one of some of the things you got to keep in mind is that if you're going to work locally and try to kind of work through your development issues as quick as you can, you're going to want to make sure that the versions of the language that you're using are as close as possible matched to what will be in the deploy environment as you can. So the advice that I give here is just if you're going to set up and do this, just work locally as long as you can. Now, you can only go so far, right? At some point, usually the set of services and interaction between things really only exists in the deployed scenario once it's in Cloud Foundry. But if you can kind of work through a lot of just the getting going sorts of stuff, you know, when you first start working on an application, yeah, you're trying to remember what... It's like we're in Python, so I got to like make sure all my spaces are right and my indentation is all right. Oh, what was that system call called and all that stuff? You can work through a lot of that pretty quickly as you go. Yeah, you can ignore things like the authorization. Just set yourself and your module up to work anonymously to start and make sure that it's definitely moving quickly from that perspective. Like I say, mimic. And then this is pretty key. It's pretty common now. You don't have to use get, but use a version control to kind of synchronize things. And one of the reasons for that it will come to a little bit later. And this works fine. So for instance, let me use my example and now it's not giving me the screen anymore. Hang on. Somebody's going to have to tell me what that setting is in PowerPoint where it doesn't want to use the alternate side. I think that's the one I want. All right. So here I am and I'm using Eclipse here. The IDE is not important. It could be other things. This just happens to work pretty well for me. The project I mentioned earlier I've brought in. So if we just take a look here, we can see that there's several components. Web is the one that I was calling app router in my description. And then the one that created the database here is called DB. Then I have one Node.js. That was that one. And then there's a Python one, which is what we'll focus on. So say I'm working on this Python component and I just want to test it. So I just right click and use the standard tools to test. So here it says it's run my Python. It's up and running. It's camped out on port 8089. So I can go to my local browser and I can pull it up now. My little example program is not as fancy as most of the ones you've seen. It's just spitting out some text of the environment so I can verify things. I've got a few little links here, but effectively it's done. It kind of told me what was going on. And what's nice about this is I can just flip back over here and I've got a full local development environment. So I've got a couple break points. I can test it just quickly making changes and saving them and I don't have to redeploy anything. I could just save and test. So what I'm going to do now is like, let's just say there's something that requires a little more investigation. Let's debug it. So now I've relaunched it and debug mode. Oh, well actually I forgot to kill it first. It was camped out on port 8089 and now it's had that in use. So let me rerun it, debug mode. All right, so now we're up and running again. I'll go back. I'll just refresh this page. And here we are. It hit this first break point. It's in this dump, pie and end environment routine and we can see all the debugging tools are available. I can step through here and trace. I can watch my variables as they're being modified and verify things are going. I mean so many times you think you know what your program is doing but there's nothing like really knowing what your program is doing. There's nothing like the actual truth. So sometimes there's just no substitute and it's usually you get to a branch and you just assume this particular variable had a value so you keep wondering why is it skipping or not doing something and there's nothing better than just tracing through it to find out, oh yeah, it's not what I think it is. So here I'm just creating a variable and adding some text to it. I created a little integer here and before I put it out though, I'm gonna, I mean you can do things like this. I'm not gonna get into all the different debugging tips and techniques but just as an example you can see here that I have my integer set to a value of five and I just went in. Now that it's been assigned I've gone in and with the debugger and I set it to a value of seven. So I'm gonna go ahead and just run this to completion now and if I go back to my output you'll see here it's actually spat out over value of seven and so we know that it's actually doing what we think it's doing. So often you get confused that you're looking at some sort of cached output. So basically that's great but again and I'll flip back to my presentation. This, oops, got a little bit, right. This only goes so far, right. It works if your data container is accessible from your local machine that may not always be the case and if your application relies on say, Redis or something else that's really not easy to replicate locally then you're kind of out of luck with that. It doesn't mean you can't get through a bunch of your coding. It just means that that portion of it is not available to be kind of tested against. So here's my local workstation and basically we're just connecting straight to the database with the copy of that Python module sitting locally. I should have got through this part before. Anyway, so what's the next sort of approach that you can do to kind of minimize your deploy time and we can do what I call piecemeal deployments which is just pushing one part of it. So in this case, we got my version on my local machine and I just want to do a single push. Now again, a lot of people they only have one module that's all they really work with so this is kind of what they do every day anyway. But even this takes time. You remember that timeline I had on the other slide. All that stuff adds up and if you're doing it this way, sure you can make a change locally, push it and when it's ready and running you can then hit it and see what your result was. But one thing that you kind of don't have to do is you don't have to rebind all the services. You can kind of push on top of something that already exists and all the bindings will stay in place unless you manually undo them. But one of the things, this process that SAP has put together with the MTA is that we express all the relations of all the different modules in this MTA YAML file and you can't really take advantage of all that stuff being specified there. You kind of have to copy it into your local push statement with all the parameters. So I've done this with one of my things and it actually took only about 40 seconds to get that push. So we're going from close to eight minutes down to like 40 seconds starting to run low on time. So I'm going to like skip some of this and skip ahead a little bit because our last session was a little bit long as well. One of the things, the caveats of this approach though is you have to be careful where your dependencies are coming from. If they're locally and you do a push and the architecture of your local workstation is like mine and OSX machine then and you're deploying it into a Linux environment. If it pulls dependencies that are architecture specific to your local machine then they won't work once it's deployed. So you have to be a bit careful about the architecture of the dependencies. And one way I get around this is I've got a local Linux docker that I've got all these tools in. I can run locally and that way it always mimics again something very close to the deploy environment. So what about, what else can we do? So we can also edit it directly after the deploy. So once it's been deployed we can actually shell into it and edit it. Everybody familiar with CF SSH? So yeah, I got some head nods. So it's great. I won't, well again I've got, what about five minutes? Yeah, so I won't burden you with that one. But there is a, that's great. But if you manage to do something that causes your session to fault then all your changes are gone. The Stager will restage your app from the blob store and anything you were doing is gone because that's what Cloud Foundry is supposed to do. It's supposed to keep your stuff up, right? So the fact that you're poking it while it's live is a little dangerous. There is a way to get around this and if you look at my project you'll see this figured out and that is I've created a little script that actually takes a form of the SSH command and it uses it to mimic an SCP of a single file. So if you look in the project again the link to this GitHub repose in the document which is downloadable from the schedule. So don't worry about not being able to find it but I've effectively created a little script here called CP to CF that will take and run this command in this specific way and the blog or the post here has all the details of how you get these commands and how you invoke them but there's effectively a special variation of the CF command that allows you to grab this one-time code and then you can use your standard SSH program in this special way and the special way is like you got to get the GUID of your application and you got to go to a certain special endpoint and all this fun stuff but the effect is that it allows you to throw one file into your live container and that's great. You can sit here back in your editor. You can make a change. You can save it, sync it and run this script and boom it's already in there. But I want to make sure to not miss the pester's instance which is tunneling home. So in this case what we're doing is actually also using the SSH program the CF SSH command in a way such that we can invoke SSH to create a reverse tunnel back to from the container running live that's already deployed back to our local machine. And once this is established what we can do is we can invoke the same debugging functionality on the live container working in deployment with all the environment that it's got available to it and debug it in sync with the code that's sitting on our local machine. And so I'm going to... So I've already established the tunnel using this command here. I'm sorry. And what this is doing is it's creating a reverse tunnel on port 5678 which is the debugger port. I'm going to go to my debugger. Sorry. And I'm going to cross my fingers. I hope this works. I'm going to start the remote debugger and then I'm going to go to my application that's deployed and I'm going to run a little routine called detach which calls from the code that's running in production back to my local machine. So when it does this... Hang on a second. I may have to kill that other debugger. All right. I know we're running tight on time here and let's try to invoke that again. Some of the problems are the tunnel can get dropped. That's a problem. That might be the problem. Let's try this again. Oh, I've been... Ever since I changed laptops I've been having this issue where it wants to debug. You can see that it's connecting, but then it immediately drops it one more time. Come on. Yeah, it's not going to behave. All right. Well, the point of it... Sorry. It's not a real presentation unless part of your demo doesn't work, right? So the point of this was when the code that's running in production calls back to your local server here through the tunnel, then you get basically the full debugging functionality that you had when you were testing it locally, but now you're doing it with the full environment available to your application. So with that, I'll close. Let's see. Presentation. Any questions? Yes. No? Yes. So it's a good question. It was, how do you hot swap your code without the orchestrator detecting a problem? And the answer that I have is that for one in Python, it's set in a mode such that it detects when there's a change to the source and then reloads itself. So I've done this with Python. There's a way to do it with Node.js using a module called Nodemon. Java, I'm sure there's a way to do it as well. But to answer your question more specifically, the orchestrator is using the health check to see if you're still going. And when you do this, the health check is still functional. It's still saying, hey, my application's up and running, so it doesn't think anything's wrong. Again, we're talking about debugging in production, right? Something you're not going to want to do a lot because it's dangerous. We're talking sharp knives here. You don't want to leave debug settings on in production. You don't want to have this tunnel sitting open in production. This is only when you're trying to get those problems that you can only figure out when it's running in production because your production environment is just that specific, right? Obviously, you would probably do this more against a staging environment or some other development environment where you can have a lot more leeway. But sometimes things happen in production that don't happen anywhere else and you need ways to get in and look at stuff that way as well. Any other questions? Well, I appreciate you all coming out. Thanks so much.