 All righty. Thank you everyone for coming and now it's late. I'm tired. I'm sure everyone's tired So we'll try not to take too much time Out of curiosity who here is a Bosch platform operator type person? Cool. Awesome All right, so I'm Morgan fine. I am a product manager at pivotal working on the Bosch team And I'm Danny Berger. I've had quite a lot of time as a Bosch engineer So get started today. We're gonna chat about a few things first take a little time to go over what we've been working on the past six to nine months just kind of call out some of that and then Take a look at some of our upcoming stuff And if anyone starts seeing Morgan swaying side to side, please give us a heads up No falling kitties or morgans One of the first things that we've tried to be more intentional about the past Six months or so is how often we are cutting releases of our stem cells and director Historically, we've had a bit more of like every several months or Longer periods where we don't cut the director for a while or do More delays there, but we've been trying to be more intentional and say every two weeks try to get the latest features out Part of the goals with us doing that is to get more faster feedback and Let people try out the new stuff. We're working on Stem cells. We also have been doing every couple weeks just to make sure we get more security patches integrated and Try to get that out there quicker for everyone Graph kind of shows some of the components like we've got Bosch Nubuntu trustee and Xenio lately and CLI You can see we had quite a few trustee releases happening in the middle of last year Which was something that made us kind of reconsider and kind of Focus on how often we really want to be spending time cutting those new releases and getting those out there So it's kind of motivated us to Officially define what our support policy is so we follow a N and N minus one policy So every time we cut a new major We'll support that until we cut two versions later So in the example here, we keep supporting and cutting minor versions of these stem cells Until there's another new major That we can then continue with Yep One of the things that we've been working on as a team from the pivotal side is that We've historically had a fairly large team working on directors stem cells CLI agents and all the other various releases Recently we've been splitting up the team responsibilities So we now have a kind of director side of it and then a systems side Director is focused on the core Ruby director component some of the Bosch IO a couple of the other smaller Repositories and products and that's led by Morgan as the RPM and then the bar systems team is taking responsibility for more of the Core shared assets across environments So things like stem cells agent DNS and a couple of the OS type releases and that's really helped us kind of Both narrow down what the team is responsible for and needing to maintain and it's really helped folks kind of Be more comfortable as they're joining the team and being more productive The system side is led by Mukesh You can find both of them on slack and then we also still have the Bosch channel for if you have questions Currently that's in the systems team This is all kind of a fairly new structure And so we're still trying to figure out where some of these Components are but if you have a question you can still raise it in the github CLI Issue or bring it up in the shared Bosch channel a couple other things we've looked at Recently we've started doing Bosch PMCs every month I think we just had our first or second one in the last month And so this is trying to be more intentional about bringing in all of the different CPIs and community members who are contributing to Bosch and Make sure there's more communication and shared vision about where we're going and what we're working on It's public if you'd like to join. There's the Bosch PMC notes that you can find on github and Join us if you're interested And then kind of along with the more regular releases We've been trying to be a bit more intentional about listing out Changes that are coming in through the release notes on github just to make it easier and also trying to more closely followed the semantic version in Concepts and so majors really do represent stronger breaking changes and minors are intended to be more features additive and Patches if there's some backwards thing but in general continuing to be kind of a rolling forward approach to cutting Bosch and then one other thing We've it's always been interesting in a bit of a struggle with the open source and Repositories and following up with the various issues. We're trying to be a bit more intentional about that So we've tried integrating it into some of our daily workflows through tracker to keep it more front of our minds and Regularly prioritize that and work through it to make sure all of the team is able to contribute and help follow up with issues So it's kind of where we've come from now turn it over to Morgan for where we're going cool, so Historically, I think a lot of folks might have seen Bosch notes as a way of keeping track of a lot of ideas and dumping ground of Thoughts and communication one of the things that that's great for is it really helps us define and think through Those problem spaces and think like a tactical level but it was hard for people outside of the Bosch team to really understand where we're going and In the last few months we spent a while thinking about what a strategy and roadmap and more thematic areas for us to invest in are and so this next section is effectively going to be talking through some of those themes and Basically features and problem areas that we're solving and hoping to get to in the coming Six to twelve months dependent on timelines and all those other fun things So one of the big ones that really comes up a lot And I think we asked this at the beginning is who works with Bosch who operates Bosch who uses Bosch to deploy a foundation and More importantly, what is that day two of the foundation really looking like? And one of the big things that we always talk about is how do you scale a foundation and keep a small platform team able to run a really large platform And so a lot of these areas are things like Improving the day two of a resurrection experience. And so I see at least Bayhont here who's been working on that a lot at SAP And making it easier than this global resurrection config to hopefully help there and give you more Nuance control when a VM goes down and bringing it back up Another bigger theme if those of you have talked to me here heard me talking about is you run a Bosch commander Do something with Bosch and it just does not do what you expect Typical one is running start or stop during a failed deploy and Bosch trying to just update the entire deployment We're making a very focused effort to make sure that we Make Bosch do what a human intends it to do and that these surprises that come with these commands are not as surprising if at all And then the other one is making it easier for I guess I flip bullets two and three Making it easier for platform folks to be declarative and what they're doing with their manifests There's lots of nuances and edge cases to sharing releases and sharing stem cells and compiled releases and patched digits and all these things That at first glance don't seem like they conflict with each other And then you do something and all of a sudden every VM is updating We're making it a lot easier to be explicit and ensure that the software you intend to run is running and doesn't get unexpectedly updated Another one which is a harder one but comes with this is just continuing to improve Faster deploys and make speed a top-of-mind thing. I think there were a few Shades and jabs at Bosch thrown about that in some of the keynotes. So gotta throw that back there Another one is I'm sure folks here are always aware of security It's really important to Bosch kind of the base should be secure And if nothing at the base is secure nothing on top of it can be secure So everything we do is always built with security first and foremost But even on top of that we're trying to think about how do we improve a day-to-experience for security? We've been working a lot with the cred hub team about certificate rotation and ways to make it easier to rotate Variables and certificates and what an interface between the two products might look like going forward Additionally, there's things where Bosch just needs to put credentials for itself onto these VMs But how can we minimize the access of those things and make sure that when you're on the VM? That's really is the least set of privileges that are required there Another model that we're starting to see as we continue to move forward is just this Untrusted operator model or how do you really reduce access to the least people who need it and As I'm sure a lot of you are aware Bosch does have credentials It does put credentials for its jobs on these VMs a lot of companies don't trust Those VMs to be there and don't trust people to be able to access those VMs What does it look like to secure against these sorts of threats of a human being malicious all the way up and down your foundation? Most recently this isn't necessarily a roadmap, but did want to call it out We've made one improvement in this area where you can store credentials on an ephemeral disk or on a temp FS This of course only works on Linux, but it does help attack against a disk being stolen because the credentials are on memory Another big theme that we've been focused a lot on and I think there's a whole other track about this is on-demand services and Data services and CFC are in all of these different areas We're starting to see a lot more customers in areas where people are focused on having a lot of foundations and having interconnection between those foundations right now Bosch has a very Holistic view of itself and considers itself the world and there's not an easy way for a Foundation to map in route to another foundation We've had a way to expose public IPs, but it wasn't necessarily easy and in an on-demand world doesn't work So we've made some changes to cloud config to support this and make it easy for an application in one foundation to route to a data service in another foundation and then last area in this bit is Providing more information during job process stopping that really turns into as you have a lot of clustered data services That in a lot of cases take longer than you might like to drain and they have to do that right now because they're naive We just call a stop command and it's up to them to drain however they want We're making a new life cycle hook called pre-stop that will hopefully provide the software The answers to these questions below Around is the VM going to be there is the deployment getting torn down and depending on some of these things the software can take a Smarter decision for updating so if the deployment is going to be deleted Who cares if you gracefully shut down just kill it if the VM is going to stick around after the update Maybe you don't need to drain it. You can just reattach to the process after you've modified a few things All of this is hopefully going to help speed up upgrades a lot in certain cases and maybe even into deletions as well Another one which is fun to think about is the Bosch director is a singleton right now Historically we've seen This come up as a concern from people But when we talk through it the director isn't actually in the critical path for applications And it's actually the applications that are what we're concerned with not the ability to create new VMs Now that we're starting to move towards an on-demand world where we're dynamically creating clusters and creating services all the time We are starting to see a need for a higher availability of the director And so what are the sorts of cases where? Having a singleton director isn't enough either during a planned deployment of it Because it's down for however long it takes to bring up a new VM and as a result your pipelines are broken or The VM or the AZ fails and you need to get up more quickly The only answer for that right now is BBR But it's something we're starting to think about more seriously and try to figure out What the right problems people see in this problem space is the most important to solve first and then expand from there Another common question we get is how do I know when to scale up my director? How do I know when Some of the certificates that are in use are going to expire How do I know like if I should use a bigger disk for my director a lot of things There's a lot of benefits that come with the logging and metric suite for Cloud Foundry itself But the director has been a layer on top of and hasn't had that benefit We're working a lot with some of the newer teams in that space to answer these questions and make it easier to hook up into the same syslog server or Data dog or Prometheus or whatever you're using the ability to get answers to these questions with the same sort of KPIs that you Would get with the Cloud Foundry installation. I think Danny is going to talk about a few more things Couple more so then on the technical side Bosch has been around for quite a while and One of the things we've been kind of evaluating is do we need some of these longer running features? And can we help kind of try to deprecate and move them towards not being used anymore it generally Causes a lot of complication in our code and maintenance headaches just keep them around so a few of the ones that we are specifically looking at our Removing support for older v1 style manifests Essentially the world before cloud config where you were defining all your networks and I asked specific configurations in the deployments themselves Another one is the NAT is kind of our central Communication message bus and so we a while ago added support for using TLS as the authentication But historically it relied on username password. And so just trying to remove that out and have less security options available and Then also related is how we deliver templates right now Or right now we support going to both blob store and through NATs NATs is kind of our preference because it's direct communication of those templates and they're not stored anywhere else Since we've been supporting that for a while seeing what it looks like to remove that So these are all things We kind of are discussing and when we're getting a little more serious we bring it up in the mailing list So I think a couple weeks ago We mentioned the v1 style manifests if you're using these sorts of things and have feedback for us Please be sure to follow those lists or chat with us on Slack so we can kind of hear more and figure out where we're going One of the things you probably heard quite a bit is references around kubernetes and how the different kind of Communities are solving problems Bosch has been around for quite a while and seen some of these problems and has some ideas on how things like what's working or Why we do things the way we do and so kind of looking for opportunities and ways that we can participate in that community One of the more recent things is like looking into you cluster API, which is something I'm starting to look into a bit more with some other folks and Try to figure out. How can we bring our experience and understand or improve? participate with the SIG groups around that and Figure out what's what are good ways for bringing up kubernetes and all the components around that and then we also have the Other component level teams working on kubernetes and cloud foundry like CFC are Rene and Istio Probably have seen several chats sessions here today so with that you've kind of survived the rest of Summit and Free you but we have some time for questions. I think there's anything want to chat about now But we're also available on Slack later. There's anything watch out about later Any questions Yeah Yeah, so the question was what's the difference between the pre-stop lifecycle hook I mentioned and the drain lifecycle hook They're very similar in that they're both called pre-stop is going to be called before drain They're both Unbounded in terms of there's no time limit for them to finish The difference is that We try very hard to be backwards compatible with everything we do as a result We didn't want to make any breaking changes to drain And so we are adding a few net new environment variables and removing a few Random oddities that get provided to drain and just not providing those to pre-stop because we're not sure why they're there And we haven't seen anyone using them So they'll be very similar. They're called in order Eventually we might consider deprecating drain in favor of this, but for now we're releasing it one other thing that's kind of interesting with this is we want to try releasing it as a beta feature and not necessarily a stable contract to get feedback on it historically, we've always released things and they've stuck even With out necessarily getting the best feedback. We're hoping to get feedback from the community and people using it and not Confirm that it is a solidified contract until we have more confidence that it works. Well any other questions? Well, I got two of them. Yes, the question was are we considering implementing the indicator protocol for the director metrics we might be to be honest the approach we were thinking of taking was working with The Prometheus team which is a new team that's been spun up in Pivotal and I think in the foundation to focus on sending more logs in metrics to Prometheus So we're trying to take the approach of provide the information to them and let them choose where and how it goes So they might be but I'm not sure if we're going to be Yeah, I Think so we're starting to see more and more like site reliability engineering practices being spun up within The foundation and across different components as a first step I want to just get metrics out and then Hopefully probably add those things Yeah, that's a good one. Are we working on improving the windows CLI windows Bosch CLI experience? We actually spent quite a while Two or three months ago, maybe a bit longer talking with the windows Bosch team and a lot of customers about that We had the assumption that like we should spend time and should improve it And then when we really started digging into it We found the majority of folks Typically SSH into a jump box and then use the Bosch CLI from a jump box most of those jump boxes tended to be a Linux based distribution and So we were kind of like yeah, we could but no one's really using a Bosch based windows CLI But this is one of those things where as we continue to get more signals and continue to hear things if you have Different thoughts, please let us know Anymore questions. Yeah. Yep. I was wondering someone's gonna catch up So Bosch create and the command to use to actually create a director uses the old V1 syntax Yeah Yep Nothing is the short answer which is because it goes through a different code path then the Director code path So you use the old syntax to create a VM But VMs that are created by a director news the new v2 syntax We haven't taught the CLI which is where create and front about cloud config and about v2 concepts And so for now, we're not planning to but eventually we might Right Right now one of those oddities with the difference between pre-stop and drain is that drain finds out the change in the persistent disk size Which is a way people determine the next state for the VM We're being more explicit in that in the next state will be X rather than you having to infer some different information Yep So we're not planning to kill drain anytime soon both will be available and you can keep using drain But for folks who want that more explicit declaration and easier parsing that'll be available shortly Yeah, I'd love to see it Any other questions there is somewhere That's another one where we're trying to figure out how long until we start working on it Things that could come up and that are Important it is up there But there tends to be things that are more pressing and provide more short-term value And so we keep having the question when should we start investing in Bionic. I think that's the exact thing So the question was are we planning to use a mysql Galera cluster for the Bosch directors database? Potentially we're still very early in what a higher available director looks like I think right now We're starting with what are these stateless processes that need to be? made Highly available and can be redundant Assuming a single database the next thing is what is the redundant data tier look like any other fun questions? Yeah, you want that one? It is expected just by the design of the create and vibration, but it definitely is it's I think technique the question was is it intentional for Bosch create M to fully recreate the VM for every change even if it's minor and Technically it probably is possible for the CLI to be clever like it can talk to agent in the same way that director does but that's just a lot more state that then has to be tracked in your local state files and a lot more complexity so historically hasn't been enough kind of Motivation to try to bring out that same logic and to the CLI But yeah expected not optimal Any last questions? Then I think that's it. Thanks so much