 Hello. Hello. Hello from KubeCon. This is our slightly delayed open office out. There's a delay actually. Yeah. Yeah. Okay. Welcome everybody. So we've, there's a bunch of us over in KubeCon. We have been here for, you probably want to, I'll put in. We are talking about Jenkins X. We've had a fantastic week. There's been a huge amount of excitement around Jenkins X. Carlos did a fantastic talk. I think there was around, how many, about 900 people? Yeah, 900. Yeah, 900 people for Carlos's talk. It has been pretty huge. We're very, very excited. We're going to have a little write up as well when we come back and show a bit more of the details. Liam's just showing some of the conference there. For anybody that hasn't been to a KubeCon, it's hugely exciting. The energy here is just immense. Positive, it's very, it's all around a fantastic conference. So yeah, we'll do a bit more of an update on that and maybe just do a blog on the Jenkins X website when we're back next week. But yeah, so far everything has been positive. There's a lot of excitement for Jenkins X and so we're obviously very excited too. Any other highlights? James, can you talk or anything? Right, James is struggling with his headphones. Rob, what's been the highlight for you at KubeCon? I think actually the highlight for me has really been Carlos's talk. Obviously about Jenkins X, so that's really, really good. I mean, it's just the general part. So it's a real excitement around the earth and people are really increased and excited about Kubernetes in general. So without Jenkins X delivery, it's great. And actually we've come across a number of projects which we think are going to work really well with Jenkins X and accelerate the growth around that. So it's good. Will you join it? Did you join it? Yeah, actually, so there was a quarter with some of the Bitnami folks actually yesterday and they come up with, they've actually now got this like an app or something, it's a distribution of Prometheus and Grafana and a number of metrics out of the box that actually they're going to distribute in a nice easy way to consume. So that seems like a perfect win that we can try and get, try and add in as soon as possible. What was that chap, James? Is it Shipper? Is it Shipper? It was Shipper and it was from booking.com. Let me just find the link. This looks very cool. Yeah, and so this chap came over. He really loved this chap, came over and started talking about what they've been doing around Canary. And there's a link that James is going to find. We think it's called Shipper. And it looks like a great fit for Jenkins X. So it's just kind of in as soon as possible. Yeah, Bob's just popped it in the Hangout link and can add it to the Google Doc afterwards as well. But yeah, maybe everybody's interested in what would probably be hopefully doing around Canary deployments as well, then have a look at that link. We should be hopefully in the new year get that in as soon as possible, really. Yeah, Shipper looks very cool. What's been your highlight, James? I think the general buzz about, well, Kubernetes, Jenkins X, CICD. Lots of people have asked about serverless Jenkins, which has been a lot of people who are maybe not ready to go full on CICD with Kubernetes, but they are using lots of Jenkins. And lots of people have been very excited about moving to serverless Jenkins. So you don't have any Jenkins Masters to manage anymore, which is awesome. There'd be lots of excitement about that, but I think preview environments keep blowing people's minds when you can see the light bulbs go off in people's heads when they realize how awesome preview environments are. But no, it's just been awesome. Lots and lots of chats about lots and lots of things. One of probably the most common question that I've had, which is actually surprising, actually, is how do we build our Docker images? And I've kind of packed a little bit every single time at the moment where you remount the Docker socket so that Gareth will be looking at a mechanical and Google Container Builder. So basically, I've been telling everybody it's going to be sorted as soon as possible, but I don't know if that leads into Gareth. Yeah, I mean, I had the time. Sorry, you cut out there at the end, James. But yeah, it's definitely... It's very, very high on my priority list, and I'm very, very close to having it all working. It works. I've got Google Container Builder. Building images really nicely, pushing them into GCR. I'm doing that. I'm just trying to work out at the moment how to integrate it into Jenkins X so we can sort of enable it as a flag when you create a cluster on GKE. Very, very close. That'd be awesome. GCR support by default on Google Cloud would be great as well. Yeah, I mean, it seems to do that natively. As soon as you enable the Cloud Builder, it just pushes to GCR. Awesome. That's really cool. And it's super fast. It's so much faster than building on our own images. I think the serverless builds, I think they create that mono repo, or each of the mono repos creates about 10 builds of about two GKE each. So you can see why we run out of disk space pretty quickly. That's very cool. So, good. So we can actually continue telling people that that's almost it. We actually, I'm just chatting to Dan actually. I think he's managing the Kanako project, I think. And he's saying that he's been adding loads of caches that they've really quite shocked at how fast these builds are actually going. So that's really exciting stuff as well. He's taking a fancy job. I'm breaking up. Okay, I'll just keep... I think I might have lost connection too. We can hear you, James. Oh, awesome. Flaky Wi-Fi at the conference. Who'd have thought? Which video will be coming to my mind? Who knew? Who knew? Okay, so can anyone hear me? Yes! Yay! I think people are getting kicked out at the moment. Some region. Make your way. So, anybody... For us. So, I'll just mention briefly the progress on adding more Git provider support to Prow is coming along nicely. I got some good feedback from the Prow SIG testing community about that. In fact, apparently some of them that are there at Cube kind of been talking about it. And people keep asking, like, hey, when are you going to support more than just GitHub? And so, it sounds like there's a nice groundswell of community support behind this. And they had asked me about it. So I'm going to present at the next... Whenever the next SIG testing call is, what I've been working on so far and try to get some feedback and just kind of keep... Make sure that I'm staying on people's radar and that kind of thing. That's all. That's really good. Yeah. Cosmin, how's the... So the vault operator looks to be just about there now, right? Git talks, Dev mode looking. Is it close? Yeah. Almost. I guess in the installation it's almost there. It works. I figure out some issues today in the upgrade platform and it also to make upgrade platform working with Vault and also to move this GPG encrypted secrets to Vault, I would say. And then in the pipeline, we need to fetch secrets from Vault as well. Currently, we use Kubernetes secrets for that. But I did a lot of refactoring in the install. Recently, also clean up Git, rework the entire Git stuff now, make it clear with the pipeline and normal user, how you set it up. Yeah. So it's progressing, I would say. But should be soon ready. That's... As soon as the Dev Git talks is done, we should be able to do multi cluster pretty quickly after that, really. Doing GitOps for the Dev platform install is the first step to really nice multi cluster. And then for me, so the app stuff is also kind of waiting for Cosmin because, again, I'm trying to do the app stuff based around the GitOps for Dev environments and almost every single app you install except really trivial ones need a token or a password to connect to the service they're going to talk to. So I'm kind of stalled a bit, intentionally stalled a bit on that because I can't make much progress. But I've got a lot of the pieces in place. So we have a working tool that will ask you questions based on a JSON schema. So for everything that's in the JSON schema, it asks as a question. So I basically have all of the latest draft for the JSON schema implemented except some really weird education, which I don't understand how to implement when you're asking questions. There's some odd stuff around nested schemas and things like that that I didn't do. But the rest of it kind of works. So I'm going to integrate that tomorrow. And I kind of took a detour down writing an app for Slack, which is based around the Slack bot that James originally wrote, which is now working quite nicely. So it will direct message you with any pipeline, any pull requests that you create with the pipeline. And then it can also tell a chat room when a new pull request is created. And it will mention anyone you put as reviewers on GitHub to notify them that there's a new pull request created. And I'll do that. I'll do that initially as a room, but then we can also do that as direct messages. So it can literally just direct message any reviewers to tell them that there's a pull request waiting their review. And then I'll do some nice filtering, so you can kind of filter which repos you on, some which rooms and things like that. That sounds awesome. Is there a way to get notifications over the master build fields? Yeah, I'm going to do that. That's the last one because I realized really to do that we need a kind of history because it's not really very useful just notifying the person who last committed with the master build fail. You kind of need to notify everybody who committed since it last passed because it might be any of their fault that it failed. We don't really have that right now in Jenkins X and that's a really classic feature in Jenkins, right? So my thought is really that it's quite easy to work that out though because you just look back through the pipeline activities for the last success. And then you can just query the Git provider for all of the... I mean, I'm hoping that I can then take the... I know we stole the last commit shot in the pipeline activity and I'm hoping that I can then use that and maybe some utility and go to basically just work backwards through the commit history and pull out all the user names and put all the names of all the people who committed since then. Absolutely. Yeah, so, you know, because I've got code now to, like, resolve Git users to Slack users, so it's just a case of, you know, running it all through that same code, basically. How's the code work to associate a Git user with a Slack user? Are we using the... So, Git Slack basically identifies people by email address. It's Slack clearly retrofitted user identity into the system or global user identity, right? They clearly started with user identity per workspace and then retrofitted the whole unique stuff in because, actually, under the covers, you really just reference everybody by some kind of ID, like, you know, 10-digit alphanumeric thing. So, whenever you mention someone, you kind of have to call a method on the Slack API to get their user ID out, but they have a thing called Git user by email address. So, you can always just use that, I think. So, what I've done is I'm pulling the email address from Git and then I'm checking the user CRD to see if... No, sorry, I'm pulling the user name and the email address from Git provider. Then I'm checking the user CRD to see if there's an alias... See if they've got their Git user name on the user CRD and if they have override it either with a Slack user name or the email address from the user CRD and just as a backup using the email address from the Git provider, which I think that will work for most people, but, like, for us, we quite will often have our personal email address as... Or, you know, if you work for a company, you'll often have your personal email address as your GitHub user email, but then you'll have your corporate one for your private Slack or whatever. So, yeah, so it's just kind of making it possible to do that. That's great. One question to the application secrets. Do you want to store them in the same... Like, in the same system, whether we store all the... I think so because the way it works with GitOps is when we do the install of the apps, they literally get installed in the same release as the platform and all the platform components. So, to me, it seems to make most sense to just keep that consistent and install them all into the same place, but I'm also, you know, open to ideas on that. And do you need to share these secrets also to, like, let's say developers because, currently, we haven't built all the user management into default. It's just one admin service account. I don't think we need to share them because the person who would add the app would provide the secret and then the app would use the secret. So, it's entirely service accounts. It's not, like, personal accounts don't need to get it, but I'm not sure if that's the question you're asking. Yeah, I'm asking if we need to give access to the VoIP developers, for instance, if they need to create secrets from VoIP and then you need to separate what is, like, secrets for developers and what are, like, installation apps in secrets. So, developers don't need to see the secrets any more than they do with the platform. Okay, so there's no sense to put them in the same port, I would say. Really, the purpose of this is more the other way, right? So, it's so that the app installer or the platform installer can provide a secret that the app or the platform can use. Okay, so that's, it's just extending to the app. Okay, yeah, I mean, if you can also do this today, actually, you just need to create, probably, an application kind of sub-path and then create an application name for something, yeah. And next week I'll try and do that. Fantastic. How's the impossible task wheel of the Git provider supports other than how you're getting on? Sorry to throw you right into that. Yeah, no, it's going well. I mean, there's a lot of pieces to that. I'm kind of, so I was working on, I was kind of working on GitHub Enterprise as like the first sort of, I don't want to call it low-hanging fruit because that feels like bad luck, but kind of like the easiest, the first obvious thing to do is add support for GitHub Enterprise. A little bit stalled on that at the moment just because Proud assumes a version of, or it uses, there's a couple of places where Proud uses the GraphQL API that only works with a version of GitHub Enterprise that's higher than the current version that we have at CloudBees that we can test on. So I'm working with, you know, some of the folks here at CloudBees to get that upgraded so we can, so I can start working on that. So that's kind of, that's a little bit stalled at the moment, but then while I'm waiting for that to get sorted out, I've been working on factoring out the Git provider logic and actually moving it into a separate repo and moving each individual provider into its own package and kind of trying to clean up some of the, right now everything's kind of just in one package, which is, you know, nice and convenient and, you know, you can just develop really quickly with all this stuff, but then there's some, there's a little bit of leakage. Some of the abstractions aren't really as airtight and as nicely encapsulated as you might want. And so I'm kind of working on getting all that taken care of and making sure that, you know, that those are all nicely encapsulated and that the individual providers aren't coupled to one another and that the provider interface isn't, you know, coupled to any of the implementations and that kind of thing. So I've been working, just working on that, trying to get that all done. Weird Git foo that I've never done before to try and pull that stuff out of the Jenkins X repo while preserving the history, which is being uncomplicated, but I'm kind of having fun wrestling with that. So that sounds fantastic. I also want to give you a shout out. So we had some amazing feedback. Bitbucket server, is it James? Yeah. So somebody came up to the booth and said, yeah, they're using Jenkins X with Bitbucket server, flawlessly. So a massive shout out because you were doing that when you actually, as a community contribution, that was before you couldn't join part of the core team. So I thought, just really wanted to give you a shout out and, you know, well done for that. Well, thanks. Yeah, I appreciate that. Any other, any other things? Anybody's got this? I see there's a few folks on the channel. I've not seen before. They want to say hello or anything? Any feedback? It's fine if you don't. Hi there. Hello. Hi. Nice to meet you. Nice to meet you. Just wanted to say thank you for this really great project. I've been using it for the last month or so on some new stuff at my work, and it's really sort of let us stand up a whole new team and get sort of all the CI CD stuff out of the way, which is something that we've been struggling with so, yeah, thank you. That is fantastic to see that. Thank you so much. Anything, any improvements, anything you have, please keep any feedback coming. I'd love to work with you and I'll always continue to improve everything that we're doing here. So that's great to hear. Awesome. Thanks for the work. Yeah. Cheers, Andrew. Thank you. Is there anything else from anybody? I had a quick question for Will, which is more just kind of something I've been thinking about, which is that with the Git stuff, quite a lot of the Git providers don't have half as richer an API as GitHub. And I've been adding things to the Git API that have been nice to have for various apps and things like that. So like what I was just talking about with the Slack app, I get the reviewers for the pull request so I can notify them on Slack. And, yeah, sure, it's fine if you don't have that. You can still use it. It just won't mention anyone for your review. But I was wondering if you thought about how we want to handle the fact that these APIs, some of these APIs, like say, we can't always use one that's really bad. I think Bitbucket's pretty crappy. At least the client we use for it, like most of the stuff just doesn't seem to be implemented. And I'm wondering if you thought about that. Yeah, a little bit. So one thing I've thought about, not so much in connection with Bitbucket, but in connection with Garrett, for instance, which only really provides the code review features. It doesn't really have very many of the social features that GitHub has. It's not an issue tracker. One thing that I was thinking about was trying to figure out, okay, so GitHub basically has conditioned us to think about all of these different things as Git that aren't really Git, like code review and issue tracking and all the social things. So one thing I thought about is, and I've been thinking about off and on, is how could we kind of decompose all of the components of Git that make up GitHub and try to figure out, okay, this is code review, this is issue tracking, this is whatever else those other things might be, so that we could do something like in the case of Garrett, you could have Garrett be your code review system, but then you could use it with an issue tracker, JIRA or something else, and that essentially it would be, to the other components in Jenkins X, it would be relatively transparent. You would still do things like update issues and create pull requests or change sets or whatever they're called in the particular Git provider, but they could be delegated to different systems and have that be kind of decoupled and those different components be pluggable rather than just assuming that it's all one big monolithic thing like GitHub. So I don't know if that gets us to what you're talking about, Pete, but I think that kind of, that at least allows us to start reasoning in that direction where we can think about, okay, how could we fill in some of these gaps and some of these providers? That's exactly what I was kind of thinking, but just supporting the Git provider like Garrett's a great example. You actually wouldn't, I mean, yeah, we could support Garrett, we do support Garrett, but like 98% of stuff won't work. So I guess the kind of more meta question was, well, is it worth it and do we actually kind of need a different approach? I think as well, we're currently using the ability to sync the pipeline status to commit some pull requests. We use that quite heavily for promotion. And maybe we should default to using just pipeline activities instead of the Git provider because, I mean, a lot of Git providers, you can Git clone and run a pipeline, but recording CI statuses on a Git provider is a fairly Git abuse thing. Bitbucket kind of does it and GitHub has a bit of that, but all the cloud hosting Git providers, for example, they're super simple. They rarely even have pull requests, never mind all the other features. So I think having a generic implementation of lots of the bits of GitHub that we can reuse on different Git providers would be really useful. They're interesting to me, like branch protection, which is kind of related to commit status. That is actually, I did look, and that is actually a really common feature. I think everyone has some kind of branch protection. So yeah, okay. Yeah, no, that's good. I'm glad we're thinking about it because it was something that was vaguely concerning me. Yeah. Awesome. So I'm probably going to wrap it up, but I just wanted to maybe finish off with pretty much a year to the date where the first commits for Jenkins X happens. Honestly, I'm kind of a little bit exhausted, but blown away personally of how Jenkins X has just taken off. It's a true community project. And it's been anybody's watching this back if they get to the end of this call. A big thank you to everybody that's been involved and hopefully we can encourage more people to continue building up and taking Jenkins X further. This year has just blown my mind and I suspect next year is going to be even better. So yeah, I just wanted to say I've finished it off with a big, big thank you to everybody. I'm looking forward to the new year, the new effort, the new features. Okay. If anybody else hasn't got anything to add? Maybe have a birthday party. It's super loud here. Yeah, but we should maybe have a... the next hangout should be on the birthday party one year ago. And then we could maybe have a beverage together, a virtual beverage. Sounds good. I'm all in favour. All right, okay. Let's wrap it up. Thanks to everybody. Catch up soon. See you online. Cheers. See you soon. Cheers.