 Here we are again Before we start with the next Speakers, I would like to thank one more person Not gonna thank you guys. I would like to thank one more one more person up there Doing something that we actually don't see around here, but we have a stream online Was last night couldn't sleep really couldn't sleep I was freaking out and I started seeing on YouTube videos and we were there for six hours Straight beautiful shots and it's all on upstairs doing a lot of work. Thank you very much mate. Thank you All right, our next speakers Jurgen and in aunt use case from the municipality of Amsterdam Give it up for them, please Good afternoon everybody amazing to be standing here. So yeah, awesome Yeah, let's start with the introduction. I'm Jurgen Alwein. I work as a lead consultant for Luminous and Also work now for the city of Amsterdam as a lead architect for the cloud platform Yeah, my name is Leland Paderkoper. I'm also an solution architect and the system engineer and Also working for the gemeente Amsterdam and my company is it impressive So, yeah, let's first start with the organization A bit of background of the city of Amsterdam. So Yeah, a bit of who we are the city of Amsterdam is has 18,000 employees we have Of course the the city council the political part, but we have 80,000 civil servants working for the city of Amsterdam we are separated in four clusters with 42 direct management teams and One of the teams of the clusters is IT and that's about 1600 people So it's quite an organ is a big organization And I'm not sure if everybody has that ID when you're talking about the city Our local government so quite big Yeah, we started with a with a new IT strategy as a city of Amsterdam because we want to move to the next phase and So we asked the usual suspects like partner to talk to tell us what we want to do Turns out that it wasn't the same as we were already planning, but yeah, if Carter says it's okay so We decided that we wanted to go cloud native and we decided that we want to have Multi-cloud strategy, but first start with one. So we focus now on Azure As our first we also have a strategy of SAS before past both before yes, so all the Azure components are mainly pass and some of them are yes small part Yeah, we use only general available features. That's very important for later in the story And We do everything as inference code Yeah, we said also as architects we set up some cloud principles and They are based on our cloud strategy. Yeah, you see the list here. I think the most of them are recognizable like You build as you run it you own it start small Automate everything but also security and service management compliance by design So that all those things should be part of all our solutions and of course We also have to comply to several compliance standards like the BO, which is the government Compliancy standard But also we are using the security control framework for Microsoft and the modern service management control framework also for Microsoft and we have the The PR which is about privacy and the government standards like Nora. So there's a lot of compliancy standards Which we have to take in account when we deploy something Yeah, then the question is of course why share take the s. That's our business case and From here we want to take a look from our workloads perspective. We have around 40. I heard most of the most of the time of you and Costs and Compliancy that's already told but also security we want to get in place And we saw a lot of pop-up being up a different kind of AKS environments without any maintenance and trouble Basically, so I want to take that into account with you. So looking to the workloads We see a lot of people that wants to embrace basically the elements and the features of AKS But also other technologies in Azure like the data platform and the data bricks and that's the one But that's for another other use case But most of them they were experimenting with with containers So we wanted to embrace them that they are doing so Also, we are doing the way of DevOps working It's more or less common nowadays, but most of the teams do have to change to that way of work And we wanted to centralize our Kubernetes experience Because there are very rare at the moment looking to the market Looking to the other elements like cost and security. I think those are most common nowadays everybody has to Try to keep the costs in in place but most of them we're trying to achieve and Setting also FinOps. That's one of the elements that we are doing also and We're trying to centralize basically all those loosing AKS clusters to have not costs running everywhere But you little bit centralized and if you have some centralized platform You have the possibility also to take into account the compliancy and the automation by itself Then the technical side we had a little demonstration of one of our speakers yesterday Christina. Thanks. Thank you for that We're going to dive a little bit deeper But most of them it's obvious of course Kubernetes of itself There were a lot of functionalities in technical preview one and a half a year ago So we faced a lot of issues like oh, we cannot solve it So for that we had to do some tricks and elements. So that we will take further in our presentation Yes, so when we started we we needed to have a plan of action. So first we started with the architecture Look at all the standards. No Then we looked at the design decisions and documented them and Yeah, then finally we had the solution that we could build So if you look at the architecture as a base we use the cloud adoption framework for Microsoft We have the we also comply to the well architected framework for Microsoft. We had last week No, no two weeks ago. We had a review from Microsoft and well architected review on security and What I think it's very cool is that we scored on the question part 84 percent compliant and on the advisory Policy side we scored even 91 percent compliant. So even Microsoft was impressed with our setups Yeah, I think that's quite nice for our first. We are also Haven compliant. That's very important in the local government Area because I think Haven is very is the thing but Haven is 15 standards So we are Haven compliant but Haven isn't not Amsterdam compliant We come much farther than Haven We are also BBM to compliance that says something about the security of data, which is used in the cluster like personal traceable data so we had to take that in care and We had to take an account of our current HubSpot model where the solution should land and we want everything to be private so our Clusters are not publicly accessible Default if you deploy AKS, it's perfectly accessible But we use private endpoints for that to make it even more secure. So basically we are not on the list That was shown yesterday I'm going to take you within with us with a lot of design decisions I've tried to spread them in a sort of logical way, but looking to the platform There were quite a lot basically so what we are trying to do is to get most out of Azure basically So authentication with Azure AD and that kind of stuff and the most important one was of course the private functionality so you can run AKS in public mode and also in private mode Meaning that you have to work with private endpoints basically and there the older fun was starting Some functionalities weren't available at that time. For instance, it was the ACR, right? That was not Very helpful, but also we had some other discussions Beside of this big list, I want to highlight some of them I tried to make them bold, but one of the key functionalities was we have to separate all our workloads basically Meaning that every workload has to be identified and must not see each other Within the cluster. So when we started when we first met I said to the guys like hey Do you know that there's one beautiful functionality coming available and that is the agent pool concept That is a functionality based on the fact that you can set in Kubernetes stains or sort of labels and on that you can also set them in How is that called again Over availability zones so you can basically Select three of them set them in an agent pool concept and Laydown a sort of Azure tag and all the costs are also identified Also, the traffic was separated completely. So that was a cool feature and I think also the most important one of this list Other components that we want to hit is the engine X so Azure offers in its Subscriptions the possibility to use the agik. That's the application gateway Ingress controller the combination of the application gateway and the ink was controlled combined At that time that we have investigated but also later this this year in in January We still still see it. It is in GA, but it's got a lot of issues It's not reacting as it's supposed to be and since we are facing 40 Teams we didn't want to look to that issue Facing the issues that are explaining on GitHub basically So we chose engine X and in that way we can deliver very fast One other element Autoscaling is of course enabled on our side and we're trying to use Azure Defender for cloud as well Looking to the automation. We are using of course the products that Microsoft is offering So Azure DevOps is one of the tools that we are using We also have taken a look to cured. I don't know if it's familiar for everybody Can I see some hands that's familiar a few of them? Okay, so cured is Kubernetes reboot demon agents. That's an agent basically working on each worker node Giving us the possibility to update our Linux operating systems and you can add our schedule If you want to for us, it's a weekly schedule So if there are updates required for a reboot, it will do it by itself Not meaning that all the nodes are coming together, but separately, of course An other topic that we are highlighted in the to-do is the automation of DNS and certificates Most of you maybe are familiar with external DNS or set manager. Those tools are mostly used in the automation But we have some manual processes. That's not allowing us to talk to the API systems yet so this is coming but Maybe in the future, but for now it's a manual process Looking on the application side, so we have used We have also taken a look to some applications One of them was the application performance monitoring that we are filling in with application insights Is it the answer for everything at the moment? we hope so but We don't have any requirements yet to differentiate on this So there are a lot of tools out there that's helping and performing in the application stack But at the moment, we don't have complaints yet in that way Besides of that, we also use VM skill sets for creating our container images required for applications to run, of course and That is also a lot of conversation about it because it's an IS solution But at the current state the only one that we can Deliver we also have taken a look to ACA as the Azure container apps Giving us the possibility maybe to create the self-hosted agents a meaningful way to provision container images to build but also to provision to our Azure environments Looking to the security, of course, there are a lot of elements that we have taken into account One of them because we are using managed of a user Agent pools is that we have chosen user managed identities a way of authorization giving to an agent pool and then later on in our slides to other resources That are specific for a workload team, but I will come back to that one Defender for clouds. I think that's obvious, of course But one element that we are missing at the moment and that's because it's not the a of course Is the CI CD scanning so the possibility to scan our code basically it is coming and We are excited to see but for now we had to change On that direction to a third-party vendor Other elements that we are using of course is the Azure policies So in the Azure policies gatekeeper, that's a functionality for a lot of people maybe familiar where you can set Specific requirements on your cluster on your total cluster or specific namespace parts Nowadays it is built in into the Azure policies Giving us the possibility to harm basically our clusters on the security side on the cluster side and Also on the on the Kubernetes versions as well The Kubernetes best practices also yesterday explained by you And other elements that we are using is calico policies. So Kubernetes offers us the possibility to go Clownative by itself by Kubernetes policies Those are have the limitation that you can add them on namespace level and Calico one of the first CNI that is GA with Microsoft Gives us the possibility to not only a limited on namespace, but overhaul the cluster So you can set your basically your networking rules on the highest level as possible Meaning that you can make it closed Unless and then open with the specific firewall requirements of your needs Looking to other elements. It's a bit of it's too small. I I do notice so we have added a lot of links A little bit of the baseline we had just followed from Microsoft So Microsoft is creating a lot of baselines nowadays specific on the topics of ACR that we also required and AKS and On that we have made in a lot of lists to just peel them down Looking to the end solution. What are we offering to our? Workload teams possibility, of course to get their own Agent pool on that agent pool we gave them the possibility to differentiate based on the VM size So if they need more CPU or memory, they can choose otherwise in this picture It's a DV DV for s V3 On we also have given them a limit on the pot rate So they are not over provisioned their Kubernetes worker nodes that is based on the VM size Because we are using the as you see and I the as you see and I gives a lot of limits on that and for now most of the workloads do not go over the 50 parts so Looking to end slash 24 subnet it should be sufficient The network traffic we already hit on and the Azure Tech, of course What does that mean for a workload team so a workload team still provisions their own resources? That's we do on purpose because they are responsible for the database or a key for for the certificates For example, but also for their persistent storage So we can deliver the the full self-service functionality to them and we are creating connectivity from our Subscription basically based on an agent pool where the user managed identity where I just spoke about can be authorized on those components what the workloads are basically creating on That way we can provision a lot of functionality out of the box They get their own Azure DevOps and on that we can create also the automation To the namespaces and an element that we heard a lot based on namespaces So we provision a namespace and the agent pool and they can deliver all the applications they need so road map Of course, we don't have everything Ready yet. We started last year still work in progress some components are not available. So Yeah workload identity. We are really waiting for Microsoft to have that GA Really happy to implement it. So when it's there Yeah, the multi listener support we already implemented at the moment But that was one of the request of our teams Part-based row routing we also implemented that was also a request of the teams Service mesh we have not implemented service mesh yet because there's no Requirements at the moment for service mesh. So why implement something which adds complexity if it's not needed but as soon as it's in as the requirements are there we will implement it and get ops Isn't it also on our list to implement somewhere this year? Maybe you can switch shortly back. I Forgot one element to add On the left side you see some components in the light blue and those resources We have created for all the workloads basically and just to to shortly Take them on the application gateway We have shared them basically on all the workloads that we have available and also you see an It's it's a Dutch word be be here after day But what we mentioned is a sort of troubleshooting a VM so that they can do and see how their workloads basically are working Future plans we have a lot of other things. We are currently investigating should we move to the Azure C&I and move away from the Azure C&I Yeah, leave the Azure C&I and move to the Calico C&I We are we are doing a proof-of-concept with that containers getting we just Acquired a tool for that and are implementing that together with the vendor Keda, yeah, also no requirements yet for it Confidential VMs is something which we are looking into because next to BBM tools Complian C standards we can also know something that's BBM 2 plus that's More strict, so we are investigating if isolated nodes could be there a solution for that Yeah There's some other of the things aka as Aks edge is also one of the things which is there if there's a requirement we will implement that That's one was released yesterday, so yeah Yeah lessons learned Yeah, I think one of our biggest lessons learned was that we it's nice to the Deploy a case but it's not only deploying a case. It's not ready the product You you have to do a lot yourself and what we found out is that our choice for running at private Limit is in some of the options that are available some of the options for private are not available In GA, but for public they are available in GA So we had some difficult this discussion sometimes about that. How are we fixing that? Are we possible to fix it? Can we fix it? So? Yeah, a lot of things to look at Yeah, one of the other things is that the reducing of costs by sharing that that's that's a big win for us Now we have some problem with convincing sometimes the teams that that is really a win Yeah, that makes it sometimes More difficult, but yeah, we have evidence that it will save costs so And also the automation, right? I mean we have provisioned everything with with vice up and all the automation tools Yeah, it's not only AKS, but it is also the application gates way the listeners and all that kind of stuff Yeah, and I think the the most important is that we will never be ready We always will have to need to innovate Because I will always new be new options on security or on Kubernetes. So we always have to innovate All right, that's it questions Thank you very much Denon and Jürgen I love that you guys have shared the wins learns and then not so successful experiences and Totally, it's totally true that in container container times container years one year is 20 So if you stop you you are saying you're saying behind do we have any questions for Denon and Jürgen Bring it on We use the self-hosted DevOps agents and the VM scale sets to solve those problems Basically, we have two one of them is specific for the image build process that we are doing on ISVM's In a skill set manner. So they are going up Until their job is done. They are being destroyed. So every time they boot up, they get the installation script for the required tools That's mostly one of them. The other one is the self-hosted agents that is currently in Somehow a public forum. So we have provisioned it in a public kubernetes environment strict Placed with a lot of network policies on that so we can deliver basically only the needed requirements to go from public to private to their subscriptions and For with Microsoft we had last week. That's really fresh We are taking to look to the ACA solution to see if they can offer it likewise But then it's for the cloud native Unless they cannot deliver the isolation that we require Next question Yeah, we use tagging we have a tag on the agent pool so all the costs of the agent pool Because of the tagging are visible and we can we can show them to the workload our the shared costs are being divided over the workloads via formula which is made by the FinOps team We have space for one more question Yes, we do Question regarding the controls that you put in place for Azure policy. How did you do the mapping? for bio to the Azure policy and Do you have some? And some best practices there how to do it because it could be a complex issue there Yeah, yeah, so we use the bio compliancy policy here initiative from Microsoft We use the SCF and we have our own security risk team that Extended the SCF and made sure that we had a translation to the Azure policies And yeah, it was a lot of work. It's still a lot of work to keep it up to date also change in policies So, yeah, we're constantly looking how to improve that and make it better A lot of remapping on all those requirements. Thank you Thank you very much, you know, and you're again, it's always good to see consumers and Their struggles and their successes. Please give it up for them Now one one great thing is that everybody is trying to keep that door closed thank you and One other great thing is that we are still on time and we are going to have Floor which is the moment. She's very cold She's going to be speaking at 330 with sustainable open source One of those high energy people is always nice to have around so please take your break 30 minutes come back with us