 So everybody, welcome again to another OpenShift Commons briefing. We're happy to have today with us Jeff Verlast from Smalls to talk about the Belgian eGovernment's journey to the cloud and to OpenShift. So I'm going to let Jeff do all of the talking and introduce himself in a minute. If you have Q&A, what we're going to do today is we're going to let Jeff give his presentations for 20, 25 minutes. And then after that, you can ask your questions in the chat room. And for follow-ups, I'll just unmute you and your Q&A will get recorded as well. So without further ado, Jeff, let's hear about what's going on in Belgium in the eGovernment today. Thank you, Diane. So I'm Jeff Verlast, architect at Smalls and responsible for getting a part of the Belgian eGovernment to our platform as a service. So today I'm not going to present you a lot of technical stuff, but we are going to tell you a bit what were the reasons for getting there, the journey that we have done so far and what we are doing at the moment. So let me first introduce Smalls as an enterprise to let you know what we are doing and why we got to OpenShift. So Smalls is an in-house ICT provider for the Belgian government. We do different stuff. We do software development, custom development analysis, everything. On the other hand, we have our own data centers in which we offer infrastructure to our members, not only the infrastructure, but also the operations 24-7. And some of these public institutions require skilled personnel, and there we have a staffing solution. So the main idea of having such an in-house service provider is creating a cost-sharing model. So we are a non-for-profit organization. And we try to deliver this cost-sharing by applying technical standards for all institutions. If we work together, we can apply some economies of scale. We can reuse code. And very important, if the project is gone, the people remain in-house, so we retain our expertise. So we are a part of the government. We are mainly focused on social security services and also a lot of healthcare stuff. So what we create as services are things like child allowance, unemployment, the health insurance, everything like vacation, pension, all those services are treated by our programs, our data centers. We are not a recent company. We were founded in 1939, so we are already delivering these services for 75 years. We're not the most known company in Belgium, but we can say that every Belgian is somewhere in our systems because it is a complete life cycle of a person. They come into contact with the solutions that we provide. So we are government, but also in the government there are a lot of challenges in IT today. So there is a very strong pressure on the budgets, not only the IT budgets, but also a lot of business budgets are under pressure. And so they are looking for IT solutions to help them out. So for us, that means doing more with less money. We have a lot of mission-critical applications. One of the examples is we have one for donor organs. So if that one is down, really some lives are at stake. So when we are going through these budget cuts, we still have to remain operational at a higher level. And of course, because all of the data that we treat is highly confidential, we are talking about income, about medical state privacy is for us very important. So if we want to improve our efficiency on that, we think we have to do that by increasing the collaboration a bit more than we were already doing. And for that, we as SMALs are participating in a larger government community cloud that is being built at the Belgian G-Cloud. So that one is really a complete solution where we are offering infrastructure as a service, platform as a service, what we are going to present today, and also a lot of software as a service offerings. So let's have a look on how we came to go to platform as a service and then more specifically also with an open source stack. Well, in 2014, we had a very traditional infrastructure which was based on WebLogic 10. WebLogic 10, which we needed to migrate from for technical reasons, it was getting end of life. So we had a very large task ahead of us and we first had a look at, okay, what our business needs before we started to do technical migrations and taking decisions. And like I said, we are an in-house shared service provider for institutions that are very linked to us, we know them very well. But in fact, you can also compare us with this car manufacturing plan. Because if you want to get a service operational in this traditional infrastructure, what you had to do was you had to take a lot of different steps before you saw any results. And all those different steps, they were performed in sequence, so it could take some time. And all these steps were rather complicated. So this means they had to be done by a specialist. But that was not that much of a problem, but of course we are flexible, we can handle that. You can pick whatever model that you like, as long as it's in our catalog. And you can tailor your solution at your needs. You can pick any color that you want, as long as it's black. And we started to feel that, okay, standardization, of course it is important to get the efficiency, but sometimes we got the impression that, yeah, if everything you got is a hammer, then your problems better look like a nail. So we wanted to make sure, are we tackling the correct problem? So our solution may be best in class. Yeah, it was when we created it, our infrastructure at the moment, like the Model T. But in the meantime, the world is changing. The customers, they want all these fancy features, like windscreens and airbags and safety belts. And we can do that because we are flexible. So we started to optimize our processes and we did some custom automation. And that is working fine, but the result is something like this. You can still see it's a Model T. We have added a lot of flexibility into it. But as you can imagine, this is not the most industrialized solution. We can't repeat that very often. It's more one-off. So we sense that we had to do something differently. So what were the non-functional needs that we gathered with our customers? Customers said, yeah, but you know my business. It's easy. It just has to be fast. And then you talk with another customer and he says, no, no, you know my business. It just has to look sweet. And then you talk with another and he says, no, no, no, for me, capacity. And you talk to a fourth one and he says, no, for me, capacity is everything. And then you say, yeah, but that's still kind of different to what the other guy was telling us. So we had a lot of different non-functional needs that we have had to cater to. Luckily they do agree on some things, security and reliability. That is clear. Everybody agrees on that. And there is even one topic that they really, really agree upon. And that's the price. So we ended up with the conclusion that if we are just going to replace our application server, that won't give us all these things, all these change that we needed. And we were very reluctant to create another factory, another car factory of a new model that in the end would not be flexible enough. So our solution that we needed was really something that is from the beginning ready for change that still remains reliable and of course is cost-efficient. And because we have these legal security requirements, we had to be able to deploy it on-premise. And we knew from the start that if we want to make this a success, we would also have to modify our processes. So in the end, we ended up with OpenShift Enterprise Version 2. And as I said, only installing the product was not going to do it. We needed to mindset shift. So before we were very machine-oriented, you had to specify in which DMZ you had to put it, what is the name, what is the IP address. But in the end, it's just the applications, the services that what our customers want. That's the only thing that they see. So we had to shift to application-oriented. Because we were machine-oriented, all our environments had the risk of being different, which means that if you test it in development and then you put it in production because it's a different environment, you have no guarantee whatsoever that it works. So we wanted to have this container environment that we are sure that if we test it in development in production, it's exactly the same. We don't want to create a lot of processes with a lot of tools like we had before. We want to have a simple self-contained application so that we could install it 100% automated. And 100% automated means no more manual interventions. Because with these manual interventions, somebody can make a typo, can forget something. It isn't done. No, it should be zero-touch deployment. Once the deployment is created, nobody touches it, and it installs completely automatically if you want to change anything you redeploy. We also wanted to get away from the one enterprise-wide solution that needs to fit all the needs. Instead, we created a standard solution but with room for extension. So this room for extension allows us to be flexible. But it also meant that because we now have standard solutions with room for extension, which is self-contained, that we couldn't work anymore with a development team that only looks at the program, the middleware team that configures everything, the database team, the server team, whatsoever. Because we wanted to get to one definition of an application, which means that you need one team to deliver a service, and that team then needs to be noticeably naming. So what were the core concepts that we used to explain to all our employees what they needed to be doing? First of all, the application should become self-contained. You don't want to stock your information in 10 different applications. This means the code, the configuration, the modules that you need, but also things that are related, like database changes. If you deploy a new version and your database has to change, then why not store your database change scripts in your application? This means you can use it during the deployment, and via the deployment hooks we can create one application deployment that does everything. We wanted to automate as much as possible, and with OpenShift V2 we saw that we couldn't reach 100% because security and network-related stuff, that is still for the operational guys. And we didn't have a lot of plug-in points or hooks in OpenShift V2, so our aim was, okay, let's do those 95% that we can do with OpenShift, and we create a solution for the 5% later. We set up that solution with all those teams that I gave you before. Let it be clear, in going to a platform as a service, it is not an infrastructure job. Because your containers have a lot of aspects in them, they have the code and the configuration, you will need the expertise of a lot of different teams, so you have to roll that out with all those teams involved because otherwise they can't support that. We went for full traceability because before we had a lot of manual changes, manual changes that we couldn't really trace, and then if something was wrong, it's complicated because you don't know if there is a change, what was the change and who did the change. So from the beginning we went completely for personal accounts and that was explicitly not for finger pointing, just to make sure you knew who did the change and you could ask them the reasons for which they did that and to look together to fix it. And because we had these personal accounts and this whole security model, again like with the database changes, we were able to reuse this same security model with all the logging infrastructure that we had, which meant that based on the security model you get an access to a certain application and automatically you also have access to the centralized logs that it produces. So we really integrated this platform as a service into our complete infrastructure. Like I said, this standard solution with room for extension, that is important. We provided some default hooks that the applications could reuse but if they had specific needs, they could change them, which on the one hand delivers us the flexibility and again it standardizes it because everybody is doing his extension in the same way. They are using the same hooks. All the information about the extension point is also contained in the application. And then we took already in the V2 the decision to make a multi-tenant setup even if for V2 we did it only for one business unit. That has learned us a lot about the complexity of creating a multi-tenant setup from the beginning and as you will see further on that were lessons that were very valuable in what is yet to come. So if we have a look, we are now operational in production for more than a year with OpenShift V2. What can we say about platform as a service and OpenShift V2 as a whole? So we really learned that the auto idling really helps us increase our efficiency. If you look at the number of applications that we are now hosting in our development and test environment compared to the simple infrastructure as a service that we had before we have a lot of more containers and we are not using more resources. So we are very efficient and we can give the message to our development teams yes, you can create a container if you really want it. You don't have to justify every container that you create. You don't have to wait several weeks before all these other teams have executed their stuff and that means that a lot of tests that before we only were able to execute very late in the development flow I'm thinking here for example to cluster tests that at the moment the people are doing them much earlier because they can create their cluster environment and they test with it two, three days and when they have finished their tests they simply throw it away and it's something that we never did with traditional infrastructure because it was simply too costly. Also, scaled deployments, they really simplify your life. Before you testing with the cluster it was complicated you had to know how to do it. Now whether you deploy to one gear, two gears, eight gears it's the same command. So the platform really lowers the learning curve for your developers to start using real clusters and real environments which greatly improves your quality of testing. We started to do this for replacing our web logic so we were looking at the JBLS application service but at a certain time some of the colleagues in-house they learned about this project and they said yeah well but I'm doing PHP and I'm doing static websites and I have the same issues. I also need to have this environment and I can't do it on the same infrastructure and yes we can because this platform directly gives you the multiple technologies in the same interface which also means that you are working to standardization on another level. At this moment you can operate a container without even knowing what type of technology is in it and that is also very useful. About the process that we took, as I said we took the decision to automate 95% and in the perception of the people at this moment all the problems seem to move to the last 5%. So the aim should really be to go the whole nine yards you automate everything or otherwise the people will still complain about it and as we will see OpenShift V3 is helping us there. The decision that we took to say okay let's take a basic solution that you can extend yourself there we got a lot of mixed reactions to it so there are a lot of people who really love it because they say okay this really is giving me the flexibility that I need but some people they really love straight answers and they come to you and they say okay with this OpenShift tell me how do I do this and when the answer is well it depends and you can do it like this but you can also do it like that for some people that is annoying so you really have to look also at your communication. Same thing in the process now that we give the people the flexibility it also means that they have more responsibility before they created the tickets it went to the experts the expert is doing the configuration and if he isn't happy with it it blocks it comes back now you have the flexibility to do the change yourself but that doesn't mean that you don't have to inform the specialists that you don't have to talk to them and there are also some people who take the flexibility and let go the responsibility so again this is not an installation of a product like I said we were trying to create a mind shift so in fact we are changing a culture and changing a culture does not happen overnight it requires a lot of communication a lot of trial and error sometimes just to get the development teams really embrace this change and start thinking differently about the infrastructure which they deploy about the model that OpenShift offers as a platform as a service we are really happy about that but what we learned already there is that being a shared service provider also requires some more or some other features than just being on premise so like I said even in v2 because we knew what was coming we started with a multi-tenant approach in mind but it still isn't that easy when you start connecting a lot of different networks there are real challenges that are there OpenShift v2 did not have an answer for everything OpenShift v3 we see the same situation at the moment we are still lacking some essential features but we are getting there when you are changing the model and especially in the multi-tenant way the security model that you map onto your product is really key if your security model is okay you can use it like I showed you with the centralized logging you can also use it outside of the platform as a service and offer one broad view to computing for your development teams really not splitting application per application but offering it as a service so thinking about how you are going to map your business and your security governance internally to the product is something that you should take a time once it is in place it is rather difficult to make major adaptations if you do it right it really works very well if you do it wrong you are bothering the people and also what we learned is that once you introduce something a new model like this container you will see that your pricing aspects become important because in our traditional infrastructure we had some pricing model which was based on how the infrastructure worked now these containers come along and some of the assumptions of the previous model were incorrect so we are really penalizing some types of some workloads so if you are changing your model you also have to look at the prices because in the end it should not be three times more expensive because then your customers will also not go and use your service so that in short our experiences with OpenShift V2 but I think that for the audience here OpenShift V3 is more important so let's say how we are looking to the migration from version 2 to version 3 what we saw is OpenShift version 2 you can classify it as let's say a classic Linux box on steroids you still have fixed IPs you have user groups that are really clear you use the DNS those are all notions that your infrastructure team know that they can handle that it's not that much of a new world for them if you are talking OpenShift V3 well that's Docker and it's not only Docker but it's Docker on steroids you have this OpenShift layer Kubernetes that are coming into they really need it because let's be clear about that the security model of Docker is really not suited for creating something that we are doing we have the software defined network that is coming along and yes that adds a lot of flexibility but like we had to guarantee also network isolation doing that with a software defined network that's hard now we have this routing layer which means that we are using the DNS differently than we were doing in V2 so for our infrastructure teams it was a learning curve that was clear luckily we already started in V2 and also during this complete process our relationship with Red Hat has evolved for quite a bit so in V2 we were really customers implementing a product like it was but with the experience that we gained there we went into version 3 so we are already some time in the Commons community thanks again for everybody who was giving us a lot of very appreciated details there over the last month and also now it's really more of a partnership we are really working together to reach our goals and I think that is really needed to get a major upgrade like OpenShift V3 rolled out in a multi-tenant environment so where version 2 was really a big shift in the mindset of our development teams so getting from this classic traditional infrastructure into containers at this moment from V2 to V3 for the development teams that's not a big change because they already have this configuration externalised they know what a container is looking like and they see the same concepts in V3 for your infrastructure teams as said V3 is a bit more of a challenge one thing that we did see now that we are going from V2 to V3 is that docker can be a bit overwhelming so everybody is seeing in version 3 well it's based on docker so what can we do with docker and what do we need to do and how do I create a docker file and how do I link to docker containers OpenShift is doing that to you so we kind of had to curb the enthusiasm of some of the development teams because the docker got some unneeded attention and they were thinking that they had to change their complete way of developing which isn't the case so let's have a look at what we really like in version 3 if we compare it to version 2 for example so first of all the packaging like I said in version 2 if you wanted to do a binary deployment like what we want to do in our production environment it wasn't really supported out of the box so you had to create your proper deployment package yourself we did a custom script for that we are really happy that now that is standardized with docker not only for us internally but we even started to use docker as kind of the interface of external software developers that they can simply send us applications developed locally at their site they send it in this docker container and we can introduce it to the platform without having differences in deployment architecture so that is really the standardization there is really helping us to accommodate a lot more different types of workloads on our system also the docker system is much faster in scaling sometimes we had long waiting times due to the F-sync that needed to copy all the files from one key to another so it's clear the packaging in v3 is clearly superior also the integration of the metrics into the console and the action that you can do in the console the logs that you can see there it's a large difference so in v2 operationally the console was not really helping us in v3 it's really a very used tool it's not only the web console that is superior in v3 also the CLI, the new CLI OC it's much cleaner and most of all it's much faster than the old one that we had in v2 and that is important because we also use the CLI to integrate with some of our systems so we really like that about the OpenShift v3 that it is really open we can integrate everything that we already had existing build systems our existing infrastructure we can integrate into OpenShift into the pipeline and we can also extract a lot of data via the API that we can then re-inject into other systems like for the capacity planning and such also one of the big issues that we had with v2 were those gear types so in v2 you had to specify your containers then you had to log that to a node and if an application came with different requirements then you had to create a different node you have a small infrastructure and you are not yet scaled out that sometimes was a bit of a problem you had to put your application into a gear type that was not really suitable for the application and you had to be creative in v3 the resource limits per container or per deployment is much more flexible it's well done and if you still remember these our aim was to automate 95% because we still had to handle security-related stuff well that in v3 is handled by the secrets so the secrets allows us to specify things like passwords, key stores, certificates and things like that we can store that in secrets and there is a really nice security model that will make sure that only the people with the proper credentials can see them so all in all v3 really brought us a lot it really fixes a lot of the limitations that we saw with v2 there are also some things that are in OpenShift v3 that are a bit less useful for us first of all we don't use source to image mainly because at our site we don't have a lot of different programming languages that we need to support so the auto detection feature is not that useful for us and also the build pipelines in Jenkins we already had that we were used to them we had a lot of quality tools already installed and the integration out of the box is at the moment not complete enough we don't have the proper control that we want so at the moment we are still building our own infrastructure but as I said OpenShift v3 really proves to be open about that and we can very easily integrate it with our existing build pipelines same goes for the automatic image updates so we really like the idea about it unfortunately because we are in a multi-tenant setup we can't use the actual Docker registry because that is not multi-tenant and is still lacking some enterprise features so we are working with an external Docker registry and automatic image updates don't go very well with that one and we also need to have fine-grained control of which triggers are executed at which moment we can't simply have these images we built and now we are going to redistribute it everywhere so the automatic image updates we handle that also externally and again OpenShift v3 is open enough to enable that and of course as with every new product we also see some room for improvement so the network isolation at this moment we were required to do a rather large setup with a lot of nodes to get the network isolation working there also we are working together with Reader to look at how they are fixing that in versions to come so we are really confident that that will improve multi-tenant storage also an issue if we can't handle the storage in a proper multi-tenant way we can't offer the storage because we can't risk that one client is accessing the data from others user namespaces that is one where at this moment if you want to install Docker images that are coming from externally we really have to explain them no you can't be rude because you have to think about the security so we have to do some education sometimes to tweak the images a bit so we are really looking forward to the user namespaces when that comes and that will make our life easier and then the features that we really liked in v2 is the auto eiling and the auto wake up as I mentioned before in v2 that really allowed us to be more efficient in our infrastructure and routers for non-http traffic would also be appreciated so to conclude as the title already said it's a journey to platform as a service so don't consider it an event we install the product and everything is working platform as a service as itself is a journey and it's part of a bigger journey that we're doing in the belgian government the gcloud so in the end that is really a complete solution from housing bare metal service hypervisor infrastructure as a service platform as a service everything will be available and it's not only small but it's working on that initiative but we are one of the partners and in the end we will have complete government clouds that can cater to every need and specifically our infrastructure team is at the moment setting up the infrastructure as a service cloud with open stack and what we see there is really those two are coming together and the line between them is sometimes even starting to blur you can do the same action in the infrastructure as a service with everything that's in open stack or you can do it in open shift again we are gaining more flexibility there which is always a good thing so that's actually maybe a great place to start doing some Q&A here Jeff because there's a lot of questions coming in and I'm actually quite thrilled to see that you're setting up open stack too as we're heading for open stack summit soon so we'll steal you and get you to talk about that story running open shift on open stack in the not too distant future but there are a couple of people if you look in the chat and we'll probably Arresh just asked and I think it's a good way to start is your current open shift environment and he's guessing bare metal but I think you said it was a virtualized environment and he asked the question about open stack which you answered for him but there was that question where is it deployed and a couple of questions about the size of your open shift cluster and the volume of transactions so if you could give them a sense of that that would be probably a good place to start with a Q&A so no we are deployed on virtual machines and the main reason for that is the network isolation so at this moment the network isolation if you want to support traffic that is coming out of open shift and that is going into your legacy network to be able to define from where it is coming you can't mix the containers from different talents so which means that our nodes are created as VMs to get all the isolation per tenant and per environment because we also can't mix the traffic from production and development environments for that so clearly once the network isolation is complete because in the 3.1 there is already a lot of improvement of the network isolation but once the feature is implemented completely yes we are aiming at the bare metal installation but at the moment the VMs are there to handle the network isolation as the size of our cluster at the moment with v2 we have about 50 applications in production and if you take all the environments about it is about 300-400 containers something like that with the open shift v3 clearly we are aiming bigger so the v2 was only for one business unit to get acquainted with it so there we are talking about a couple of thousand containers so we are clearly not Google but we are also not a really small enterprise neither so so there is another question here too and a lot of people ask questions about the migration path there is a question about what was the migration process for going from WebLogic to OpenShift and how painless or painful was that and how is your migration from OpenShift v2 to v3 going so the migration from WebLogic to J-BOSS was quite challenging let's say that because we were changing a lot of things at the same time so because OpenShift was based on Git we did a subversion to Git migration then WebLogic 10 to J-BOSS that really meant that we had to they take a leap it was a change of the Java version of the Java EE version nearly all of our libraries changed so if you change a lot of your libraries a lot of things that simply worked stop working it's standardization on an application server level it's not perfect you see that if you start doing that you have things that work with one specific version of a library maybe it was an error there and now you change it with a newer version I don't really think that it's the migration from WebLogic to J-BOSS if we would have done from one WebLogic version to another we would have had very similar problems the main thing that you need to do if you're going from a traditional infrastructure to containers is you want to have your containers like I said to be self-contained and exactly the same in all environments so we couldn't allow changing the artifacts in the containers image even though in version 2 it was not really an image like it is in Docker but we really stressed from the beginning it should be the artifact that is built in development that goes into production so that means that everything that you still had in the scripter files that you changed manually or via a Maven build you have to externalize that so that you can add it next to the image so that was the main the main mind shift that the developers had to take which honestly was one that we were really happy about because before it was sometimes also complicated to see where is it configured because you had a lot of descriptors and now with everything externalized in the self-contained application all the environment relevant configuration is in one place applications that already did that migration so that have this configuration externalized they migrate really well to OpenShift v3 because you take the base image for the JVOS container and all the configuration you simply store it in a secret and it works the same way so really my feeling is that version 2 was really coming from a traditional infrastructure to containers yes that is taking some developer effort once you are in a containerized system it doesn't really matter what type of container you are in because you already have the data in the proper places all right and there was a Resh asked another question here which I think I am trying to get through some of the quicker questions are you using IPv6? no I didn't think so and Mark who is one of my friends here in the identity management space is asking for some more details on how you are managing access to the projects for instance how do you make sure that only authorized developers and admins have access do your customers have access how are you authenticating them given that your customers government what kind of regulations do you have to deal with from an authentications requirement audit requirements, multi factor authentication can you speak to that at all? yeah so for the developers authentication we don't need two factor and things like that that is mainly used for our applications themselves they clearly have that and in Belgium we have a very nice identity card which is a smart card so we do a lot of integration with that Belgian EID with the development teams because we are an in-house provider for nearly all of them we have trusted network zones circle of trust and things like that so we don't need the two factor authentication at that level it's more who can access and who can make changes to a production environment who can't and so it's more the proper mapping of the roles the access roles that was the challenge then really changing the standard authentication mechanisms offered by OpenShift all right and there is one more question here from Aresh and it probably goes back to what we talked about last week with service catalogs and service linkings or the week before last do you want to build an app catalog with OpenShift for your e-government customers? Yes, but not specifically for OpenShift but the aim is even more to have one service catalog for the complete cloud so that clearly is something that we still need to address at the moment the main focus is on getting these custom built applications on the platform these types of applications they don't really need to be in a catalog it's more the software as a service offering that we see that we want to be able to offer them in the catalog and then directly deploy them either on the infrastructure as a service as an expanded VM or if we can create that as a container directly into OpenShift All right well I think that might be giving them a minute another few seconds to type in more questions in the chat but I am incredibly impressed at the depth of the work that you've done around OpenShift and going from v2 to v3 and I'm looking forward as I said to your journey to OpenStack as well so that'll be another interesting story and hearing about those integrations and I am quite grateful for you to take the time today to be with us Jeff so we hope to have you again soon and all of the things that you are asking about v3 and looking for are all things that are in the Trello boards I've seen every one of those as a topic and are being worked on by various groups of people and so if people want to look at them or comment them they can look at the Trello boards for OpenShift and add to the commentary and the use cases that are there but this was as everyone is saying in the chat a great presentation and we are really thankful for you sharing your insights and I look forward to having more in the future so thanks Jeff Okay thank you for having me and I'll see you soon