 My name is Paul Graham. I'm from Matrix Software, Directive UK Engineering. We're a Silicon Valley-based online charging company, heavily involved in Cloud Native and 5G. It's great to be here. The presentation so far has been excellent, lots of useful information, and I'm pleased to say my presentation exactly lines up with what's already been said. And I feel this is a great way of ending the day because I will be emphasising some of the points that's already been said. So the gender will be the goals we set out when we created our CNFs, the experience, telco requirements, our involvement with the test suite and then lessons learnt. So our original goal when creating the CNFs was we're taking a 4G online charging and putting it into the strange new world of 5G in Cloud Native. That involves implementing the 5G rest day SPA as always already been highlighted, containerising the services, and then moving it to the fully compliant Cloud Native network functions. As we've already heard today, this is challenging, there's issues that come up along the way and that's what I'll be discussing today. So our experience was, first of all, assimilating the existing online charging and I'll show the diagram later of how we took the components, what we know with bare metal, RPM installs and put them into containers and then into Kubernetes to make it 5G. So it's what you know as a 4G monolith on bare metal that the boxes in VMs as discussed earlier. We then involved in multiple trials and installations, so companies like Dish in the US with a 5G, AT&T Mexico, Telstra, TPG in Australia, Big Glow and Swisscom to name but a few. A lot of experience there, so we are a vendor, we have to work what the operators want, so we have to work in all sorts of Google, Cloud, Azure, AWS, OpenShift, we have experience of working everything. And this is a constant learning experience and a constant improvement. And what I've done in this presentation has really highlighted some of the pain points, some of the learning exercises and what we need to do in the future to keep moving along. And I say I particularly like some of the earlier presentations from Joshua and Michael because it really emphasises these points and we're right in there. So where are we as a charging company business? Well, this is the 5G network architectures we all know and seen before, the user plane, control plane and we are what is known as a CHF but it's much more to than that. Working with the SMF, you know money what used to be the packet gateway GGSN and of course everything is now changed but it's kind of the same. What is missing from this diagram is a lot of the legacy components we still have to support. I started my career 28 years ago doing IN and voice fixed line and that evolved to mobile. So this concept of separation of control plane to user plane, I'm familiar with that and there's lots of angles and aspects from the previous generations that still need to be supported in this world. So even though we have 5G core, there's things are missing like the voice support, things like Sigtran and other aspects like charging for messaging and those sort of things. So this is the challenge we have. So when we took the existing online charging system and containerised it, we ended up with this. This is a simplification of course, it's very much complex but this gives an indication how complex the problem is. We're not talking about one CNF, we're talking about taking many parts of the system and having to put it into our own cluster and this is an ongoing activity. To the left we have the networks, we have the support. So even though we're moving into 5G, we still have the support voice and going back a few generations to 3G and circuit and that of course means Sigtran, Cap and Camel. We have a talk to the OSS and BSS systems and it's interesting being here in the charging world where do we sit because we have to work with the network side and we have to work with OSS, BSS and these two domains can have very different requirements in the cloud and cloud natives and a sormon of other systems like billing, payment gateways and the things that typically come with a complete charging and billing system for telco and we do find occasionally, it's an afterthought. Companies build up their, operators build up their 5G core, they get the data working, the voice working and then they have to bring in charging and we speak with all these different types and that's our challenge. As we know, if we've spoken about before, we are in telco so we have a list of these requirements we absolutely must adhere to. We have high availability, we talk about the five lines but that's taken very seriously, no downtime. We have years and years of legacy requirements that each generation we have to pull along with us. Lots of business roles in the OSS systems, a lot of networking, a lot of interworking. This all has to be accommodated as we move to the next generation of telco. It's 5G, it's highly regulated, that's already been mentioned, you know, where we put data, how we treat data in charging, it's monetisation, even more regulation on top of what we already have in telco. So that presents a significant challenge to us as well. Standards based, we have to head here to the standards and sometimes in the other world's web scale, for example, you can invent your own things, you absolutely must adhere to these standards which we have to keep taking forward with us and it straddles IT and networks which have different cloud requirements and we could all sometimes be in deployments where the networks are in one type of public cloud and the IT and OSS systems are in different other public clouds. So this is the challenge we have when dealing with charging systems. I mean, out of interest, who here would say they're in networks? Anyone want to raise your hand? You're in networks, yeah. Anybody else in the IT space want to raise your hand? Yeah, okay, so we have a mixture of people to kind of understand the different domains we're playing in. So approaches to cloud native, again we deal with lots of operators and there are lots of different approaches and never do we see like a big bang approach. Let's go 5G. It's a bit like doing playing some surgery on somebody running a marathon, right? Your operator is moving, they have a, you know, high performance, high transaction network which must be highly available and you need to migrate it into this new world as you go and you cannot do a big bang approach. You've got to take the pieces and move them along, which presents a significant challenge to moving to cloud native. So it's usually a gradual migration and we've seen hybrid approaches where in that diagram you saw that the networks are 5GSPA. Well, that started off in cloud native because the nature of it with the rest interfaces but some of the legacy stuff stays on bare metal in the previous approach. You end up with a kind of a hybrid architecture which is gradually moving to pure cloud native. One approach, another project, build a green field and migrate all the data across. Don't really see that because it's all gradual migration or have one aspect of the solution on bare metal and have another aspect in different domains in cloud native and we see all these approaches across the operators. There's no right or wrong one but they do present a challenge to us as we're trying to move the solution between 4G to 5G and adopt the kind of 5G cloud native approaches. So decisions that have to be made, how quickly do you move to cloud? You know, we're at KubeCon cloud native. We want to move that straight away but we know we're in telco. It does take time to move there. Why are we going to cloud in the first place? Well, we know in telecoms innovation can be slow so we want to be able to adopt the kind of the CRCD approaches, rapid upgrades, not those six month year installation projects. As somebody said earlier, do the upgrades, install it, leave it there for a couple of years, come back again. The reason we're doing this is we want to move to a more rapid upgrade approach and so we can be introduced more innovation into the services quickly and things like data. Where do we put the data on prem and on cloud? Again, these are things that prevent us moving as fast as we would want to more quickly. So, as I said, we've got challenges on the telco side of things, the traditional telco. I don't know how many people are here are familiar with cat protocol with Sigtran. SCTP has been mentioned earlier on and the issues around that. This is a big issue in the online charging system. We'd like to think we're just dealing with data and everybody's going to move the WhatsApp for voice calls and these sort of things. That isn't going to happen. Across the world and all the operators, most people are still on voice. A big number in telco is still in emerging markets where they support voice and these protocol and the traditional aspect still have to be supported but in cloud native. SCTP in Kubernetes is getting there but we can't say it's in every variant in every public cloud that we come across. So, this proposes a significant challenge but say that's already been mentioned today. It's been supported as service in 120 and it's getting more mature but as we mentioned we've got the multi-homing and multi-pathing issues with SCTP that have to be supported and things like multiple CNIs in containers using tools like Mortis are being brought in but they're still not there yet. So, in my domain of moving traditional telco this is one of the biggest challenges I see at the moment and we still have to kind of go forward cautiously, look at every use case and see how we're going to implement it and say it's work that still has to be done. Does anybody come across this problem themselves at the moment? SCTP and yeah okay yeah yeah so we know all about it yeah yeah yeah. This is the work we have to be doing right? We have to be as a telco group in cloud native in Kubernetes. I think we have to be championing this ourselves. Make sure this gets done. The other big issue in telcoms is state and this is where the pure cloud native people start chucking things at me because you know we want everything to be stateless but realistically when you step back into the legacy world of telco there's a lot of state in there and that point's already been highlighted I think by Michael this morning that you know if you have a a system with lots of components microservices like I put up earlier and you want to upgrade one then you've got these dependencies because you have state throughout the system. If you're doing voice calls voice calls have state the voice calls are sorted with a charging session that's two sessions you have in the system the voice cap the charging reservations that you typically have and then you still have the state in the system. This needs addressing what do we do with it where do we store it and yeah it's a big challenge that we need to sort out in cloud native but I think we're sort of working on it. It will hinder our advancement but yeah it's work that has to be done. Now an interesting thing we did we did develop a pure cnf so this is this is kind of like a brand new component that was developed as a pure cnf and that ironically it's a charging bridge to help us migrate from the old world into the new world so as I said typically when you build your 5G core sometimes you find components left behind and this can be the OCS so you have a 4G OCS which still supports diameter and then suddenly it's not compatible with your 5G core so a product we developed was a what we call a charging bridge made of a couple microservices you have one microservice acting as the SBA rest interface into the 5G core you then have a diameter client up into your OCS but interestingly this was a good way of building a pure cnf and then being able to engage with a conformancy suite which we'll talk about a bit later and it's already been discussed today and here at matrix we've played a big part in the the conformancy suite because we had a pure cnf we owe to start from so where does the charging bridge sit in the picture well based on my previous diagram you have the 5G core you then have your legacy OCS in the 4G domain that would typically speak well they always speak diameter but you want your diameter speaking OCS to communicate with your 5G smf so you need a function in the bridge and I'm certain this probably isn't the only bridging function you're going to need as you migrate from 4G environments into 5G but it's a good example but what it does mean is you can build something that is purely cloud native it doesn't have states it's it's maintaining between the smf and 5G and the diameter based OCS and in its simplest form it looks like this so you have your 5G sms function talking your n40 smf interface in the SBA it goes through whatever your service mesh into diameter clients talking to your OCS and this was able to be developed as a pure cnf which was able to be compliant with test suits and we use this as a way of taking the cloud native telco test suits running the tests against it and working with the team to develop the conformancy test suits so render with the conformancy test suits so for you not familiar with it it's a project that's been running it's a set of auto testing scripts and it adopts all the principles of cloud native function so it's a demonstrate conformancy implementation of best practices it does mean if somebody wants to develop a telco cloud native function they can get this test suite out of github and they've got kind of a something to build against it gives them a sort of a framework to try it so anybody out there that is working on a cloud native cnf try running these tests against it you know it checks your helm charts persistence storage liveness probes those sort of things and you can build it into your pipeline and this has been quite a fruitful exercise because it means you've got a kind of reference point you can work between organisations and it's a point where you can agree on how these cns are going to look like and if you have an agreement on what they're going to look like then you can have a way of putting them into your pipelines and a standard structure of using them so it becomes quite effective so what lessons have we learned well the first one is this is an evolutionary process there is no big bang to 5g cloud native we have so much legacy in there that we have to kind of keep evolving and you will end up with a kind of frankenstein looking installation where you will have some components as cloud native functions some maybe sit on bare metal as you pour them through particularly in the kind of the signaling side of the networking sctp and as sort of Kubernetes and the ecosystem catches up eventually you'll be able to replace everything and yeah we're waiting for Kubernetes to approach maturity for telco and that's us we're working on that and we're pulling it forward and keep working on it another one it's already been mentioned is the logging observability and telemetry and uh yeah we are dealing with very complex systems and we want to better see what's going on in them like a typical voice call you've got a cat protocol coming in it then sends you've got a kind of a go to the charging things are set up you've got rating going on you want to but us see that end to end and in telecoms that's a key thing telco has been very good at building systems that have good observability all the way through and we need that kind of similar set up going forward in cloud native so it's something else we need to continue working on we're going to find that there's components that we're going to need to develop ourselves as part of the open source community for telco I think today we've probably identified them when you've stopped building something for real and you hit a gap or something you think well this isn't quite working as we want to work we're going to have to work on developing these things and pushing the community to to build them for us or work together to build them and uh yeah test automation again that's already been mentioned in your pipelines I think is increased it's key to getting that cadence you know we're moving to 5G for a reason that reason is we want to put things in the cloud we want to increase the rate that we can do upgrades installations and that enables quicker innovation and coming back to that sort of six months installation upgrade project it takes so much effort we then have to leave a couple years come back to it that hinders innovation we want something where we have a CI CD pipeline the components are rapidly changing when we have those bugs we want fixing we want a bit of fixing quickly and that will move the industry along quickly with innovation so yes building common sets of tests that work across all the cnfs will increase cadence as we're moving along so what's the road ahead well yeah we've got to fill these gaps required by traditional telco if we're going to move everything into cloud native right um things like the protocols sctp that's a bit of an anchor for us at the moment it's dragging us we need the support of them but it's going to stop us going pure cloud native we need to gain trust to move to serious cd models right i guess we're a mix of operators and vendors we've got to develop trust where we can have that rubber upgrades where the cns get pushed out go into automatic testing and then get deployed you know we've got all work together to develop that system so we do get that increased rate of of change and upgrades which is you know the biggest problem in the industry how long is it going to take to get us to 5g at the current rate because all the work that has to be done and the final point really yeah enhance the test suite further and increase adoption because if we have a common reference point a common test a set of tests that we can use to bring into our environments then that will increase the rates at which things happen so plenty of work to be done and uh yeah let's make it happen it's collaboration that's why we're here we share the ideas today lots of good ideas and lots of areas where we need to be doing more work so any questions oh yeah thanks for the talk oh yeah very interesting um i have a question regarding you mentioned like voice for example and how that's being kind of cannibalised from you know vibe or whatsapp yeah like even and um charging is one of those components that sits very close to the product yeah and then also you mentioned there in the end regarding the how do we you know get that by and how do we accelerate the adoption of this cloud technology i mean putting all that together do you not see one of the missing components here is that the business needs to see that the end customer isn't just the consumer with the sim cards the plans and everything that is actually like the public cloud is the developer and that that's servicing into over the top offerings like whatsapp if you could sell in an ims solution that you could build in carrier grade voice in these over the top offerings and then from a charging perspective you can you know model those technical products towards developers and that's only available because or enables the operator because if they're built on cloud native technology that they can actually do this you know do you not think that there's something there yeah i i do absolutely like no yeah because the whole point of cloud native it's not just about building a 5g core it's having a cloud that you can put other things into and once you've got into you build up the trust and you increase the performance and reduce latency right because everything's banged together um telco doesn't like anybody just going into their networks and changing things yeah and there's a there's quite significant dmz between the telcos and their charging systems and they're over the top and at the moment it's a gap this big yeah i mean and yes we we're an online charging absolutely right we want to bring in the um the the whatsapps of the netflix is and be able to charge combined into your um telco plan yeah at the moment yeah telco is just charging um data it should be in that micro payment space working with these over the top and i completely agree with you um because you know telco keep their their their stuff in their um the silos in their bunkers it's their machines you've been nowhere you can't get anywhere near it right a third party cannot get the the apis into that telco space yeah this however a cloud native i mean there is a um network exposure element in the 5g core that can be used to open up to third parties like whatsapp and over over the top and we can do that with charging as well yeah absolutely that what we're doing right in this room yeah it shouldn't be just telco people it should be those guides as well yeah because we need them on our side you know oh we do otherwise they still are lunch yeah yeah yeah because i mean they've got no more let's say horses anymore beyond the sim card the piece of plastic the plan and everything yeah if they know that you can enable developers through building on cloud native technologies and that becomes an offering a technical offering then they would pull the network people into saying hey we need this you know and and so the monetization i'm talking about you know i'm i understand yeah yeah yeah i totally totally agree monetization of telco and monetization of everything else is too separate yeah because the telcos don't give access to these other third parties to use those monetization systems yeah right now what you want is a kind of apple relationship right what apple provides the third parties with the charging platform yeah and uh telco should do that too certainly yeah but but anyways you know just to wrap up like i'd love if there's some critical conversation with you know the group that cncf has kind of developed in this cloud native thing because what's missing there is the developer and the developer that has that relationship with the consumer you know which is kind of eaten away from the operators because right now operators just have infrastructure in the brand and that's it yeah they're just the pipes which are getting cannibalised but are not monetizing on top of that cannibalization you know and i'm wondering how we can service back into them with the two i do know what i think the answer is okay i mean i've been in telco 28 years right and i've been in research in r&d and everybody comes up with these services but nobody thinks how do i monetize that service yes if we had monetization at the beginning of the process and give the developer the access to the monetization platform while they build the service that would happen yeah yeah true but we don't have the we just don't have the structure or the apis to be able to do that yeah as you say it's not in the developer mindset yes if somebody goes out and builds the next whatsapp they're not thinking how do i charge for this from day one exactly uh i mean you can ask all yourselves a question anyone here from the gsm a anyone so we're not making an impact and that's what i just wanted to do so that's the thing if we enable the gsm a for example to take ownership on this then we'll be winning the kind of let's say the telco onboarding on this you know because uh so but yeah thanks very much for your open i don't want to longer but if anyone wants to continue the conversation more than happy to share my views on it and yeah well there's something we should be considering right absolutely yeah i mean i think the dream of putting this in the cloud absolutely is so you can have everybody else building on top of it yeah but we have to build it first and make it available yes um i'm not sure yet in that third party app development space who's even aware of 5g the 5g core and what's available to them yeah um it's us and telco with our 5g core it's those people over the top building their stuff yeah right we want to combine both yeah okay cool thank you thank you