 All right, we're going to get started again here. Thank you all for your patience with our process and everybody on Facebook. We're really pleased to have with us my Canadian Telco Telus Digital here to tell us their case study. We're mixing it up a little bit and changing the tone. And then we'll get back into upstream stuff. So without further ado, the folks from Telus Digital. So hi there. We're thrilled to be here with you all today at OpenShift Commons. Our topic for today is how we're staying ahead of our avalanche, how we're surviving our digital transformation at Telus. So my name is Phil Dufault, and I'm the product owner for our delivery team at Telus Digital. And with me are my two teammates that are crucial in our journey, Tom and Alex. Hey, folks, I'm Tom. I'm the cloud platform architect at Telus Digital. And more or less, I make websites. And hey, guys, I'm Alex. I'm a tech lead at Telus Digital, and I spend my day automating things. So I'd like to start us off with some context about Telus since we're down in the States. Telus is one of the largest Canadian telecommunications companies. We've got 48,000 employees and lots more vendors and contractors. We're also into the TV and health markets as well as the telecommunications space. And if you look into our history of mergers and acquisitions, you'll find actually Telus is more than 100 years old. So for our enterprise IT, this may sound a little familiar to some of you. We run our own data centers. We've got tons of bare metal. We've got IBM mainframes. We've got systems of record from the 80s and even 90s. Legacy Java apps, maybe even Coleball hidden away in the back, waterfall business practices. And we have some definite challenges with config and release management. And this ultimately all results in some long release cycles. Now, what about Telus Digital in particular? Telus Digital has created us a disruptor as a startup within our enterprise. Our intention is to push the envelope with digital and mobile first enterprise or experiences. And we value outcomes over outputs, quality over speed, data over opinions, and empowerment over governance. So where did we start? Our first solid attempt at digital transformation was Telus My Account. And the idea with that was to enable customers to check their bills, usage, and call history, et cetera. So true self-serve web experience. It was our first stab at agile and scrum practices. As well, we dipped our toes into the cloud using DevOps practices, infrastructure as a service, Ansible provisioning for our infra. And it was our first mobile first experience. So this was delivered just in time for the explosion of smartphone growth. So at the time of launching, had 15% smartphone penetration. Now it's over 75% of our traffic is coming from smartphones. It was also a highly accessible site. So we are WCAG 2AA compliant. This basically means it's highly accessible and fantastic for those with visual impairment, works with screen readers, and everything else. We offloaded a ton of call center volume, which saved us buckets of money, and thrusted us to have the highest customer satisfaction of the Big Three Canadian telcos. However, at the time, writing code on this platform was a quixotic experience. It never felt like we were done. It never felt like we were doing anything that was that great. It was built on a proprietary Telus Digital PHP framework, which was an extension of Code Igniter, which at the time was already somewhat obsolete. And the architecture was dependent on lots of secret lore, which was passed verbally between the development teams. And that was also difficult to keep aligned because we are split between half of our development team being in Vancouver and half in Toronto, three hours apart. So architecture was also full of lots of weird sacred cows and lots of shortcuts and tech debt that rarely got paid off. So with lots of different tech stacks and reinventing the wheel, this was really difficult for new users to get onboarded into our platform, often taking weeks to get started in development. And overall it was just a very confusing development user experience. And then in terms of releases, we had no automated build system. Integration typically happened very late in the game. We had Gitflow, lots of feature branches and to compound the issue, we had proprietary bash and Python scripts that would pull together some 50 odd GitHub repos into several monoliths to be deployed. The releases often took a week to coordinate. And while this was better than our former quarterly releases, there's still tons of room for improvement. And this was largely due to there being a lot of manual testing, several days of it for every single release. We do a full manual regression test. Our automation tests that we did have were very bloated as well. They took a number of hours to run. And there was very poor uptake on unit testing. So if you've ever heard of the testing pyramid, we had the testing ice cream cone anti-pattern, lots of manual tests and very few unit tests. So one broken commit in the whole release would grind the whole thing to a halt. You have hundreds and thousands of commits, sometimes just parked. And in summary, that meant for unhappy devs. The tight coupling in our framework meant that there was friction between the teams. And the monolithic architecture and weekly releases was not really conducive to having autonomous data-driven teams. So we finally realized our architecture had seen better days and decided to make a massive leap of faith in order to move on to new and better things. So we had to start re-imagining our team structure in order to build a culture of architecture. And so what we did was the Singletal's digital development team was split up into four different outcome teams. These outcome teams are mobility, my account, home solutions, and business. And so we also had enablement teams. And these enablement teams were there to help enable the outcome teams to ship software more efficiently and more productive. And that's us, which is delivery. We have design, we have API, we have content, analytics, security, and many more. So institutionally, we recognized that we needed a good day-to-day user experience for our technologists. And happy developers means wonderful experiences which creates happy customers. So we decided to listen to our engineers because they had some excellent feedback for us. And as a thought experiment, we mapped our path to production. We wrote down all the things we needed to do to bring an idea from concept to production. We brought together leadership and representatives from all different practices. And we found countless opportunities for optimization. And so this kickstarted a dialogue about possible architectural solutions to our problems. And out came the reference architecture. And basically what this is is we found it really important that to have organization alignment. So we wanted to make it easier for developers to move between projects. And we wanted to be easier for developers to share their experiences among each other. And we wanted to support a large amount of applications. We're a telecom, there's a lot of microsites and other things. So the operational component was very important for us. And how did we get there? With a lot of experimentation. So we have a culture where we were able to fail fast and pivot. So we played around with a lot of different tech stacks. We played around with PHP, Ruby, Java. Ultimately we ended up on JavaScript. We really wanted that cutting edge web and development experience. And also on the infrastructure side, we played around with things like Ansible, Terraform, we managed our own Kubernetes cluster. Ultimately we decided OpenShift dedicated was the best for our use case. We did originally do infrastructure as a service. It was a lot of work and we only had a few AWS experts in house. And we decided that platform as a service was more what we needed. So that's why we ended up selecting OpenShift dedicated because it allowed us to outsource our server operations so we could focus on shipping experiences to our customers and focusing on tackling some of our automation bottlenecks. And so because we love our customers and our technology, we are evolving it with full transparency. And as you can see, we have a public wiki. You can see our reference architecture, it's public. Anyone can give us feedback. Anyone can contribute to it. And it's just called reference-architecture under our Telus Digital Organization GitHub. So to grow our architecture, all the enablement teams worked together to build a shared digital platform. So after we created the reference architecture, we set many teams loose on it and they started building it from the ground up. But we had asserted so many opinions on so many tools and so many paradigms that essentially it was like herding cats. And we ended up with lots of duplication of effort, varying code quality and non-uniformity in our apps. So to focus the teams on building cohesive on-brand experiences rather than lots of boilerplate, we built the digital platform. So the Telus Design System is a suite of common React components that give a consistent look and feel across all of our applications. The analytics layer gives an end-to-end strategy for assessing the actual needs of the consumer. The content platform allows developers and marketers alike to update content without having to redeploy the application. Our API platform is a microservices network which brings discovery and authentication to the table. And finally, delivery, our team was responsible for making it fast, easy, and most importantly, fun to build and deploy Telus software. So there's many technologies that play in our delivery platform. We use Terraform to manage our classical AWS infrastructure. We still use things on AWS like RDS for our databases. GitHub stores our source code. Pretty self-explanatory. Vault stores our, HashiCorp Vault stores our secrets and our access control lists. Jenkins runs our delivery pipelines. And of course, OpenShift runs our apps and our builds. So to truly expedite the development on the digital platform, we needed a platform for application architecture as well. And this is where our starter kits come in. So what are starter kits? They leverage the two strongest tools in the modern developer's portfolio. Copy, paste, and find and replace. So show of hands, who has used Stack Overflow? Come on, there you go. So you all know this. So starter kits are a reference implementation of the reference architecture. So our architecture is now enshrined as code. They're designed as loosely coupled, self-deploying, continuous integration and continuous delivery automaton such that none of the teams get interrupted by another team's deployment. Their code is fully self-shipping. And we attempted to not fall into the framework trap. So we had bad experiences building our own custom frameworks before. So the starter kits are a collection of managed code duplication, which is mostly configuration. The frameworks themselves are React, OpenShift, et cetera, made by people much smarter than us. And so the starter kits are a functional, ephemeral, declarative, item potent, translation, cloud native, configuration boilerplate that glues together all of these frameworks. So any proprietary shared functionality is managed by individual libraries for each specific context. So for example, our accessibility and SEO testing is all done with Telus proprietary libraries, some of those open sourced. So our backbone starter kits, which account for about 95% of our OpenShift cluster currently is the server-side rendered React user interface, as well as the API microservice starter kit. So in order to achieve ludicrously quick deployment times, we had to set goals. So these were set at the outset of our delivery team kickoff and our goals were to have under 10 minutes from commit to production, as well as to onboard new developers on their first day and have them also pushed to production, something we had seen from a lot of bleeding-edged companies like Facebook. They also follow the KISS principle. This means keep it simple, stupid. But basically, no lore, just open-source software which has a large backing community and lots of free support and common sense for our developers. So all the starter kits, as well, revolve around their build pipelines. So the various build pipeline steps, we have the checkout phase. So every single commit triggers a checkout of your code for that commit. We apply the secrets stored in Hashicorp Vault and mount them to OpenShift for being read by the apps. We apply the OpenShift templates for both build and deployment and this allows us to couple our code together with our infrastructure into the same commit such that we can make changes in parallel and test them as one whole. The build phase builds all of the relevant containers to the various life cycle steps of our applications. Some of them are just for testing purposes. Others are for full runtime and allows us to couple our application dependencies with the code base. The testing phase is, of course, the most important. The test, the quality of your testing is basically what gives you the confidence in your pipeline. So if you have poor testing, you have low confidence and you can't ship quickly. So we obviously wanted to have absolutely the best testing we could possibly get. So we have everything from linting, unit testing, code quality measurement with SonarCube, node package security scans, performance tests, basically you name it. The deployment phase is basically, as you would expect with Kubernetes. We've got auto scaling and rollbacks and all of that jazz. So for every deployment, we also run an end-to-end test and this is not just for functionality but also we test for accessibility, SEO. And finally, we also have contract testing, load testing, device testing, et cetera. So an analogy that we like to use for our deployments, it's like bowling with bumpers. You keep safely rolling the balls down the lane until you knock down all the pins. But our game doesn't just stop once you knock down the pins and get a strike. We also have instrumentation for runtime. So for SEO optimization, we have server-side rendering. We also support logging with our Kibana stack. We have New Relic for monitoring. We have pager duty and incident management teams on call. And analytics, feature flagging, security, time series metrics, really the world is your oyster. So all of this comes out of the box. You copy, paste your Fort Knox grade Hello World and this means you flip the paradigm, you're no longer getting your production instance on the last day right before you ship. You start on day one with a production instance and you make small incremental commits and leverage the CI CD pipeline to test and deploy every single change to production. So exposing the site to customers is as simple as toggling the feature. You just show the site, embed some links and away you go. So we're easily able to get an application from commit to production in under 10 minutes but how do we get users engaged in our platform? So as Tom mentioned, the starter kits have enabled our reference architecture and digital platform to scale out very quickly. But there are some other organizational bottlenecks that we needed to tackle as well. Onboarding for us at TELUS has traditionally been a slow and very painful process. We have some automation now to onboard people onto our OpenShift clusters and additionally the HashiCorp fault side. We call that Chippy. We distilled a TELUS specific domain model for our applications and the users or squads that are building those. And when our users are added in, we automatically provision them onto the various clusters and tools. So we have a CLI and an API right now to manage this onboarding and offboarding of users and it also assists in the deploying of the starter kits onto our OpenShift clusters. So it's a simplified single interface to our various platform tools and it helps convey our architecture, our culture and our documentation. Beyond simply onboarding and delivering our code, we also need to get all of our technologists to contribute. We found that we really needed to respect our technologists and their craft. We had great opinions and we needed to listen to them. We found that architecture really mattered and our architecture is continuously improving because now we're never satisfied. So to help with that, we now run inclusive guild meetings for all of our respective disciplines. So we have design, delivery, API, architecture, testing, security and content. And for each of these practice meetings, we keep decision records so that we record the challenges of then and as we move forward and evolve, we can challenge those assertions so that we can continue to grow. We're also updating these starter kits with these new and evolving standards and these updates are now coming from the outcome teams themselves using an inner-source pull request model. And these outcome teams are great canaries for us and now they're generally doing the trailblazing for us. So these bets that are paying off for those outcome teams can get merged back into the starter kits and be leveraged by the whole. So how are we keeping the momentum going? A crucial turning point for us was just getting the leadership team to buy into our reference architecture. And then we received updated mandates with some success to align beyond Tellus Digital into Tellus, Kudo and our public mobile brands to the reference architecture and the culture associated. Some other learnings we had was that our hero culture ultimately had to stop that our key individuals weren't scalable, that our DevOps stopped being a team and it became a cultural practice and that the developers are the subject matter experts for their applications and as a result they need to be on call for that so they can triage and escalate as needed. And those outcome teams are now graded and measured on their application experiences and uptime. And our OpenShift dedicated cluster is responsible for keeping the stability of our platform at large and it's doing a really great job for us. So now we wanna talk about the juicy part, the growth and the results. So we've had 120 applications deployed on our platform since March. We have now 400 plus deployments per day. In the old world we were doing about one deployment per week and this is a huge, huge upgrade for us and absolutely incredible that a telecom is able to ship 400 times a day. And the developers at Tellus Digital are super proud of this because the platform really helped us achieve this. Our cluster size is doubling every few months. We're just getting a lot of people from other brands and teams that really wanna join our cluster. We have happy customers, we're able to ship updates all the time very quickly throughout the day. And we also have happy developers because they don't have to focus on the operational aspect of their application anymore, it just deploys and runs. So a bit of a case study we wanna talk about the iPhone launch. So recently on the platform we launched the iPhone 8 and the iPhone X and in the old world there was a lot of fire drills, a lot of stress, people were up all night. This is a huge release for Tellus. Website problems, just a lot of stress. And in today's world with the new platform this is all gone, there's no stress, no one was being paged, everything just released. We actually released a minute after midnight. We're able to start accepting our pre-orders for the iPhone, which is absolutely incredible. And we really wanna replicate this major release for all other major releases. So as you can see, we've received some great feedback from our customers on Twitter about how they're able to pre-order an iPhone and it was a very smooth experience and we're very humbled by this and we know we're doing the right things for shipping great experiences to our customers. So our success has not gone unnoticed. And now we're embarking on a most critical and crucial journey and that's expanding the platform and the culture to the rest of Tellus. As Phil said, Tellus is 40,000 plus people. And we have other teams already, other teams and brands already on our platform and we've received a lot of great feedback already from other teams. And I think now what we need to do is we really need to expand our reference architecture to support other technology stacks because there are teams that are more familiar with Java and Ruby and we really want to receive their contribution to our reference architecture so we can expand our reference architecture and make it better for everyone at Tellus as a whole. So what's next for us at Tellus? Well, the future is friendly. So our journey over the last year has been incredible. Ultimately, if you look at where we were and where we are now, we've coalesced a lot of our disparate technology stacks. We've defined a single reference architecture and turned it into a digital platform with tons of reuse. And with all of that uptake, our projects are actually getting to the market quicker. We're getting about a third of the time to build those experiences and deliver. And we're not gonna stop until our applications are running themselves. Paying it forward, true to the Tellus core value of giving where we live, we're actively participating in the Node.js and the open source communities and our outreach now is a beacon for hiring. On the new stack, many of the best and brightest technologists are actively seeking us out as word of our tech stats are spreading. So there's many pathways for us to venture down. The sky's the limit. Big data and AI to process the deluge of data that we're generating seems like a natural progression. Supporting alternative deployment paradigms such as functions as a service. What's in the shipy future? I definitely think a React UI, so a web experience, slackbots for sure, and maybe even new connected interface. Okay, shipy, create me a new app. So some key lessons that we want you to be able to take away. We weren't just building a platform, we were really building the cultural movement. As technologists, we can get really obsessed into the details and forget about all the people involved and exercises like mapping the path to production were really, really valuable in sharing that understanding amongst ourselves. Establishing the culture of architecture and enablement. There can be no enablement without the architecture to support it. Anything less is gonna result in anarchy and instability and no architecture can succeed without the enablement side. Anything less is an ivory tower. So codify your standards, automate all the things. This is about expediting your operations architecture. We're focusing on minimizing the time from ideation to our customers. So perhaps now you can understand a bit more about us at Tell Us Digital and how we're tackling our mission of enabling Tell Us team members and customers to do what they wanna do easily. Thank you very much for your time.