 Okay, so it's 415. Let's get started with the introductions. My name is Mohsen Ahmed. I'm a senior systems engineer In the cloud engineering team at Ford Motor Company Hello, I'm Shaji Thomas. I'm team lead for PCF operations at Ford Motor Company Hi for those of you that don't know me. I'm Hayden Ryan. I'm an advisory solutions architect for Pivotal I work in the customer delivery team and have been out in Dibble and working with these guys since well October I'm Dave Waller F and I work with Hayden. All right So let's get into the presentation So it's an exciting time to be working in the automotive industry There's a man immense activity going on whether it's in the field of connected cars Or it's in the field of self-driving autonomous vehicles Ford Motor Company is at the forefront of research and innovation in all these fields Continuously working on making products and services that people want and value Going further on making people's lives better Ford recently announced its transformation plan with the creation of Ford smart mobility a fully-owned subsidiary To expand from an automotive to both an auto and a mobility company This strategy will allow Ford to stay focused on Strengthening and focusing on its core business, which is Manufacturing cars trucks and utilities and at the same time aggressively pursuing emerging opportunities Through Ford smart mobility to transform the customer experience to a new dimension Transforming Ford's customer experience is core to our strategy The launch of Ford pass is a part of Ford smart mobility initiative And the beginning of Ford's transition into auto and mobility company Ford had teamed up with pivotal to deliver Ford pass that we launched earlier this month over cloud foundry in Azure public cloud The popularity of Uber and Lyft clearly demonstrates the shift in consumer preferences But they are moving towards mobility solutions as opposed to auto ownership The automotive landscape is changing and Ford's initiative with the launch of Ford pass With will boost the company's growth in future Ford pass is not just a cloud native application Rather, it's a platform of integrating digital physical and personal experiences With four main parts, which is going to change and transform the overall customer experience Ford marketplace for mobility solutions Ford guides who are always there to help and assist Ford appreciation where members are rewarded for their loyalty and consuming the services and Ford hubs where consumers can experience and explore our latest innovations It's available both on Android and iOS platforms and is open to everyone whether you own a Ford or not Just by registering online We believe that great experiences build long-term relationships from a business business standpoint It will drive better loyalty bringing new consumers and Accelerate Ford in becoming a serious player and a leader in the mobility services With the launch of Ford pass the journey has just started towards making for an auto and a mobility company Next I like to share a short video That would show the future set of Ford pass and give you an insight of the platform Driven by a passion to make people's lives better. Ford is going further to change how the world moves To do this, we've created a new product platform Ford pass an ecosystem of digital physical and personal solutions That empower you with more mobility options so you can go further The digital engine that connects to every feature of the ecosystem is the Ford pass app It lets you book a service appointment with your local dealer with just a few clicks Or borrow the vehicle you need when you need it Or find and secure parking spaces in a busy city and Gain access to the parking garage using only your phone When your car is done being serviced the Ford pass app will let you know it's ready for pickup and provide your service records virtually Integral to everything is your supportive Ford guide who can solve or facilitate any mobility need you might have initial Ford hubs in New York London Shanghai and San Francisco invite you to see Shape and experience what the transportation of tomorrow could be Imagine Ford pass guides and hubs all working together with cities to provide remarkable mobility solutions and Best of all we can now show our appreciation in a few ways with Ford pass points an Opportunity for us to say thanks for your loyalty with points. You can earn and redeem over time and with Ford pass perks Benefits from affinity partners like McDonald's 7-eleven or Spotify that can help make any journey more enjoyable behind the scenes We've built a new way of working together to create this Ford pass ecosystem to give our customers of today and tomorrow What they need to go further than they ever thought possible Impressive, huh? So Ford pass is all hosted on cloud foundry. Thank you So that was just the glimpse of the Ford pass and it would continuously grow and evolve There are more features being added like look waiting your vehicle lock unlock and remote start of your cars right from a stock smartphones the scope of rest of the presentation is around the infrastructure design and Development and implementation of cloud foundry on the Azure public cloud that provided a platform to host for pass Next my colleague Shaji Thomas will walk us through the plan build and operate model And how we transform the IT side to achieve our goal of launching for pass on cloud foundry Shaji Thanks Mohsen So how was Ford company a traditional company able to develop a Ford pass app using the PCF infrastructure? This is much different than how we traditionally do it We're a very process-driven company with lots of meetings to go through and Processes to follow The old way we would have the app team would come to our solutions customer solutions team And they would work with our global fulfillment planning team who would work with our global fulfillment implementation team and Then they would end up working with our engineering team who would give us the final Checklist to do a deployment But we wanted to move to the zero time zero downtime upgrades And to be able to deploy within minutes So we started moving to more of self-service tools to give Our cut our end customers meaning the app teams More self-service tools like the APM manager And they were able now they're able to do their CF push themselves as opposed to involving our team the Operations team they can roll their changes from dev QA to prod So Ford motor company was very risk adverse Failure at a automotive company is can mean bankruptcy it can you write some bad code and 15 years later can come back and Through the NTSB they have recalls. Why is that emitting so many so much emissions? And you have to explain some code bad code that you rolled 15 20 years ago So and then you have to go through a whole recall process. It's a very time-consuming and painful process So as I said it we use the plan-built operating model So planning and architecture will build an idea then engineering will make sure it works And then they hand over a checklist to us to To fulfill to build it this produces a high quality final product so What are the processes we had to work through was how do we coordinate three companies working together? So traditionally before we had an incident manager that would coordinate all the different teams and If we had outage they would coordinate the global cloud operations teams with the PCF operations team And they would with the incident manager would interface with met upper management and explain when what's happening But now we have Microsoft Azure. We have pivotal. We have our team and all our sub teams so now we Integrated that with our with what the incident manager team that we have and we integrate that with the critical situation team that Microsoft has and then the on-site engineers that pivotal has so So how do we move forward to this new agile environment so we had to go through and break our normal operations We traditionally if we had any scheduled operations activity had to be done During our maintenance window, which is usually 3 a.m. In the morning to 11 a.m. on Sundays and To do an upgrade during that time We would have to fill out what would we what's called a geek or a global infrastructure change requests all global infrastructure change requests Have to be filled out 10 days in advance So there's a waiting period and then in those 10 days it gives any manager or an app team Opportunity to reject the change and push it off So if we had a patch it would have to wait until it gets through the whole system They're also have we also have these traditional Freezes they have the Christmas freeze. We have an audit show freeze. We have a quarterly freeze for doing the financials But now with the with using CF. We're using the IT advisory method. So basically you sign up for a Bulkmail list and when we do a deployment or an upgrade we just sent an IT advisory out that the system is getting upgraded dramatically reducing the time that it goes It takes to go through an upgrade So on this slide I wanted to talk a little bit of how Pivotal what Pivotal gave us and how we made it work it for so we had to kind of blend the two Two approaches so with monitoring and alerting Pivotal was successfully using data dog And they were starting to use the Elk tile for monitoring But at Ford we had a lot of experience using Splunk so we were able to create dashboards using Splunk model after what Pivotal was Doing using data dog. We put syslog relay servers and heavy forwarders in the cloud both on East and West and Azure for production and pre-production And then we would move the jmx data back to Fort the Ford dashboards and Fort's network is closed off. It's a private, you know from the internet So we had to put another syslog relay server and heavy forwarder in our DMZ to forward that data back to our dashboards The next thing that we had to blend together was the jump servers the template that was provided to us from From Microsoft was using a bunto but at Ford were a sledge shop So we had to go through and and rework it So We have a ton of tools that use sledge the part our password vaulting tool which would go in and automatically change Password all these things so we needed to get a sledge tool So we're currently working with our Linux engineering team to come up with a Ford OS version for the sledge in the cloud We're also trying to recycle our processes for incident management requests requests and then problem management So now we're moving on to our global launch Which will soon be deploying into China on the 21 binet cloud as well as Europe and Asian Pacific So I'm gonna talk briefly about what we chose to deploy at at Ford We deployed PCF one five at the time one six was out But one five was a little more stable and going forward at the very bleeding edge as we are on Azure We want to give them something a little more stable We deployed rabbit and my sequel to support the spring cloud services for services discovery and config management We deployed the Gemfire PCF tile for data sync and op-symetrics for the JMX endpoints One of the first major discussions that we had was which load balancer to use Ford uses f5s on-prem. They have very good knowledge with it They're very used to doing it But they wanted to try something a little different when they moved to Azure Obviously, we could have gone HAProxy tried and true, but you can't get Bosch managed HAProxy as your edge device without with zero downtime So in the end we decided to go with the Azure Primitives in particular the Azure load balancer as our edge device as you can see here So the Azure load balancer sits on the edge and forwards traffic to HAProxy Which are then Bosch managed to give us that Bosch goodness where we terminate SSL and forward traffic into the go-riders So this is the rough diagram of what a single foundation looks like Because then we had the next apology question of single or dual deployment Obviously, we wanted to go as many deployments as possible for HA, but that leads to the questions Ford has the vehicle SDN, which is one of their products that a lot of the apps are gonna need to integrate into That's only available in certain regions. This influenced our design as to where we're going to deploy Also, there were certain latency requirements that we had to take into accounts We decided to go at the end of the day with the active active east-west deployments again Utilizing the Azure primitives of the traffic manager for the GTLM DNS That points to the Azure load balancers feeding traffic and and using the gemfire WAN replication to handle data sync between the regions The active active allows for many good things including the maintenance We can take down one side and still have full traffic on the other side for the troubleshooting maintenance that needs to happen Downtime does happen. This is the real world things happen. So it's good to have both sides active should one of those things fall down Hey, yeah, thanks Okay, so Let's let's jump into a little bit of a primer around some of the best practices lessons learned And an advice we would give to people that want to deploy to Azure We're gonna cover a couple mini topics here. I'm gonna keep this recently brief John Sermon from Microsoft and I have a session tomorrow Which is basically an open forum where we're gonna be discussing a lot of these in a lot more depth and throwing it open to questions So we invite you to come along and ask us more questions based on what you're about to hear here So the things that I want to cover Availability sets. These are not availability zones that you'd probably be aware of They use fault domains and we'll talk about those in a minute talk about the TCP keep alive, which and I think everybody on the stage and Everybody at Ford and Microsoft and quite a few of you from Pivotal will agree. This is one of the most pivotal excuse the pun Features to enable when you're deploying to Azure I Also, I want to talk very briefly and just kind of touch on the Bosch Azure CPI And then talk a little bit about storage accounts and what they are Okay, so at Pivotal we talk about the four levels of high availability One of those is using availability zones to provide resiliency when you've got multiple jobs So for instance, if you've got HAProxy you want to have that in a separate availability zone or a separate fault domain to provide resiliency in case That fault domain or availability zone goes down Azure has a slightly different concept and it's pretty cool. So what it what happens is it's actually perpendicular to a fault domain So Azure is super smart and all you need to do is say hey this job is part of an availability set And then Azure will take those VMs and automatically deploy them across fault domains. So pretty cool tech One of the unfortunate things about it being so smart is that Bosch also expects to be in control All right, so that causes a slight conflict But as you can see when we go to the next slide, it still works pretty well All right It's important to note that Azure treats a single virtual machine quite differently to a virtual machine as part of an availability set All right, there is a lot more notification that happens if it's a single virtual machine So we still do have singleton jobs in our deployment So we left those as not part of an availability set. So not part of a availability set of one if that makes sense But yeah, it's it's important if you have highly available jobs that you're deploying more than one instance of To put them into an availability set Okay, so what happens in an availability set if you specify something here for instance, let's say these are NATS VMs And you specify an availability set called AS-CF-NATS If one of these VMs goes away As you will automatically bring it back Probably in another fault domain All right, so it automatically handles that for you You don't need to split your resource pools into two sections and define different regions or availability zones It's simply a One-line entry into the cloud properties of the resource pools So hopefully everybody recognises this part and parcel of doing Bosch deployed CF all right, these are some of the Cloud properties that you have with Azure. So you have an instance type which defines the size of the VM obviously Availability sets and then you can define storage accounts that back these virtual machines. So we'll talk about that in a minute One of the key moments for us as I mentioned before was when we found that there was some networking issues that we were observing when deploying CF on Azure and this is basically because of the way the Azure load balancer works, so We were advised by Microsoft to include a TCP Keepalive, which Dimitri very quickly made a Release for and we deployed it using a the Bosch update runtime config All right, so we say that this is required for any Azure environment Personally seeing it go from not stable to stable right in terms of networking issues Networking issues to no networking issues. All right, so paramount I've also included some information around How to actually deploy this again, thanks Dimitri, Ben Black and everybody involved and then again just touching on the CPI Microsoft has been super responsive to all of the feature requests and all of the bug reports that we've put in and Rapidly iterating over their CPI With the release of CPI v11. There are some important Patches that have gone into that that we recommend that everybody uses But be aware that when deploying with Azure you will be iterating over what CPI you're using quite regularly All right, this is a good thing because We want rapid development right rapid development is fantastic Okay, so let's touch briefly on storage accounts currently the Windows Azure storage system that backs All of the cloud foundry deployments Does have some limitations in storage accounts. So the storage accounts for regular storage have an IOPS limit of 20,000 IOPS All right, this means that each storage account can service approximately 40 virtual machines All right, so for a POC, that's fine because if you're in the Microsoft talk a little bit earlier today You would have heard that they deployed about 36 VMs All right, that's fine But as you start to scale in a large enterprise environment You need to be super cognizant of what your storage accounts are but back your virtual machines All right, so at Ford what we did was each deployment had a separate storage account So we deployed Gemfire that had a separate storage account unless it was a super big deployment of Gemfire And then it had multiple storage accounts Same with CF we deployed CF with one storage account for everything except for the runners And that was a separate storage account. So again, that's how you define a storage account One of the final recommendations that we received from Microsoft was to use premium storage over regular storage We saw a 25 to 100 percent real-world increase in performance Apparently there is a lower fault rate. We didn't see any faults. So that was great and There is a documented migration methodology All right that we can we can run through at the end if we have a little bit more time So I'll pass it back to Mohsen for some of the engineering challenges So so we encountered a number of challenges and those of you That were part of the keynote in the morning I could relate to the comments of the speaker for all state that it wasn't all rainbow and butterflies We had a set of challenges that we work towards resolving and finding the workarounds So starting off we had pretty much issues around compute storage and network For example on the compute side the ratio of the memory to cores Azure didn't have a flexible offering Especially for Jamfire. We were looking at a higher memory VM as opposed to a higher core But we ended up consuming higher core count and since we are being charged by the core There's a cost implication for that as Hayden mentioned we transitioned and migrated from standard to premium storage and And there is still we're waiting for the availability of availability zones Around the network side the whole process of SSL termination Encrypting the data in flight and at rest and the overall failover and replication Using the security For wildcards SSLs and DNS that was a requirement by pivotal to be supported which was new to the fort Since we've never done it and there was some concerns which were valid so we did work around those by developing a solid onboarding process for developers and Putting controls in place for promoting code from lower to higher environments the open source components Ford has a very stringent review process for Contracts with the licensing and terms and conditions and with pivotal using all open source We had to go through the the entire review process with the legal team That has been now streamlined to allow for quick approval On the Azure side from our experience. We've seen there is a lack of robust tool set on the for pivotal on the Azure space The product is more mature in AWS and VMware And we have some enhancement requests that have been worked on especially around the availability of ops manager in Azure On the documentation side, we we saw some gaps. So there is a potential for some improvement adding some more details and alignment for logging Since as Shaji mentioned we are using Splunk and when we initially Configured firehose to syslog that opened up the floodgate for logs On the Splunk side, we are charred by the amount of data and it's expensive So we had to do the filtering at the syslog level to filter the logs For monitoring, we did a lot of work with the GMX and ops metrics integration with Splunk Creating dashboard and alerts for our operations teams On gemfire, we've seen that there is some scalability With the plans that you can have in the manifest For scalability, we are going through an exercise right now for doing the right sizing for the environment For ADFS It's about getting the right metadata and using the proper claims for authentication. I would say right now The pivotal cloud foundry the phase we're going through is high maintenance because of the continuous iteration on the platform side as well the the frequent release of Bosch CPI stem cells and CF services Like rabbit MQ gemfire and spring cloud services Requires us to continuously patch upgrade and rollout We are working on automation and overall patching process to streamline it That concludes our Presentation and before we open the stage for open up the floor for Q&A. I like Hayden to give a final message to the audience Thanks, Mohsen So I basically wanted to say thank you to everybody that's been involved at the Ford account across Ford Motor Company Pivotal and Microsoft right I See very much deployments on Azure as being awesome and only getting better from here on in right so the fact that we have this concept called availability sets as we start to use this more and We see other people use this in addition to availability zones, which is coming and manage disk Which will remove all of the limitations with storage accounts Azure is actually going to be able to provide Probably a fifth level of high availability Right on top of all the other levels that cloud foundry automatically provides for you so again, thank you to everybody involved and Shout out to Duncan win keep calm and see a push on Microsoft Azure So how are we going on time? Do we have time for questions? Cool. No questions at all I'd love to commit Ford to that Yeah, so again big shout out especially to John sermon, please come and see our talk tomorrow ask us lots of questions It's essentially an open forum Yeah, thank you Thank you for coming