OK. Thank you for coming to our presentation.Today, we'd like to talk about our private cloud strategyto expand scale and how deep our technologies of our private cloud.And so, first, I introduce ourselves shortly.Our company, Entity Dogomois the largest mobile operators company in Japan.And so, we manage OpenStack-based private cloud,which is used for in-house users.And so, we use all co-components for our cloud,and so, like Nova or Neutron or so on.And so, I'm Jun Ishii,mainly workers operator and consultant for in-house users.I talk about our cloud strategy and some GPU know-hows.And the second speaker is Hiomichi Ito,the CTO of Batchatech Japan.He works for a long time in our project over two years,and he will talk about the layer-to-get-way technologies in our cloud.And the last speaker is Kojo Aman.He has plenty knowledge of Entity Dogomo's security policies,and so, his topic is about reference model,which is based on our company policies,and so, plan for automatic security update in our clouds.The first half theme is focus on strategy.How do we expand our cloud private cloud?So, we launch our private cloud system just one year old.And when we started the cloud,we only have one-ditch center on about 50-100 cores compute mode.It's a little bit too small to manage or maintain systemsat the point of TCO,total cost.As you know,running costs are affected by scale.So, systems should be large to some extent for the costs.By some reasons,which we mentioned later,we succeed in scaling our cloud.So, just now,scale of our system is about six times largerthan last year,by launching new-ditch centerand set up more nodes to last-ditch centers.So, on,we are constructing two more-ditch centers nowand they will be launched in next March.And so,cores will be over 335,000,so it's a big worry.Then,now explain our strategy.To expand our clouds,we aim to accommodate a large forest first.So,large-scale system,which uses so many coresis like a forest.These four-size systems enable us to beapproval for expanding our clouds.After obtaining authorization to create a large forest,we'll enrich our private clouds functionto accommodate various types of systemsfrom tiny to medium-scale systems.From next page,I try to break downthis first-and-three methodto three important challengesand mention these details.So,these are details.First point,we obtain the budgetto create new-ditch centersand scale up our system.In Tecom company,there are so manyin-house systems,like operational systemsor business systems are working.We succeed to invite some of these systemsto migrate for our clouds.As you know,to create a large-scale private cloud,one of the most important programsis how to obtain the budgetto create them.So,fortunately,our private cloudhas decided to use newsystem-based.This is one of the example stories.A system,which working onpremise environment,was moved to our system.It's time to end our hardware life cycle.We suggest them to use our cloudsbecause there are so many trends.As a result,CAPEXand OPEX can be much reducedby using our cloud systemsand construct new on-premiseby their own.Moreover,these systemsare designed for distributed architectureand familiar withopenstack-based cloud system.Thus,they decide to use our cloud systemand we can expand more compute nodesand make more debt centers.So,not only these systems,but alsosome in-house systems will be migratedto our cloud system.So,our cloud system goes larger and larger.Next,short delivery timeto launch new debt center can be a strong point for us.Quick deploymentenable us to reduce CAPEX andOPEX and increasepossibility to satisfyurgent large-scale requirements.So,we always have some rulesfor operating cloudand one of the rules isnormalization and automation.So,based on don't repeat yourself dry rule,almost all knowledge are automatedby unsealable playbooksnormalized by knowledge-basedweekly.So,all informationare written by unsealable orknowledge-based.These know-how and how we storedknowledge are mentioned in the lastpresentation in Tokyo.So,if you want to know details,please check it out in YouTube.Then,this normalizationenable us to shorten period ofdeployment.So,we can do configuration settingsand network settings andso,install OpenStack componentsand start courty assurancetest in just ten daysby five people.First of all,these operatorsare not experts of OpenStack.They are just beginners of OpenStack.Are knowledge enoughso precise for beginners?Ifit can also be an advantagefor expanding.Last details,so,novell challengesare mainly focusing onexpand our clouds with various systems.So,after creating forests,filling rooms or spaceor by littlea bit smaller systemis needed for useeach nodes fully.When inviting small scale systemto cloud,so,many systemspreferred to usepublic cloudbecause,they haveso many functions.To satisfythe various users'will,we paidattention to what functions are neededto our company users.Actually,Layer 2Gateway is usefuland large scale systemalso needed to use.This is the key technologyfor connection betweentwo regions,the centers.We decided to add this functionto our cloud for the first stepof expanding.GPU is alsoimportant systemsfor machine learning usage.So,some of our usersare research and development sectionmembers.They eagerto use GPU nodesto analyzevarious kinds of big data.We judgedGPU isenoughmatured on work on OpenStrikeand can deploy by ourselvesand tried.Last two challengesfocused on our in-house rulesor policies.When constructingsystems,there aretoo many guidelines to apply for.Reference modelreduced these cost to applyin-house policy.Security update is justtesting report.However,it willenable us and our system usersto reduce maintenance costsfor security management.Okay,thenlet's see each technicaloverviews and know-housein last half.And first entry isenable us to cloud so farjust a cheap joke.So,thenwe talk about how deep in our private cloudso technical issues.So,first of all,Hiromichitalks about the layer 2 gatewaytechnics in our clouds.So,Hiromichiplease talk about it.Hi.Hi,everyone.I would like to talkaboutour layer 2 gateway systemwho connecting existinglarge-scalenetworks and inter-cloudnetworking.First ofall,let me talk aboutour user.Our userhas a large-scale existingnetwork and proprietarycomputer systems.Thisnetwork system has a greatability that provides layer 2connectivity to nationwide.However,thisproperietary computer systemside does not have enoughflexibility.Userneed rest API andサービスモビリティSo,they decided to migrateto Opus stack onthis renewal timing.But,they requestthat network system sidemigration must be minimal.And our user requestednew 2 network services.The first request is thatconnect the tenant networkbetween the two data centers.The second request is thatthe instance can communicatewith existing equipment.This request showsthe overview ofuser systems before renewal.The network run-triptime is about20-40msbetween thetwo data centers.There is a large-scalewide area network.And the dedicatedequipment is connected.This request shows the overviewof the system after renewal.We deployed theopus stack on both data centers.And weadd the new link foropus stack to wide area network.I will explain laterin detail.Our user'srequirement summary has threeitems.The first requestis about high availability.Our users requestedthat do not sharecontrol service betweendata centerbecause they would like toprevent cascading failure.So we choose regionzoning model.In region modelall service is separatedcorrectly.And our usersrequested thataboyed single point of failure.Hortulentlyour baseopus stack deploymentmodel avoids single pointof failure already.The second request is abouttechnical limitations.Our user requestedthat do not changeIP addressing and routing architecture.We can deploy that on theoverlay network.Nextour user request that do notuse network addresstranslation.Networkadress registration is masstechnic for floating IPand connecting the externalserver.But this system doesnot request the floating IPadress function.We donot anything for that.Last,our userrequires it abouttechnical limitations thatconnects instance andexisting equipment by layer2.This limitation camefrom existing systemtechnical limitations.Our baseopus stack deploymodel is using layer3 equal cost multi-passfabric and vxlan.So we choose vxlanlayer2 gateway.The third request isabout performance.Our user requested theperformance target about totalthroughput and averagepackage size.In termsof cost performance,thattarget does not match thesoftware-based solution.Sowe choose using hardwarebased vxlanlayer2 gateway.Upto here we have spokenabout user and requirements.Now let me moveon to the topic of hardwarelayer2 gateway.Wecarried out the equipment selection about hardware-basedlayer2 gateway.It's verylong name hereaftercalled hardware-layer2 gateway.Modern layer3 switch chipset has hardwarelayer2 gateway functions.Inthe example,interand broadcom have that functions.So we triedexamining both interfm6000and broadcom triedin two best L3 networkswitch.Finally,we compared three vendorslayer3 switches.As a result,wechoose vendor 8L3 switch becausethey support vxlan withina multi chassis run deployment.We chooseneutronetworking L2gateway for managing hardwarelayer2 gateway.Sowe carried out the proof ofconcept with using thecombination of networkinglayer2 gateway and hardwarelayer2 gateway.Asa result,we meet severalminor bands.Wenotice the missing of futurethat are required for the productionenvironment.Andwe created a patchthat add missing of futurethat areSSL support and multi-casthandling.Andthen we passed proof of concept.Thisshows the logical viewof proof of concept.TheL2 gatewayagent controls the hardwarelayer2 gateway byOBSDV protocol.Wepassed proof ofconcept.Next,we carriedout the pilot test asscaled as the productionenvironment.Asa result,we encounteredseveral critical bands.Wehave not encountered thesebands when carried outthe proof of concept.Inthe hardwarelayer2 gateway side,wheninserting a large number ofregord at one time,OBSDV server has crashed.Hochunetly,thisissue already fixed bythe vendor.Inthe networkingL2 gateway side,we have encounteredseveralcritical bands that arehard to reproduce.Whenhit these bands,L2 gatewayagent stopped.AndL2 gatewayagent recoveryfrom a crash state isvery tough.L2 gatewayagent always think statebetween neutron databaseand OBSDV.Hochunetly,whenL2 gatewayagent crashedor stored,these twodatabass sometimes lostsink.So,wehave toresisterL2 gatewayconnections settingsmanually whenmet these bands.L2 gatewayagent trouble occurredwithout missing a week.TheL2 gatewayagent does notwork correctlyafter a few dayswhen users runlong test.Theinstance could notcommunicate another regioninstance and existing equipmentin many times.Our user said,we don'twant to use this unstablesystem whole layer to getaway.So,we decide to notuse networkingL2Gateway in productionenvironment for thetime being.Becausewe could notreproduce bug.Sorry.Wecould not replace connection troubles between OBSDVand theL2 gatewayagentthat occurred weekly.Wecould not fix all criticalbugs that we encountered.Andwe met the scalabilityissue ofL2 populationthat is needed for networkingL2Gateway working.TheL2population does nothave enough scalabilitywho over several hundredknowed yet.Andwe must keep adelivery date because user'sside project depends onlayer to getaway in thetest stage.Andour project reader saidwe must keep delivery date becausedelay means ourproject death.Forthat reason,wecreated manual procedure tomanage layer to getaway atfirst.Wedo the manual operation tohardware layer to getaway andopen voice through tables whenuser request creates ourdelete and instance.Andwe created an automationsystem based ondelete procedurefinally.Thissystem working iscorrectly now.Letme summarize theproof point of ourL2Gateway project.Weprovide stableL2Gateway which is connectingthe two regions,instanceand existing equipment.Wepassed all test criteria provided fromthe client.Ourproof point ismobile flexibility by theopen stack.By using thelayer to getaway,the existingnetwork configuration waskept that our userrequested.Finally,wewould like to talk about thenext challenge of ourlayer to getway project.Thefirst challenge is that wefix known issue of networkingL2Gateway because we wouldlike to provide layer togetway service by standardAPI.Thesecond challenge is that weimproved earth populationscalability.Andthe third challenge is that weinvestigatedEthernet VPN for providinglayer to getway servicemore widely.That'sall for my section.Thankyou.Thank youHiyomiichi forL2Gatewaytalks.Andthe next is a GPUinstance.How do we dealwith them?However,thereis no time.Ineed to leave time forKozuro,so I speakshortly.GPU nodes havesome difficulty to dealwith.And there are somelimitations work on privatecloud.Need to knowprosent cons before youintroduce GPU nodes toyour private cloud.Soprosent consshortly.Andwe choose GPU nodesshown in here.Sothis server includesNVIDIA M40cards and four cardsinside of the servers.DeFroy isquite easy.Soneed to enabletwo functions.So PCIpassthrough and IOMMUfunctions.And so how topastthrough is written inOpenStackWiki.Just see it.And so IOMMUis also simple.Just writedown,grab file andreboot the server.Andoplate is a little bitdifficult for me.As a result of our verification.SoIOMMU allocates all memorywhen the instance launches.If you set flavor sizelarge enough and launchmaximum number of instance.SometimesOMKiller try tokill these process.Andso instance automaticallyshutdown.So swappingdoesn't work well becauseso these IOMMUaggrates these areas sofastly.And memory ballooningdoesn't work.Sowe need to set some workarounds to these problems.Sotake enoughmargin for the first OSand good for OS.Oneis a reduced flavor memory size.Howeverso memory sizeaffect to thegpu users.Sometimes it'stoo uncomfortable for users.Andanother workaroundis a so setreservedhost memory megabyte inNova file.Set toenough large size.Italsoaffect other flavors.Andso it cause decreasemaximum number of instancesfor the first OS.Sothese are just workarounds.Soif you have any othersolutions,please tellafter these sections.Andso how should we offergpu flavor to theprivate cloud users.Sothere are some pros and consjust writing downand so I have no timeto explain so justsee it.And theconcrete is so corporate withgpu instance user isso important forprivate cloud providers.Soit's my conclusion.Andso last twotopic is spoken bycozio and difference modeland security update.SoThanks my introductionsand I'mcozio Amanos.AndI would like toexplain difference models.SoI would like toexplain difference models.Soin order tomigrate some ofinhouse app to ourclouds,we havethe security policies whichmeans over 100 securityguide lines for securitygovernance.Soand then a lot ofeffort by usersrequired to meet thepolicytes when usersreconstruct app on ourclouds.Sohere is part of our securitypolicytes.Forexample redundant seats,log file like that.Andin order to reduce users'tasks,we proposepredefined models.Andpredefined model is calledreference models on ourclouds.Reference modelis a system architecturebased on many of securitypolicytes.Andreference model deploys someof servers sets of open sourcesoftware stacks that havebehably tested on ourproject.Solet me show you a demohow to play reference models atfirst.Sothis is demo.Andfirst users can downloadheat templates.Andusers can confirmto share the image whichwe prepared.Thiscan be used inorchestration service.Andsome of server image afterthat we used.Andthis is orchestration service.Andinput them heat template filesinput them.Andthis is the heat template.Andnext.Andsome setting like the securitygroup or those serverimage passwordsor key pairsand like thatand launch.Andafter the three minuteshere we go.Orchestration can be complete.Andyou can see the networkproject.This isreference models networkproject.Butit is a little bit complicated.So back to the presentationsand I would like to explainin details in systemarchitectures.Sothis is system architecture.This is based on Web3 tier models.Andwhich means three networksegments.Andin each segment there areunique servers.Andthis is covered withsome of security policiessuch as network segmentsand enforcing securitygroup like that.AndI will explain Web load balancer serversand VPN servers.I will explain for example.So first Web and load balancer servers.Andopen source software appacheis installed by defaultfor Web servers.OrLVS that is software loadbalancers is installedfor load balancers LV servers.Andsome of security policies such as dummy certificatefor HTTPSor Web basic rulesfor IDS.Andin this time the key pointis to not only installopen source software but alsocomplete those defaultsettings about securitylike HTTPS like that.By the way we did not useLVS as a service load balanceras a service which calledLVS.We did not usethis is because LVSV1 in Junotes does notsatisfy with use caseof our users.Ourusers require tofor example set the securitygroup to load balancersor terminate SSLat load balanceror provide storage pagelike that.So that's why wedid not use LVS.Andnext VPN serversfor secure remote accessweopen source softwareopenvpn is installedand makes SSLvpn.Andthe defaultsetting can be doneand then we prepare toolsfor operation of SSLvpn such as createand revoke certificateand these enable usersto reduce users tasks.And same as LVSwe do not usevpn as a servicevpn us this is becausethe algorithm forauthentication in IK phase 1accept it onlywhich will beencryptions losing safetyassurance.So wedid not use.By the wayindecent version newtonsvpn as a service accept itsh8256.So noproblem about that.And this is summarylastlyas we showed different modelsreference model is coveredwith some of security policiescurrently.Andcurrently covered area is60% in this picturespink areas covered areas.So and our futureis we will updatethis models by addingthe missing parts in this gray partsabout security policiesand aims to cover 100%and aims to reduce userstask and more and more.So Reference modelis end and nextis security updates.So at first we willexplain about current dailyoperations,manualies.And most of the operationour operation is manualcheck vulnerabilities orrisk assessment of vulnerabilitiesand management to do listupdate our crowds.And thosemanual operation can causehuman errors for get tocheck vulnerabilities.Andtime are a little bit longtime hours are days.And moreover as ourcloud explains it will becomeimportant operations ofsecurities.Sothat's why we want thisthis operation can be automatedin order tomanualize my errors.So and in order toconsider automatic processso we will explainin details current operations.So first check vulnerabilitiesand we check vendor sitemanualies one by one.And then risk assessmentthis is semi-automaticprocess by ourscript.And afterthe risk assessmentmanagement to do listsales and afterthe update,after themanagement,we updatesemi-automatic process byunsettles.Andfromthose processeshuman errors can behappens.Sowe aretesting ourproposal ways.We proposehow to be automatic.Sonow we areconducting our proofof our concept.Sonow let me showthe most how while we aretesting in automatic processand each process.Sofirst check vulnerabilitiesautomatic process.Thisis our proposed wayis based on CVEand CVSS.AndCVE is attached IDand CVSS isscratch vulnerabilities.Andinstead of vendor sitefor check automatically,APICVEsearch is used.Andlet me showshort demos andautomatically process.Sothis is our tools,our testing tools.Andthe list of vulnerabilitieswill be shown here in thisscreens.Andnow CVEsearch APIis executedbackground.Sothe listis shown.Forexample,CVE is left andCVSS scores are shown.Andalso summaries orlast update like that.Soby using APIs,it can beenabled as to automatic process taking vendor sites.Okay.So next,disk assessment.The key point is CVSS.Thedisk assessment byCVSS is not alwaysmuch with our environment.If the version vulnerabilityis not related,the risk will be announced or the hostin the network will be targeted.The risk will be lost.So like that.So that's why we needto reevaluate the CVSS scoresfor each host regardingits environment.Soautomatically reevaluate the processlet me show the demo.So confirm toautomatically reevaluate the CVSS process.And first choose the host.And package listin host are shown.Hereis that.And rest is package nameand urgencyand install versionlike that.AndCVSS is automatically reevaluatedand from some pointof views.And shown as urgencies.And this isokay is no vulnerabilitylike that.Sosad management to do list.Of course do not forget vulnerabilitieswhich have high risk untilthe patch is upright.And also moreimportant thing is evenif the CVSS score is上昇it will sometimes become highscore in our environment.So we need to checkthe same vulnerabilitiescontinuously.Andautomatically processdemo.Quickshort.So this ismanagement to do listabout vulnerabilities anddemo.And same asprevious.Choosethe urgency and the listand host packageversions and relatedCVEs isshowns.Andthe result ofCVEsearch isreflect once a dayand automaticallyreflect in these few yearsrelated CVEs.Solast update our cloud.This is only semi-automatic proceduresbecause we mustcare about the influenceof the user's instance.Andit is necessary to havemanual integrations forcheckpoints.Sothat's why we decidedsemi-automatic procedures.In this part isupdate cloud proceduresafter the check, please.And the futurefor the future we willapply our proposed this waysand we willpublize the environmentfirst.And alsowe will extend our toolsfor user'ssefchecks.Sofuture works, I think.Sothis is end.Thank you for listening.Soif you have any questions,please.So nothing.Sookay.This is end.Thankyou for attention.