25分钟怎么了25分钟哎你知道这个吗我打电话没有哎你不在乎吗你不在乎吗嗯哦嗯你不在乎吗嗯哦你不在乎吗你不在乎吗嗯嗯你好好好我还没参与所以我给你打电话你还没有参与吗没有我没带过因为我还没参与你知道吗嗯我带过的帮助是 tôi带过这个帮助我带过这个帮助你带过这个帮助你带过这个帮助嗯所以我带过这个帮助我带过这个帮助同样哦真的吗嗯好你會到,你假的我會到,我會到我會到今天是我的生日我現在在基督教館今天是我的生日我需要兩個人我需要兩個人我需要兩個人我需要兩個人我需要兩個人沒有,我已經有很多人我已經有很多人我已經有很多人我過去了剛才我說了我剛剛說的第二次我沒 forestall給你看一下再講你酒店的儲物沒啥我猜她在whale我們怎麼使用和phroudoraI think it starts with wailing by people. I don't know.Oh, you assume that we need to change the mode.I'm not sure, but...Yeah, I have the slides on the USB drive.We can try them.Yeah, we can try them. Thank you so much.So it didn't work out straight out?No, it didn't work out.My teammate is going to try on his computer to see if it's going to work.If it's going to display, then I will put the slides on this computerand display it on the right.Oh, yeah, yeah.It's going to display something.What's the assistive wheel?Is this teleblogger?Yes.Thank you anyway.I think it's going to be possiblebecause currently the device was thought onwithout connecting the devicethe machine actually works.Yeah, we've had other sessions in here today.Yeah, we did.I wonder if it displays resolution.Do you see any representative?We can try the mode display.You can do it.Yeah.Can we come out?Yeah, I just came to display.You want to try it?It detected it last time.OK.Can you help?I can do that.It's not projecting.It's not going to work.So I think it's going to work.Can you see the screen?Oh, yeah.It's in English on your free phone.Oh.Oh, yeah.You got it.Thank you very much.I still wonder if it's in...Because I have a demo.I mean, we call it DVD.It follows the time.And the...I only connect to a DVD as a supplier to stream the video.So, yeah.Somebody like a volunteer to be back there streaming the video?I have the on the side.I can see the family is allowing me to do that.See the motion.Yeah, it's working.That is good.Possible unboxing of the PowerCycle?Um,You know, it means that...It means that they share...I'm going to call somebody.So,We're having some issues in Paris.Because the project is just not really working.We need a PowerCycle.I'm still not displaying anything.So,Is this a call spider?You could call.Okay.Do you have a number?Yeah.Yeah.Yeah.Okay.Okay.Do you have a number?And two actually.So,replace that?Yeah.We're going to try cables.And if anything,we'll just have somebody bring in another projector.Okay.這樣我一點都不緊張.You too.Oh,the video stream that we recorded,they will just cut up from time to time.I mean for some reason.No,anyway.So,I guess...I have the local video.Let me check.Yeah.I should have the local.Oh,yeah.I have the video local.In case,I probably out there to a local video.Oh,what do you mean?I have the video,not on the YouTube.I put in the local directly.Okay.So,in a demo time,I will exit from the slideshowand the stream is going to work.Yeah,I think probably that's the best option.So,avoid any disruption in between?Yeah.Yes,yeah.Seriously?Are you able to solve the problem?Yeah.How long ago was that message?Just now.This way.And the projector.So,yeah.So,here from podium?Yeah.Is there someone over there on the wall?There's one right here.You just go in directly.Yeah,it's still going.Oh,is there?Yeah,maybe we need to do that stream.Yeah.Well,yeah.They use another projector.It works now.Yeah.Yeah.Yeah.Oh,yeah.The stream is fine.You're not the final stream.It's just...Okay.So,on the stream?The stream is fine.Yeah,sorry.No,it's fine,it's fine.It's one of the...I know,it's...Yeah,I'm going to have to ask them to just give me some...You're going to bring them up again?你那邊有 timer?會讀計時的就到10點.Do you read the...the one?Um...I need to read the time.I don't know.I can put slides in that computer.That would be better.Yeah.Okay.It's not going to give me anything.That computer?Yeah.I think it's just walking on the computer all the time.Yeah.Do you have another deck setting?Uh-uh.It's a little wet,well,okay.Yeah.So,it's got delays.Yeah,unfortunately,we just have to...to continue.We have to continue.Oh,yeah,okay.Yeah,do you want me to grab something?Like,just grab...Yeah,yeah,yeah.Do you have a preference on something?No,no,nothing.Anything?Anything.Okay.That would be...So that...So much...That would be...Yeah.No.No.Yeah.No.No.No.No.No.No.No.No.No.So right now, we're just going to do the projectors with content.It will not be there, so we'll have to shoot the content of the slides and see the slides.So right now, so we are not able to bring in just the content.It's going to be the camera, so people will see you and the slides through the camera.But before we just have the content, but for now, we just have you both in the slides.Is that okay?Yeah, I'm fine.Because we just have to, you know.I'm not waiting anymore.Yeah, right, yeah.So we just have to move back to the object.I assume if I have a problem, other people will also have a problem.Right.So I can start the talk?Yeah, you can start the talk.Let me see if I can do this.Okay, finally, we fix it.Yeah, and thank you so much for the patience.And let's get started.And my top up, let's first introduce myself.I'm Wen Wen.I've been working in Red Hat for about two years.And primarily, I'm working on the network loop.And it's an Ansible module.And I'm also working on some network-managed stuff.And the topic for today is modern network configured by Ansible network loop.And I just wanted, how many people use Ansible before?Can you show me your hands?And how many people use the network loop before?Oh, you do use it?Okay.Because network is so, so, so powerful.But it's underutilized.I'm glad that someone here is actually using it.And Ansible is a tool that can make the IT automation so easier.Because imagine you have a huge fleet on machine.If you want to configure on this machine,or those machines might be very OS,and other system components.So with Ansible,it can configure them all at once.And with zero downtime.And drilling the updates with load balance are easier.And while the network loop is Ansible loop,it can be used to manage.And configure the networking settingsto manage host.And the machine, of course,might be varying OS and versions.And providing such a toolto automate the configuration process.And because considerably,the requirements and standards might be changed.But with the tools,it can just everything happen automatically.Because usually,it's a declarative API.So users need to provide the settings.So the module abstract like a black boxand config setting for the user.And before the invention of the network load,and there are already some other modules,like other networking tools,like network manager,or network MCI.And the power of Ansible is can,because using the Ansible facts,get the status info from various machine.And in the inventory,you can custom some settings.In the black box,you can retrieve the settingsand pass as a parameter to the module.So it's very convenient.And it's automationso that you can do some parallel configuring.And which may makethe configuration processjust be simplified.And the network also providevery good error handlingand just all error reportingand also can do the checkpointand rollback whenever happens.And because I know some usermight still stick to the old legacy wayfor configuring the networking.Now a network still have two providers,bothnetworkmanager and inescripts.And currently,there are two primary modulesin a network row.The one is network connections.The other one is network state.For the network connections,they have two providers,inescriptsand network manager.And it basically create the connectionprovides for the device.And activating the connection provides.And for the network state,instead,it just configures the device directly.And it'saccept all themstate state examples.Okay.Here I draw a very vivid comparisonbetween these two modules.Because network state justimplemed me I think a couple of months before.It's just brand new.For network connections,you cannot do partial configuration.You have to track all the configurations.But for network state,you can.And the providersand network state,I mean,the thing behind it,it's just still using thenetworkmanager.But the network connection can usebothinescripts andnetworkmanager providers.And network connection canconfigure the device.It cannot configure the devicequery.It's always creating theconnection provider at the first stage.And thesupportive feature,I think of various interface typessupported forthe network connections.And recently we justimplemented various new featureslike policy routing,various bounding modes,options,etc.dnsate.But for the network state,since it's back in thisLiberM state,and all the featuresin theLiberM statethat are also supported onthe network state.Now that for the network state,it's a full-fledged.So I assume that almost all thefeatures are supported.And here is an examplefor configuring the IP addressfor the Ethernet via the network connections.Since the standsport,we have to specify the hosts.We need to include the joe.And we declare the variableswith network connections.And what the nameis,the profile nameand the title to beunboot.And then weconfigure for IP,configure IP for them.And there'sthe other way thatconfiguring networking,which isthe network state.Similarly,we need to import the row.We declare a network state.And we specify the IPconfiguration.Originally,the network connectionwas invented.It'svery goodfor the userstick to theinit scripts provider.Because the network statecannot use the init scripts provider.It can only usethe network manager provider.Well,with the bothprovider for configuring thewith both module,you canconfigure the IP addressfor the Ethernet.And forconfiguring the rows andnetwork statehave someglobal config,butactually it matchesthe interfacesin the system.And config onthe device,I meanper device will havethe rows and the dnssetting.Similarly,youcan do about the same,Imean,achieve the same effectwhile the network connections.And now,you canconfigure theinit scripts provider.And wejustseverify IP,choosetheinit scripts provider.Andwe justseverify IP,choosetheinit scripts provider.Andwe justseverify IP,choosetheinit scripts provider.Here,I was choosingthe SAE,WPS repersonalforostimification.Here,it's a complex them for configuringthe policy routing.It's justimplement recentlyinhere,it'swe're using thewe add different rows tothe routing tables.We define theroutine rows from,likesauce routing,and thesauce port range,so that it canforwarding or field toport packet corresponding.Andthis is configuring thebonding options.Also,use networkconnection module.So,utilize thebond,so wecan have a feature likelinker aggregation.Andif any failurehappen,can failover toanother link.Andwe cancreate thecontroller device andadd the ports tothem.Andchoosing different boundingmodes,and defineImake a demo for thepolicy routing.Solet's take a look athow toconfigure the policy routingthrough the network.Let's firsttake a look at thetesting profiles.Inthe testing profilesspaced by host since it'ssensible.Andwe initializethe testing interfaces.It'swe import thedraw,andspecify the networking configon the network connectionvariables.Andwe specify theprofiling interface namestate to be on.Andthe type to be isn't it.It'sautoconnected.We canfigure both IPv4 and IPv6 services.Wecreate the draw toand add the draw tothe demo.We alsoconfigure the policy routingthrough the network.Let'sfirst take a look at itin the...Hi,everyone.TodayI will give you a demohow toconfigure the policy routingthrough the network.Let'sfirst take a look at thetesting profiles.Inthe testing profiles,spaced byhosts since it'ssensible.Andweinitialize the testing interfaces.It'screated by the IPcommand.Andwe import the draw.Andwe specifythe networking configon thenetwork connection variables.I'lltry to explain.Oh.Andwe specify theprofiling interface name.Stateto be on.Andthe type to be isn't it.It'sautoconnected.We canfigure both IPv4 and IPv6 services.We createthe road.Andwe add the road to theroutine table.We alsoconfigure the policy routingthroughs.Inother terminal,let's actually runto see the effect.HereI was using the QCOP2imager test.It's QEmocopy on white.It's veryconfigure convenient.Andsee policy routing.Wecan see the testing.Inother terminal,let'sactually runto seethe effect.As you know,we have touse it all the way.Everyone,todayI will give you a demo onhow to configuring thepolicy routing.It's better.Let'sfirst take a look.Okay,Ican explain another way.Well,theresolution on my computer isbetter than this one.Butanyway,weinitialize the testingfaces.It'screated by the IP.Soweinitialize thetest interface.Wecreated the vests bythe iKIMP command.Then we importthe row.We definethe profile name,interface name,statebr,and configuringthe IP,the root.Andtheroutine row.Theroutine row.Andthis isbasically forpolicy routines.Andin other terminal,actuallyusing qco2img,qmocopy on-ride.It's veryconvenient for testing purposes.Andwe runthe row.CurrentlyI was using the networkconnection modules.Werun in the row.Andokay,it starts running.Becausethe row has some default tasksrunning.Maybe install severalpackages.Depends onthe module I'm using.Soit's short tackle.Butit's going to be veryquick.Andhere are someassertion states,just like I said before,it just install several packages.Basically,what I did is just afterconfiguring,our SS addsinto the machine,use different iKIMP command to verify what I did iscorrect.So I get thepredentials.Nowthe ansible playbooksactually passed.Sono errors.So I get inside themachine.I use IPSshow eth20.Andit displayed.Ididn't block it.It displayedthe IP address exactly the same asI just configed.Andit's veryconvenient as a user,because it justdo not need to memorize all the commands,justby the settings.Andlet's take a look at theroad that be added tothe table,series0 to 00.And another roadadd to the table,series0for 00.It's alsomatched with the road Ispecified here.Andfinally,I need toverify theroutine roads,thepolicyroutine routes,still usingIQQ command.I mean IPshow tableto find all thepolicyroutine routes.You can clearly see it'sa source routing andfrom all sourcepods128 to 256.So it'scorrect.Andalso the traditionalrouting for the packetgoing to the address9851105awas also added intotheroutine routes.YeahI think the primary,that's it.I have not a media,butgiven the fact thatit's not complaining,butgiven the fact that it limits timeand it's kind of full resolution.I will stop for now.Andthank you so much for the patients.I willanswersome questions,if you have any.So,let'stry to repeat a question.Youtry to use an open b-switchto make all the nodesmanuals at leastcan talk to each other.What's the module?Oh,okay.I see the point.Sothe machine is kind of isolated.We cannot talk to each other.For now,as you know,you're talking about the tune and tapinterfaces.Currently,thethat's not supportedfor now.Yeah.Did you try Network Manager?Did you try Network Manager?Yes,yes.For your problem,currentlythe road cannotconfigure the tune and tapinterfaces.But I haven't tried it outin the network state.I don't know.But at least fornetwork connection module,it cannot.Yeah,it's just released.I think one,probably a coupleweek,one month.AndI will keep that in mindto see if network state canconfigure the tune and tapinterfacesand create the VMsso that the VMscan talk to each other.So,your question iswhen we reboot,ifthe configurationisthat'swell,here,this thingI want to mention is thatfor thenetwork state module,wedo not do any validationarrow handling.Firstly,we justpassing whatever the parametersuser providing and feeding to theLibern state.So,I would assume thatLibern statewill do the eitherrestore,like you saidor save the configuration somewhere.Yeah,yeah,but the roledidn't do that.Yeah.Any other questions?Okay.If nowthen probably that's all for today.Yeah.Thank you allfor the patienceand the participation.Yeah.Thank you so much.謝謝Are you outside?Yeah,I'm outside.I just finished it.This is recording,right?I wishI wishYeah.That would be likebecause this cannotparts in the regular time window.Okay.So,itshould still be mainstream becausethey're not stopping.Was this the one Ithis is the one I love,yeah.Still,in the same time,youhave a couple of numbers.Yeah,yeah.So,I meanyeah,I didn't have timeto play all the game modes.Well,yeah.That's whatwe're going to do right now.We're just going to come through andyou had lunch?Yeah.I'll be fine,butit's fine.Yeah.You had lunch,right?Yeah.How to work a featureMake sure I didn'tthat for anything,otherwiseyou might be in trouble.謝謝戴上 everybodyso it's that kind of presentationI'm used to working remoteWilliam Brace the new test to herbut thenthe men will end up hearing youI'm gonna do it with my kidsI mean they should be hereso it isI think no problembut it's a nice excuse to watch themtheir lives will go on the whole timesoHello, my name is Mr. Turkal, and the first talk is a virtual talk, and thank you for waiting.Yeah, I'm very happy to meet with people who are passionate about software and the best of work.Yeah, so we'll have them hop on call at the end.So today we could change your practices.And of course discuss about ways to insert efficiency and minimize the cost in our workflows.I'm a quote.Yeah, this is a sound that's coming out great in the other ones.I'm a quality assurance engineer, and today we will talk about different quality assurance practices.And of course, we think will not be the quality, not only the outdoor.I think the next volume.Oh, hold on, because I have it turned down there.This is the virtual ones, yeah.To meet with people passionate about software development and the best development practices.And of course discuss about ways to improve the efficiency and minimize the cost in our workflows.I'm a quality assurance engineer, and today we will talk about different quality assurance practices.And of course ways to improve the quality, not only in our product,but also in our processes and the life cycles, the workflows that we follow as well.So I'm sharing a journey and a story, including lots of different initiatives that we've performed in our teams.And as we will see, these initiatives are performed by a strong communicationand a collaboration with the other teams or the other team members,because quality assurance is not only testing,but including lots of different stages,starting from the analysis of the system,analysis of the requirements and specifications,the customer expectations or the end users expectations,until the maintenance stage,which is raising the issues or the improvement pointsor the vulnerabilities that we find to be fixed or to be improved.So we will try to cover all the stages one by one,and we will cover all the rings of the whole chain,because quality assurance is not achieved by individual efforts,but it should be a holistic approach.So we will see lots of communication channels performed and maintained by the other teams or the team members,and we will see how closely the quality assurance members should workwith the other teams,with the product managers,with the product owners,with the development leads of the development or the development team colleagues,development team members.So as a result of all these initiatives,we will see how the outcomes will improve,and how we will increase the visibility of the quality assurance activities and the practices.So starting from the system under tests,clarifying the system that we test and we develop mainlyis very important as a starting point,because if we don't know all the aspects of the product,all the specifications,all the features,all the use cases,or if we don't know what the customers or end users are expecting from this product,then we will not be able to check if the actual behaviors are meeting with the expected behaviors.So making all the requirements and the features visible and understandableis very important from the first place,because assuming that most of us are working with agile practices nowadays,and as agile suggests,the working product is more important than the comprehensive documentation,but it doesn't mean that we don't need any documentation at all,we will still have to put some effort on making some documentationto clarify the features,and at least making them understandable and visible to everyone,because otherwise we may have some misunderstandings.To make everyone on the same page,and to have the same understanding,we have to make a basic documentation,it doesn't necessarily have to be a comprehensive documentation,but at least some documents explaining the usagesand the features,the aspects of the productwill make everyone be clear about the product and the expectations,and in this way we can even improve the testability,we can figure out what kind of testability issues we have,and we can figure out what kind of coverage rates we have,so these were the very first initiatives that we have done,and one more thing,we have agreed with all teams,including the development teams or the product teams,was to encourage the early QA involvement,because if quality assurance members are not aware ofwhat is being implemented at the moment,then they cannot start their preparations in the meantime,because in agile workflows,the efficiency is very important,we have a very limited testing window,so we have to already start our testing preparationswhen the design is already started to be set,and if we are aware what is being implemented,then we can already discuss about the testabilitiesand the old aspects of the requirement as well,so what we have agreed with the other team membersto building a strong communication channel,and whenever they start a new development activity,the quality assurance members were invitedto the discussions of the requirementsand the other aspects of the product,so after this stage,all the system levelclarifications and the announces of the requirements,the next thing that we do in the software testing lifecycleis designing the test cases,designing the test cases is again very important,to cover all the use cases,because otherwise if we have some coverage issues,if we don't properly cover all the use cases,then we may have some production errors or some bugs,if we test some features with only a specific data setor a specific data type,then it may not be sufficient to figure outall the possible problems or all the possible issueswhich may raise in production,because if the feature is supporting lots of different data typein various data ranges,then we should prepare a comprehensive data input,test input,and cover lots of different regions.Let's say if we are developing a user interfacewhich is supposed to execute on lots of different browsers,then we have to do the course browser testing as well,so we have to figure out if there is a compatibility issue,then we have to fix in advance,not to have any surprises or any issues in production,because if we find bugs or the vulnerabilitiesin later stages,then it would be much more costly.If we find them in the early stages,then it would be easierand of course cheaper in terms of the effortwhich is needed to fix,make the fix,or at least the system will not be down in production.So these kind of issues can be managedor avoided in the first placeby having a good test coverage.So we should consider all different supported platforms,all the execution environments.So if we have any hardwareor machines operating systems,then we can try lots of different versionsand lots of different platforms likewe can test on desktop platforms as wellor on mobile platforms as wellso if we test on desktopmaybe we may miss some issueswhich are raising on mobile executionsor if we are testing on only one of the browsersthen we may miss some other issueswhich may raise on the other browsers.So not to have any coverage issueswe can utilize lots of differenttest design techniqueslike maybe boundary value analysisto consider all the corner casesor all the positive scenariosnegative scenariosor we can do the exploratory testingto figure out what kind of other use cases there areor we can use lots of different test design techniquesbut eventually we will have a good test coverageand we will have a good test input set as welland after having all the design test caseswe can document themand we can build a traceabilitybetween the use casesbetween the requirements and the test casesand in this way we can figure outwhat range or what rate of coverage we havewhat rate of test coverageand the automation coverage we haveand we can figure outwhat kind of testability issuesthere arein the project or in the productthe testability issuesI wanted to emphasize specificallybecause it is very importantthere may be various reasonswhich are blocking us to be able to testall the aspects of the productfor example if we have any instabilitiesfor any reasonlike if we are having some AMV testing activitiesand we are deploying some different versionson to our instancesthen we may not know or we may not be surewhich version we will hitwhen we are doing the testing activitiesor the behavior itself may be a conditionalwhen the user data falls into somespecific rangethen it will trigger some expected behaviorsbut otherwise it will trigger some other expectationsso this kind of instancesis a challenge to do our testing activitiesor if we have some hardware modulesit is really difficult to maintainthe testing activitiesat least the automation will be much more trickybecause manual testingmaybe we can do somehowbut the automation willmaybe require some simulationor the test harnessesor we may have some dependenciesif we are requestingor if we are pulling some datafrom other subsystemsand if they don't have some publicapisor some public interfacesthen some data is already lostin communicationfor example if I have tosee only the items or the componentshaving some specific ideasand that IDlike the feed IDor any other IDsis not visible to methen I will not be ableto do or checkif all these componentshave this IDso what we can do isand what we did in our projectis to go over all thesekind of itemsand figure out what kindof testable issues we haveand then weperformed another initiativeand we tried to improvethe testabilityone activity wasthey implemented someinterfacessome apispublic apisonly for testing purposeswhich will not be usedby the end usersbut just to improvethe testabilitythe development teamimplemented some new apisand in this wayI was able to findthe dataor the attributesthat I need to testnormally whichI was not able to checkin the software testinglife cycleafter the testsI've designedthe next thingisto execute test casesand we can executetest caseseither in manual waysor with anautomated mannerso as we all knowtoday test automationis very importantbecause most of usare doingcontinuous testingand we are integratingour test executionsinto our CI pipelinesnot to break anythingif we have any changeor any updatewhich is appliedto our code basesto repositoriesthen aftereach mergeor beforeeach commitwe run our test casesto be surethat these changesare not breaking anythingso to be ableto do this continuoustesting activitiestest automationis very importantand in the meantimeit has lots of otherbenefits likeit is reducingthe executiondurationit is reducingthe manual effortand sometimesit is opening the wayfor usfor what we cannotdoin manual wayslike load testingand concurrentrequest testinglots of different thingswe can dowith the helpof test automationit is greatit has lots of differentadvantagesbut it is not a silverballotit has lots of differentchallengesas wellin the meantimeand we haveto maintainthe code very wellbecause otherwiseit will besome other problemsand issuesthe code qualityis very importantwe have tomaintainour test codein lots of differentaspectsthe reliabilityis veryimportantthe robustnesstheunderstandabilitythemaintainabilitythe re-usabilitythese kindof aspectsare differentpropertiesof the codequalityand forthisto improvethe qualityin our testframeworkwe needstrong qualitygatesbecause wefeel iteverythingto immersein our pipelinesthey willraise somepotential issuesand problemsthey willraise some travelto usin ourlaterworkflowsand for this purposeof coursethe maintenanceis very importantand maybethe peer reviewsand some otherstated codeanalysiswe can utilizeand one more thingthat we havedoneto improvethemaintainabilitywith the helpof the otherdevelopment teamespeciallyfor UIautomationlocating the elementson the webpagesare very importantand somenormal we usedifferent selectorsto do thatlike we can usethe class namesor we can usethe expertsof the elementsbut sometimesif the webpagesdesignis updatedby the developmentteamsthen sometimesour test casesare brokenthis is a verycommonly knownissueby the test automationengineersto doafter an agreementwith the developmentteam membersis use the commonlocatorsin all appswe are usingand we addsome testdata IDsin this waywe don't haveto deal withless ofdifferent longand complexcomplicatedexpertsbut insteadwe just addthe testdata IDsand evenif thepetsor theid or theclass ofdata IDis stillsameand ourtest caseis not brokenanymore afterthese changesand one more thingis of coursere-use aboutthe systemthat I'm testingthere are lotsof different applicationsand the sitesbut theyhave somecommon front-endand commonbackend librariescommon backend modulesso thereare someuse caseswhich canbe testedorperformedsome querieslike they canput somedifferent keywordsor locationsand they cantry tosearch somerelated jobsso thiscan betested onall sitesso Idon'thave towrite thesametestcase foreach ofthesesites undertestso whatI cando isre-useabilityand itwilleliminatethe waste oftime or maintenance effortsso whatIthat isIhave toIhave toexcluteall theamount ofvaribalsand theconfigurationalsettingsapartfromthetestcaseitselfand thenif Iexecute thistestcaseand ondifferentsitesand in thisway were-use tomaintenancecost becauseif one ofthetestcase should beupdatedthennowI don'thave togo tolots ofdifferentclassesbut insteadIapplymychanges ononlysinglesort oftruthwould漫全可以createforthesetanddétend如果我們在進行一個激情測試我們可能只能選擇一個正常的測試或是我們只能進行一個激情測試因為每個激情測試它是不太可行的並且它並不需要使用所有的激情測試但我們還可以選擇正常的激情測試讓我們沒有什麼打破任何功能或是在工作的系統上我們可以改變效果因為激情測試時很重要在每個激情測試或在每個激情測試如果我們的激情測試花太多時間了大家就會開始懷疑對嗎所以我們要可能選擇正常的激情測試而如果我們能做這個測試當然正常的激情測試是很重要的而我們要和製作人和製作人決定正常的激情測試以及正常的激情測試所以我們要創造一個標準在哪裏有不同的方式例如激情測試或是激情測試或是激情測試或是激情測試而另一方面有影響如果有這個標準會有什麼影響如何重要這個問題如果整個激情測試或是只有有些空間的視角有些標準有些有些標準這些類型可能不太重要但可能更加能夠看見所以決定這些不同的方式我們 finalizePriorities我們創造我們的測試甚至我們選擇這些不同的測試例如我們有一些激情測試在哪裏我們沒有任何短短的所以我們可以選擇更多的測試但每一個測試這三個判斷我們測試我們選擇小的測試以及測試測試最後最後但不少的是測試因為我們完成所有測試測試測試他們的測試測試他們的測試但我們測試這個測試我們可以把測試測試測試目前不是一般我們測試測試測試在測試測試測試我們測試測試測試我們測試測試測試測試測試所以我們在這個程度下,我們減少保持性能量和提供的努力,明白如果失敗是否正確的立場或是真正的立場還有一件事,當然,我們繼續減少減少減少的努力我們可以減少減少的減少減少或減少的時間例如,如果我們有些減少在我們的減少程度上,我們可以減少它或是我們有些減少的減少我們並不需要減少我們的期待的狀況我們可以減少它我們可以減少減少的減少但更加上減少減少的努力很重要的是,我們要減少減少減少的努力因為,我們沒有要等待在減少減少的減少程度上我們需要等待但我們可以減少減少的努力我們可以減少減少的努力我們可以減少減少的努力而在每個程度上,我們可以減少減少的努力但當然,這是一個團體的努力這些不是一些努力可以減少的努力所以,在每個程度上團體的努力我們應該跟其他團體的努力和我們應該建立一個強大的交流頻道我們必須要理解每個程度或每個努力都要很清楚和理解很棒我們有任何問題嗎?或是想問什麼?我們現在沒有任何問題如果有任何問題Mesut在社交平台上所以,你必須要問他在社交平台上對不起,這是一個技術的問題希望你會享受這個討論好,謝謝大家我們在這裡,抱歉在前一陣子有一個討論有一個討論但是,我很高興來到這裡可以討論一個問題我非常興奮是發展以及發展所以,謝謝我們在這裡會有短短的時間三分鐘我們會做一些討論所以,我們會嘗試展示你一個討論如果能夠展示我今天謝謝Mesut謝謝你只是一個 quick introduction我的名字是Rubro我是社交平台的發展方式專注於社交平台、討論整個社交平台以及基本上我喜歡旅行餐飲歷史我是一個西方人在地區所以,我住在地區不需要旅行我現在已經在發展多於25年每個社交平台發展社交平台的生活我是原來電子系統工程師所以,我在電子系統和電子系統因為我們在Machine Vision2002所以,我們很很無奈的技術但是,我們在這裡我們在傳掲程度社交平台以及這種我們會來看一下一些學習和圖書還有與他們的需要現代社交安排社交社交社交頂升這些新的能量都在最多新的要 Different around so we need to enable computing and compute capabilities where they are needed.So it is a perhaps say, desatur on where you need to deploy some servers and computing power there to you know, spread insurance resources, or just to keep continuing providing services perhaps during the course ship.甚至在電腦開發中,我們需要學習軟體,做過發展,尤其是用電腦發揮,這就是為什麼H為每一天更加重要而當我們討論EHCOMPUTY至少從我今天討論的 perspective它們最主要是這兩種特性首先,我們討論了Small Servers所以今天我們會討論手機和 microcontrollers因為它們通常都很類似的例如英文、SDK和其他雜誌的雜誌所以我叫它們更特別我們討論了Small Servers例如Intel Nook Server或者Limited Servers我們討論了2 cores、1 core還有2-4 gigs of RAM然後,另一個部分已經在我們 remote location也就是說,一些雜誌的雜誌都沒有連接到網路或者是連接到不可靠的網路或者是連接到所有的時間但這就是最少的情況所以,這就是我們討論的原因所以我們說,沒有電話的位置所以,在用來討論的情況我們討論了我們討論了我們討論了也就是說,如果我們要討論討論了我們討論了我們短註的這個地方其實我們討論了我們討論了我們討論了這就是我們討論的有些人已經在搜索搜索的產品例如Cubanet那些類型的產品然後更多的資料中心將它們搜索到不同的地方所以你的手提起如果你是一個發展你的手提起如果你是一個發展好 完美所以我們會正在討論所以一件事我們在討論搜索搜索搜索搜索搜索搜索那些類型的產品我們需要更多的知識在語言這是比較低級的語言沒有那麼多的策略像是手提起手提起但是我們還可以學習很多東西有很多人在做很有趣的有些項目的項目你會有Pyp and Code有些項目的項目很有趣但是在這個項目的情況下我們會講 more on the general purpose type of applications你能夠使用你的實際知識从你的背景,比如说,像我们说的,我们在最多的时间,在 cloud 基础上,我们会谈到这些,所以,我的主意是,如果您在做,如果您在做,发展,发展,发展,发展,发展,发展,在 cloud 和 AWS 和 any 这些,我们可以 translate  regular 专顿,发展,然后发展到了吧!电脑 jeritlub介绍到哦!让我介绍一下,我们可以役集内部的好的功能和需要。我们要把这些 Emmaே像是一隻蜘蛛一樣像是蜘蛛一樣有幾個人可以進行我們在學習的部分就是CICD可以使用應用的方式和效果影像的效果昨天我們討論了有些人會在街上用這個技術的設計他們會在街上它们会想起重新进行进行更新和改善这些计划很简单最重要的东西是有一个专业的建设所以我们会调整为何EDA是重要的和为何有一个障碍你没有建设的关系是可以的你需要有另一种方式分享資訊或資訊資訊我們會談論新的東西在所有的語言例如NATIVE BUILDS或Java還有最重要的HARVITAL CONNECTIVITY我們開始一件很有趣的事情這件事是關於Java的有多少人都喜歡Java在過去的好 完美你也知道Java所以很多時候你用Java有很多人都喜歡Java因為Java的印象是10、15年前它是很慢的很重的所以這就是我看過我的構思所以Java的構思我試著建立這個建築結果它已經超過了主要是在 serverless train這件事是很有趣的因為當Race of Lambda和serverless movement start它們需要有些特性的頁面和頁面的頁面會在 serverless 上寫所以我們需要有call start所以我們需要可以開始頁面的頁面的頁面的我們需要使用 less resources從 CPU 和記憶體因為我們要付出每一頁所以這很重要有一個頁面可以建立頁面的頁面的頁面所以Java was not doing that如果你是在Java你會記得所有的頁面像Wesphere和其他頁面的頁面的頁面的很痛苦所以很多人都要移除Java不過JavaFolks和Java頁面的頁面開始計劃這些頁面開始做一些一件事他們開始做的是這幾頁Micro ServicesFocus Frameworks那些至少從我們的頁面在 Red Hat是QuarkusQuarkus是一頁讓我們有有可能去建立頁面的頁面使用Java Language使用Java Enterprise Frameworks但同時可以建立頁面的頁面有更多的頁面的頁面有更多的頁面的頁面在頁面在頁面的頁面這樣我們可以減少頁面的頁面使用的頁面減少頁面的頁面減少頁面的頁面減少頁面的頁面減少頁面的頁面減少頁面的頁面所以在這個頁面我們可以可以轉換頁面使用的頁面使用的頁面使用的頁面還有有趣的東西我們可以從頁面的頁面使用的頁面是頁面的頁面在從頁面來的頁面也可使用有系統的方式方式免費也能適合用低儲備製作並接続裝置這也就是說我們用的頁面在頁面上卸貨也就是我們可以把所有的資源從我們做的那些在 cloud 和企業的發展中成為成為成為成為成為成為成為成為因為我們不僅可以用 Java 語言而可以成為JVMJava Virtual Machine但我們也能夠用那種用這個 同的用這個同樣的以這一樣的作用以這一個成為你的主動使用這一個用這兩個用這個用這個用這個用這個用這個這一個用這個用這個用這個用這個用這個而是有條件,由雞肉器做到順利上升於解鎖度即使是雞肉器,它是那麼重要的用雞肉器來解鎖的,才會有其他方法所以,我們會看到那些大小的那些雞肉器污染上雞肉器,我們會看到那些大小的雞肉器我們會看到雞肉器,我們會看到這些雞肉器这些计划的计划能够成为一个计划这些计划的计划能够成为一个计划你需要计划一个JDK或是Java Runtime Environment在计划的计划中这些计划不需要计划因为计划的计划正在拆除你的计划你的计划的计划正在拆除你的计划我们从计划的计划用Http Server的计划从计划的计划计划的计划在我的计划中现在我们能够计划计划的计划让我们能够计划的计划在同一个计划中这些计划的计划就是Gralbm的效果Gralbm是一个计划与一些人在计划的计划的计划我们在计划的计划我们在计划的计划中与计划的计划互动现在我提示我们是如何调整计划我们有两种计划最有趣是我们可以计划一个计划我们可以计划计划有利用用的用户或是可以以我們國內的用户成績作為主題並且可以用户成績作為主題如果在網上發行可以發行例如有一個單單開啟或是發行一部門的新的單單開啟例如MICRESHIP或是單單開啟或是PUBMAN 或是DUCTKER如果有一個單單你能夠達到所有的能量證明能量、可能性能量、能量等 depending on how much management you need在你的 local server你的 edge computing server你能夠解釋什麼樣的 infrastructure你想要的如果我有兩張照片我會先給大家看你可以看到這是一個很簡單的REST application所以如果你可以看到在左邊我的天啊我的天啊我的天啊我的天啊好這是一個 application如果你可以看到這是一個傳統的 Java applicationPOM XMLBasement Maven它有一個 code它是基本上是 Java code用記憶體、CDI你可以用任何的應用而這個 benefit here就是傳統 code這個 code可以運用 Lambda可以運用傳統 application server可以運用這些 native builds所以在這個情況下我們也有這個 benefit我們也有一些 Docker files可以幫我們創造這些Java based machines所以我們有這個 base imageJava development kit然後我們可以我們也有這個 option這個 Docker file可以讓我們創造Native build然後用minimum image從UBI 8映出映出REGISTER from Red Hat然後可以運用這個 application所以當你當你運用這個 application它是有時候可以獲得這個 kind of deployment所以在第一部分我運用這個 application在JVM你可以看到我們有資訊在JVM run你可以看到這個這個 server開始 in less than one second所以在0.8 second這也讓我們能夠運用這個 service然後運用port 8080這就是這十二頁我鐘我 16給我這個還是我可以不要我要它是在 your computer running on your application所以我会再次转转所以你能看到它开始转移如果我只是想转转一圈就会回到朋友的信息这是一个例子你能够利用一些网络的网络的信息如果我們看到這頁面的頁面是用來建立的我們會看到這頁面有兩種頁面這頁面是用來建立的所以這是NATIC的頁面這是第二頁面這是一頁面在JBM的頁面這頁面是用來建立的如果你看到不同的頁面它是很難看的它是一三頁的頁面是一種傳統頁面我們說了一頁面我們需要把這些頁面傳送到網頁這頁面的頁面是用來建立的用來建立的用來建立的150MB用來建立的這頁面的頁面是用來建立的這頁面是用來建立的如果我們沒有建立的頁面這頁面是用來建立的我們可以看到如果我們看到這個頁面我們說了一頁面這頁面是用來建立的我們可以看到我們可以建立一頁面這頁面是用來建立的這頁面是用來建立的這頁面是用來建立的我們可以看到這頁面這頁面是用來建立的但這個頁面是用來建立的我們可以看到這頁面是用來建立的如果我們用這個頁面我們可以說是TargetCode with wercus跟snapshotrunnerrunner這個頁面會用來建立用來建立MacMachine在NativeModeok so it'syou don't like that oneok well sorry about that it didn't workI think I replaced that with the Linux versionbut this is basically a single file, a single binaryso this binary file if you are running on a Linux machinethat it's deployed on the edgeyou can just copy that filethat you can see it's just 40 megabytesand you can just move that file systeminto your golden imageif your ensue imageit's gonna be configuring the serviceso it's a very interesting way toto get it started with this kind of applicationsso this is Brawl VMand so we're saying we can generatedepending on what type of containermanagement you want to useif you want to use containeryou can use podmanyou can use your native fieldsyour container imagesand suchand the idea is also when you haveno internet connectionyou can do some thingslike even driven architectureso even driven architectureallows you to implement patternsfor delivering datawhen you have a disconnectionbetween your service and thoseand most of the most commonways to implement EVAit's for example through eventsso you have used Kafka in the pastor any type of GMS brokeror traditionalmessage queues and topicsit's a way to implement patternslike start and forwardwhere you can process informationlet these broker, these bossesbeen able to take ownership of themessage or eventand then if there's a connectionbeen able to transfer that informationall the way to the next stageso in this casewhen we're talking about EVAyou can have things and benefitslike a single node Kafkaso remember if you're using Kafkaon the cloudyou get all the benefitsof these streaming application frameworksstreaming APIsand scalabilityand such kind of thingsbut sometimes when you're on a deviceyou can get a lot of trafficbecause of telemetryand devices sending informationinto your serverbut most of the timewhat you want to dois really take the advantageof the streamingso in this casewith the new developmentsin Kafkawe havewe have things like thissingle node Kafkait's basicallyit's the latestdevelopmentin the Kafka communitythat allows us to have thisthis server runningand this server is justone single containeron the serverso if you see the container runningwe only have one that's Kafkaso instead of havingall thesethree streamkeepersall these three nodeseverything that requires a Kafkayou can have all the benefitsof Kafkawith just one single imagethat can runeither as a containeror as a JBMand then you are abletoexecuteon thatthat serviceso if I just accessmylocalhost1992I can see that there'sone node brokerthat I can useobviously you lose things likehigh availabilityand the possibilityto spread acrossdifferent partitionsand replicationbut insteadyou get all the benefitsof Kafkaso you can still use thingslike a KafkaStreamCPIand othersso now that'sthings that I wanted toshare with youand there are other thingsthat you can addcamel extensions for Quarkusso it's basically frameworks runningon top of Quarkusthat get all the benefitsfrom thatwe talk about the differenttypesofobashipsand Kubernetesthat you can deployon this kind of applicationswhen you're runningon topof a Kubernetesyou get all the benefitsof things like operatorsthat allows youto implement newdevelopmentpartitionsregardingfunction as a serviceas well asall the benefitsof serverlessin your local systemso you canhave a mini-shapebeen able to run a containerthat allows youto run snippets of codeso just toto end the sessionand then perhapsopen the platformsome questionsthe idea is thatwe have been doinga lot of great thingsfor targetingthe cloudoriginallybut if you'rea developerthat has been doingapplication developmentin the cloudor web applicationsyour knowledgeis still valuableis still usablewhen developingedge applicationsso you cancreateapplicationsleveragebenefitsofcloud-nativedevelopmentyou cancreateand getapplicationsat a scale to zerousingthings likethere are a lotof frameworkthat we can reusethat areedge-friendlylike the onethat I mentionedlike Apache Camelthat it's embracingthings likeQuarkusand other kindof frameworksthat allows youto runon these kindof newbuildsand obviouslythe benefitsofthe data pipelinesthroughEDAlikeApache Kafkaor a traditionalgms brokerthat we cancreateand loadtype ofcarriersothere's a lotof interestingthings to doIencouraged youto joinfor exampleourdeveloperssidesodevelopers.reka.comthere's a tonof contentregardinghow to useQuarkushow to getstarted withKapkayou can use iteither as acloud serviceor you canthank you very muchI really appreciatethis quick chatandthere's informationon thetutor handlerif you want to follow mesmallbut I'msorry about the formatif notI'm going to be therein the standwith thedevelopmentprogram teamand if you haveany questionswe can certainlyadder them thereif notif you havesomething you canask nowI think we stillhaveanyquestioncommentis this usefuldid I showyou somethingthat was notin your mind beforeok coolawesomewho hastriedKapka yethave youtriedsorryKapkayeahQuarkusokcoolperfectso nowyou canstart blending themKapkais that stillbeing ableit's basicallya codethat islike 10 years oldso it'svery difficult tostart toport all thosekind of thingsbutthere's other thingsyou can stillrun thatin the containerand thenhaveQuarkus applicationsbeing ableto consumeand connectto thatsingle notekapkasoas you canseethere'sa ton of thingsyou canumthat would probably be goodsosurethat's perfect thank you very muchwhy did Isign outyou're trying tostack overwell and of coursegoogle chooses right nowto sign me out ofgoogle chose right nowto sign me out of everythingso I have torelog inyou this weekit's the waywhere did you choosealright and it's notgonna get me in herewell fair enoughthat'll be finealrightgo ahead and get goinggood afternoonmy nameis eric munsonand I'm hereto talk to youabout unicernal linuxthis is the subtitleis applyingunicernal optimizationsto a generalpurpose osor ours goes to 11first I want tosay thank youto the unicernal teamso I'mI'm on the listthere down thesecond rowwith that top rowwithout themnone of uswould be herethey bang their headagainst this problemfor a number of yearsto prove that itwas possibleDO staticonmoderndeploymentstypicallytake asingle usersegregatesegregateoh Wheels DOallto thatoverhead beinga general-purposeoperatedcleanUNICURNAL 會想到我們已經將一種應用程式進入 virtual machine所以為何我們不將 kernel 進入而是做一種大部分的大部分供應用程式並將它們連接到同樣的進行程式我們會創造一種 binary應用程式和 kernel 程式進入一種應用程式然後我們會直接進入Virtual Machine Monitor 或 Hypervisor我們會移動,我們會創造但它們不會創造Linux的軟體所以你可能會想到它們的應用程式不一樣如果需要一種應用程式它們不會進入你必須自己寫運用程式是很重要的你必須確保應用程式的應用程式是很適合的如果你是幸運的在你的應用程式中你必須實現你的應用程式有時候他們會說你必須寫一句話也許運用程式可以讓你想要重新重新重新重新重新例如在FRT 文件中他們選擇重新重新Memcache D API在FRT 上重新重新重新重新所以你必須用一個應用程式功用程式 turner 動一下作為尺寸能力Memcache D API在FRT 上Runtime debugging, if you've ever logged into a system and tried to say strace, or perf, or ftrace, or top, or use one of these tools to debug what's going on in your system, if you don't have the notion of a process, you aren't doing that. It's not happening.And when we work with the Linux operating system, we get millions of CPU hours of production time. This is code that has been tested through whatever it is you want to throw at it. Somebody's done it on this code base already.That's not true of a from scratch unicolonal implementation. Very few of them have any production time at all, and those that do, it's dwarfed by Linux's production time.So there was a lot of work put into making Linux into being able to use it as a unicolonal, and Allian, as I said, Allian and Tommy put a bunch of that effort in and made it happen.But towards the end, they started deciding, well, are we asking the right question, right?Do we really want you to have to choose between a general purpose operating system and a unicolonal and make that a binary choice?Or would we kind of prefer to have something a little bit more analog, someplace where we could pick and choose the bits of unicolonalness that we actually want to use for our project as opposed to saying it's on or it's off?Well, that led to a couple of design priorities when actually doing the UKL implementation.And these design priorities, they are, we want to maintain as wide an application support as we possibly can.We don't want to force you to reimplement everything in OCAML.We don't want to give you parts of the API. We want to give you the whole thing.And we don't want you to have to reimplement everything depending on what we have provided.A big one here is Linux's HCL is huge.Modern applications don't just use a CPU and memory anymore.They're going to be touching things like storage arrays or GPUs or TPUs on the side.And if you don't support those in your unicolonal, you're not going to be working with these modern applications.And we want to keep Linux's HCL, so it's good to just, if it's already there, go ahead and make use of it.And finally, there's a wide ecosystem of things that come with supporting Linux, including a huge development community.So, somebody out there knows the subsystem you're having trouble with.Whatever that subsystem is, it can probably help you fix it if you need it.There is a rich set of tools and libraries already built on top of Linux.And we don't want to throw any of that away.And finally, those debugging tools I mentioned knows this admin is going to stay working for your company.If you tell them they can't use any of those tools to debug production problems.So, with that, we come up with the UKL base idea.This is the base configuration supported by UKL.Now, you'll notice a couple of things.First of all, we have the application marked with, actually linked with the kernel,as we did in the other unicolonial illustration.However, you'll notice that in the previous one, that standard C library wasn't there anymore.But here it is.We maintain compatibility with glibc.And so the interfaces you're used to working with are still there and still usable.And finally, we don't discard the notion of the process.In UKL base, you are still allowed to start unprivileged user space processwhatever they would normally.So, you want to bring up the network? fine.You can do that.You want to start systemD.Great.You can go ahead and do that, too.File system mounting, not a problem.All of that can still happen while you have your privileged application running in kernel space.Now, once we have this base model working,this allows us to explore a couple of unicolonial specific optimizations.And the four that we looked at for the paper are bypassing kernel and entry exit code, avoiding stack switching,using ret instead of iret for all transitions from kernel space to user space,and then applications interacting with the kernel directlyas opposed to through the system call interface.For this talk, we're only going to talk about the first and the last.And I know they sound similar,but they are very different in what they require from the developer to actually implement.So, first an explanation of bypassing entry exit.I know there's a lot here.Just know that the call chain is long.And we get fromthis is the call your user space process made,read down to doing the actual work of TCP received message.The stuff in the middle,just know there's a lot of boxes.Okay.So, when you make a system call and you transition in,the kernel does several things before actually dispatching the system callthe thing you've actually made.It does things like checking to see if you need to call the scheduler,checking to see if there's any pending schedule,I mean,signals,doing RCU epoch handling,and a number of other bookkeeping tasks as well.However,these bookkeeping tasksdon't need to happen on every transition.You check to see if they need to happen,but you don't necessarily do work.So, with our bypass technique,we look something more like this,where read calls directly to case is read,skipping the bookkeeping work.Now,it's not quite like that,because remember,we wanted to maintain 100% compatibility.So,this actually happensis a combination between the libGlibCand the kernel itself,but it doesn't require the user to make any changesto their codebase whatsoever.You can control the frequency of this skip,but we'll talk about that in just a second.Okay.So,that wasn't if you didn't requireany code changes from the userand we got,protentially some performance benefit.Now,for the deeper optimization,for one that requires more user input,more developer input on this,we talked about calling directly into kernel functions.So,in that call chain,which we'll get back to in a second,there are six or eight functions in the middlethat are just disambiguation.They're getting from the fact thatthe read and the write system callssupport the notion that everything is a fileor a file descriptor,and underneath it will decidewell,what kind of file descriptor is this,what kind of socket is this,etc.as we get down to the actual codethat's going to do the work.If we know,a priori,what this write callis going to resolve to,why don't you just call it directly.Just skip all of the middlemenit's not quite that simple.Sure,you have to transform from the file descriptoryou had to the sock structor the iovex or whatever it is you need to do,but you certainly can skipmost of that intermediate code.So,you left with our same call graphfor read here,and instead of our bluewhich jumps a little bit,we jump all the way downinto calling tcp receive messageinstead of calling read,which can leadto a real significant savings.So,let's look at some benchmark results.So,for this graph,we have takena benchmark called lebenchwhich is really just made to measurethe how long it takes to execute a system calland integrated it with ukland three different,well,with two flavorsand run on a baseline.Now,the three setups we ranare straightlinux,ukl just linkedthe application into the kerneland finally we turned on the bypassingthat I talked about at first.So,this is the skipping the entry exitfor nine out of ten system callinto interactions.So,you'll see that just by linkingthe application,we see a modestimprovement and performance.These are average system call timesjust to make that one system call.When we get to actually doing the bypassand we're skipping those checks.Now,we're not skipping it entirely.We're only skipping it for nine out of ten interactions.We see a huge performance improvementin all of these system calls.But the real magic,and we'llget to this in a second,is actuallya complicated graph.This is going to be talking aboutrunning redis on linux on uklbypass and then also on theshortcutting.But what you needto take away from this are thesetwo numbers.These are the twonumbers that we care about.Theyare the average or p50 latencyfor a request and the p99 orthe 99th percentile latency.Right.Sowe see we have about two millisecondsand about three and a quartermilliseconds on linux baseline.Whenwe just link redis in anditempercent reduction in p99as well.But when we decideto reach in and call thoseTCP functions directly,we see23% improvement in the p50latency and a 22%improvement in the p99.Andthe other real win that needsto be called out here,as you'llnotice that with each of theseoptimizations,the distancebetween those two numbershas shrunk.So we are cuttingdown on the window of latencieswhen we run with thissetup.We also ranmemcashd with a number ofthe same setups,the linuxbaseline,the ukl linkage,thebypass and the shortcut.Andwe see similar improvementsalong that spectrum.Right.Withbypass and shortcuts showingreally significant improvementsin the number of queries wecouldsatisfy under ourhalf a millisecond SLA.Okay.Sowe talked about ours goes to11.So let's get to themultiparty computation applicationdeveloped here at BU.Butthe key thing we need to knowfor our work is that it doesa little bit of processing andexchange a very small tcpmessage at every round ofprocessing.Right.So it doesa lot of very small traffic.Right.Whichis normally actually reallyhard on linux because you bumpinto all kinds of things likecongestion control and thenwhether or not you're awakeand can receive the interruptionfor the three setups.We ranlinux the baseline,thebypass and the shortcut.Okay.Nowlinux performed okay.They werebutthis the operator that wechose to test had been sidelinedby the secrecy group because itis too slow.We can't actuallyuse this in clear planningbecause no one is going toto rely on what we got out ofthis.Right.Now for 100 rowswhich is not really all that bigwe saw really good improvementfor other things that we'dexpected to see anywherebetween 25 and 18%18 and 25% of improvementfor the bypass.Howeverwith the shortcut we didn'tjust see a few percentagepoints.We saw a 95 timesreduction in runtime.Right.Nowthis sounded crazy.Likeno way.That's not real.95xthat's not a thing you couldactually see.So we didsome investigation and wefirst noted that wenoticed that dramaticreduction in the number ofinterrupts being generatedby theNIC.Right.Andwhat we came to find wasthatbecause we were movingalong at such a rapid paceand neverschedulingtheNIC never got a chanceto get out of polling mode.Soit would say just feed datato us as we came back.Itwas ready.You go aheadand take it.No need to putthe application to sleepand then wake back upagain when there was dataavailable.So we skippedpaying the cost of all ofthat and just ran tocompletion effectively.Okay.Sowhat's next.The UKLproject is going to beso we're currently workingon getting the patchesready to be posted forupstream.We hope to dothat within a monthor six weeks.It'staken some beatingand we're going to havesome people to convincedefinitely.Butwe're ready to dothat.We think two weeks.Okay.Myadvisor says two weeks.Sotwo weeks.But some ofthe more interesting workthat we have here islikenow that we havethis result with secrecyhow can we goeven further?Right.Canwe make use ofmemory management withinthe kernel instead ofinstead of usingkeeping user spacearenas and mallocinvolved.Can we justcall vmalloc directly?Right.Weare looking atevent-drivenapplications.Sothat's very generic.Let'stalk a little bit morespecifically.The kernelin its nature isan event-drivenapplication.Thingshappen,the kernelresponds to it.Right.Itdoesn't do things unlessit's been told to.Itjust sits idleotherwise.Nowmany applications forinfrastructureare writtenthe same way.However,because we presentthis notion ofyou're just runningon a vax.Andthere's this impedancematching betweencernel eventsand user spaceevents that goesthrough the threadingand the scheduler.Butwhat if we tookthat out?Right.Howcould we improvethe performanceof these event-drivenapplications by tyingthem directly intothe event interfaceof the kernel,as opposedto having to go throughthe C-layer.We'relooking ataccelerating containerdeployments.Specifically,we're interestedin talking withacautic containers folksabout how we mightintegrate that with UKL.Andfinally,we're alsolooking atinterpreted languagerun times.So,forthis specifically,we werelooking at JavaScriptand WebSM.Butwe're alsointerested in Pythonbecause we feel likethese might beinteresting notionsfor things like serverlist deployments.Allright.Well,thanksvery much for coming out.Nowdo we have any questions?SoI haven't done anylooking into it,butwe're open to talkinganyone that wantsto talk.Yeah.So,Ithink thatthis notionofcateric containerseems like areally coolman.If youwant to dothe baseenvironment,I'mnot sure.Okay.Theone thing thatIthinkEric didn'ttalk about is thatthere may bethe notion ofit's just genericallythe abilitynow to makethe applicationcoceternal.Weneed to usefile system.Thatmight be abeautiful waythat wouldaccelerate thatifit's no waywe canaccount totheactiondirectlyWin0.Sothefile systemcoceternallydateswereallyriddatedthemask.Sowehaveworkedwithsecrecy.Weranthe benchmarkingtests.Wehave gottenNGINX running.Wehavedone我們也在說著我們在說到The SethRados Group在他們進行誰派的人但是他的要求是打造對於UKL但是不那麼大致布土地區沒有提示UKL是為了範圍 而解決 nasty你看這個第一點不只是能開始這是一個一個想法解決範圍的磚段這就是So the point I tried to make with the calling off the two different optimizations that we covered was that you get the baseline and it's really very little effort for developers, right?There's some relinking, some recomplation relinking you have to do and then you've got it, right?But if you want to reach deeper, you can put as much effort into going fast as you want, especially now that you have access to kernel internals, you're not limited by the API presented to user space, you call whatever you want.Yeah, so this is an idea that this is an idea that the team has kicked around about adding entry points that would easily take the same API as presented in user space but do the short cutting for you for things like PCP connection.We don't, we boot bare metal, right?If Linux supports it, we still support it as well.So if you want to bring in a bare metal, most of the testing for this talk was done on bare metal machines, right?But it also can work with a hypervisor, but I think it would actually be a very interesting testbed, right?Because you care about all of the things that Unicernal can provide.I think it wasn't really fun, but not about the benefit of such, but it's actually real fun.Well, this gives you a lot, like we found that it's just naturally more deterministic.It's easier to control those things, plus as an application now, you for example, have direct access to RCU, you can do it like that.So we think that there's, that could be one of the potential gains is to allow, you know, all of the links to this, but now we have to have an application that's actually got more deterministic response.So to talk about, to talk about this a little bit in more depth, the error bars on this graph are the coefficient of variation for the runs, right?And so you'll notice that as we apply UKL techniques, they get, there's some variation in the very small ones, but the Linux variation is giant compared to either of the UKL ones.So you are paying significantly less in jitter and in for every call that you end up making.It's much more deterministic in how long it's going to take.So we currently only support x8664, but I mean, especially once that we get upstream, we hope that that effort will look much like the KVM effort did, right?KVM got upstream for x86 only at first and then branched out into Power and R.So we hope that follow much the same path.So we looked into it for the paper and unfortunately a bunch of the OpenCL stuff still requires proprietary dynamically loaded libraries, which are not something we support.We have to at least be able to recompile the code and relink it, right?And so we couldn't get NVIDIA or AMD to turn over those libraries to us.So I don't know, if anybody's got an in there, we'd be happy to talk to them, but.We didn't have to do that. We asked them to create static and link that.Right.And so they would have to relink that.But if I could just recompile them in a way that they would require that.Right.No, we're not asking them to change their open source policy and release all this,but just a relink and providing us with a static object we could use.And so far they haven't taken this up on it.That would be huge.Yeah.If you know somebody that could do that.Perfect.Yeah, I mean, we can even give you, give them an experiment to run where they.Certainly.So, even if they won't give it to us, we really like to show them.Right.Because theoretically it's all there, right?We haven't actually verified the GPUs or a function thing yet.Yeah.Okay.Where do you see this going on?So that's,that was actually one of my motivating,like why is this work cool and why I wanted to join this lab was thatmost of the AWS stuff is exactly this kind of deployment.It's one application you care about,perhaps with some supporting user space,but generally it's one thing that you're doing.Right.And I could easily see this being the way that people choose to deploy that one application.Right.Indeed.Yeah.All right.Do we have any other questions?Fantastic.All right.Well, thank you all very much.Well, thank you.Thank you.Are you going to have audio?Do you have audio in your presentation?Like videos?No,no.If you do,you have the...Okay.Sure.Thank you.Thank you.Yeah.Okay.Thank you.Thank you.Thank you.Thank you.Thank you.Thank you.Thank you.Thank you.And there's a,there's a lab.You just wanted to walk around.Chasing,testing.I just like to set up here.Do you need an appointment?No.Thank you.OK.Sure.Thank you.Sure.Thank you.OK.Welcome everyone.Thanks for coming to our session.So.My name is Ur and here is Amin.我們今天會展示你們的合作資訊 for object storage這是一個項目我們其實是一個大團隊我們是學生從巴斯大學學院的學生我們也在這個項目中在Red Hat Fox with the SEF team所以data centers like typically have someform of like hierarchical networkand asand bandwidthget constraintas the communication has to traverse higher levels in the topologymassive like demand of data today as a result of massive demand for data todaydata centerscreate a single shared mutable data lakesand these data lakesdesigned to store all the data for all the clusters running in the data centerand theyefficiently store the data running on high capacity and slow disksso this like popular architecture single shared data lakecan result in like a poor performancethen all the clusters compute clusters access the data lakeover the limited very limited like bisectional bandwidththey see performance aggregationsalso these data lakes like typically designed for or optimized for particular access patterntoday we know that there's like different applicationswith very diverse demands they're all shared thesedata lakesand finally as I mentioned they run on likehigh capacity but slow diskssocomput clusters like frameworks or clusterresult this issue by deploying cachingand these caches are great becausethey bring locality they cache the data however they have two problems the first one isthe first one is they don't allow efficient sharingso if both cluster access the same data we have to redone the cache this data on these expensive cache resourcesand in today's like virtual environment the clusters like grow and shrinkgrow and shrinkto match the demand of applicationsand if we have for example a dirty data on one of the caches let's assume we have a one terabyte of dirty datait takes tens of seconds to write that data back to the data lake before releasing that dedicatedcache resourcesso like in 90s early 90s like 2000s there was this idea of cooperative caching which is like initiallydesigned for distributed file systems to extend the file systemacross the machines that access itso rather than like deploying clusteror framework cachingbasically in our research we explore can we use the same techniquesto extend the data lakeover the data centerso we implementor wedesign tocache architecture called D3 and D4nand basically what we did is we distribute SSDs all around and both design like we distribute SSDs all around around the datacenter and form a cooperative cachingso this designnaturally caching the data on the access side of bottlenecks sowe provide localityit's naturally support sharing because if somebody already bring the data to the cacheremaining clusters can access the same data so we are not wasting any expensive cache resourcesand also because these caches are not dedicated or silo to particularcluster we cannaturally support cluster elasticitysowe integrate both of our design in theSEF data lakeand SEF provides a piece of software called Rado Skatewayand Rado Skateway is basically provides an S3 API for users or applications so users can access the object storeand what we did is we take thisRado Skateway and explore can we integrate the caching functionality in itand then we can distribute this Rado Skateway all around the data centerand form the cooperative cachesowhen we like start this project we weretry to like explore can weintegrate this caching into the Rado Skateway because Rado Skateway is not designed for being caching it's designed for something elseand we want to make sure we don't introduce any major changes to the code we want to make sure we don'tmake any changes to the existing flowobviously we also want to make sure is this going to like perform wellsowe come up with a very cleandesignbasically we avoid to maintain any global statewe go with a multi-layer architecture like a CPU caches so we have a layer one cachelocal to your rack layer two local to yourcluster and we have a layer tree which is likeshared across the entire data centerand we use consistent caching to locate the blocks on the higher levelsand we avoid anyand we avoid maintaining any global stateand the D3 and alsoonly supports a non-durableright cachesowith this like clean design and takes the RATATself engineers we were able to like upstream the codeand by running a lot of experiments we also show that actually our implementation was super fastit was able to saturate the like maximum speed of SSDs and NICand we also run like some real workloads we run a trace fromfacebook map reduce clusterand we we show that like D3 andcan improve the performance of workloads up to three times it can reduce the reduction on the data lakeby four timesand then the next obvious question alrightnow we have the D3 andcan be now for example support write request because there are a lot of like intermediate data setit doesn't have to go all the way to the data lake and we likekeep those data sets make sure they live and die in the cacheand can we also support wide range of applications because applications havefor example serverless applications access like in small granularitybig data applications access the data in much larger granularityso in D4n wedesign a resilient distributeddirectory which maintain the global stateand we also provide a durable write tierso instead of like implementing the write tier from scratch what we did is we take the existing OSset OS decode implementation anddeploy it as a write tier and layer on top of the data lakeso this way we were able to replicate the data redundantlyeither using erasure coding or replication on the write tierwe were able to use the global state in the directory and we implementanoble algorithm where we can able to compare the value of block in each different cacheandas a result we see significant performance improvements sowe run the same facebook map reduce traceand in here you see we compare the D4n with D3n alsowith the existing state of art solution which is alexioand these like yellow and red bars are alexio with two different configurationand the performance improvements of D4n come fromstuttings first it swing write tier with erasure coding which supports writeand provide very high throughputthe second performance improvements come from the novel algorithm that we usebut which exploit the information that we store in the global directoryso D3n as I mentioned is already upstream so you can access itit's on thethere's already documentation page on the web pageand right now we are on the process ofupstreaming D4nand we are trying to integrate the D4n with the RGW's new feature called Zipperso hopefully it will be available toeveryone's or brother community soonso as Ur explained D3n and D4n are storage cachesourcite cachesand this puts us in a unique positionwe have access to a deep sea of informationinformation like global state through directorywe have past and current access patternswe have also information about environmentwhere is the datawhere is physical location of the datanetwork band if you sayand all those informationbut before we get to have to exploit this informationwe should consider one important thingthat we are out of cluster nowand we are losing some informationthat we can get from clusterslike information from usersframeworks schedulersso the question iscan we make it up for itturns out we canwe developed a carieswhich is a caching mechanism for analytical frameworksas we knowanalytical jobs like spark jobconsist of graphs of tasksthis graphs or tagsshow what data is going to be accessed in the futureand what is their dependencyarys extracts these tags from the schedulerand through a simple APIask d4n to pin the datathat is already in the cacheand pre-fit the datathat is going to be accessed in the futurewe compared carieswith two state-of-the-artcaching policies for analytical frameworksMRD and CPCCMR is our main caries algorithmand we ran a TPCH querieson a datasetwe inspired by a tracewe got from an industrial collaboratorand as we can seein different types of queriesseeing ad hoc linear starall the different kinds of queriescaries gets better performanceso now that wesolved that problemthe question ishow can we exploit the storage site informationfirst of allwhen a cluster is shut downit loses abilityand resources to keep the informationof its access patternsbut as a storage site cachewe can keep this informationwe can store the metadata along objectsand we can have this informationfrom creating an objectuntil the current timeso that's a huge opportunity for uswe can design informed caching policiesfor example for tableless frameworkswe know that there are a lot of small rightsso we can use a coalescingto make a big rightone big rightinstead of small rightsseveral small rightsand get better performanceswe also can develop intelligent algorithmswe canas I said we can have past informationso we can knowwhat data will be accessedin what time of the dayinstead of having a passive algorithmlike LRUwe can have active algorithmsthat knows what data should be cachedor purified into the cacheand these are all not old theoryD3 and D4 and CARESall are developedand deployedon MOCsMOC Alliances Infrastructurewe have 50 petabytes of storagethousands of serversand all of them are running on MGHVCCwhich is a data centerso it is realand as UR explainedD3 and it is upstreamD4 and it is getting upstreamso you can get informationfor these projects through these linksand we are here forif you want to discuss more about these projectsthank youone thing is I think we are going to layabout these togetherhopefully we get D3 andis so whenEric was talking aboutbeauty kernels stuffall RGWis a serveris there any event requestswe think we can actuallyget this all running on UKLand it is an optimized serverbecause it is just especiallythere is going to be assisteesand performance is going to be really criticalso it is a really natural targetbecause this thing is doing very fast operationsto local assisteesas a result of tons and tons ofrequests coming inover the networksoit is going to be kind of tied togetherthat is one ideafromhow many of you doesn't understandthat we are talking aboutbasically we are doingthe same thingwith the logic source actionis there any impactcan it impact likemary planningthat we are takingwith the localsportscowards,maryI meanright now that is ourwhat we are trying to dowe can trying to exploitthe existing informationstoring the directoryfor schedulersin the like sparkor Hadoopso they can maybe rearrangetheir schedulingor query planningif you knowif there is already existing datacached dataso maybe they can prioritizethose queries or tasksso that is our likekind of theone of the directionsthat we want to goactually I meanto be really closethere are people that work on thosebecause if you go both directionsrightis that if you knowwhat is beingwhat is in the cache right nowthen you can usedesign your querycache to offlineright now we are onlygoing in one directionwe are extracting the dagof what decisionit is already madeabout thethe stuffso we have asmall visibilityof the futurecaching is waysyou are in the futureso we have asmall visibilityfutureand that is theadventurebut actuallyit can go both directionsrightif there are someand exposingwhat iscacheand where iscacheand Idon't havemodifiedbutthe interestingabout itis thatit can actuallyrightcachethenit can actuallyrightcachethen it cancachethen it cancachethen it cancachethen it cancachethen it cancachethen itcachethenit cancachethenit cancachethenit cancachethenit cancachethen itcancachethen itcancachethen itcancachethen itcancachethen itcancachethen itcancachethen itcancachethen itcancachethen itcancachethen itcancache你喜歡在另一邊能夠提供資訊進入資訊在其中一些計劃所以Locality今天被提供了在開設工作的方式在開設工作的方式唯有Locality是用機器來調整至少在我們的電腦中有限量的資訊在外面Locality是很重要的在電腦上或者是在群体中的建造所以我想要把这些订阅到我们的订阅和我们的订阅一些更多的订阅在在 Microsoft是在发生了一个硬件的就是说我们就要做一个特别的硬件在硬件上是有一个有一个硬件的What's the right interface to expose that with guidance?Yeah, and I thinkYeah, in like today's like people doesn't likeGo and just stay in one environment like they haveThey open computational different regions, different cloud providers and you knowAnd I think this is like little bit hard problemI should be prefetched data and if you're going to prefetch itOr we can be prefetched from maybe other cachesThat sits on other regions rather than back endOr should we ship the computeSo I think that's an interesting area that all of us like alsoMonoburg and explore moreIt's just bringing the pictureI think you do it right nowAlexio things like that just in these cluster cachesThey don't have visibility anything outside of thatThat's what we're saying at the beginningWe wish on the data center levelSo we have storageWe can actually have the physical locations and stuffAnd see things across themAnd some of the colleagues we're working withFor example, Two SigmaThey have these really complicated workflowsWhere they actually pull data inFrom the market at a certain timeIt's a well defined time where that data is going to processOr everybody is now going to access this data setRight, so all this stuff is invisibleIf you kind of introduce caches within the clusterAnd like one of the things I mean point toIs looking at the history of the analyticsBased and came out of IBMThere is very very reputable behaviorsThat happen over large time scalesYou know that everybody is ignoring right nowWe're just caching things withLRU or UDA and stuffAnd when in fact at the data center levelThere's these real workflowsWhere something creates something at a certain timeIt's processed by a bunch of things the next dayAnd massive amounts of data accessAre going to create the workflowThank youI admitted to getting this done beforeSoYeahI meanThis thing we have a lot of lecturesIn a couple of yearsSo we're just going to take thoseConvert them and start any infrastructureI thinkI hate working for other peopleWe don't knowBut writing this isI want to show you that's the pointIt's reallyMost of the time it's said so farYeahSo thisI meanI mean it's so proficientRightSo there's thisThere's thisRightThere's thisRightThere's thisYeahSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSoSo