Hi, my name is Yuki.I'm a principal architect from Datastaxand a long time Patrick Sandler Committer.Today I'd like to show my workthat I've been working on for quite a whilewhich is an open telemetry integrationinto your Patrick Sandler.So how many of you know aboutthe open telemetry project?And do you already deploy open telemetry to your environment?No? Okay. Yes, thank you.So yeah, this is the talk aboutusing open telemetry to sendall the telemetry from Patrick Sandlerand I'd like to show my demo in this sessionso you can maybe in the futureif you deploy Patrick Sandler with open telemetryyou can get much more insight in one place.So open telemetry, this is an open source projecthosted on CNCFand historically it is a merging projectbetween the open tracing projectand open sensors project.It provides an observability frameworkso you can use open telemetryto instrument your applicationit is a language agnosticso you know if your application iswhether it's written in Java,Python or Goyou can use open telemetry to instrument your applicationand it also provides a specificationto send those telemetries outto any APMs that supports open telemetryso this is taken from the other websiteand yeah, it's just what I saidand the other important thing for this projectis it's vendor agnosticwhether you use open source promisesyega for tracingor if you prefer to use commercial offeringslike Datalog and Splunkthey also support open telemetryand choose between those productsto monitor your applicationso here's why I decided towork on this my own personal projectso as a consultant at Datastaxpart of my job is totroubleshoot various problemsin customers' clusterso I need the information from the customers' clusterso I ask a DBAwhat query the application is runningor can you show me the metricsor can you get me the logsand often times, yesthey set up the monitoring systemto get all the metrics from thesunder clusterbut when I ask about the application querythe people I interact withmostly the DBAsand they don't know the answerso they need to ask the app teamfor the schema and querieslead and write patternsand that takes time to get me the informationand the logsif the project is reachingbudget they have a log collectorsystem set up in their applicationso they can go into the central placeto look for the logbut most of the timethey don't have those luxury stuffso people need to log into eachnode in the clusterand get me the logso before I start analyzing thingsthere's a huge gapover all the timeto collect those informationand when those are in my handstart piecing all those togetherto do the troubleshootingsoyeah, thoseapplication logsI mean the query logsthe metrics and actualsunder node logsthose are the key tothe systems of observabilitysoif you look up the observabilityand search for the webyou can see these keywordsrepeats of observabilitywhich are the tracing metricsand logsandthese are the thingsactually already Cassandra hasin a long timefor the tracingwe have a tracing on capabilitysince Cassandra 1.2andif you turn on the tracingand run your CQL queryit logsall the internal stuffinto the Cassandra tableyou can take a look atwhat's going onwhen you learn the CQL queryall the metricsCassandrais instrumentedwith a metrics libraryand it hasa huge amount ofmetrics you canextract it fromandbecause the Cassandra is Java applicationthose are by defaultexposed throughjavasyou knowtheformal jmexmethod to get the dataand the logs Cassandra hasa logback librarybuilt into it and by defaultitemitsthe unstructured logto the filegetting all those data outfrom the Cassandra clusteras I describedit's very hard for all the people sometimesand tracingthere's no way to exportin the current stateit is in the Cassandra tableyou can just dump thosestuffinto csvbut still you need tocollidate with other stuffby handit is exported to jmexbut you need tosomehowinstall jmex exportersto get the data outand the logsyou need the filetailingstuffto extract the logs as welland it is unstructuredso sometimesif you get the exceptionas all datayou needstuff to actually preservethe stack place from the loghere comes the open telemetryas I mentioned earlieropen telemetry definesthe way to exportthese 3prs of telemetrythrough thestandardized protocol so integratingopen telemetry into Cassandrawill make our life easierto extract thosereservability telemetriesinto theexternal systemso yesif you have open telemetry capabilityyou can exportthese 3prsof observability toany APMs you are usingkindlyso I'd like to show the demothat I builtor I've been buildingfor a year nowandsorry this is a small diagrambut you can see this in mymy GitHub repositorythat I can show you laterbut I set up the Kubernetes clusteron AmazonI deployed a 3 nodestandard clusterthat I also instrumentedwith open telemetryand I deployedthe web applicationwhich is built usingand all thoseapplication and standard nodedeployed withopen telemetry collectoras a sidecar soeach application serversor each standard node can sendtelemetriesto that collectorand there's also acentralized open telemetrycluster thatfor those telemetriesto outsidethe systemand for this demoI set up thiscentralized open telemetrycluster to sendthe jazelle telemetryto dataloghoneycomband aawscloud watchso I'd like to demomy applicationhereso this is aspring bootapplication andit talks to the ksandra andtheseproducts informationfrom the ksandra cluster soif Iclick on the t-shirts category it showsall the t-shirts and if I click on the product thenyeah it shows the product andthese data are fetched fromthe ksandraand every time Iaccess these URLsit generates the telemetryboth from application as wellas a ksandra nodeandthis is thedatalogdashboard that I set upand it isgetting the data fromthe open telemetrycollector in the other time and if Ifor example accessa productsomaybethis is atrace from the applicationto get the product pricingand if I go into thisas you can seeyou can see the tracingfrom your application as well aswhat's happening in Apache ksandraand you can seethe query that this applicationisexecuting as well aswhat's happening oneach node when the ksandra isdoingwhen executing this queryin distributed wayright nowit isemitting the same information asa Apache ksandra tracingthings likeleadrequest messagedeceivedthose are the exact same eventsthat you can see in thethingif you go intothe metricsyou canget the various ksandrametrics in the samedatalog applicationsoyou can showthe compactionby the compactedand you can create the dashboardfrom these metricsand I also send the log filetheksandraall the log filethat's emitted from every node in theksandraso you know using thisopen geometry instrumentationyou can sendthe three keychametries toonecentral placethe same geometry is actuallysend to othergames like honeycomband you can seethe tracingin honeycomb as wellso these are theother url accessand as you can seeit shows the application tracingas well as the ksandra tracingin one placethe same thing for cloud watchyou canget the tracingfrom AmazonXAjust like I showedin the otherAPM applicationssoyou can seeintegrating open geometry into ksandrahasmany potential foroperators of ksandrato troubleshootthe internal of ksandra in one placeopen geometry also has a capabilityto coordinateof these telemetries to the otherso if Iput more effort into this demoand put moreimplementationto thisI can actuallycorridate tracing metrics and logsin more meaningful wayso the highlights of the demo isapplication and ksandra send telemetriesby openstandardized open telemetry protocoland the collector will export it tovarious APMsso you can monitor these telemetriesin one placethe code for this demo isavailable under mygithubksandra 5.0modification with open telemetryand the application side as welland the last onealso has the same diagramthat I show in this sessionso let's take a look atthe implementation sidethe goal of thisimplementation is actuallyenableoperators to easily turn onthis feature and sendingthe key telemetries tooutside worldI'd like to addonly one ksandra democonfiguration open telemetryenable andif the operator turnson thisksandra can immediately sendtelemetryoutsideto the clusterbut of coursethere's a couple of things you want toconfigure so I usethis open telemetry SDKold configure extensionso thatwhen necessarythe operators can customizetheir open telemetrypropertiesas they deploy the ksandraso in my demo I useda environmental variablesto specify for exampleopen telemetry service nameor some sampleconfiguration like thisbut you can also usea java properties toconfigure things soyou can put the sameinformation in jvm.optionsfile toconfigure these behaviorsand to enablea open telemetry tracingI need to change a couple of thingsso the first thingis takingthe same tracing contractfrom the applicationso for thatapplication needs tosend the tracingcontracted informationto ksandra and ksandraalso need topropagate the tracing contractand there are a coupleof places that needto be modifiedone is a native protocollevelbut fortunatelycurrent native protocolsupports adding customheader to itsmessaging soI can just usea standardized tracingcontracted format thereto get the contract from theapplication and propagate itto ksandra nodeandfor the messaging betweenthe ksandra nodeswe already have a ksandra tracingcapability sowhich does the similartracing propagationbetween the nodes soall I canall I neededto do was to add the newmessage typecontracted andjust to realizethe necessary attributesinto the messaging headerjust like a currentapplication tracing doesto putthese tracingcontracted in the sametracing IDand for the metricsI don't thinkchanging everythingand instrumentthe open telemetry takesvery very long time andit needs tomake huge changeto the current codebase sowhat I did here is todisten tothe metric registryand as the new metricis registeredto do this currentmetrics libraryI can justput the listenerto listen to the metricsandthe open telemetry SDKyou can justpull the current metricsand expose it tothe open telemetry protocol soI use that tonot change thecurrent codebase butonly add one morelayer to extract the metricsand for the logsthe ksandraI use logback library towrite the logs to fileand open telemetry java projecthas this logback adapterto actuallychange the logbacklog event intothe open telemetry log eventobject and send it tothe open telemetry protocolsoI added one lineinto the logback XMLto use this open telemetrydataand that's alland it just starts sending the log tooutside worldso all of thesechangesthat I mentioned in today's sessionI'm drafting the officialCEPCEP32 to make itthe default ksandrafeaturein the future release of ksandrait is still in the draftstate I started writinga year agowhenthe open telemetrystandards are stillevolving soI need to updateto the current state of open telemetryand I need to update my codebasetoapplyto ksandra trunkbrunchbut all the informationthat Igive todaymore described in thisCEP that I'm working onsothat's my sessiontoday and I'd like tohear more from youabout thisCEPand you canleach me with thisYUKIM handleon the patch stackaudit tab or twitteror you can just email meto the Apache mailso nowe can discuss moreon the topicto makethis official ksandrafeaturethank you very much for listeningand if you have anyquestions I cananswerthe questionis about the latencyimpact when you enablethe tracingI say yes butI didn't do anyperformance impact analysisbutas Ishow here you canactually tweak yoursampling rate for exampleto get the tracinghow frequently you wantto generate thesemetrics soyou can tweak thattomake the performance impactminimalthe question is does this workon 4.xwhen I started thiswork it wasfor 4.0I have a 4.0version of demofrom last yearbut I updated this touse 5.0for this year'spresentationbut I don't have any4.1implementation for thisbut it can be easilyadaptedI think theporting from 4.0 to 5.0was very minimalchange soif there'sno questions thank you very muchfor coming to my talksee you soon