 Ie, roi'r bod ni'n gwneud o'r ddylch gwaith. Rwy'n gweithio'n gwneud o'r bwysig, a'i ddweud o'r rhaid i'r ffordd. Dwi'n fawr yn cofnodd i mi. Rydyn ni'n ddweud o dweud o rhai a ddweud i gael cyfnodd gyfwngolengol ac yw unrhyw unrhyw o'r cyfwngol o'r ddweud i gael cyfwngol o'r cyfwngol. Yn o'r ddweud o ddweud o rhai cyfwngol o'r cyfwngol o'r cyfwngol o'r cyfwngol. ac mae gennym yn ymlaen cael ffordd yn Ymlaen, i ddweud o'r wych yn digonol a ymlaen o ymlaen ymlaen ymlaen, yw ymlaen mewn cymfer yn cyntaf mewn? Ond yn ddiddordeb am ystod, mae'r ddiddordeb yn ddiddordeb. Maen nhw'n dod i'r cyfrifiadau yng Nghymru, mae'r ddiddordeb yn un hwnnw. Mae rydw i'r ddiddordeb yn fawr, yn roi'r gynllun o'r cyfrifiadau yn ddiddordeb, mae'r ddiddordeb yn yn niferfynion. Oherwydd mae'r ddiddordeb yn ymdigiddordeb digital designer doing embedded software and digital design but what I found myself doing is spending most of my time installing the CAD tools, debugging embedded builds, reading other people's scripts and so I left that area and founded Alexis. Alexis is a company that specialises in tracing Linux applications and profiling file IO in order to solve installation problems, build problems, dependency issues and so on. The main thing is we work out why your application doesn't work where it should, i.e. works fine in your environment since you deploy somewhere else something goes wrong and we try and take the guess work out of working out why it's gone wrong. What am I going to talk about? I'm going to talk about profiling and debugging technologies so I'm going to mention a few tools but I'm going to focus more on the technology underpinning them. I'm going to include a bit about our tools and how they work as well and I'm going to talk about gaps in the tool ecosystem so what is missing, what do we need and hopefully by the end we'll have a bit of an idea about where we need to go in the future. So this is what our tool looks like. We pull out tool hierarchies, a program called hierarchies. What you're seeing there is actually a trace of Firefox. The root node is always shown in green. We show scripts in blue and programs in orange so what you're seeing there is that Firefox is actually an alias for a script. It calls two other scripts and a bunch of programs in the system. It then finally calls the Firefox binary itself which makes a bunch of network connections shown in purple. So this was originally designed to help solve problems with scripted flows particularly in the semiconductor industry but really anywhere where you find yourself reading scripts and executing them in your head to find out what's going on. But it's been extended over the years to look at more system level problems such as library dependency issues and so on. This graph really illustrates what I mean by application level tracing so we're not looking at debugging any process in particular. We don't look at the algorithmic details. There are many, many debuggers that can help you solve algorithmic problems out there. It's more about when we pull the whole application together whether that's the installation process, the startup procedure and all the tools around it. That's what I mean by application level debugging. The way breeze is normally used is in comparing two applications. You would make a trace of a tool on a system that works, a trace of a tool or a scripted flow on a system that doesn't work and you get to compare the two. That could be working out what's different about a particular machine, what's different about a particular user. It's so common that you put together a scripted flow handed to someone else and it doesn't work. You need to work out why it doesn't work, whether that's a missing dependency or whether they're just using it in a different way. Are the arguments are passing in the same? Do they have some user settings that interfere with the problem? There are lots of tools out there that can help you make sure that your Linux installation is the same every time, same packages installed, same libraries and so on, but nothing can really prepare you for changes such as a user misspelling an environment variable. We help you take the guess work out of working out what's wrong by looking at the application instead of looking at individual programs or just the system. In 2012 we also added file.io profiling for remote file systems. That means that you can not just work out what your dependencies are, but you can also work out how often different areas of the file system are being accessed. This graph here is a build with multiple targets and you can see each orange spike is accessing an area of the home directory. Each green line is accessing areas of slash temp so you can very clearly see the build cycles for each target in the system. In this way you get a very holistic overview of what's going on. I've mostly tracked slash temp and slash home in this graph but you could set up a breeze to look at your database installation, your tool installation and track what it's doing in those ways. This really highlights where we are going as a company and where we see the industry going in that increasingly we're being asked to match up physical resource use with application status. That's a theme that we see time and time again and that's really what this talk is about. It's about making sure that if your file system is being overutilised, if you're seeing spikes in the network then trying to get back and work out which application is actually causing that particular behaviour. It could be something as simple as a global pattern matcher in a script can actually cause applications to trawl the entire file system looking for matches to that regular expression and it's one of those things it's very easy to do, it's very difficult to actually look at the performance of your file system and work your way back to which script, which line of which script is actually causing that to happen. I've talked a bit about our tool breeze but I want to talk about what else is available and how does it work. We've had instruction level debuggers for years, some of them are really excellent, many of you will be familiar with GDB. GDB is actually quite funky in that when you set a breakpoint it actually swaps in a special breakpoint instruction into the code so that when in hardware when you hit that breakpoint instruction everything halts and GDB is alerted and you can take it from there. That's an example of hardware and software code design. The software wouldn't work without the hardware support but for that reason it works very well indeed, you get very good performance but it's still not really any good if you're trying to work out what's going on at the system level. Once you've got it working and sent it to your customer if it suddenly slows down at one particular point GDB is not really going to help you, you can't really go into the customer site and debug it on site, you probably don't have any debug symbols in there anyway. Valgrind is another tool which is extremely powerful for doing instruction level debugging. For those of you who don't know Valgrind is a tool although I'm reliably informed it's pronounced Valgrind actually. It's a tool for spotting memory leaks in your application so in in C that's as simple as making sure that you've got a free for every malloc but it can do a lot more powerful things than that and Valgrind actually works by rewriting your binary and rewriting the allocation and free statements so that they can actually track all your memory use throughout the application. It's a really phenomenal tool give it a go if you've never used it before but again it's more at the application development stage when you're up to elbows in code it's not really about working out what goes wrong once you've actually got your application running in the in the wild. So there are various challenges as with any tools as well it's not only is it difficult to deploy this in the field but it can be difficult to make sure that the debugger doesn't change the application as well. I'm sure most of you have been trying to debug a race condition which disappears as soon as you attach debugger to it any kind of change even with the most minimal overhead can potentially cause a problem and I've just talked about some some very old well-loved instruction level debuggers but there are a few new ones on the market I'm not going to list them all but just a couple here for example undo software take snapshots through the running of your application so you can in fact run it backwards no you no longer have to sort of hit a breakpoint and and try and guess what scenario led up to the state you can actually use their tool to step back in your c or c++ debugger similarly a linear can debug parallel code across multiple machines so if you have an embarrassingly parallel problem with many many processes running your algorithms then you can optimise your functions using using their tools so the the tools available have really come on they're innovating all the time in this industry and there are there are dozens out there to choose from but it's still it's still not about debugging the system it's still not going to tell you why your database is a bottleneck one um or or why your your network that you so carefully carefully planned out isn't isn't up to scratch um a lot of these tools focus on embedded but these days when my phone is about as powerful as the computer I took to university um but what has embedded me in any way so uh many of you will also be familiar with a tool called ptrace uh ptrace is built into the linux kernel um and gives you access to to system system level library level calls as the application is running um l trace and s trace are both built using using the ptrace api and they they show system calls and library calls respectively so here I just want to have a small pause and clarify exactly what I mean by system call and library call um the the terms are often often used interchangeably with good reason because a lot of the library calls have the same name as the system calls are mapped directly but many of them don't when your application runs irrespective of what it is there's a very good chance it links into the glibc library in order to access to the next kernel so that will be for everything from opening a file to connecting to a network to spawning a a child process everything has to go um to into a into the kernel via system call and most of the time that's made easier for you um using the glibc it's um much easier to call um some of the some of the high level constructs rather than work out um some of the low level uh mechanics of of the i o but they're still in there for for you if you want to want to use them and l trace uh gives you library calls s trace unsprizingly gives you system calls both these tools dump out a huge amount of information they're often used to solve system level application level problems but it's it's really like sipping from from a fire hydrant when you when you try and um sort out what's going on with these applications and they don't cope very well with very very large applications so it's very very quickly you end up with gigabytes and gigabytes of data trying to sort through and find out which process you want to look at and uh which call is interesting can be very time consuming so now i'm going to talk about another technique um breeze our tool and others like it use ld preload instead of p trace um this is just for a show of hands um how many of you are familiar with ld preload techniques excellent excellent so ld preload is a technique that allows you to override library calls it's most commonly used to override um glibc or to replace a system library but you can actually preload any any library you want and override any any library in the tool so if the if your application has got proprietary application specific libraries you can even override those if you know the if you know the um function signatures and the way it works it's very simple you set the ld preload environment variable that's why it's always in capitals it's not that we are so excited about it we need to shout um the then once the the uh linker sees sees your library in the environment it will actually preload that ahead of any other system library so that your application is linked against and by preload i mean exactly that i mean any calls that the application makes go to the ld preloaded library before going on to any other system library and it's really up to you whether you in fact call into the glibc library um or return something else uh so this is this provides a very very powerful interface um and it allows you to manipulate the application so unlike using ptrace or strace it doesn't just give you data it allows you to handle all kinds of scenarios that uh strace won't won't cope with because uh you can't inherently change the application so this is what um a preload library signature might look like uh you simply match the applet the function signature um so here i'm overriding the read function in glibc and the first two lines might look a little bit like gary pokery but all we're doing in the first line is defining a function pointer and using this special function dl sim to get the next read function in the in the chain so in my illustration here we only have um uh the breeze preload preload library and the glibc but you can chain together a pretty much as many as many applications as you like i'm not aware um that there's any limit if i'm sure i'm sure some of you can find it um so all you do is ask for the next read function um do whatever it is you want to do in your wrapper so here we're just printing information that this application has read um and then finally we return uh we call next read um which would be the real read function as it were so here we're not changing the behaviour of the application materially we're just adding a print statement so similar techniques can actually do more powerful things such as swap file systems or suspend and resume a job so here we have um another uh another library wrapper um this time i'm wrapping the open call and what i'm doing is uh replacing uh prefixing all the path names with uh slash alt meaning alternate file system so in that way whenever your application tries to um access a file in the home directory it will instead be diverted to a file in slash alt uh slash home for example um so this is not totally unobservable from the point of view of the application because if it reads slash proc to find out if it's open the right file it will find it's it's actually open something else but not many applications actually do this uh so most of the time you can very very safely just swap um one file for another and the application will continue to run provided it sees the data that it's expecting so here we just return next open with the alternate um path name and so we can you can use this to to hot swap a different file system in in at runtime and uh many of you may be familiar with the yokto project uh the yokto project actually uses a tool called sudo um in order to do just this so in this call graph you can see that um in the in the yokto build they use a build system called bit bake uh which then makes a call to sudo which is just a wrapper strip that sets the ld preload library so this is used to fake root access um the during the build the application thinks it has root access to the file system in fact um the file system is swapped out for a database so here you can see that whenever the uh whenever the application tries to access a file it in that open or read write call is in fact redirected to the database and the data it's expecting is returned so that means that you can actually do a yokto build without being root and now this all sounds sounds very simple i've touched on a few things that can go wrong um but in fact it's a minefield of things that can go wrong um newer applications actually are much easier to trace than older applications but we have a legacy of older applications so i don't think there are very many applications that don't use for example bash or make um uh in order to in order to indeed start um and you have to make sure you get absolutely everything right so for example if you're trying to run breeze and sudo at the same time breeze needs to be first but sudo needs to not know that it's not first and you also i've talked about just overriding one function in the glibc in fact the glibc has got multiple versions of each function you need to get the right one it's no good if you start returning the wrong value the api is mostly clean but some of the data structures returned have changed over the years so if you get the wrong one it's a complete disaster so from from that point of view it can be very difficult to get right so it seems like a sort of a silver bullet solution but in fact you need to be just as careful as with anything else so why should you care about application tracing system design is getting more and more complicated we heard in the keynotes um today all about cloud um i'm sure if anyone doesn't know what that is then i'll suggest you're lost um it used to be that we had a single application um running on a single operating system on a machine that had local storage but that's not the case anymore now we have the application and the operating system sitting on top of job scheduling virtualisation on top of a host operating system as well as a whole bunch of other things in the stack designed to give you more options and abstract away from the complexity but at the same time it breaks us further and further apart from the real system resources and the applications so if everything goes right it's amazing when something goes wrong it it just makes it so much harder to work out what the problem is we now have multiple applications running on the same machine that have to be um separate from each other they may share they may have their own core um but they're still ultimately running on the same ship and have to have to share i o resources for example even if you've abstracted your applications so that they run on separate machines they may share a network connection to a shared file system so it's very very difficult to keep everything separate and keep this abstract bubble of virtualisation running um in an ideal world we would also have homogeneous hardware so that it doesn't matter where you deploy your application you get the same performance every time but realistically that's not cost effective most companies will buy a bunch of machines and then upgrade at some point by buying some more machines those machines will not necessarily have the same uh applications stored on them they won't necessarily have the same performance and at that point we start to see performance bugs such as um race conditions appear on some machines but not others um and i just want to put it down in the bottom corner because actually this is a company mainly working with windows and mac at the moment but this gives you an impression of just how complicated it's going to get um bromium is a company that's working on creating a virtual machine for every single process in your on your computer so for example if you're editing multiple word documents or browsing multiple tabs each one of those run in their own operating system stack um so from a user point of view this is amazing you can click on absolutely anything and you know that you can't infect your computer with a virus because it's that tab is trapped inside the operating system um in for just that just that tab you also um because you're customising the operating system stack you can actually avoid many of the many of the problems that arise from having a very predictable stack it becomes it very very difficult to write viruses as well so i just wanted to talk to them talk to you about them today not because this is a technology that is coming to Linux immediately but because it's the kind of thing we can expect in the future basically um we're just going to have the most unbelievable amount of complexity in the software stack and everything is going to get harder so now i've got a couple of case studies to show you what can go wrong when you have too much abstraction so we did some work with ARM recently um on profiling their file system now they have a huge cluster for compiling their chips they run a lot of verification computationally it looks a lot like mapping the genome or or doing financial trading it's lots and lots and lots of of operations done over and over again in order to verify their system and performance is key the uh they have a complex file system hierarchy in order to make sure that all their needs are served some super backed up some super low latency and it's critical that users or that their developers use the right area of the file system so their applications um are largely speaking uh third party um CAD tools that have been bought in in order to do the verification and then they put their own scripted flow around that like like many others in that industry and others so that means they introduce a whole new levels of application dependencies that are designed by some local engineer and not necessarily controlled by a third party tool um they have license settings so the license server is also critical because if you're running lots of short jobs high license latency can be a bit of a disaster i'll come back to that again um but their main question was is everything being stored in the right place and unsurprisingly the answer was no it is not always being stored in the right place so this is actually some result i hope you can all see that it's come out quite well on your screen look terrible on my screen um the this is actually the first job that we traced um with them uh we looked at two areas of the file system slash temp where you hope there is loads and loads of activity because you want to be doing all your temporary calculations on local storage you don't want to be hammering the network and sending sending data back and forth unnecessarily um scratch was a area of the file system used for results um so fairly low latency it's just designed literally to spit out for the application to spit out a log file and it hangs around for some time um but you can see from this trace uh we looked at the number of rights to different areas of the file system you can see that slash temp is hardly used at all whereas scratch is is used quite heavily so in that sense they were they'd spent all this money on optimising their applications optimising uh their file system buying the best machines they could and then the whole thing was being tanked by some script uh putting the putting the data in the wrong place so that's an that's an an example of how abstraction doesn't always work you do really need to know what's going on if you're going to get the best performance it's far too easy to to tank performance with uh just one simple mistake um here's another another example um um the uh this was a uh company that we worked with uh again very short running applications um and they had a bunch of different license servers listed in their environment variable very easy to set up um and the license software just goes away and finds an available license but if you look at this particular trace um the red lines here I hope you can see the uh the connect line across the middle of the trace the red lines are actually failed connections so you can see here it's the application spends 200 milliseconds out of a run that lasts less than a second spends 200 milliseconds just hanging around waiting for failed connections before finally getting a license that's a disaster you've just thrown away more than more than 30 percent of your performance so there's an example where just really something simple is reordering the license servers in the uh in the the settings for for obtaining licenses um can very quickly give you a lot more performance and here we have another case study this this one's a bit different um this is an example where one user couldn't type in anything into their login systems um so the it manager traced a star tech session for that user um and he saw that uh xm bind the key binding uh application was sourcing the wrong key bindings so here we've got a picture of that we've got a trace of star techs at the top uh you can see xm bind being called and you can see that it's accessing the uh bind aliases in slash grid um now that's an application specific area of the file system there's no way the the system should be getting their tool bind key bindings from there um they were overriding the the local key bindings and preventing the user from typing anything so that was another example of just simply swapping the environment variables around so that the system key bindings were sourced first instead of these ones in slash grid um so this time it was easy it was obviously due to a lot of system level knowledge from the it manager in question but how long can we can we do this what if it's not a problem with your application it's not a problem with your virtual machine it's in fact a problem with the host operating system if you're running your application in some third party cloud how on earth are you going to debug anything from there so where do we go from here one of the problems is um how do we trace a distributed application um i've the examples i've looked at here have largely speaking been applications running on linux systems what if you have um a linux processing system with a linux database you've got some java application which could be running on anything to um to be honest and then you've got the user accessing this system through the internet from any device they want how are you actually going to profile that it's you can potentially set up a few uh load benchmarks um in your in your own system but once it's out in the wild how are you going to cope with unexpected drops in performance you can use a whole variety of system level monitoring tools in order to see which part is being hammered the most but how are you actually going to tie that back to what the user is doing how do you know how much of the um when you've spotted a bottleneck for example in your database how do you know which calls from which users actually caused that that spike in in performance it's not just a question of uniform load obviously under under some uh system with a very simple api um you know it's just a case of lots of traffic but that's not what real applications look like in real applications you can have some some operations causing very little load on the system at the same time as some few application few uses causing a very high level of load on the system and so how are you actually going to create that story from the user right through the whole system accessing every part of your of your application back to what they actually experience so uh even if we could create um a tool that would do lots of different tracing with lots of i know there are there are lots of good tools out there with open apis that let you create and share information across different systems easily but how do we actually marry them all together because it's not necessarily going to be fit for android or a mac front end um as well as a a java application and the second problem is of course what shall we measure um it's hard to combine different datasets even with open um open open data and open apis for different tracing tools and debugging tools we're never going to align the whole community so how do we actually pull out the bits of information we need um and we certainly don't want to measure everything so how do we work it out so that we can pull out just the bits that we want at that particular time obviously if we measure everything we're just going to tank the performance of everything that we've been uh we've been working for and there'll be an unacceptable delay when you click like on uh on your on your girlfriend's kitten pictures so um we don't want to reverse engineer uh every system that we need to profile we need to find a better solution for that so one solution is designed for profiling we've all heard of designed for test um and hopefully we all design effective testing methodologies um whenever we go about architecting a system we all know how to create correctness in our code make sure that our functions can cope with with all the possible available inputs without throwing unnecessary exceptions but what about design for profiling because performance bugs are still bugs if it doesn't work at the end of the day it's a bug um so we can build programmes with debug symbols and debug code um but often turning them on um is a uh is a measure everything scenario and that's not always what we want we actually usually need something highly configurable um and that becomes another engineering challenge in itself um it's normal to have a design team for the application and a design team for test but maybe we need to start having a design team for observability as well so critically profiling frameworks should really be chosen at the start and not attached later on so i wouldn't be doing my job if i didn't tell you what we offer um these solutions are available for creating custom profilers and custom debuggers um but i'm not going to pretend that this is some um panacea for the for the problem um our breeze technology is based on top of a tracing framework called star trace um we can make custom debuggers and custom profilers using that and we also have um uh just started to support an open source uh solution for doing similar levels of inspection to Java applications so by combining the two you do get a lot of visibility into the system matching up for example calls in the application with uh network level information and and so on um but we're still a long way from creating a framework uh and an ecosystem that can tell us anything we want to know when we want to know it um one solution is to try and monitor everything and correlate to actually expect that a large proportion of the performance is going to be given over to measurement um and observability that means that we focus more on meeting deadlines rather than getting the best performance for our money um it's always going to be difficult to get fine-grained coverage without harm harming performance um but with enough options in there then hopefully we can we can gather the information that we want to see um and this gives um an opportunity rather than to go in for pinpoint debugging approaches i.e. is the correct number being read from that file the fifth time is it is accessed that that level of debugging we can go for a more holistic view of debugging where we can solve performance problems without looking at the application architecture in detail so um trying to work out where we get the what kind of behaviour leads to um spikes in the network or spikes in in memory use and perhaps one day we could actually start to predict what our applications are doing i know a number of the larger vendors are already looking at that i passed a um a big screen on my way to the station um to come here which is all about microsoft's um efforts in uh correlating uh data within their cloud by measuring and trying to predict what's going on and and i think the there are a number of people doing similar work in in the Linux community so now it's time for your ideas now it's time for your questions so yes um so you're you're asking about whether um breezes open source breeze isn't an open source tool um at the moment we can make open source ld preload libraries four companies if they need an open source solution um but of course it won't be the it won't be the full technology um uh so so yeah the the um the configurability in our ld preload library at at the moment it's largely been on a consultancy basis but we've got plans to open up the api so that you don't get the source code of our um ld preload technology but you can use it to make ld preload libraries yourself as in you can uh essentially supply your code to to a build infrastructure and then and then make your own so from that sense um we are opening it up to the community so yes you're looking there uh the existing tracers uh i'm the outer one of the main outer of ld i mean we've solved those problems solved those problems already i mean i'm wondering about the overhead that really the approach has uh because we're basically doing system problem tracing in ld from within the terminal we are in perfection within the applications with user-level tracing uh we have uh uh cloud uh cloud support basically some synchronization of traces across an entire cluster we are also at that synchronization of traces between user space in the kernel so pretty much everything that you're listing here as work that should be investigated are actually things that have been solved a couple of years ago so you might be interested to look into those solutions mainly for the racing back end because i think what what's interesting here is that you bring interesting analysis on top of the the data extraction but a lot of work has been done already on the extraction yeah so the the question was why on earth why didn't i why didn't i give a talk on lttng is is largely your question um there are a number of talks later on where you can learn about lttng and the marvellous work that um that has been done in filling this space it is one of the tools um that does solve a lot of the problems um that i've been talking about today um but it is again by no means a solution for for everything um it's a very powerful tracing tool but it doesn't necessarily give you the same flexibility to uh change the application so for example it's very difficult to um uh manipulate an application and uh change the argument so we can set up tracing differently on a job that's been launched remotely depending on what the parent did um we don't just monitor the whole cluster um because we don't want to monitor the whole cluster we can't monitor the whole cluster because we're monitoring applications that are running against running in systems with applications that have uh that can't be traced because they're they're in production they're in the middle of taping out a chip for example um so a combined approach um would be a very good solution for a number of problems but again there are still problems out there uh which it doesn't solve um yes yeah it's certainly a technology that is um working very hard to fill a number of gaps but but yeah as I say please go to the talk on LTTNG if you want to know more about that um any other questions yeah oh wow what a question um there are a number um of issues I suppose that my developers would uh would come up with and say that we could do this and that's to to make life easier but by and large um the um uh it gives you enough rope to hang yourself basically so you can pretty much do what you like as long as you know what you're doing and that's um that from from our point of view that's great but it also means that we've needed to produce the star trace libraries to allow people to pick up the technology and run with it quickly so um yes and no no system's perfect but it's certainly it's certainly fine fine ease but by no means that as we've already said they're the only technology in in in the game and interesting even more in the Java part of your work how it's progressing as I mentioned you are starting to do this in progress yeah so we're starting to support an existing open source technology um it's been developed by a research group in Switzerland and at the Italian University of Switzerland and um again it's a case of their system is brilliant um it's very very powerful it gives you a loss of control about what you monitor in Java applications um but it gives you enough rope to hang yourself basically so what we're doing is offering support so that people who want to develop something quickly and easily um can ask us to help them do that um but yes it's the the technology is fully available for anyone to download um and take a look at um if you want to look at the the technology itself it's called uh diesel d isl um and will be adding links to their commercial support later on this year uh on our website so yes uh yes there are a number of security um technique um uh problems that can prevent it from working and um we've circumvented most of them and we're not going to tell you how um so uh yes it's it's an incredibly powerful technique because it lets you change the application so of course there are security measures built into Linux to prevent you from doing that but um if you think hard enough most of it's not not not a problem so one last question okay i think oh yes so um the uh question was about uh visualization and the fact that when you change from instructions tracing to system level tracing you change your visualization and it's a very clunky transition i think that's that's one of the one of the problems most of the talk has been about the tracing technology but actually visualization is just as hard a problem um and uh i know lttng has got some uh very good visualization tools with with open apis but um uh what we find is usually when it's it's a mixture so some data can be easily represented by by tools that are already out there but some data isn't and it's just a question of having that expertise at your fingertips and just being smart about it thinking about this comes back to the uh design for profiling think about what you want to see and how you're going to see it um i think it's the only way so okay thank you very much