 Rydyn ni'n Richard Palefwrp. Rydyn ni'n Susay Cynulliadol Cynulliadol. Rydyn ni'n dweud gwneud i'r cyfnodd, nid yw'r cyflwyno'r Cywain OpenQA yn ein bydd jadw. Rydyn ni'n mynd i'n gweithio'r cyflwyno, ond rydyn ni'n gweithio'n gweithio'n gyfrân. Rydyn ni'n gweithio'n gweithio'n cyfrannu. Rydyn ni'n jadw yn cyfrannu, rydyn ni'n gweithio'n cyfrannu, rydyn ni'n cyfrannu. Rydyn ni, ofeed rydyn ni fydd i'n mynd i na hm Wolfrodd Awnnolgyniadau pawdau Bikegeeol neu gallw'n gweithio'n gweithio'n acefredi gennym. Rydyn ni wedi'u bod yn brin insightful gael Cyffr lunch'rBLEEPxgo'i sits uwch? Dwi'n fath eich rhai, mae'n wneud eich bwysigau y Ffaintwyr yn ddwy'r ex-Yn-Z. Gwyddo yn ymgyrch yn ymgyrch yn ymgyrch. Yn ymarhwych yn ymgyrch. Ond rwy'n gweithio'r gwybodaeth. Ond fyddwch i'n fath yw'r gwybodaeth eich gwybodaeth eich gwybodaeth i'r gwybodaeth unigol. Mae'n gwybodaeth i'r code cyllid ymgyrch yn cyffredin. Yn gyfynig ychydig ymgyrch yn ymgyrch yn ymgyrch, a gyda'r cwestiynau, yma, a'r cymdeithasol. Rwy'n amlwg a'r cymdeithasol, a'r cwestiynau, a'r cymdeithasol, a'r cymdeithasol iawn gweld cymdeithasol, a'r cymdeithasol iawn gweld cymdeithasol, rydych chi'n gael Cascale oedd y canwyd yn bachol. Yn y gallu cymdeithasol, yr cymdeithio'r cwestiynau oherwydd ychydig iawn gyda cyrcaf, a rwy'n credu roi'r clasau o'r ddweud. Felly mae'r llangwydau gweld ymlaen yw'r cyfnod yn fwy o'r cyfnod. Mae'r llangwydau i'r ddweud, felly mae'r ddweud wedi bod yn gweithio, ond rydyn ni'n gwneud ddim yn oed o ddataethau, nad oeddwn i'r projiexio, felly rydyn ni'n gwneud o'r syniad o'r ddweud o'r cyfnod o'r ddweud, o'r pifon, o'r ddweud o'r gyfnod, ac rydyn ni'n fath ychydig y ffrind ond rydyn ni'n ziwbod gynnig Rydyn ni'n gwybodaeth eich cyfwyr isaeth gyda gwybodaeth eich cyfwyr I'r thwybyn hollu cyfwyr a'i gwybodaeth eich cyfwyr Efallai rydyn ni'n iddo i gyd, rydyn ni'n gwybodaeth eich cyfwyr i ddod o'r awtfod Felly, yn ddau'r Corroedd ac yn yr gorfod y system ychydig yn ymwneud, y cyfnodd y dywed y cyfnodd yn ydy i'r gweithio os ydych chi'n gofyn. Mae'n meddwl yw'r ffordd ffilm yw'r ffordd ffordd, a'u gydigwch ymwneud i'r ffordd ei nifer i'r hynny o'r ymddangos cyfnodd. felly mae eich cyfwyrdd, ac mae'n cael eu botonnwch i yn cael eu hunain oherwydd ac mae'n tynnu ar y bydd. Felly mae'n dweud eich cyfwyrdd o'r ysgolwyrd yn cael eich cyfwyrdd. Felly mae'n dweud eich cyfwyrdd yn cael ei ffordd yn awdurdod o'r tîm, ac mae'n ddysgu'r cyfwyrdd. Rwy'n d carving, mae'r byw yn gennym, oes i'ch cyfathon x86 armarch, powh PC. Rydym yn cael ei fathos i'r byw'r methel, oherwydd yn y Bender, oherwydd mae'n rhan cyhoedd y degolodyddau bair yn cerdd, ac mae hyn yn rhan i'r amser iddyn ni i'r pwyllfa bwyd, oes i'r gweithl deddyn ni'n gwahanol. Rwy'n gwneud o'r ddod yn cyfnod L-Q-A'r ddod i ddiddor yn bwynt arno ddiddor i mi wedi ddod y ddod yn gweithio'r cyfnod ar bobl oedd cyflogom yn bwysig wedi'u gwybod yn cyfrannu lyfr A hefyd yn gwneud o sylfaen, gydy'r cyfan y byw y botl o'r drwy'r cyffredinol y gynhyrch leol, dwi'n rymrwg â cyfrannu ffordd mewn yma, mae hwn yn golygu i ddiogelio'u bwysig sy'n gwneud o gwybod, amser a'r cyfrannu eirau mawr i bandol L-Q-A, Here, setting up all the infrastructure that OpenQA requires. OpenQA also displays test results and it allows you to comment on test results and it has a scheduler for scheduling jobs. It has all kinds of stuff going on in there which probably should be split out into different projects. Sort of like how you saw earlier with the Kernel CI. Ac mae'r gweithio modulau a'r gweithio'r modul o'r mwy bwysig. Gweithio'r modul o'r meddwl chi'n gweld unrhyw, ac mae'r gweithio'r gweithio'r dylunio'r tylu, ac mae hynny'n fawr i'r gweithio'r gweithio. Mae'r gweithio'r gweithio, ond mae'n gweithio'r gweithio'r gweithio, mae'n cael ei fod yn ddwy i, ond mae'n gweithio'r gweithio. Cada Previously is this framework I created. Here we have listed what I currently use it for so the only thing people really appreciate is the bug tag propagation or what I'll show you in a moment. This is probably the primary thing that worked. I also have some other test reports and stuff that I did which I'll show you at least the test status difference matrix. That's nice to look at It's nice to look at, but the actual usefulness of this stuff is maybe not so much. Because fundamentally if your data is noisy, all you can really do with that is filter the noise. And there's not really so much you need for doing that. In fact all you need to do is really look at your, whatever dashboard you have, go OK. Half the tests say they're failing for infrastructure issues. Then you can just at that point say okay, we need to go and fix that. And you'll get a lot more value for your time by just going and fixing that stuff instead of creating a data analysis framework and then trying to create some reports for that. This is my original motivation for creating JDP. I've kept this slide the same from when I started it. some of this stuff's still true. OpenQA, Bugzilla, all of these services, they're often very slow, throwing more hardware at it doesn't work because the code is slow. And fixing the code is difficult because there's such huge frameworks with so much complication. And yeah, as I said before the JDP is a general purpose framework in principal at least. So you can take data from from anywhere, it will put it into a local data cache, it will normalise it and then you can create reports on that reasonably quickly because it just works within memory data and you don't have to worry about say Bugzilla's API or anything like that so long as somebody has created a connector for you. I'm just going to now show you OpenQA. This is the OpenQA dashboard just for a single image build. So this is for, I can't see the thing now, I guess this is for the tumbleweed, oh no this is for silly enterprise Linux and each one of these is an OpenQA job module. And then inside each job module we have sort of a graphical representation of each stage of the test and something I have to point out is that in our case we're actually wrapping LTP in OpenQA. So normally in an OpenQA job you have a test module which is written in Perl and it has some logic for like send this key press, expect this response from the user, expect this image to appear and this kind of thing. For us because we're wrapping the LTP we've essentially written a test runner inside OpenQA that then displays each OpenQA LTP test and we don't use any of the graphical features we just do text matching. So basically up here you can see the machine booting and you can see here that there is an initial bit of screen matching but then we turn that off straight away and we just go to matching serial output because it saves a lot of performance and is much faster. And here you can see the output of one of the LTP tests. In this case it's one of the CVE tests which tests for some bug which had a security implication and we have the output there. And of course you can get the raw logs and other assets like images, the job might produce and everything like that. You can also do some basic filtering on the test results so you can just show the failures. One thing OpenQA does but doesn't do so well is bug tagging. So when you have a test case that's failing you may know that there's a particular bug for that test case, a product bug. And while that's being resolved you want to list somewhere okay this test result has this bug and you want to have that automatically propagated to new test failures so that you're not constantly just looking at the results and going okay this test failed again and again and again and we know why it's because of X, Y or Z. The same goes for if the test is simply broken itself. You may just want to tag the failure to say that okay this test is broken we want to keep running it but we also don't want to waste time each time we have a new build looking at this and saying this is a known bug. OpenQA itself can actually do this to a limited extent however I extended it with JDP and this comment just here is actually from JDP saying that we previously found this bug tag on another job that looked similar and so I've propagated it. And there's also some bugs which it didn't propagate for example say maybe the architecture doesn't match one bug to the other because sometimes you get a bug in ARM and it won't happen on X86 so you don't want ARM bugs being propagated to X86. So JDP has just recommended some other bugs for investigation but hasn't actually tagged them and the reason I didn't implement this into OpenQA itself is because essentially we could have lots of other test runners. For example we could start using the upstream kernel CI and then we'll have the same LTP test cases failing in upstream on a different testing framework that are also failing on our OpenQA test instance and really we want to share tags and propagate tags between those two instances. So we want a third party service which has some rules and say reads the BigQuery database and shares tags between those and JDP can do that quite well. And this is the thing that actually works very well and as we start to use more testing frameworks, I say start to be able to use more testing frameworks we can propagate these tags between them. The test status difference matrix this is something that's more questionable and I mean I've seen people do something similar to this quite a lot as well where okay so you have a lot of failing tests and you think to yourself well a lot of these are just failing week in week out I know that somebody is looking at it or whatever. I'm just going to look at what's actually changed because okay this thing is probably not interesting because it's not new so I'll make a report which just tells me what's changed. The problem I found is that basically the report which tells you what's only what's changed basically still just tells you the same thing which is your tests are noisy and that you have tests that are randomly failing. Most of these tests here will be randomly failing for infrastructure issues or as there's merely just a dash listed these tests didn't even run so the reports just saying this test result was missing from this tumbleweed build. So this is a Suze tumbleweed image and it's saying this test wasn't even run we expect it to be run because I've seen it before but it wasn't there. And really what this reports just telling is just telling you something you could see just by clicking on a few different builds and looking and saying yeah that's broken. Having said that I guess people still do look at this report from time to time and it can send out notifications when something changes which is another thing. People I guess think we want but actually when it comes down to it if you send out a notification either the person who receives it doesn't want the notification in which case they'll just block you. Or if it's really useful to you to get this notification then it means that you're not checking your results often enough. I mean similarly you can just send out a notification just saying this test failed and that's quite a good notification. If most of your tests always pass then one notification your test failed then that's fine. So yeah I'd advise steering away from this kind of stuff. On the other hand theoretically speaking if say 100% of our data was signal and we were just trying to do some analysis of the signal then things would be different. I mean in theory you could create something which automatically scans bugzilla then scans the error logs and matches errors in the logs to bugs in bugzilla. And this is something that people have done before. OK so JDP itself I decided that I wanted to make it as flexible as possible. So I decided I want to manipulate data in memory with just ordinary code. No SQL no other fancy no SQL query languages or you know anything that resembles SQL. I just wanted to take the language it's written in which because it's Julia it's fast so it's not like doing a query with raw Python which will be something like 30 times slower. You can you know if your algorithm is reasonably good then it should be able to do it at the same or even better performance than an ordinary database. But you have the freedom to use something which simply doesn't fit into SQL. And JDP has a whole bunch of different connectors to other data sources like bugzilla. Yeah open QA red mine QADB which is some internal thing we have get hopefully Jira at some point. And it could plug into all sorts of other things. And then when you're writing your query even if you want to access all these different providers and also syncs then you can just use one API you just have to use the JDP API. And yeah it also has a users redis as the data back end and redis is able to replicate the data between everyone's node. And it's quite easy at least in theory to set up a local node and do your work locally. So if you have unreliable data sources and even if JDP central instances disconnected for some reason then you will probably have a local copy of the data which allows you to keep working which is the same principle behind get essentially. And this is very useful when you say have an office in Beijing which has a really bad data connection between officers. So yeah and JDP is designed to not be a source of truth so I'm not using this as the primary data storage unit for test results. They live somewhere else they either live on open QA or they live upstream somewhere so that we keep the freedom of being able to simply delete the entire JDP database and redesign it. So that it can better perform its one function of simply doing data analysis and not acting as a reliable data store for the long term. OK and to be honest at the moment I'm taking a break from this because it's useful. This is questionable for our current problems our current problems resolve more around the testing itself so for now I'm just keeping an open mind and. Obviously we'll keep using it for tagging test results and that's all we'll see what happens again this is just you know he was involved so. I guess I might have finished quite early so people have questions please ask OK I mean so the question was how do you match failing test results against bug results. Like what are the properties or parameters and the answer is it's arbitrary and there's basically two ways that it can be done or more than two ways. I mean one obvious thing which is used sorry I'm getting in a mess here. One obvious thing that was used is actually just the test name. So when somebody writes a bug tag in think better did it here. You just write something like this like the DHCP D6 test and the DNS mask six test. They've been tagged with this red mine issue. OK and so in the if you don't provide any more information than just this then it will just match all tests with this name that are known to be in the LTP test suite so it has some name spacing for tests. So if you have some other random test in a in some other test suite with exactly the same name it won't tag those. It can figure out from the open QA context that this is an LTP test. Furthermore not sure we can see it here. JDP reads the bug data say from bugzilla or from red mine and it passes the title and it reads what architecture what products what component listed in the bug and then it decides whether basically if it has a failing test with the correct name. But say the bug is only listed against another product or another architecture then it won't tag it instead it will just post a tag recommendation which is the bottom square down there. And then if you realize that actually that book is really occurring on say more architectures or more products and you can update the bug and say actually this is this has a wider scope. And also the opposite if you see something being tagged if you see an obvious on bug being tagged. On the next 86 and you can set the architecture to in the bug correctly. OK does that answer your question. Anything else OK. OK so the question is is JDP being asked. Sorry is JDP just doing bug tagging or is it being used for other things. Well test execution all of that is handled by open QA or something else. Storing the test results and storing the bugs on a permanent basis something else does that JDP just caches the data from open QA or bugzilla or wherever and then performs some kind of workflow on it some kind of report. And those reports and workflows are arbitrary. They're just scripts. You can either use say a Jupiter workbook if you've ever heard of that to create a report or you can just write a Julia script that uses the framework. Practically speaking at the moment we just use it for both tagging. And I guess the status difference report which you can see here. This is actually just a static view of the Jupiter workbook. The Jupiter workbook is interactive so. You can actually connect to this and start changing the queries and start changing the way the report is displayed. And all of the code which makes up the report excluding the library functions is in the report. I won't even attempt to try and explain what it's doing. Partially because I haven't looked at it for a while and I'm not sure what it's doing. But basically you end up with this nice matrix view and I'm very proud of this matrix view. It's just maybe not the most useful thing practically speaking. So is that answer your question? Yeah, I mean this JDP also runs on a constant basis and actually runs in GitLab as well. It has a bunch of GitLab pipelines and these periodically regenerate the reports, re-update the local cache and do the bug tagging. Honestly no because, oh sorry. So the question is, is there a standardised input format for creating a tracker? JDP is built under the assumption that the rest of the world is a mess and you can't expect anybody to use your format. So it will go and query Bugzilla or OpenQA, use their API and bring the data into JDP, then normalise it and turn it into a nice format for the end user is writing the report. I guess in theory you could write a tracker which provides an interface like that. Lots of people have opinions on which test format to use and that kind of thing and I don't have an opinion, I'll just use whatever people give me. So the question is, let's say you have a build and some tests are mysteriously missing from that. Is that what you're saying? Say if there's an infrastructure issue and so... Okay. The answer is that the more systems you provide you'll have a smaller and smaller set of normalised data. So that's just a consequence of if you have lots of data sources with different formats. JDP also allows you to pass through the raw data. So you have a choice of doing whatever level of normalisation you want to do. So if a particular bug tracker gives you a lot more data to work with which is pretty much universally true, all the bug trackers have slightly different fields and some of them simply have things missing. If you want to use that information in your tagging algorithm for example you just have to insert some special code to take advantage of that. And I'd say again that's another thing where Julia makes that kind of thing really easy because it's so flexible. So I mean I think the answer to that is just yeah it happens that you end up having a tiny common set of information and all you can really do about that is create some extra codes to deal with these things in your algorithm. I think that's not really a thing that JDP could solve I don't think.