 Hello, good morning, good afternoon everybody, my name is Eric Ogé, I work at Red Hat in the virtualization team, and today I'm going to talk about KVM unit tests and KVM safe tests with a special focus on ARM. Après une introduction, je vais introduire les deux test frameworks, et je vais essayer de comparer les deux. Je vais aussi parler des stats de GIT, et nous allons avoir un look à l'état de current pour les test codes basés dans les deux frameworks. Ensuite, je vais vous donner quelques exemples d'advertir les frameworks, je vais vous donner quelques exemples, je vais vous mettre sur les highlights pour les uns d'entre eux. Je vais parler de mes expériences personnelles, des lessons que j'ai apportées pendant les tests de KVM unit. Ensuite, je vais parler de quelque chose qui me semble très important pour moi, c'est de comment développer les tests en temps, en utilisant les modèles ARM MVP, et de comment développer les tests de KVM unit et les tests de KVM self-test avec ces environnements, ou c'est aussi possible avec les tests de QMU TCG. Ensuite, je vais parler d'une autre très bonne feature qui est parmi les tests de KVM unit avec les tests de QMU, qui est la capacité de test de migration, et éventuellement, nous allons conclure. Alors, pour l'introduction, je voudrais dire que l'expérience de virtualisation pour ARM maintenant est une réalité, nous avons des systèmes en production. La bonne chose aussi, c'est que nous avons stabilisé des zones historiques et difficiles. Je pense aussi sur le V-Geek, mais aussi sur les timers, etc. En même temps, nous avons un code basé de carnel, et nous avons beaucoup de trafic sur la liste de mail. Lorsque nous avons vu des réunions énormes, comme les réunions de code réunis, et surtout, je dirais que nous avons beaucoup de features de KVM, qui viennent avec ARM V8, quelque chose d'une réunion en spec, pour laquelle on ne généralement n'a pas de hardware pour tester. Et c'est nouveau, et c'est un problème, en fait. Donc, maintenant, nous avons vu la statue du système de KVM pour ARM. Maintenant, si nous regardons la statue du framework de KVM, donc la situation est un peu différente. Nous avons en fait très peu de tests unitaires qui sont contribués. Généralement, de nouvelles features viennent sans aucun test unitaire. Elles sont directement fonctionnellement testées avec un espace utilisé, KVM ou KVM. Et éventuellement, les tests unitaires généralement viennent trop tard, et n'ont pas de très signifiant coverage. D'ailleurs, nous avons très peu de producteurs de bug, en spite que nous avons des bugs. Donc, j'ai voulu cette présentation pour être une petite temps pour l'introspection, essayer de comprendre pourquoi nous sommes dans cette situation avec respect aux tests unitaires, et surtout, comment nous pouvons improvez les choses. Alors, maintenant, nous allons entrer dans les details. Et nous allons voir la comparaison entre les deux tests frameworks, qui sont KVM, celles test, sur l'un end, et KVM unitaires sur l'autre end. Donc, le premier test de KVM sur l'autre end, est le plus récent, il a été introduit en 2018, et nous avons armé les portes depuis l'année 2. Ce qui est très beau, c'est que les tests KVM sont partie de la Linux 3. Donc, avec ce framework, le testeur va s'occuper de l'application fonctionnelle de KVM. Et aussi, il y a la possibilité d'avoir un code guest, qui va rester très simple. C'est écrit en C, un code assemblée. Et il se débrouille au niveau privilège de l'EL1. Donc, avec ce framework, vous n'avez absolument aucune dépendance. C'est-à-dire, le testeur va directement cliquer sur l'application fonctionnelle de KVM, sur l'autre end, et évoquer le code guest. Donc, il n'y a pas de dépendance sur l'usage de KVM, comme QMU ou KVM2. Et c'est le plus grand différence avec le test de KVM. Donc, ce que le framework donne, c'est en fait, des rappeurs, des aidants, les plus utiles de l'application de KVM. Il y a aussi un nombre de codes délicatés pour l'allocation dans le KVM, l'allocation physique, l'allocation virtuel, et aussi la map des services de translation entre les différents adresses spécifiques. Et avec respect au code délicaté sur le guest, il y a très peu de services par le framework. C'est-à-dire qu'il y a une capacité d'essayer entre l'usage et le guest. Donc, maintenant, let's have a look at the KVM unitess. Donc, ce framework est beaucoup plus ancien. Donc, il vit dans ses propres repositories depuis 2010. Et il y a un port armé depuis 2014. Donc, comme je l'ai mentionné, comme opposed au test de KVM, le test de KVM unités n'est pas standalone. Ils sont réelés sur un espace vm-usage. C'est l'un des défauts qui est QMU. Rien que KVM ou TCG, mais il peut aussi être un tool de KVM. Et le testeur, cette fois-ci, va focusser sur le code guest seulement. Il va utiliser le code guest dans les assemblées de CN. À TL1. Et récemment, il a été supporté aussi pour développer à TL2 pour Nestid. Donc, ce n'est pas abstrime, mais c'est en flight. Donc, cette fois-ci, le framework va vous donner services au niveau de la code guest. Donc, vous allez trouver basic OS services, Vector Handling, SMP, UART, Devise Tree, Interrupt Management, etc. Some Lipsy Functions, and test specific utilities to report errors, etc. And besides those services for guest code writing, the framework also brings some interesting batch scripts that will develop some of them later on, which allows to configure the test to group them and to also test migration with QMU, for instance. So, now we have seen that the part that is tested is different in KVM self-test and KVM unit tests. So KVM self-tests are very adapted to the test of existing and new KVM user APIs. So it's meant for that. And the test will involve very simple guest code. So it's very practical also to test init sequence for instance. And I think about that because we have suffered a lot with VGeek, PMU init sequences and it could have been really fruitful to have such kind of test in this framework. And it's largely used also for nested in other apps. Now, for KVM unit tests so it's different. So you focus on guest code writing so you can write much more involved guest code that cares for interrupts, for timers and what is also part of the software being test is the VMM user space. So it would be QMU or KVM tool and so it's a major difference compared to KVM self-test where the tester directly calls the KVM user APIs. So it's totally different. So KVM unit test is very adapted to the test of internal emulated devices and it's largely used for that. So the majority of the test code is aiming at that today. But it's also used to test microbenches. You can also test migration which is a difference compared to self-test as well. And it only works with QMU because QMU is the only user space I mean easy the user space that supports migration compared to KVM tool and it's largely used also for nested testing. So now let's have a look at GitStats. So it's not meant to look into the details but I wanted to emphasize the big difference between x86 commits in the different frameworks compared to the other arch. You can see that the number of commits related to ARM are very slow. Of course you have some commits that are related to framework stuff very small number of commits related to ARM in both frameworks. So if we look now at KVM's test, so it's quite simple actually we have no ARM specific test. We only have the framework itself that supports ARM and we have few tests which are shared between different architectures. For instance we have stolen time, maxVCPU, maxMEMSLOT test demand paging. Those tests were generally initiated on x86 and were ported to ARM later on. So on KVM unitest the picture is a little bit different we have ARM specific test so you will find some of those here so SMP, GIC, IT, SMSI controller, PMU, RTC, some cache test, microbenches, PSCI but still the coverage even for those devices or some of those devices the coverage is insufficient and we don't have so many tests actually. So now I'm going to enter into more details with respect to each kind of test and I'm going to introduce some examples actually because I it was easier to understand what the frameworks allows you to do. So for KVM safe test I chose the still time test example so this test allows you to test that stolen time value reported to the guest by KVM matches the PROC FS SCAT stat value. So what this test does on the host is that it's calling a set of KVM user API function so definitely it's going to create a VM, create name slots, create some guest VA, guest PA mapping within the guest create the VCPUs what is very interesting is that it's going to test the API for stolen time so it's going to check the stolen time capability it is going to configure the device attribute for stolen time and test different kinds of errors and eventually we will run the guest so what runs on the guest is very simple code that call SMCC that does SMCC calls to read the stolen time value and write the rate value at some places that is readable by the host so there is a mechanism to share to share the variable between the host and the guest and of course since the goal is to steal time from the guest so the test is going to launch a thread with the same affinity as the VCPU that steals time from the guest and eventually the host is going to read the guest stolen time value read by the guest and compare it against the procfs value so this is a good example of what can be achieved with the KVM safe test so I wanted to highlight a few things if you were to remember only a few stuff so there is no need for any user space integration so you do not need to wait for a given feature to be integrated in QMU or in KVM tool so you can write the test directly once the KVM series is available what is very nice is your testing efforts are going to be easily visible from the kernel community because it's in the same tree it's a very nice way to learn the KVM user API of course the setup is easy and you can develop tests very fast because you don't boot any guest real guest so I mean it's very fast iterating so there are crying needs with respect to self test because we have seen there are very very few tests actually we have even for the tests which are shared with other architecture some of them are not actually implemented for ARM so I'm thinking about memslot related tests so this can be done quite easily I think and of course we have no ARM specific test yet so logically any new KVM API, KVM user API should be unitary tested with KVM self test framework before any user space integration so there is a lot of work already there is a need for fuzzing test this was reported by Marc Zangier we need to emulate ilbef user spaces we can write some ilbef guest as well trying to use features that the host is not supporting there is a support there is the testing of nested virt which is looming and potentially also I think there is some need for improving the documentation for the framework itself to make the ramp up maybe more faster I mean now let's have a look at KVM unit test so I chose to take two examples because I was involved in the development test so I'm going to talk about ITS MSI controller test and the PMU test so both of those tests are related to internal emulated devices so the goal of the ITS MSI controller test was to to be able to program the translation for for LPI and to trigger some interrupts for instance and make sure that the right interrupt was triggered eventually so the drawback of this kind of test is that actually the programming of the device requires quite a lot of logic to set up the translation tables because it does some translation like an IUMMU for instance so writing such kind of logic comes to somehow rewriting a quick and dirty driver so it was a lot of code and it took quite a lot of effort to put in place all the headers the helpers that were required to initiate the first test also what was not very practical with this device is that the hardware device practically ignores most of the wrong programming so it's not very tester friendly in the sense that you cannot easily check that when you apply a wrong configuration this one is properly properly handled also I must confess that my test came too late in the development process so I knew that but I had a bug producer for a bug that was reported to me related to ITS migration and I revived the series I wrote quite a long time ago so since the test came too late so actually I did not find any bugs and it's also due to the fact that I did a lot of efforts putting in place the environment and unfortunately we tend to stop at this stage so what would have been I should have leveraged on this infrastructure to write other test on top of that and I think it would it's still time to do that but so it's costly so well we are all busy but nevertheless it was a good opportunity for me to enable the migration testing on ARM and I will talk about this feature later on so for the PMU so it was more so the programming of the PMU is much more low level so it's more a matter of writing into the registers enabling the counters setting up the proper filtering triggering interrupts checking the interrupts are properly so it was much more easy with less logic in the test itself also there were some existing tests related to the cycle counter so it was more an incremental approach and what was very interesting with this PMU test is that you have a fine-grained control on the registers as opposed to if you were to use the perflayer which adds some abstraction so those tests arrived on time because there was a new feature chain counters which had been introduced that so much time before so I was able to find bugs and to fix them in the kernel and also what is interesting is that we have some new versions for the PMU in 8.5 which bring 64 bit counters and I think it would be very easy to leverage on those tests and increment them to test the new version I discovered that some tests were not passing on the hardware so it's always interesting to notice that and also those tests were useful to validate QMUTCG PMU even counter support so it shows that if you write test it can be used with QMUTCG and QMUTCG so you must remember that when you write QMUT test you also test a user space you focus on guest code you have interesting scripts that enable to automate things and is the CI integration this test is going to run on TCG or KVM it relates to ARM32 bit, ARM64 bit you can test migration you have also a nice framework to handle ERATAS so typically this checks that you have a specific commit and if you have this commit you can adapt the test so it also helps the CI integration and we have also crying needs we need more fuzzing test we need to test more vCPU feature there is nested coming we need to improve the coverage of existing test and I think maybe we should think about better advertising the framework from Linux and Alexandre made a good suggestion I think which consists in adding a tag reported by KVM unitest it may be interesting to improve the visibility and give credits to KVM unitest testing now I wanted to spend some time shedding the light on the fact that we need to develop test on time and for that we can use ARM models of course we can use TCG as well but in my case I chose to experiment usage of the AVP base model so those are free of charge models I started with a blog written by Andrew Murray which I think is very interesting to start with for me the most difficult part was to find a good image which must be light but rich enough to compile the user space in case of KVM unitest and also what was not so simple for me was to find a good device tree I needed to hack it actually and of course you need to get familiar with the numerous options of the model to enable VRTIONET, NIDP so I eventually got a successful setup so this is definitely not maybe the best setup you may find but actually it works so I compile my host and the test on natively on an ARM machine I share them with NFS on my laptop so I have the model which is only X86 I build the boot wrapper image which will be loaded by the model I have my file system and to make possible fast iteration so I share the test which were compiled natively on ARM through NFS and 9p so this is my manner to share the test easily with on the guest so I wanted to emphasize the fact that if you want to run a KVM unitest so for me running QMU was something that was not really bearable so it's very slow also generally it's difficult to compile so I think we really need to continue our efforts to shrink the executable and go in a direction that looks like to KVM tool in a term of size and execution speed by tweaking the configuration and making so we will need to make it as light as KVM tool eventually if it's possible so with KVM tool I was able to develop test without any hardware so those were the statistical profiling extension test of course I'm mostly interested in QMU integration and with KVM tool I'm not testing that, I cannot test migration but I think it's a good alternative to be able to develop test on time I wanted also to emphasize that with KVM tool you directly launch your test with KVM tool so you do not have the same automation as the one as you have with QMU but nevertheless it's very helpful so I don't want to spend much time on this slide so it's for you to see the kind of model invocation I use you can see the console of the model with my SP test running you can see the KVM tool command line which is used in the guest and the way I mount my shared directories the directories that share test KVM unitest and KVM self test sorry now the last feature I wanted to to talk about is the capability to test migration with KVM unitest and QMU user space it's a very nice feature we give you a great flexibility so it's very easy to define such kind of test you need to put your test in the migration group then the framework will launch automatically the source QMU and the destination QMU transparently and then you choose the instant when you start the migration from the guest the guest code initiates the migration by outputting the migrate keyword put migrates this is interpreted by the scripts and you can this notification the QMP migration command will be send then the guest code needs to wait for the migration to complete so it's very easy and very practical because typically for me it helps me to reproduce the bug on the ITS migration where I need MS size to be pending and so it works and so it was not that easy but with that framework it was very very easy to reproduce so as a conclusion I would say we have two really state frameworks which are largely underused on ARM and it's really OPT I think we need to pay attention to develop test on time that's where they are the most useful and for that I really think we can we can use QMUTCG or ARM models in an efficient way so what's very nice with those frameworks is that you can develop things with fast iterations you do not need to wait for the guest to boot I wanted to emphasize also that those frameworks are largely integrated in distro CIs so regularly those tests are run to check there are no regression so we still find developers using their own frameworks but it's a pity because it could be CI integrated and also it's a very nice way to learn and it's a easy way to contribute and to keep up with the pace in the KVM for ARM subsystem it complements the review process you can put the ants in the dirt and pick and poke the registers tweak the guest code so it's very nice and so I mentioned that many times we have crying needs we need bug reproducers we need the fuzzing and as a last comment I would say that I think we need to continue our efforts trying to get QMU lighter to be renewable on the ARM MVP model I know some people run QMU on the model but still I think it takes too much time executing and we can we can do better and so this is the end of my presentation so of course feel free to contact me if you have questions on the setups and so on I would be very happy to discuss that with you so thank you very much for attending