 Hello everyone, my name is Konstantin Kostyuk, I am a software engineer at Denix. Today we are going to talk about HTK ACI enabling CI for Windows Guest Paravirtualized Drivers. During the presentation we will talk about Windows Drivers for Virtio devices, driver testing and certification processes, automating these processes, challenges we faced while working on the CI. Virtio Win is a set of Windows Drivers for Paravirtualized CUMU devices based on Virtio specifications. All Drivers are open source and published on GitHub. Now Drivers for the major Virtio devices are implemented, some Drivers are still in the development stage. We want to be open to contributions from other developers, while being sure that pull requests do not break driver functionality and follow Microsoft quality guidelines. Manual setup and preparation of Windows Hardware Quality Labs are not easy, this leads to a high entry threshold for new developers. So we need a way to automatically run WHQL testing on our PRs. Windows Hardware Quality Lab testing also known as Windows Logo Testing is Microsoft testing process which involves running a series of tests on third party hardware or software. And then submitting the log files from this test to Microsoft for review. Before submitting a driver to Microsoft for certification it needs to be tested. Certified Drivers have some advantages. First, Drivers certified by Microsoft are generally more trusted by the users and do not track your traditional installation steps. Second, it is the only way to use automatic driver installation via the Windows Update channel. Third, Uncertified Drivers cannot be used with secure boot enabled. The device won't boot properly. Hardware Lab Kit Replace Hardware Certification Kit in Windows 10. This kit provides utility to test a driver using the WHQL test playlist. HLK consists of two main parts, controller and client. It shows OS crashes or other unexpected behavior, so it is required to have a separate workstation to orchestrate clients and tests, which is called controller. Controller performs driver installation, specific setup for each test, run test cases and collects results from connected clients and decides on the test set results. Tests may record more than a single client workstation, so there are two main client types, device under test and support device. If the test records more than one workstation, a driver under testing must be installed on all of them. The device under test is a single primary client that executes more test steps. Support devices used as helpers to emulate the record environments. For example, to test a network driver, environments need to have another device that will communicate with the device under test via network using the provided driver. Such support device will verify network packets received from the device under test. The certification process can be split into device driver certification and certification for the platform itself, SVVP. Some types of driver can have specific tests. For example, Andy's test for network driver or DirectX test for graphic card driver. All drivers also need to pass driver verifier tests. During the test process, devices and client workstation will be subjected to stress loads to ensure system stability. Server virtualization validation program is a policy that allows administrators to receive technical support for Windows system running on certified virtualization technologies, other than Hyper-V. Such certification is designed to verify the hyper-visor quality and stability. Prior to Windows 10, Microsoft used HTK to run WHQL tests. HTK is a single kit designed to run on any of the supported systems. So this is possible to use a single controller setup to run any test on different clients in parallel. Starting from Windows 10, Microsoft releases new version of HLK for most new Windows builds. The new release schema has a big limitation that only one HLK controller version can be installed on a single workstation. So additional work is needed to configure a test environment for all record Windows versions. There is a sample configuration for network driver testing. Some tests record to clients, so the support device in present in the environment. Control bridge is used for communication between controller and both clients for transfer test cases and run test software on it. Test bridge is used for communication between network drivers under test. For many other devices one client is enough, so in this environment a test bridge and support device are absent. We need to have the ability to run certification as part of the CI process. Provide a mechanism for individual developers to test their source code before sending pull request. And the ultimate goal is to use HTK CI for actual certification in production. Virtue HTK is a set of shell scripts for running Microsoft hardware certification kit test for a variety of Windows drivers in a virtual machine environment. The development of Virtue HTK began as utility to help developers check their code. It creates an isolated virtual network environment for HTK controller and clients, virtual machines running on top of QMKVM. Creates images snapshot and allows to run of various HTK tests in automated mode. Additionally, fast recovery from OS corruption is possible. Just delete the current disk snapshot and create a new one from an original image. And the results are expected to be very consistent, no physical hardware that may malfunction is involved. Except possibly they tested the device itself if it is indeed physical. To automate the process of WHKL testing it is necessary to solve several problems. To develop a utility for remote control of HLK environment. To develop software for VM and network orchestration. To develop a system for managing test and their results. Auto-HTK is a framework for HTK CI. Auto-HTK has a modular architecture that allows easily to add a new supported test manager or new feature in general. There are four main components. Gas communication layer, engines, setup managers and result uploaders. Also Auto-HTK has an auxiliary library for all components. Engines execute main logic and control other components. Currently there are two engine tests and install both of them are for HLK environment. The test engine run HLK controller and perform WHKL testing. The install engine prepare HLK environment from the scratch. Gas communication layer provides the ability to execute command and call API function on guest OS. Currently AirTools-HTK is used for this. Setup managers provide communication layer to hardware. Currently there are two managers QMA-HTK and FIS-HTK. QMA-HTK creates a virtual environment with QMA-HIVM hypervisor. FIS-HTK communicates with real hardware. Results uploaders provide API for managing test results. Uploads the log-to-cloud storage and updates GitHub commit status. The guest automation tool is called Tool-HTK. It is a PowerShell-HTK-HLK controller API wrapper that wraps and exposes the API to users as a shell. This tool can choose a target test driver, run a test on a device from a pool and collect all results into a package. Let's discuss how to use tools-HTK, how a manual automation tool. Before run test the machine pool and HLK project should be created. To create a pool using the create pool command with the pool name as an argument. The same for project creation. The next step is to select a target device for testing. To select a target using the create project target command with the following arguments. Hardware device ID, HLK project name, destination test unit and machine pool name. When the target device is selected, HLK can run tests. To run the test using the create test command with the next argument. Test ID, Hardware device ID, HLK project name, destination test unit and machine pool name. To find the test ID use HLK test reference documentation or the list test command. To get the test results using the list test result command with the same argument as a queue test. When all tests are finished the HLK result project can be exported. Use the create project package command for this with the next arguments. HLK project name, optional argument is the package path. If the package path is specified then results will be exported to this file. In another case will be exported to a temp directory. So there were base commands that used for running tests. Of course some step was skipped for example driver installation. Rtools.htk is a RubyGAM toolkit based on tools.htk for managing and maintaining the htk setup remotely. It provides all tools.htk features for the host machine. Rtools.htk use windows remote management protocol for communication with guest OS and TCP protocol for communication with tools.htk. WinRM allows to run PowerShell commands on remote windows machines and provide file manager between host and guest. htk.ci use file manager for uploading filters and playlists to guest and unloading test results and windows mini dumps after client blue screen of death to host. After htk prepares the host environment to run VMs with the configuration according to the selected driver. In the beginning it creates guest images that will be used for the test. For each test run it creates proper network topologies and image snapshot for each VM and then run VMs with the appropriate device connected. Test managers control the configuration of OS and platform that will be used as a guest OS. It assign an htk playlists for the corresponding OS version and applies htk filters for finishing tests. During test execution the manager control guest OS status and htk test results. A report manager has several important tasks. It parses htk results allowing to receive results of tests without htk studio installation. A report manager can upload results to cloud storage. Currently htk has only Dropbox support implemented. If blue screen of death occurred during the test run a report manager will save mini dump files. Also it can use github integration to update pull request status during the CI process. The server records a lot of resources to run htk CI. We should calculate the CPU and RAM of all VMs that will be run during the test process. The controller VM run with 2 CPUs and 4 GB of RAM. Client resources depends on guest operation system and driver requirements. For example the RSS test for network drivers records the number of CPUs to match at least the number of RSS queues. Also for some drivers 2 clients should be run. Recommended CPU and RAM can be found in platform config file. htk CI can work with CPU over commit but for smooth performance the host should have enough CPU. CPU with hyperthreading feature is also supported. Auto-HTK had been tested on the following host OS Ubuntu 2004, Fedora 32 and Red Hat Enterprise Linux 8. Internet connection is used by many system components. During installation for the hlk online installer the offline installer is also supported. Downloading extra software. During the test process for downloading the hlk playlist and filters. Additionally auto-htk upload driver binary, logs, test results and updates github commit status. There is possible to run htk without an internet connection but all online features will be unavailable. All source code are available on github. Auto-htk repository contains installation readme but in general installation is easy. Auto-htk is written in the Ruby language so first of all Ruby should be installed. The minimal version is 2.6.5. Then there are 4 repositories that should be cloned. The next step is configuration. htk framework has many configuration files but almost all of them are ready to use from the repository. All files are in JSON format. The JSON format is versatile, it can be easily edited by a user, at the same time it can be generated on the fly. This allows making simple configuration and adds new automation. These files can be split into 3 groups. Global configuration, setup manager configuration and engine configuration. There are 3 files of global configuration. Config.JSON contains a general auto-htk configuration. Driver.JSON contains information of all drivers which can be tested by htkci. This information includes VM device name, INF file name, drivers installation method, hlk target name, pre-test commands, list of hlk tests which should be ignored. SVP.JSON contains specific configuration for system certification. Currently, htkci has 2 setup managers. Their configuration is present in the following files. QMMachined.JSON contains information about QMM virtual machine setup manager. This file includes paths for QMM binaries, default VM settings and network settings. This htk.JSON contains information about physical setup manager. It includes the hlk kit version and controller server IP address. htk framework has 2 engines. Their configuration is present in the following files. hlk install.JSON generic configuration for install engine. This engine has some specific configurations. ISO.JSON contains information about Windows ISO for each htk hlk platform. This file contains the only template so it should be filled in according to the manual. QMM.JSON contains information about each htk hlk kit. It includes kit name and version and installer to float your app. Studio Platform.JSON contains information on which platform should be used for each htk hlk kit. Platform.JSON contains information of each platform. htkci can be integrated with optional software Jenkins and Sentry. Jenkins is an open source automation server. Jenkins allow configuring a lot of different triggers to run out of htk automatically. For example, Jenkins can pull github pull request and run htk testing for new PRs. Sentry is a service that helps you monitor and fix crashes in real time. The server is in Python but it contains a full API for sending events from any language in any application. Sentry has an alert mechanism. Alerts provide real time visibility into problems with your code and impact on your users. There are several types of alerts available with customizable threshold and integrations. Alerts can be sent to email, Slack or other alert providers. Sentry sends notification to your Slack channel for the alerts and workflow updates you define. After htk can create guest OS images automatically. Automatic installation is performed by using answer files. This is Windows ability to install and configure the system from the scratch to the full end. The unattended XML answer file contains settings for each Windows setup page. When a setting for a specific setup page is configured, Windows setup skips that page. When a setting for each page is configured, Windows setup runs without any user interaction. htk framework uses predefined answer files and additional hlk setup scripts to configure the system. To run installation, use the install autohtk command line command. Autohtk will create three images, controller and two clients and perform the installation. What happened when htk ci in running? Autohtk was triggered to test the iWishMem driver. There you can see logs from the host OS. If connected to the controller VM via VNC, hlk studio can be opened and attached to the current testing process. There is a screenshot of a client VM where some kind of test for iWishMem device is running. Two projects that use htk ci for upstream driver testing. It's Virtio Win and OpenVPN. In conclusion, we have the hlk ci that running for several upstream projects has driver testing and SVP testing automation and it can easily be updated with new features. In our plans, we want to add other hypervisors and result storage. So, thank you very much. If you have more questions, please ask me in the chat or send me an email with questions and comments. I will be happy to answer them. Thank you very much. Bye.