 Good morning, everyone. I'm Smitha from Samsung. Today I'll be presenting Unraveling Hub Firmware Interacts with the Linux app system. Agenda for today will be introduction, types of firmware request, firmware cache, direct file system lookup, built-in firmware, fallback mechanism, firmware lookup order, and example using MFC. Have you all wondered how you're able to load the device driver firmware onto the device microcontrollers or microcode for CPU Erata, or even get device driver information on calibration data, e from overrides, etc. All these are possible due to the firmware API provided in Linux, which enables the kernel to request files for necessary functionality from user space. Also, with the increased SoC complexity, where number of multiple CPUs are new normal, like NPU, other hardware accelerator, firmware system has become extremely important. Now, there are two types of firmware request, synchronous and asynchronous. The use of these calls depends on your requirements. The thumb rule is we should opt for asynchronous APIs unless you are already using asynchronous initializations which will not stall or delay booting. Even if loading firmware does not take a lot of time processing firmware might, and this still can delay boot or initialization as such mechanism such as asynchronous probe can help supplement drivers. So, these are the firmware structure and important terminology. Now, struct firmware has size and has data and firmware loader. And we also have an enum which has all the important firmware operations. So, we have firmware opt event. It enables the fallback mechanism to send a K object event when the firmware is not found. Now, user space is in charge to load the firmware using the surface loading facility. Firmware opt no wait. It is used to describe the firmware request as asynchronous. Firmware opt user helper. It enables the fallback mechanism in case the direct file system lookup fails at finding the firmware. The firmware opt no want. It's quite and it avoids printing warning messages. Firmware opt no cache. It disables firmware caching. Firmware caching is used to cache the firmware upon suspend so that upon resume, there's no race against firmware file lookup in the storage. And it is used for calls where the file might be too big or where the driver takes charge of its own firmware caching mechanism. Firmware opt no fallback surface. This is used to disable the surface fallback mechanism. It takes precedence over firmware opt event and firmware opt user helper. Now we have firmware opt fallback platform. It enables fallback to device firmware copy embedded in the platform's main firmware. If both this fallback and the surface fallback are enabled, then this fallback will be tried first firmware opt partial. It allows partial read of firmware instead of needing to read the entire file. So the base code for firmware is a driver's base firmware loader. You would typically load firmware and then load it into your device. So firmware flow is first requesting firmware and then copying it into our device and then releasing the firmware. If you use request firmware and it returns, the driver has the firmware image accessible in FW underscore entry with the data and size. And if something went wrong, request firmware returns non zero and firmware entry set to none. So once your driver is done processing the firmware, it can call release firmware with FW entry to release the firmware image and any related resource. Synchronous APIs. So synchronous firmware request will wait until the firmware is found or until an error is returned. So in the upcoming slides, we'll discuss various asynchronous APIs which are present. First we'll be discussing the request firmware. It sounds it sends firmware request and we'll wait for it. So we have arguments to this firmware P, which we use to return a firmware image name is the name of the firmware file device is the device which is requesting the firmware. So for this request firmware APM, it will have firmware flags as firmware opt U event. This indicates that the fallback mechanism is to send K object event event. And it should be called from user context where sleeping is allowed. And the name variable will be used as firmware in the event environment, and should be distinctive enough not to be confused with any other firmware image for this or any other device. The caller must hold reference count of the variable device and the function can be called safely inside devices, suspend and resume callbacks. Next is the firmware request no one. So it is a request for an optional firmware module. And it has the arguments firmware, which is a pointer to the firmware image name, it is the name of the firmware file devices for which the firmware is being loaded to me. So this will have firmware flags as firmware opt U event and firmware opt no one. So you can, as mentioned, we'll have fallback mechanism to send object, K object event and without any warning messages. So this function is similar to request firmware. The difference is that it doesn't produce warning messages when the file is not found. This fallback mechanism is enabled if direct file system look up fails, but failures to find the firmware file with that are still suppressed so that the driver has to check for the return value of this call and decide when to inform the users of errors. Next will be the firmware request platform. We used to request firmware with the platform FW fallback. It also has a firmware with pointer to the firmware image name of the firmware file device for which the firmware is being loaded. And the request firmware will be called a firmware opt event and firmware opt fallback platform. So it states that we have fallback mechanism to send K object event and as mentioned earlier to device firmware copy embedded in the platform's firmware. So this call is again similar in behavior to request for when except that if direct file system look up fails, it will fall back to looking for a copy of the requested firmware embedded in the platform's main firmware. For example, UEFI, which we'll discuss later. Firmware request data. It loads firmware directly without user mode helper. It also uses firmware P pointer to firmware image name of the firmware file device for which the firmware is being loaded. And the flash which are sent to the firmware request is firmware opt event firmware opt no one firmware opt no fallback. Sysfs which indicates we have fallback mechanism to send K object event without any warning messages and it also disables the Sysfs fallback mechanism. This function also is similar to the request firmware but it doesn't fall back to user mode helper even if firmware couldn't be loaded directly from the file system. So it is useful for loading optional firmwares which are which aren't always present and without extra long time whatsoever. Next is request firmware into buff. It is used to load the firmware into previously allocated buffer. It takes in firmware underscore P which is pointed to firmware image name of the firmware file device for which the firmware is being loaded and the DMA region allocated. Buff is the address of the buffer to load firmware into sizes size of buffer. This function again pretty much works similar to request from them but it doesn't allocate a buffer to pull the firmware data. Instead the firmware is loaded directly into the buffer pointed by variable buff and the variable firmware underscore P data member is pointed at variable buff. So this has firmware op no cache and firmware op you went request firmware. No way. So asynchronous firmware request allowed driver code to not have to wait until the firmware or an error is written function callbacks are provided so that when the firmware or an error is found the driver is informed to the callback request firmware. firmware no wait is only asynchronous firmware API and this cannot be called an atomic context request firmware no wait is an asynchronous version of firmware of request firmware. So this takes in module. Which is our model requesting the firmware you went which sends you went to copy the firmware image if this flag is non zero as the firmware must be copied manually. Name of the firmware fi device for which the firmware is being loaded GFPR the allocation flags. Context will be passed over to a function and firmware may be null if a request fails. And haunt has a function which will be called asynchronously when the firmware request is over. So color must hold the reference count of the variable device. And this asynchronous variant of request firmware we use for user context if sleep for as small as a small periods of as possible. Since it may increase kernel boot time of the built in device driver requesting firmware during the probe methods if the GFP set to GFP kernel. And it can't sleep at all if the GFP set to GFP atomic. So we pass a request firmware API with flags firmware of no way where no way tells that it is an asynchronous request. And based on the Boolean variable you went it sends either K object you event as fallback mechanism or fallback mechanism in case of the direct file system lookup fails at finding the firmware using firmware fallback surface which will be discussing further. When Linux resumes from suspense some device driver request firmware lookups to reinitialize devices. During resume there may be a period of time during which firmware lookups are not possible. During the short period of time firmware request will fail. So timers of essence though and delaying drivers to wait for the root file system for firmware will delay the user experience with device functionality. So in order to support this requirements the firmware Instra infrastructure implements a firmware cache for this device drivers for most of the API calls and it is automatically called behind the scenes. The firmware cache will make use of certain firmware API calls, which are safe for to be called during device driver suspend and resume callback. And users of these API calls need not cache the firmware by themselves for dealing with firmware loss during the system resume. So the firmware cache works by requesting for firmware prior to suspend and caching it in memory. Upon the resume of the device drivers using the firmware API will have access to firmware immediately without having to wait for the root file system to mount. Or dealing with any possible race issues with the lookups as the firmware file system mounts. So the implementation details about the firmware cache setup. The firmware cache is set up by adding the dev res entry for each device that uses all asynchronous calls. This is for accept request firmware into bug. When asynchronous call is used the firmware cache is only set up for a device that the second argument that is that you event for request firmware no wait is set to true. So when you event is true it request that a key object you went to send to user space for the firmware request to the surface callback mechanism if the firmware file is not found. The firmware cache is determined to be needed as per the above criteria which I mentioned the firmware cache is set up by adding a dev res entry for making the firmware request. Now the firmware dev res entry is maintained throughout the lifetime of the device. This means that even if you release firmware the firmware cache will still be used on resume from suspend. So the fallback mechanism is temporarily reduced to 10 second as firmware cache is set up during suspend and the timeout is set back to the old value had configured after the cache is set up. So upon suspend any pending non you event firmware requests are killed to avoid stalling the kernel. This is done with kill request without you events and the kernel calls requiring the non you event need to implement their own firmware cache mechanism. So you must not use the firmware API on suspend. So using this firmware cache mechanism there is special optimization that can be done on boot. So some devices have an optimization in place to enable firmware to be retained during system reboot that is not requiring the loading of the phone when. When the optimizations are used, the driver must ensure the firmware is still available on resume from suspend. So this can be done with firmware request cache instead of requesting firmware to be loaded firmware request cache with cache firmware for suspend. So resume can use it. So this uses some the argument name name of the firmware file device for which the firmware should be cash for. This helper is not compatible with drivers which use request firmware into buff or request firmware no way with no you event set from the search path. The following search parts are used to look for firmware onto the root file system. So we have firmware path para. This is a module parameter which default is empty. So this is normally ignored. The other search parts include lip firmware updates UTS release lip firmware updates lip firmware and so on. So the module parameter path can be passed to the firmware class module to activate the first optional custom firmware path para. And this custom path can only be up to 256 characters long and the kernel parameter past would be firmware class path which is a customized part. There is an alternate to this customize the path at runtime after boot up. You can also use the file since module firmware class parameters path. You would echo into your custom path and the firmware requested will be searched there first direct file system lookup. This is the most common form of firmware lookup performed by the kernel. The kernel looks up for the firmware directly on the root file system in the path discussed earlier. And the file system lookup is implemented in firmware get file system firmware. So it uses common core kernel file loader facility kernel read file from path. The max path allowed is path max which is 4096 characters. It is recommended you keep lip firmware path on your root file system avoid having a separate partition for them to avoid possible raise with lookups. Or to use custom fallback mechanism again which will be discussed further. So drivers which are built into the kernel should have firmware integrated as part of the init ramfs used to boot the kernel. Otherwise it will lead to raise with loading the driver and the real root FS may not be available still. So stuffing the firmware into the init ramfs results this race issue however we need to use init rd which will not suffice to address the same race. So there are circumstances which justify not wanting to include firmware into the init ramfs such as dealing with large firmware files for the remote proc subsystem. In such cases a user space fallback mechanism is the only viable solution. The user space only can know for sure when the real root FS is ready and mounted built in firmware. So firmware can be built into the kernel. This means that building the firmware into the VM Linux directly to avoid having to look for the firmware from the file system. So you can directly look it inside the kernel itself so you can enable the built in. So using the kernel option config extra firmware and cost pick extra firmware there. So advantages of building your firmware into the kernel with config extra firmware speed. And firmware is needed for accessing the boot device and user doesn't want to stop the firmware into the boot in a ramfs. And disadvantage is this firmware is a non-GPL compatible thing and some firmware might be optional. Firmware upgrades are possible so a new firmware would lead to complete kernel rebuild. And some firmware files may be really large in size like how we discuss about the remote proc subsystem. And the firmware may need to be scrapped out from some device specific location dynamically. For example calibration data for some Wi-Fi chipsets. This calibration data can be unique for each device. Fallback mechanism a fallback mechanism allows you to overcome failures to do a direct file system lookup on the root file system. Or when the firmware simply cannot be installed for practical reasons on the root file system. So why do we need fallback mechanism when there's a race against access with the root file system upon boot up or where is the race upon resume from suspend. This is resolved by the firmware cache but the firmware cache is only supported if you use new events and it's not supported for request firmware into bug. Another reason is firmware is not accessible through typical means that is either it cannot be installed into the root file system or when critical info is installed on a separate flash partition other than where the root file system is provided. Now the firmware provides very unique device specific data tailored for the unit gathered with local information and example as mentioned earlier is the calibration data for Wi-Fi chipsets. This calibration data is not common for all units, but it is tailor made for each unit. So kernel configuration options to support this firmware fallback mechanism. First one is the conflict firmware loader user helper. It enables building the firmware fallback mechanism and most distributions enable this option today. If enabled, but if the conflict firmware loader user helper fallback is disabled only the custom fallback mechanism is available and is further request firmware no wait call. Now if you have conflict firmware loader user helper fee fallback it force enables each request to enable the K object U when fallback mechanism for all firmware APS except for request firmware direct. But this is disabled in most of the distributions today. The call request firmware no wait it allows for one alternative fallback mechanism if this K conflict option is enabled and your second parameter to request firmware no wait that is the event is set to false you are informing the kernel that you have a custom fallback mechanism and it will manually load the firmware. And when we have conflict firmware loader user helper is set to yes and conflict firmware loader user helper fallback set to no the K object U event fallback mechanism will never take effect even for request firmware no wait when you went to set to true. So before we proceed further let's discuss on the firmware surface to understand the types of fallback mechanism. In order to help device drivers upload firmware using fallback mechanism the firmware infrastructure creates a surface interface to enable user space to load and indicate when the firmware is ready. The surface directory is created via firmware create instance. This call creates a new struck device named after the firmware requested and establishes it in the device hierarchy by associating the device used to make the request as the device is parent. The specifics directory file attributes are defined and controlled through the new devices class that is firmware class and group that is a firmware dev attribute groups. This is actually where the original firmware class module name came from, even that originally the only firmware loading mechanism available was the mechanism we use now as the fallback mechanism, which registers a struct class from a class. The attributes exposed are a part of the module name, the module name firmware class cannot be renamed in the future to avoid backward compatibility with older user space. To load the firmware using the surface interface we expose a loading indicator and a file upload firmware sys dev part loading sys dev part data and to upload firmware you will echo one onto the loading file to indicate your loading firmware. You then write the firmware into the data file and you notify the kernel, the firmware is ready by echoing zero onto the loading file. The firmware device used to help load the firmware using sys sys is only created if the direct firmware loading fails and if the fallback mechanism is enabled to your firmware request. So this is set up with function firmware fallback sys sys. It is very important to reiterate that no device is created if a direct file system lookup has succeeded. The echo one into sys dev part loading, this will clean any previous partial load at once and make the firmware API return an error. When loading firmware, the firmware class grows a buffer for the firmware in page size increments to hold the image as it comes. And the firmware data read and firmware loading show, I just provided for the test firmware driver for testing. They're not called in normal use case or expected to be used regularly by the user space. Now firmware fallback sys sys, this is used as a fallback mechanism to find firmware. This takes in firmware pointer to the firmware image, name of the firmware file to look for, device for which the firmware is being loaded. OPT flags are options which are used to control the firmware loading behavior as defined by the enum firmware R. Then we have RIT, which is the return value from direct lookup, which are triggered by the fallback mechanism. Now firmware fallback sysfs is called if direct lookup for the firmware failed, it enables a fallback mechanism for user space by exposing a sysfs loading interface. This sysfs fallback mechanism may be disabled completely on a system by setting the proc ctl value that is ignore sysfs fallback to true. If this is false, we check if the internal API caller set the firmware of no fallback sysfs flag. If so, it would also disable the fallback mechanism. Now a system may want to force a sysfs fallback mechanism at all times. It can do this by setting ignore sysfs fallback to false and force sysfs fallback to true. So enabling force sysfs fallback is functionally equivalent to building a kernel with config firmware loader user helper fallback. Types of fallback mechanism. We have three types of fallback mechanism K object event custom and EFI. K object event and custom are the only two fallback mechanism which are using the sysfs interface as a loading facility. K object event. Since a device is created for sysfs interface to help load firmware as a fallback mechanism, user space can be informed of addition of the device by relying on the K object events. Now the addition of device into the device hierarchy will mean that the fallback mechanism for firmware loading has been initiated. For implementation, we need to follow firmware load sysfs fallback in particular use of DevSec event suppress and K object event. Now the firmware load sysfs fallback loads a firmware via the sysfs fallback mechanism taking inputs as a firmware sysfs which is the firmware sysfs information for the firmware to load and time out. Which is the time out for the loading. Now this is basically in charge of constructing a sysfs fallback interface for firmware loading. And if the flag firmware opt event is set to to then the DevSec event suppress function, it will pass the device structure along with another argument false. First this it will call the object K object event with K object add parameter. This will notify the user space by sending a new event. It takes K object struct that is a K object that the action is happening to. And another parameter action and that is the action happening like K object add K object remove K object bind unbind etc. It returns zero if K object event is completed with success or corresponding error when it fails. It then calls K object underscore ENV which sends an event with environmental data. This takes in a key object to struct again which to which the action is happening to and another parameter for action in the is happening. And ENV P underscore EXT which is a pointer to the environmental data. This returns zero if the K object U event environment ENV is completed with success or the corresponding error when it fails. We need to mark add event so that we can make sure we deliver remove event to user space during automatic cleaner. And this is done by setting variable set add U event send. If the object did add an add event remove will be automatically generated by the core if not already done by the caller. Now the variable pass to user space with K object. Events are firmware timeout and a sync which tells whether the call is synchronous or asynchronous along with the timeout value and the firmware. The kernels K object event mechanism is implemented in live K object event dot same it issues events to user space. And as a supplement to K object events Linux distribution could also enable config event helper path, which makes use of core kernels user mode helper. Which is the UHM functionality to call out to a user space helper for K object units in practice though no standard distribution has ever used the conflict. You and help up but this means that most Linux distribution today are not using or taking full advantage of the phone with fallback mechanism provided by the K object units. This is proven due to the fact that most distribution today also disable conflict firmware loader user helper fallback and by default dev path is set by the internal kernel K object infrastructure. Now let us look at an example example of the K object event script where both the dev path and firmware are already provided in the environment. Now my firmware directory is inside live firmware and we equal one into the system path loading, and we copy the entire firmware into the system path data and post that we equal zero into the system path loading. Now custom fallback mechanism users of the request firmware no wait call have it another option available at their disposal. And they rely on the surface fallback mechanism, but request that no key object events be issued to user space. Now the original logic behind this was that the utilities other than new dev might be required to look up firmware in non traditional paths, that is paths outside what was discussed in the direct file system lookup. This option is not available to any of the other API calls as you events are all based force for them. And since events are only meaningful if the fallback mechanism is enabled in your kernel. It would seem odd to enable you events with kernels to do not have the fallback mechanism enabled in the kernel. Additionally, we also rely on the event flag, which can be disabled by request firmware no wait to also set up the firmware cache for the firmware request. And as discussed earlier the firmware cache is set up only if you went is enabled for an API call. So although this can be disabled the firmware cache for request firmware no wait calls users of this a page not use it for purpose of disabling the cache as that was not the original purpose of this flag. Not setting the event flag means you want to go for a custom firmware fallback mechanism, but you want to suppress the key object events. So you have a custom solution which will monitor for your device addition into the device hierarchy somehow and load the firmware for you in the custom path. On some devices the systems EFI code or ROM may contain an embedded copy of firmware for some of the systems integrated peripheral devices and the peripherals Linux device driver needs to access this firmware. Device drivers which need such firmware can use the firmware request platform function for this. So we should note that there's a separate fallback mechanism from the other fallback mechanism like key object event and the custom. This does not use the surface interface. And the device driver which needs this can describe the firmware it needs using an EFI embedded firmware destruct, which is as shown in this slide. The EFI embedded firmware code works by scanning all EFI boot services code memory segments for an eight byte sequence matching prefix. If the prefixes found then it searches over the length bytes and if that matches it makes a copy of length bytes and adds it to the list with the found firmware. To avoid doing this expensive scan on all the systems we started using DMI matching and drivers are expected to export the DMI system ID adding with each entries driver data pointing to an EFI embedded firmware descriptor. So to register this array with the EFI embedded firmware code. So a driver needs to always be built into the kernel or store the DMI system ID array in a separate object file which always gets built in. Add an extra external declaration for the DMI system ID array to include inside Linux EFI embedded firmware.h and the DMI system ID array to the embedded firmware table. Add selected EFI embedded firmware if EFI stub to its kconfig entry. The firmware request platform function will always first try to load the firmware with the specified name directly from the disk so that the EFI embedded firmware can always be overridden by placing the file under libfirmware. Fallback firmware fallback timeout. So the firmware fallback mechanism has a timeout. If firmware is not loaded onto the surface interface by the timeout value and error is sent to the driver. By default the timeout is set to 60 seconds if you events are desirable. Otherwise this max Giffy offset is used which is the max timeout possible. So the logic behind using max Giffy offset for non events is that a custom solution will have much as much time as it needs to load the firmware. So you can customize the firmware timeout by echoing your desired timeout into the file sys class firmware timeout. So if you echo zero into it it means max Giffy offset will be used and the data type for the timeout will be in. So firmware lookup order. Different functionality is available to enable firmware to be found. This slide shows the chronological order of how firmware will be looked for once a device driver issues a firmware API call. So the first will be the built-in firmware is checked first. If the firmware is present we return it to the device driver immediately. If not then the firmware cache is looked at next and if the firmware is found we again return it immediately to a device driver. Then the direct file system is lookup is performed. If found we again return it immediately to the device driver. Then the platform fallback firmware fallback is performed next but only when firmware request platform is used. If found again this will be returned to directly to the device driver and if no firmware is found then we use the fallback mechanism. So we enable the SysFS interfaces created and with that there is a key object event type of fallback mechanism and that firmware is returned to the driver or we will have a custom firmware fallback mechanism and that again will be returned to the driver. So example using MFC. I'll be taking MFC driver as an example to explain the firmware interaction with Linux for this specific IP. So Exynos MFC is Samsung IP used for video encoding and decoding. It is expanded to multi-format video coding and it is used to encode and decode various codecs like HEVCH264, MPEC4, VP9, etc. And it has capability of handling multiple streams at once. MFC mainly is used in video playback pipeline that is the decoder functionality or camera pipeline that is the encoder functionality. And it has a firmware running on a risk which mediates calls from the device drivers to the actual MFC hardware. So during MFC call of S5P MFC probe or open, the firmware initialization happens from the device driver of MFC. So via this we call S5P MFC load firmware which in turn calls a request firmware API with flags, firmware opt U event which is set as the fallback mechanism. This in turn calls a request firmware prepare which prepares firmware and firmware buff structures. It returns zero if a firmware is already assigned and one if it needs to be loaded or a negative error code. This further checks in for firmware request built-in buff which look for firmware in the built-in kernel. Else it calls for a lock lookup firmware which checks firmware request with the same name. Post this it gets the firmware via firmware get file system firmware where it loads the firmware file from the mount name space of the unit and the end it calls assign firmware and adds a firmware name into a device so that we can auto-cache and un-cache firmware for device. Now don't cache firmware handled without U event and after caching firmware image is started. Let it piggyback on request firmware and at last pass page buffers to the driver via firmware set page data. So you can get back to me for any questions. Thank you.