 Hello everyone and greetings. Welcome to the presentation on V4L2 controls from the perspective of reader capture devices. Myself Satyakam Mehtawaram, the co-author for this presentation is Inbaraj Ilengavan. We are from Samsung semiconductor India research that is based in Bangalore in India. Inbaraj is my colleague. We are working together on VPCSI2 controller development for various Samsung SOCs. We implemented CSI2 drivers based on V4L2 that have multiple sensors serializer and DC laser combination. We prepared this presentation based on our experience with capture devices and the V4L2 controls being used. So, here is the agenda for presentation. We first looking to the structure of V4L2 framework and then we will see what is a V4L2 device, what is a V4L2 sub-device, what are the video device nodes and what are the V4L2 control objects provided by Linux kernel. And then we move on to the topic of controlling V4L2 devices. That is why controls are needed, the control methods provided by V4L2, how the user applications can leverage these controls and we prepared few nodes based on our experience and implementing the controls. Finally, we will look into the application with, we will present a sample capture application with the control methods that we discussed so far. Also, we present the V4L2 CTL utils provided by the V4L utils package. At the end of this presentation, we hope the audience will gain fair knowledge of V4L2 controls and using these controls for the capture devices. Typical video streaming involves multiple devices. In order to explain the structure of V4L2 framework, here we introduce a sample system that consists of camera sensors, serializer and DC laser combination and a host cluster interface to the cameras. For this topic, we assume the host cluster interface to be MPCSI2 interface which is found in most of the systems. In this configuration, cameras are connected to the serializer which then aggregates the camera streams and then transmits over them to the DC laser. The DC laser inter transfers the image streams to the host cluster where the host cluster interface. Here we introduce the term bridge device that is responsible for transferring the image data to the memory using the DMA transfer. Any off chip device or on chip IP are considered as sub devices. In the next slide, we will see how V4L2 framework abstracts these devices. V4L2 framework provides the canal data structures to describe the devices. Struct V4L2 device describes the top is the top level data structure that describes the bridge device and is responsible for managing the child devices. Struct V4L2 sub device describes the sub devices attached to the bridge device. These can be the camera sensors or the MPCSI controller that we shown in previous sample system. Struct video device describes the video device nodes and exposes these device nodes under the slash div directory for the user applications. Struct V4L2 CTRL describes the control properties and keeps track of the control value. Struct V4L2 CTRL handler keeps track of all the controls within the device that are registered with that handler. There can be multiple controls for a device and these controls which are registered with that handler, the V4L2 CTRL handler keeps track of all those controls. Here is the block diagram for the sample system which we discussed in the previous slide. First, we have the struct V4L2 device that represents the device instance and we have the struct V4L2 subdev that describes the sub devices that are attached to the bridge device. Like the sensors, the serializer, the decelerizer or the control interface and we have the video device nodes that are exposed to the user under slash div directory as a slash div video x or slash div V4L sub div x where x is the device number. The total of 256 devices can be registered. Okay, so why controls are needed? Advanced screening systems usually have multiple devices or IPs such as the image sensing processor or the multi-format codec that will be encoding and decoding of video screens. Often, these devices provide configuration parameters that are controllable by user and controls provided can be specific to the device which can be vendor specific and device control needs can be application specific as well. V4L2 framework provides methods to set these controls such as the standard controls, extended controls, custom controls or the private controls. The capture devices can implement all of these controls or some of the required controls. Each of the available controls are identified by the respective control ID values. V4L2 framework arranges these controls into classes which serve as the base for this control ID such as the user controls, the impact controls or the camera class control all the way up to the RF and the detection control. V4L2 framework provides standard controls with predefined IDs. The user class serves as the base for this control ID. The standard controls are defined for brightness, contrast, saturation and all the way up to the color effects. The last one V4L2 CID last P1 is not a control ID access but serves as the tag when applications are carrying for the controls. The standard controls provided by the framework were not sufficient enough as the hardware capability of devices became more sophisticated. Personally intended for complex drivers like encoders the controls were extended for other classes as well such as camera, FM tuner, RF tuner and others. The extended controls for camera class are defined for auto exposure, pan and tilt controls all the way up to pan speed and tilt speed. The extended control I have tells expect a pointer to struct V4L2 X controls which has a pointer to struct V4L2 X control. This allows multiple controls within a class to be grouped together as an array and configured by the applications. Aside from the controls available in framework depending on the system sometimes additional controls are required. These are defined using the custom control IDs. Coming to the capture devices this can be for sensor app for specifying sensor parameters or controlling the test pattern generation or to set the number of data lanes and MAPC SI2 interface or set per stream control such as errors are shown or the DMA controls for each stream or any other controls that are diverse specific. V4L2 also provides methods to inherit the control that is controls defined by one handler can be added to another by calling the API V4L2 CTRL add handler. This adds the controls defined by the control handler passed in the second argument to the control handler passed in the first argument. A filter function can also be defined to specify which controls can be added. The fourth argument specifies to the framework the controls that the controls being added are actually defined by another device. Inheriting controls is useful when the controls implemented by two devices are same thus avoids the code re-reimplementation. This is useful for example when the controls defined by gain brightness or exposure are already defined by sensor which can be reused by the bridge device. There can be occasions when adding the controls some of them might not be needed to be added for example when a debug feature is implemented by a sensor such control is not always needed to be added to the bridge device node. In such cases struct V4L2 CTRL provides a bitmap variable eSpyVeq to inform the framework to exclude the control to be added. Also the controls that are added from sub-device can be overwritten by the bridge device driver during the sub-dev registration call. Setting the eSpyVeq parameter prevents the controls from being added from sub-device to the root device during the V4L2 device register sub-dev call. So far we have seen the control methods provided by V4L2 and below are the control flags which the drivers can use to inform about the controls to application. When a control is no longer being used or no longer sorry when a control is no longer supported it is advised that the driver set the V4L2 CTRL flag disabled so that the applications can ignore such control. When a control is being used by one application and cannot be reused by another the driver can set the control as grabbed. If any other application tries to reuse the control the framework will throw an eBZ error. The read only flag is used to indicate that the control cannot be modified such as the link frequency or the blanking periods. There may be occasions when a control is being modified it can also update the control value of others. In such occasions in such occasions the drivers can set the update flag. Also a control may not be applicable when the device configuration is changed. In such cases the drivers can set the control as inactive. This can be the case usually when the inputs are being switched. The write only flag is meaningful when setting the control triggers an action but reading the control is not meaningful. The control can be changed by hardware. Example setting the control for auto gain or the auto exposure. In these cases the volatile flag should be set for the control by the driver. If user can set a new value for such volatile control the flag execute unwrite should be set by the driver. Finally a control that changes the buffer layout such as for the rotation controls the drivers can set the modify modify layout flag for that control. V4L2 framework provides control notification method wherein any change in control value can be notified to the other device. This is particularly useful when controls are inherited such as when bridge device has the controls of sensor sub device. Any changes in control value by the sub device can be notified for the bridge device. Only one notify function should be used for control handler. You can set the notify callback using using the V4L2 CTRL notify API. Before we discuss about integration of V4L2 controls with the Linux media controller framework we like to introduce the basic terms. As the number of devices participating in the media streaming pipeline and their functionality increased, Linux provided the media controller framework that helps in device discoverability and configuring these devices wherein the hardware devices are represented as a media device consisting of media entities and media pads. The media device is represented by the struct media device. The media entity is any basic hardware block that is participating in the media streaming pipeline. These can be off-chip devices or the on-chip IPs such as ISP. These are represented by struct media entity. The media pad is a connection endpoint through which the data is transferred. So the pads can be a source pad or the sync pad. These are represented by struct media pad. A media link is a connection between two pads either on the same entity or between different entities. The media links are represented by struct media link. To illustrate the terms described in the previous slide we present here a sample media device which has two media entities. This camera sensor and the host and the host processor interface to the camera. In this case we assume that to be MPCSA2 interface controller. The sensor is the source for the image data which has source pad number 0 and is connected to the input source pad of CSI entity on the sync pad 0 of CSI. The CSI controller also has source link which can be input to another entity in the media device such as DDR memory or another IP such as ISP. The connection between these two entities the media link is represented by the black arrow. The media controller framework abstracts the V4L to kernel objects as below. The struct media device pointer mdev abstracts the V4L to device. V4L to sub devices and the video devices are seen as media entities. The type field of struct media entity is set to media entity type V4L to subdev for sub devices or media entity type video device for the video device nodes. Drivers initialize the entity pads using media entity pads in it and register the entities with the media device by calling media device register entity. V4L to driver initializes the media devices within struct V4L to device using media device in it. Each entity driver initializes its entity and pad arrays using media entity pads in it. Thus the controls set by V4L to driver are applicable for the media entities. So, the media controller framework leverages the controls provided by V4L to. So far we have seen the control methods provided by V4L to and now we will look into how the user space applications can leverage these controls. The framework provides IAC terms to enumerate the controls that is the query CTRL and for extended controls VDAC query XCTRL. Applications can set the can get the control value by using VDAC GCTRL and for the extended controls using VDAC GXCTRLs. Applications can set the control value using VDAC SCCTRL and for extended controls using VDAC SXCTRLs. Applications can also try setting the control value by using VDAC TRYXCTRLs. Drivers must implement these actiles whenever the device supports one or one or more controls. Custom control IDs must be exposed to the applications from the header file under the directory include slash give API slash line X. Here is the code snippet for applications to enumerate the controls. Applications can query for all the controls using CTRL ID V4L to CID base and all the way up to V4L to CID last P1. Applications can enumerate the private controls provided by driver starting with the control ID V4L to CID private base. When changing the control value applications can first query for the desired control using VDAC GCTRL. Here we intend to change the value of a custom control. If the control is available then get the control value using VDAC GCTRL and then go ahead by setting the control value using VDAC SCCTRL. Here are a few notes on implementing controls. Driver implementation of controls need to be documented that is under documentation slash USPAC API slash media slash drivers. This is particularly useful when controls are implemented when custom controls are implemented. V4L to subdev control can be overwritten by V4L to device during subdevice registration. So decide which controls need protection and set the ease private flag for such controls. Another point is adding controls to V4L to subdevice after the device registration will not have any effect. So add the required controls for the subdevice prior to calling V4L to device register subdev. A good practice is to initialize the hardware with default values for the controls that configure the hardware. In such cases the drivers can call V4L to CTRL handler setup and initialize the control values for that hardware. This helps in applications to avoid setting the hardware to default values. So far we have seen the structure of V4L to the control methods provided by V4L to an integration of the controls with media controller framework and notes on how to implement the V4L to control. And now we present a sample capsule device to illustrate the control. The media device consists of two cameras connected to serializers. The image screens are regulated by the serializer and the serializer combination and are received by the host and the MTCSI to RX port. The green boxes here represent the media entities. The camera sensors each have the source pad connected to the sync pad of serializer 0 and 1. The serializer entity connects to the source the serializer entity connects to the DC laser. The DC laser has two source pads connected to the each of the RX port. The yellow boxes here represent the media device notes. So how do we add controls? Drivers need to first fill the structure V4L to CTRL config and V4L to CTRL OS. We will see in the next slide the contents of the structure. After initializing the EBO structures, controls can be added for custom controls by calling the V4L to CTRL new custom and for non-menu standard controls by calling V4L to CTRL std. For standard menu controls, the controls can be added by calling V4L to CTRL new std menu. Here are the contents of the structure V4L to CTRL config. The desired operations for the control can be set in the V4L to CTRL OS parameter. The control id in the id parameter. Drivers can set a meaningful name for the control in the name parameter. The type of control can be set in the type parameter. The minimum control value allowed by the control can be set by the driver in the min parameter and the maximum value allowed by the control in the max parameter. For integer type controls, a step value can be specified. For example, the number of lanes and MPCSA to interface. The default value for the control can be set in the left parameter. Here are the contents of structure V4L to CTRL OS. The GCTRL gets the new value for the control, generally relevant for the volatile or the read-only controls, such as the signal strength. If not set, then the currently cached value is returned. Try CTRL test whether the controls value is valid. This is only relevant when the usual min, max or the step values, step checks are not sufficient. SCTRL sets the control value. SCTRL is compulsory. Here is the example for initializing V4L to CTRL OS structure. For the sample device we discussed in the previous slides. The CSI captured driver uses the SCTRL operations to set the number of lanes of the MPCSA to interface by using the custom control ID here. And here is the example for setting the contents of V4L to CTRL config. The OS parameter is set to capture CTRL OS shown in the previous slide. The ID value is set to the custom control ID which sets the number of lanes on the MPCSA interface. The type of control is set to type integer since the number of lanes is an integer value. The minimum number of lanes configurable is 1. The maximum number of lanes configurable is 4. And the step value to increment or decrement the lanes is 1. The default number of lanes is 1. Once the structures V4L to CTRL config and V4L to CTRL OS are initialized, the driver can add the controls, add the new custom control by calling the API V4L to CTRL new custom API. The drivers can implement a control handler by adding the V4L to CTRL handler structure to the top level structure or the driver's private structure. For V4L to drivers, the control handler can be added to the at the same level where the V4L to device is present and initialize the control handler and all the necessary controls for the handler as discussed in the previous slides. Optionally, any control that is interacting with hardware, the driver can force initialize these controls and free the control handler when the device is leaving or removed. And here are the steps for adding the control handler to the sample capture device. Struct capture dev is the top level structure describing this bridge device. This has the structure V4L to device to describe the device instance and has the control handler. We initialize these structures as described in the previous slides and call V4L to CTRL handler in it with the number of controls handled by the control handler. The custom control to set the MPCSI to number of lanes is added by calling V4L to CTRL new custom. And finally, V4L to CTRL handler setup is called to initialize the default number of lanes for this device. The control notification can be set by calling V4L to CTRL notify API with the control handler and the control ID for which the notification is required. For example, here is a sample code where the sensor driver after adding the controls to its handler requires a framework to be notified of the changes for setting the number of lanes and MPCSI interface. The sensor CTRL notify function is called whenever the application changes the number of lanes and the video device mode or the sub device mode. And thus we have shown the sample capture device using V4L to controls. And now we will look into the utilities provided for controls. The V4L utility package provides command line tool to control the devices. The list device argument lists all the V4L devices along with the video device number and the device name. The driver and device information can be obtained by the arguments all followed by device followed by the device node name. And the controls owned by a device can be obtained by the arguments list CTRL followed by the device node name. And the command can be used to get or set the control value for the device nodes. And the current value of all the controls owned by a device can be specified by the argument log status for the device. With that we conclude our presentation on V4L to controls from the perspective of capture devices. We have seen the structure of V4L to framework and the control methods provided. We also seen how applications can leverage these controls and nodes and implementing these implementing the controls. Also a implementation of sample capture device with the V4L to control methods. We hope this presentation provides fair knowledge of V4L to controls and they are used by capture devices. Please let us know any questions that you have. Thank you all for attending this presentation and see you at the exciting events. Thank you.