diff options
Diffstat (limited to 'configuring-pipelines.rst')
| -rw-r--r-- | configuring-pipelines.rst | 759 |
1 files changed, 759 insertions, 0 deletions
diff --git a/configuring-pipelines.rst b/configuring-pipelines.rst new file mode 100644 index 0000000..440be2a --- /dev/null +++ b/configuring-pipelines.rst @@ -0,0 +1,759 @@ +.. SPDX-License-Identifier: CC-BY-SA-4.0 + +Configuring Pipelines +===================== + +Allowing fine-grained control of formats comes at the cost of increasing +complexity for applications. Where an application used to only have to set the +format on a video device, it must now set formats on all pads in the pipeline, +and ensure that the result forms a coherent configuration. + +As previously stated, the examples in this section will be based on the NXP +i.MX8MP ISP device: + +.. graphviz:: imx8mp-isp.dot + :caption: Media graph of the NXP i.MX8MP with default configuration + + +Anatomy of a Pipeline +--------------------- + +.. _media-ctl: https://git.linuxtv.org/v4l-utils.git/tree/utils/media-ctl +.. _v4l2-ctl: https://git.linuxtv.org/v4l-utils.git/tree/utils/v4l2-ctl + +The `media-ctl`_ utility can be used to display the formats on all pads in a +graph. Let's start by disecting the output of ``media-ctl -d /dev/media0 -p``: + +.. code-block:: none + :linenos: + :lineno-start: 1 + + $ media-ctl -d /dev/media0 -p + Media controller API version 6.0.0 + + Media device information + ------------------------ + driver rkisp1 + model rkisp1 + serial + bus info platform:rkisp1 + hw revision 0xe + driver version 6.0.0 + +media-ctl first provides summary information, to quickly identify the hardware +device corresponding to the media controller device (``/dev/media0`` here): + +- the driver name +- the device model name +- the device serial number (when available) +- which bus the device sits on +- the device hardware revision (in a device-specific format) +- the driver version (identical to the kernel version) + +The ISP integrated in the NXP i.MX8MP is also found in Rockchip SoCs. They are +both handled by the ``rkisp1`` kernel driver, as shown by the driver and device +model names. + +media-ctl then prints detailed information for all entities in the graph. +Entities are sorted by their numerical ID, which doesn't necessarily follow any +logical order. To improve readability, the entities below have been reordered +in source to sink order, as shown by the line numbers. This corresponds to the +top to bottom order in the graphical representation of the media graph. + +.. code-block:: none + :linenos: + :lineno-start: 73 + + - entity 34: imx290 2-001a (1 pad, 1 link) + type V4L2 subdev subtype Sensor flags 0 + device node name /dev/v4l-subdev3 + pad0: Source + [fmt:SRGGB10_1X10/1920x1080 field:none + crop.bounds:(0,0)/1945x1097 + crop:(12,8)/1920x1080] + -> "csis-32e40000.csi":0 [] + +All entities have a numerical ID and a name. The first entity in pipeline order +has ID ``34`` and is named ``imx290 2-001a``. Names are chosen by drivers but +should follow a set of naming rules. For devices controlled through an I2C bus, +as is the case here, the entity name is made of the device model (``imx290``) +and bus information (``2-001a``) separated by a space. The I2C bus information +is itself made of the bus number and the device address (in hexadecimal), +separated by a dash. + +This particular entity has a total of 1 pad and 1 link (line 73). It is a V4L2 +subdevice whose subtype is a camera sensor (line 74). The subdevice is exposed +to userspace through the ``/dev/v4l-subdev3`` device node (line 75). + +For each pad of the entity media-ctl then displays the pad index and type, the +available links, as well as pad information specific to the entity type. Pad +indices start at 0. The IMX290 camera sensor entity has a single pad, thus at +index 0 and named ``pad0``, whose type is a source pad (line 76). + +Links are displayed on lines starting with a ``->`` (for outbound links, on +source pads) or ``<-`` (for inbound links, on sink pads) sign, followed by the +link target (``"csis-32e40000.csi":0``) and link flags (``[]``). The link +target is a pad of another entity, expressed as the target entity name +surrounded by double quotes, followed by a colon and the target pad index. The +supported link flags (with their numerical values) are: + +ENABLED (0x1) + The link is enabled. In the absence of this flag, the link is disabled. + +IMMUTABLE (0x2) + The link is immutable, its ``ENABLED`` flag cannot be modified. Immutable + links in practice always have the ``ENABLED`` flag set. + +As the list of link flags is empty, this specific link is thus disabled, but +can be enabled as it isn't immutable. The graphical representation displays +this link as a dotted line. + +We will ignore for now the pad information specific to the entity type +displayed on lines 77 to 79, this will be explained further down. + +The next entity, in graph order, is the MIPI CSI-2 receiver that the camera +sensor is attached to: + +.. code-block:: none + :linenos: + :lineno-start: 64 + + - entity 29: csis-32e40000.csi (2 pads, 2 links) + type V4L2 subdev subtype Unknown flags 0 + device node name /dev/v4l-subdev2 + pad0: Sink + [fmt:UYVY8_1X16/640x480 field:none colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range] + <- "imx290 2-001a":0 [] + pad1: Source + [fmt:UYVY8_1X16/640x480 field:none colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range] + -> "rkisp1_isp":0 [ENABLED] + +The CSI-2 receiver is an IP core internal to the SoC, modelled in the kernel as +a platform device. Entity names from platform devices are more free-formed. In +this case, the driver follows the usual recommendation, which uses the platform +device model name (``csis``) and the platform device name (``32e40000.csi``), +separated by a dash. On machines using Device Tree to describe the system (as +is the case on ARM and ARM64 machines), the platform device name is typically +make of the device's bus address (``32e40000``) and device node name (``csi``). + +This entity has two pads, a sink pad and a source pad, with one link each. The +sink pad, ``pad0`` (line 67), is the target of the link originating from the +camera sensor, and is thus without surprise displayed as connected to the +source pad of the camera sensor (line 69). The link flags being a property of +the link, not the pad, they match the link flags displayed in the camera sensor +entity. + +The source pad, ``pad1`` (line 70), is connected to pad 0 of the ``rkisp1_isp`` +entity (line 72). This link is enabled but not immutable, thus displayed as a +thin plain line in the graphical representation. + +The next entity represents the core of the ISP itself: + +.. code-block:: none + :linenos: + :lineno-start: 14 + + - entity 1: rkisp1_isp (4 pads, 4 links) + type V4L2 subdev subtype Unknown flags 0 + device node name /dev/v4l-subdev0 + pad0: Sink + [fmt:SRGGB10_1X10/800x600 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range + crop.bounds:(0,0)/800x600 + crop:(0,0)/800x600] + <- "csis-32e40000.csi":1 [ENABLED] + pad1: Sink + [fmt:unknown/0x0 field:none] + <- "rkisp1_params":0 [ENABLED,IMMUTABLE] + pad2: Source + [fmt:YUYV8_2X8/800x600 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range + crop.bounds:(0,0)/800x600 + crop:(0,0)/800x600] + -> "rkisp1_resizer_mainpath":0 [ENABLED] + pad3: Source + [fmt:unknown/0x0 field:none] + -> "rkisp1_stats":0 [ENABLED,IMMUTABLE] + +Its name doesn't follow the previously described recommendation for entities +corresponding to platform devices, which shows that applications can't rely on +a particular format for entity names in general. This may be considered as a +kernel bug, but is unlikely to be changed as entity name changes may break +existing applications. + +This entity is also a V4L2 subdevice, with four pads, two sink pads (``pad0`` +and ``pad1`` on lines 17 and 22 respectively) and two source pads (``pad2`` and +``pad3`` on lines 25 and 30). Pads 1 and 3 are connected to entities +representing the video device nodes through which ISP parameters are supplied +and ISP statistics are captured. We will ignore them in this example as they +are not strictly required to operate the device. It is however worth noting +that the corresponding links are enabled and immutable, and thus displayed in +the graphical representation as thich plain lines. + +Pad 2 is a source pad that connects the ISP core to the resizer, which is the +next entity in the pipeline. + +.. code-block:: none + :linenos: + :lineno-start: 34 + + - entity 6: rkisp1_resizer_mainpath (2 pads, 2 links) + type V4L2 subdev subtype Unknown flags 0 + device node name /dev/v4l-subdev1 + pad0: Sink + [fmt:YUYV8_2X8/800x600 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range + crop.bounds:(0,0)/800x600 + crop:(0,0)/800x600] + <- "rkisp1_isp":2 [ENABLED] + pad1: Source + [fmt:YUYV8_2X8/800x600 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range] + -> "rkisp1_mainpath":0 [ENABLED,IMMUTABLE] + +The resizer is also a V4L2 subdevice, and has nothing noteworthy at this point +compared to the entities previously described. Its source pad (line 42) is +connected to the last entity in the pipeline: + +.. code-block:: none + :linenos: + :lineno-start: 46 + + - entity 9: rkisp1_mainpath (1 pad, 1 link) + type Node subtype V4L flags 0 + device node name /dev/video0 + pad0: Sink + <- "rkisp1_resizer_mainpath":1 [ENABLED,IMMUTABLE] + +The ``rkisp1_mainpath`` entity, unlike all previous entities, is not a V4L2 +subdevice but a V4L2 video node (line 47). Its device node name is +``/dev/video0`` (line 48), corresponding to the output of the pipeline. This is +the device node from which frames originating from the sensor, processed by the +ISP, and scaled by the resizer are captured. + +Let's try to capture frames in NV16 format with a 800x600 resolution: + +.. code-block:: none + :emphasize-lines: 2 + + $ v4l2-ctl -d /dev/video0 -v pixelformat=NV16,width=800,height=600 --stream-count 10 --stream-mmap + VIDIOC_STREAMON returned -1 (Broken pipe) + +The kernel driver returns the `Broken pipe` error code, which indicates that +the pipeline isn't properly configured. The first possibly culprit to +investigate is the links. + + +Link Setup +---------- + +.. _EBNF metasyntax: https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form + +We have seen that, unless immutable, links can be enabled or disabled. Enabling +or disabling links allows configuring data routing through the graph. For +instance, a device that has multiple camera sensors connected to an ISP, but +can only capture from one of the sensors at a time, could be controlled by +enabling one of the links from the camera sensors to select the source. + +In the default configuration of the device, the link from the camera sensor to +the CSI-2 receiver is disabled. The pipeline is thus missing a source, which +causes the capture to fail with the `Broken pipe` error. To fix this, links +must be setup before starting capture. This is done with the media-ctl utility: + +.. code-block:: none + + $ media-ctl -d /dev/media0 --link '"imx290 2-001a":0 -> "csis-32e40000.csi":0 [1]' + +The ``-link`` option (or its shorter form ``-l``) takes a list of links, +expressed as follows (using a syntax close to the `EBNF metasyntax`_, found in +the output of ``media-ctl --help``): + +.. code-block:: ebnf + + links = link { ',' link } ; + link = pad '->' pad '[' flags ']' ; + pad = entity ':' pad-number ; + entity = entity-number | ( '"' entity-name '"' ) ; + +In the above example, we enable (``[1]``) the link from the source pad of the +camera sensor entity (``'"imx290 2-001a":0``) to the sink pad of the CSI-2 +receiver entity (``"csis-32e40000.csi":0``). Links can also be disabled by +clearing the ENABLED flag (using ``[0]``), and multiple links can be setup with +a single command, although this is discouraged in scripts as the resulting +syntax would be hard to read. + +After running the above command, ``media-ctl -p`` reports the link between the +camera sensor and the CSI-2 receiver as enabled: + +.. code-block:: none + :linenos: + :lineno-start: 73 + :emphasize-lines: 8 + + - entity 34: imx290 2-001a (1 pad, 1 link) + type V4L2 subdev subtype Sensor flags 0 + device node name /dev/v4l-subdev3 + pad0: Source + [fmt:SRGGB10_1X10/1920x1080 field:none + crop.bounds:(0,0)/1945x1097 + crop:(12,8)/1920x1080] + -> "csis-32e40000.csi":0 [ENABLED] + +This is also displayed in the graphical representation, with the corresponding +link now being a thin plain line instead of a dotted line: + +.. graphviz:: imx8mp-isp-links-enabled.dot + :caption: Media graph of the NXP i.MX8MP after enabling links + +.. note:: + + It is recommended, before setting up links in a pipeline, to reset all + non-immutable links to their disabled state with ``media-ctl --reset``. This + helps starting from a known state and, and avoids depending on a previous + configuration of the media graph. In this specific example, the links + between the CSI-2 receiver and the ISP, and between the ISP and the resizer, + would get disabled, and would need to be enabled manually to create a + complete pipeline. + +Let's try to capture frames in NV16 format with a 800x600 resolution again: + +.. code-block:: none + :emphasize-lines: 2 + + $ v4l2-ctl -d /dev/video0 -v pixelformat=NV16,width=800,height=600 --stream-count 10 --stream-mmap + VIDIOC_STREAMON returned -1 (Broken pipe) + +The kernel still returns the `Broken pipe` error, for an entirely different +reason: we haven't taken care of formats in the pipeline. + + +Formats in the Pipeline +----------------------- + +.. _Media Bus Format: https://linuxtv.org/downloads/v4l-dvb-apis/userspace-api/v4l/subdev-formats.html> + +It is time to stop ignoring the pad information specific to the entity type +that is printed by media-ctl. Let's have a look at the camera sensor entity +again: + +.. code-block:: none + :linenos: + :lineno-start: 73 + :emphasize-lines: 5-7 + + - entity 34: imx290 2-001a (1 pad, 1 link) + type V4L2 subdev subtype Sensor flags 0 + device node name /dev/v4l-subdev3 + pad0: Source + [fmt:SRGGB10_1X10/1920x1080 field:none + crop.bounds:(0,0)/1945x1097 + crop:(12,8)/1920x1080] + -> "csis-32e40000.csi":0 [ENABLED] + +For entities corresponding to V4L2 subdevices, media-ctl prints the active +configuration for each pad (lines 68 to 78). The configuration is made of +multiple properties: + +- The media bus format, expressed as + + .. code-block:: ebnf + + media bus format = "fmt:", fcc, "/", size ; + size = width, "x", height ; + + where ``fcc`` is the `media bus format`_ code. Format codes are listed in the + ``linux/media-bus-format.h`` header file of the Linux kernel, and expressed + here as a string without the ``MEDIA_BUS_FMT_`` prefix. + +- The field, for interlaced video: + + .. code-block:: ebnf + + field = "field:", v4l2-field ; + v4l2-field = "any" | "none" | "top" | "bottom" | "interlaced" + | "seq-tb" | "seq-bt" | "alternate" | "interlaced-tb" + | "interlaced-bt" ; + +- The color space, expressed as four properties, reporting the V4L2 color + space, transfer function, YCbCr encoding and quantization range: + + .. code-block:: ebnf + + color space = "colorspace:", v4l2-colorspace, " xfer:", v4l2-xfer-func, + " ycbcr:", v4l2-ycbcr-enc, " quantization:", v4l2-quant ; + v4l2-colorspace = "default" | "smpte170m" | "smpte240m" | "rec709" + | "470m" | "470bg" | "jpeg" | "srgb" | "oprgb" | "bt2020" + | "raw" | "dcip3" ; + v4l2-xfer-func = "default" | "709" | "srgb" | "oprgb" | "smpte240m" + | "smpte2084" | "dcip3" | "none" ; + v4l2-ycbcr-enc = "default" | "601" | "709" | "xv601" | "xv709" | "bt2020" + | "bt2020c" | "smpte240m" ; + v4l2-quant = "default" | "full-range" | "lim-range" ; + +- The compose and crop rectangles: + + .. code-block:: ebnf + + rectangle = "(", left, ",", top, ")", "/", size ; + size = width, "x", height ; + + +In order for a pipeline to be valid, the configuration of pads on both sides of +every link must match. Unless drivers implement custom validation rules, the +default constraint is that the media bus format on both sides must be +identical. For interlaced video, the fields must also be identical, or the +field on the sink side must be equal to ``none`` (in order to support +interlaced sources connected to sinks that only support progressive formats). +The crop and compose rectangles are not taken into account when validating +pipelines, as they are internal to subdevices. + +The default validation is performed by the ``v4l2_subdev_link_validate_default()`` +function in the kernel. If dynamic debugging is enabled (through the +``CONFIG_DYNAMIC_DEBUG`` kernel configuration option), extra messages can be +enabled through debugfs to diagnose the problem more precisely: + +.. code-block:: none + + $ mount -t debugfs none /sys/kernel/debug/ + $ echo 'func v4l2_subdev_link_validate_default +p' > /sys/kernel/debug/dynamic_debug/control + +After trying to capture frames with v4l2-ctl and obtaining the `Broken pipe` +error, the kernel log (obtained with ``dmesg``) provides useful information: + +.. code-block:: none + + rkisp1 32e10000.isp: Validating pad 'rkisp1_isp':0 + rkisp1 32e10000.isp: v4l2_subdev_link_validate_default: width does not match (source 640, sink 800) + rkisp1 32e10000.isp: v4l2_subdev_link_validate_default: height does not match (source 480, sink 600) + rkisp1 32e10000.isp: v4l2_subdev_link_validate_default: media bus code does not match (source 0x0000200f, sink 0x0000300f) + rkisp1 32e10000.isp: v4l2_subdev_link_validate_default: link was "csis-32e40000.csi":1 -> "rkisp1_isp":0 + rkisp1 32e10000.isp: Link 'csis-32e40000.csi':1 -> 'rkisp1_isp':0 failed validation: -32 + +This indicates a mismatch between the source pad of the CSI-2 receiver and the +sink pad of the ISP, both in the resolution and in the format code. The CSI-2 +receiver is configured to output UYVY8_1X16 in 640x480, and the ISP to receive +SRGGB10_1X10 in 800x600, leading to a broken pipeline. + +To fix this, formats must be configuration through the pipeline with a process +referred to as `format propagation`. + + +Format Propagation +------------------ + +Format propagation is a pipeline configuration mechanism where the format is +first configured on the source of the pipeline (the camera sensor in this +case), and then propagated downstream through all subdevices until the video +node at the end of the pipeline. It involves two different types of operations: + +- To ensure that formats on both ends of a link match, when configuring the + format on a source pad of a subdevice, the same format must be set on the + connected sink pad at the other end of the link. This is known as `format + propagation across links`. Subdevices do not automatically propagate formats + across links within the kernel in order to avoid hardcoding use cases in + drivers and to simplify driver implementation. This format propagation is the + responsibility of userspace. + +- Within a subdevice, the kernel ensures that pads configuration stays valid at + all times. When an application configures a format on a subdevice pad, the + driver adjusts parameters that are not valid to the closest supported values. + For instance, resolutions are clampedto the supported range, rounded if the + device has alignment constraints, and unsupported format codes are replaced + with supported values. This is known as `format negotiation` and is similar + to how the kernel handles format setting on V4L2 video devices. + + In addition to the format negotiation at the pad level, the kernel also + ensures that the configurations of all pads within a subdevice remain + coherent with each other. When an application configures the format on a sink + pad, the subdevice driver also adjusts the configuration of the source pads + if the new sink pad configuration would otherwise result in an invalid + subdevice configuration. For instance, the i.MX8MP resizer can scale images + but can't convert between bus format codes, so if an application sets a + format code on the resizer sink pad, the driver will automatically set the + same format code on the resizer source pad. This is known as `format + propagation inside subdevices`, and is the responsibility of the subdevice + drivers. + + Unlike across links where the formats on the source and sink sides of a link + must match, formats on the sink and source pads of a subdevice may be + different. For instance, the i.MX8MP ISP interpolates the raw Bayer data + provided by the camera sensor, and converts it to YUV. The format code on the + source pad of the ISP will thus differ from the sink pad. When userspace + configures the format on the ISP sink pad, the driver will propagate the sink + resolution to the source pad, but will adjust the sink Bayer format code to a + YUV format code on the source pad. + + Format propagation within subdevices ensures that their configuration remains + coherent, but doesn't save applications from having to configure formats on + source pads. Using the resizer example again, the driver will ensure that the + format on the source pad gets adjusted to a value compatible with the format + set on the sink pad, but applications will likely want to then configure the + format on the source pad explicitly in order to change the resolution and + select the scaling ratio. Format propagation within subdevices always operate + `from sink to source`. When userspace sets the format on a source pad, the + driver will adjust it to values compatible with the current configuration of + the sink pads if needed, but will never modify the sink pads configuration by + back-propagating the format from source pad to sink pad. + +Subdevice pad formats can be set with the media-ctl utility, using the +``--set-v4l2`` option (shortened to ``-V``). The syntax is identical to the +format printed by media-ctl with the ``-p`` option. The first step in our +example is to configure the format on the sensor source pad. We use the default +format reported by the media graph (10-bit raw Bayer in 1920x1080), which +corresponds to the sensor native resolution: + +.. code-block:: none + + $ media-ctl -d /dev/media0 -V '"imx290 2-001a":0 [fmt:SRGGB10_1X10/1920x1080 colorspace:raw xfer:none ycbcr:601 quantization:full-range]' + +Running ``media-ctl -p`` again, the format on the sensor source pad has been +updated with the newly selected color space: + +.. code-block:: none + :linenos: + :lineno-start: 73 + :emphasize-lines: 5 + + - entity 34: imx290 2-001a (1 pad, 1 link) + type V4L2 subdev subtype Sensor flags 0 + device node name /dev/v4l-subdev3 + pad0: Source + [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range + crop.bounds:(0,0)/1945x1097 + crop:(12,8)/1920x1080] + -> "csis-32e40000.csi":0 [ENABLED] + +The format is otherwise the same, as we have selected a format code and +resolution identical to the previously configured value. However, this is not +the only change in the media graph. The configuration of the CSI-2 receiver is +now different: + +.. code-block:: none + :linenos: + :lineno-start: 64 + :emphasize-lines: 5,8 + + - entity 29: csis-32e40000.csi (2 pads, 2 links) + type V4L2 subdev subtype Unknown flags 0 + device node name /dev/v4l-subdev2 + pad0: Sink + [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range] + <- "imx290 2-001a":0 [ENABLED] + pad1: Source + [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range] + -> "rkisp1_isp":0 [ENABLED] + +The change in the sink pad format seems to contradict what we have seen before, +as the kernel isn't supposed to propagate formats across links. The reason for +this lies in the media-ctl utility. When setting a format on a source pad, +media-ctl automatically reads back the format from the subdevice (to take into +account possible adjustments applied by the driver) and sets that format on all +sink pads connected through enabled links. + +The source pad format is also modified, due to automatic format propagation by +drivers within subdevices. As the CSI-2 receiver can't transcode or scale, the +source pad format has thus been set to match the sink pad format. + +These two effects simplify the format propagation procedure, as formats on sink +pads don't need to be set manually. Instead, we can read back the format on the +source pad of the next subdevice (the CSI-2 receiver in this case) using +``media-ctl -p``, and set it on the same pad to let media-ctl propagate it to +the next subdevice: + +.. code-block:: none + + $ media-ctl -d /dev/media0 -V '"csis-32e40000.csi":1 [fmt:SRGGB10_1X10/1920x1080 colorspace:raw xfer:none ycbcr:601 quantization:full-range]' + +We expect the next subdevice, ´´rkisp1_isp``, to output an image in a processed +YUV format with a 1920x1080 resolution. This is not what happens: + +.. code-block:: none + :linenos: + :lineno-start: 14 + :emphasize-lines: 6-7,13 + + - entity 1: rkisp1_isp (4 pads, 4 links) + type V4L2 subdev subtype Unknown flags 0 + device node name /dev/v4l-subdev0 + pad0: Sink + [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range + crop.bounds:(0,0)/1920x1080 + crop:(0,0)/800x600] + <- "csis-32e40000.csi":1 [ENABLED] + pad1: Sink + [fmt:unknown/0x0 field:none] + <- "rkisp1_params":0 [ENABLED,IMMUTABLE] + pad2: Source + [fmt:YUYV8_2X8/800x600 field:none colorspace:raw xfer:none ycbcr:601 quantization:lim-range + crop.bounds:(0,0)/800x600 + crop:(0,0)/800x600] + -> "rkisp1_resizer_mainpath":0 [ENABLED] + pad3: Source + [fmt:unknown/0x0 field:none] + -> "rkisp1_stats":0 [ENABLED,IMMUTABLE] + +The output format code (``YUYV8_2X8``) is fine, but the output resolution has +been set to 800x600. This is due to the crop capability of the sink pad. While +the crop bounds have been adjusted (line 19), the crop rectangle itself (line +20) hasn't been adjusted as its previous size (``(0,0)/800x600``) is compatible +with the new sink format. We must thus set the crop rectangle manually on the +sink pad: + +.. code-block:: none + + $ media-ctl -d /dev/media0 -V '"rkisp1_isp":0 [crop:(0,0)/1920x1080]' + +This improves the situation: + +.. code-block:: none + :linenos: + :lineno-start: 14 + :emphasize-lines: 7,13-15 + + - entity 1: rkisp1_isp (4 pads, 4 links) + type V4L2 subdev subtype Unknown flags 0 + device node name /dev/v4l-subdev0 + pad0: Sink + [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range + crop.bounds:(0,0)/1920x1080 + crop:(0,0)/1920x1080] + <- "csis-32e40000.csi":1 [ENABLED] + pad1: Sink + [fmt:unknown/0x0 field:none] + <- "rkisp1_params":0 [ENABLED,IMMUTABLE] + pad2: Source + [fmt:YUYV8_2X8/800x600 field:none colorspace:raw xfer:none ycbcr:601 quantization:lim-range + crop.bounds:(0,0)/1920x1080 + crop:(0,0)/800x600] + -> "rkisp1_resizer_mainpath":0 [ENABLED] + pad3: Source + [fmt:unknown/0x0 field:none] + -> "rkisp1_stats":0 [ENABLED,IMMUTABLE] + +The crop bounds on the source pad (line 27) now cover the full resolution, but +the source crop rectangle (line 28) hasn't been adjusted, for the same reason +as previously. Fixing this is simple: + +.. code-block:: none + + $ media-ctl -d /dev/media0 -V '"rkisp1_isp":2 [crop:(0,0)/1920x1080]' + +This gives us the configuration we want for the source resolution: + +.. code-block:: none + :linenos: + :lineno-start: 14 + :emphasize-lines: 13,15 + + - entity 1: rkisp1_isp (4 pads, 4 links) + type V4L2 subdev subtype Unknown flags 0 + device node name /dev/v4l-subdev0 + pad0: Sink + [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range + crop.bounds:(0,0)/1920x1080 + crop:(0,0)/1920x1080] + <- "csis-32e40000.csi":1 [ENABLED] + pad1: Sink + [fmt:unknown/0x0 field:none] + <- "rkisp1_params":0 [ENABLED,IMMUTABLE] + pad2: Source + [fmt:YUYV8_2X8/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:lim-range + crop.bounds:(0,0)/1920x1080 + crop:(0,0)/1920x1080] + -> "rkisp1_resizer_mainpath":0 [ENABLED] + pad3: Source + [fmt:unknown/0x0 field:none] + -> "rkisp1_stats":0 [ENABLED,IMMUTABLE] + +Finally, we set the format on the source pad to let media-ctl propagate it to +the next subdevice (``rkisp1_resizer_mainpath``): + +.. code-block:: none + + $ media-ctl -d /dev/media0 -V '"rkisp1_isp":2 [fmt:YUYV8_2X8/1920x1080]' + +The resizer sink pad format is configured correctly: + +.. code-block:: none + :linenos: + :lineno-start: 34 + :emphasize-lines: 5,7,10 + + - entity 6: rkisp1_resizer_mainpath (2 pads, 2 links) + type V4L2 subdev subtype Unknown flags 0 + device node name /dev/v4l-subdev1 + pad0: Sink + [fmt:YUYV8_2X8/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:lim-range + crop.bounds:(0,0)/1920x1080 + crop:(0,0)/800x600] + <- "rkisp1_isp":2 [ENABLED] + pad1: Source + [fmt:YUYV8_2X8/800x600 field:none colorspace:raw xfer:none ycbcr:601 quantization:lim-range] + -> "rkisp1_mainpath":0 [ENABLED,IMMUTABLE] + +The resizer source pad (line 43) matches the resolution we want to capture +(``800x600``). However, this is not achieved by scaling, but by cropping the +input frame (line 40). This isn't the desired behaviour, so, even if frame +capture would work with this configuration, we must adjust the sink pad crop +rectangle: + +.. code-block:: none + + $ media-ctl -d /dev/media0 -V '"rkisp1_resizer_mainpath":0 [crop:(0,0)/1920x1080]' + +The source pad format is unchanged and still set to ``YUYV8_2X8/800x600``. If +we wanted to scale to a different resolution, we would need to set the format +on the resizer source pad at this point. + +The next entity in the pipeline (``rkisp1_mainpath``) is not a subdevice but a +video device. There is no need to configure pad formats, as formats on video +devices are set through the classic V4L2 API. Care must be taken to configure +the video device with a pixel format compatible with the source pad of the last +subdevice in the pipeline, and an identical resolution. Which pixel formats +qualify as compatible is device-dependent, as explained in the +:ref:`v4l2-subdevice-formats` section. For the i.MX8MP, the DMA engine at the +output of the pipeline can write the YUV 4:2:2 data it receives from the +resizer in the semi-planar NV16 format: + +.. code-block:: none + + $ v4l2-ctl -d /dev/video0 -v pixelformat=NV16,width=800,height=600 --stream-count 10 --stream-mmap + <<<<<<<<<< + +The pipeline is valid, and 10 frames have been captured successfully. + + +Summary +------- + +Pipelines can be inspected with ``media-ctl -p`` and must configured by +userspace before being used. + +The first configuration step covers link setup. Links should be reset with +``media-ctl -r`` to avoid any influence from a previous configuration, and +individual links should then be enabled with ``media-ctl -l``. + +The second configuration step covers format configuration. Formats are set with +``media-ctl -V`` and must be propagated through the pipeline from source to +sink. The combination of automatic format propagation across links in media-ctl +with the automatic format propagation inside subdevices in the kernel +simplifies format propagation. The generic procedure is as follows: + +#. Set the format on the source pad of the first subdevice in the pipeline. +#. The kernel adjusts the format, and media-ctl automatically propagates it + over the link to the connected sink pad of the next subdevice. +#. If the next subdevice has crop or compose rectangleon its sink or source + pads, sets them as necessary, in the following order: + + - Sink crop + - Sink compose + - Source crop + +#. The kernel automatically propagates the format to the source pad of the + subdevice, adjusting it if needed. +#. Retrieve the propagated format on the source pad of the subdevice, modify it + if required (for instance to configure scaling), and set of on the same + source pad to let media-ctl propagate it across the next link. +#. Repeat this procedure from subdevice to subdevice, until the last subdevice + in the pipeline is reached. + +After configuring the pipeline, frames can be captured from the video device at +the pipeline output with the classic V4L2 API, in a pixel format and resolution +that matches the pipeline configuration. + |