.. SPDX-License-Identifier: CC-BY-SA-4.0 Configuring Pipelines ===================== Allowing fine-grained control of formats comes at the cost of increasing complexity for applications. Where an application used to only have to set the format on a video device, it must now set formats on all pads in the pipeline, and ensure that the result forms a coherent configuration. As previously stated, the examples in this section will be based on the NXP i.MX8MP ISP device shown in :numref:`media-graph-nxp-imx8mp-default`. .. _media-graph-nxp-imx8mp-default: .. graphviz:: imx8mp-isp.dot :caption: Media graph of the NXP i.MX8MP with default configuration Anatomy of a Pipeline --------------------- .. _media-ctl: https://git.linuxtv.org/v4l-utils.git/tree/utils/media-ctl .. _v4l2-ctl: https://git.linuxtv.org/v4l-utils.git/tree/utils/v4l2-ctl The `media-ctl`_ utility can be used to display the formats on all pads in a graph. Let's start by disecting the output of ``media-ctl -d /dev/media0 -p``: .. code-block:: none :linenos: :lineno-start: 1 $ media-ctl -d /dev/media0 -p Media controller API version 6.0.0 Media device information ------------------------ driver rkisp1 model rkisp1 serial bus info platform:rkisp1 hw revision 0xe driver version 6.0.0 media-ctl first provides summary information, to quickly identify the hardware device corresponding to the media controller device (``/dev/media0`` here): - the driver name - the device model name - the device serial number (when available) - which bus the device sits on - the device hardware revision (in a device-specific format) - the driver version (identical to the kernel version) The ISP integrated in the NXP i.MX8MP is also found in Rockchip SoCs. They are both handled by the ``rkisp1`` kernel driver, as shown by the driver and device model names. media-ctl then prints detailed information for all entities in the graph. Entities are sorted by their numerical ID, which doesn't necessarily follow any logical order. To improve readability, the entities below have been reordered in source to sink order, as shown by the line numbers. This corresponds to the top to bottom order in the graphical representation of the media graph. .. code-block:: none :linenos: :lineno-start: 73 - entity 34: imx290 2-001a (1 pad, 1 link) type V4L2 subdev subtype Sensor flags 0 device node name /dev/v4l-subdev3 pad0: Source [fmt:SRGGB10_1X10/1920x1080 field:none crop.bounds:(0,0)/1945x1097 crop:(12,8)/1920x1080] -> "csis-32e40000.csi":0 [] All entities have a numerical ID and a name. The first entity in pipeline order has ID ``34`` and is named ``imx290 2-001a``. Names are chosen by drivers but should follow a set of naming rules. For devices controlled through an I2C bus, as is the case here, the entity name is made of the device model (``imx290``) and bus information (``2-001a``) separated by a space. The I2C bus information is itself made of the bus number and the device address (in hexadecimal), separated by a dash. This particular entity has a total of 1 pad and 1 link (line 73). It is a V4L2 subdevice whose subtype is a camera sensor (line 74). The subdevice is exposed to userspace through the ``/dev/v4l-subdev3`` device node (line 75). For each pad of the entity media-ctl then displays the pad index and type, the available links, as well as pad information specific to the entity type. Pad indices start at 0. The IMX290 camera sensor entity has a single pad, thus at index 0 and named ``pad0``, whose type is a source pad (line 76). Links are displayed on lines starting with a ``->`` (for outbound links, on source pads) or ``<-`` (for inbound links, on sink pads) sign, followed by the link target (``"csis-32e40000.csi":0``) and link flags (``[]``). The link target is a pad of another entity, expressed as the target entity name surrounded by double quotes, followed by a colon and the target pad index. The supported link flags (with their numerical values) are: ENABLED (0x1) The link is enabled. In the absence of this flag, the link is disabled. IMMUTABLE (0x2) The link is immutable, its ``ENABLED`` flag cannot be modified. Immutable links in practice always have the ``ENABLED`` flag set. As the list of link flags is empty, this specific link is thus disabled, but can be enabled as it isn't immutable. The graphical representation displays this link as a dotted line. We will ignore for now the pad information specific to the entity type displayed on lines 77 to 79, this will be explained further down. The next entity, in graph order, is the MIPI CSI-2 receiver that the camera sensor is attached to: .. code-block:: none :linenos: :lineno-start: 64 - entity 29: csis-32e40000.csi (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev2 pad0: Sink [fmt:UYVY8_1X16/640x480 field:none colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range] <- "imx290 2-001a":0 [] pad1: Source [fmt:UYVY8_1X16/640x480 field:none colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range] -> "rkisp1_isp":0 [ENABLED] The CSI-2 receiver is an IP core internal to the SoC, modelled in the kernel as a platform device. Entity names from platform devices are more free-formed. In this case, the driver follows the usual recommendation, which uses the platform device model name (``csis``) and the platform device name (``32e40000.csi``), separated by a dash. On machines using Device Tree to describe the system (as is the case on ARM and ARM64 machines), the platform device name is typically make of the device's bus address (``32e40000``) and device node name (``csi``). This entity has two pads, a sink pad and a source pad, with one link each. The sink pad, ``pad0`` (line 67), is the target of the link originating from the camera sensor, and is thus without surprise displayed as connected to the source pad of the camera sensor (line 69). The link flags being a property of the link, not the pad, they match the link flags displayed in the camera sensor entity. The source pad, ``pad1`` (line 70), is connected to pad 0 of the ``rkisp1_isp`` entity (line 72). This link is enabled but not immutable, thus displayed as a thin plain line in the graphical representation. The next entity represents the core of the ISP itself: .. code-block:: none :linenos: :lineno-start: 14 - entity 1: rkisp1_isp (4 pads, 4 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Sink [fmt:SRGGB10_1X10/800x600 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range crop.bounds:(0,0)/800x600 crop:(0,0)/800x600] <- "csis-32e40000.csi":1 [ENABLED] pad1: Sink [fmt:unknown/0x0 field:none] <- "rkisp1_params":0 [ENABLED,IMMUTABLE] pad2: Source [fmt:YUYV8_2X8/800x600 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range crop.bounds:(0,0)/800x600 crop:(0,0)/800x600] -> "rkisp1_resizer_mainpath":0 [ENABLED] pad3: Source [fmt:unknown/0x0 field:none] -> "rkisp1_stats":0 [ENABLED,IMMUTABLE] Its name doesn't follow the previously described recommendation for entities corresponding to platform devices, which shows that applications can't rely on a particular format for entity names in general. This may be considered as a kernel bug, but is unlikely to be changed as entity name changes may break existing applications. This entity is also a V4L2 subdevice, with four pads, two sink pads (``pad0`` and ``pad1`` on lines 17 and 22 respectively) and two source pads (``pad2`` and ``pad3`` on lines 25 and 30). Pads 1 and 3 are connected to entities representing the video device nodes through which ISP parameters are supplied and ISP statistics are captured. We will ignore them in this example as they are not strictly required to operate the device. It is however worth noting that the corresponding links are enabled and immutable, and thus displayed in the graphical representation as thich plain lines. Pad 2 is a source pad that connects the ISP core to the resizer, which is the next entity in the pipeline. .. code-block:: none :linenos: :lineno-start: 34 - entity 6: rkisp1_resizer_mainpath (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev1 pad0: Sink [fmt:YUYV8_2X8/800x600 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range crop.bounds:(0,0)/800x600 crop:(0,0)/800x600] <- "rkisp1_isp":2 [ENABLED] pad1: Source [fmt:YUYV8_2X8/800x600 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range] -> "rkisp1_mainpath":0 [ENABLED,IMMUTABLE] The resizer is also a V4L2 subdevice, and has nothing noteworthy at this point compared to the entities previously described. Its source pad (line 42) is connected to the last entity in the pipeline: .. code-block:: none :linenos: :lineno-start: 46 - entity 9: rkisp1_mainpath (1 pad, 1 link) type Node subtype V4L flags 0 device node name /dev/video0 pad0: Sink <- "rkisp1_resizer_mainpath":1 [ENABLED,IMMUTABLE] The ``rkisp1_mainpath`` entity, unlike all previous entities, is not a V4L2 subdevice but a V4L2 video node (line 47). Its device node name is ``/dev/video0`` (line 48), corresponding to the output of the pipeline. This is the device node from which frames originating from the sensor, processed by the ISP, and scaled by the resizer are captured. Let's try to capture frames in NV16 format with a 800x600 resolution: .. code-block:: none :emphasize-lines: 2 $ v4l2-ctl -d /dev/video0 -v pixelformat=NV16,width=800,height=600 --stream-count 10 --stream-mmap VIDIOC_STREAMON returned -1 (Broken pipe) The kernel driver returns the `Broken pipe` error code, which indicates that the pipeline isn't properly configured. The first possibly culprit to investigate is the links. Link Setup ---------- .. _EBNF metasyntax: https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form We have seen that, unless immutable, links can be enabled or disabled. Enabling or disabling links allows configuring data routing through the graph. For instance, a device that has multiple camera sensors connected to an ISP, but can only capture from one of the sensors at a time, could be controlled by enabling one of the links from the camera sensors to select the source. In the default configuration of the device, the link from the camera sensor to the CSI-2 receiver is disabled. The pipeline is thus missing a source, which causes the capture to fail with the `Broken pipe` error. To fix this, links must be setup before starting capture. This is done with the media-ctl utility: .. code-block:: none $ media-ctl -d /dev/media0 --link '"imx290 2-001a":0 -> "csis-32e40000.csi":0 [1]' The ``-link`` option (or its shorter form ``-l``) takes a list of links, expressed as follows (using a syntax close to the `EBNF metasyntax`_, found in the output of ``media-ctl --help``): .. code-block:: ebnf links = link { ',' link } ; link = pad '->' pad '[' flags ']' ; pad = entity ':' pad-number ; entity = entity-number | ( '"' entity-name '"' ) ; In the above example, we enable (``[1]``) the link from the source pad of the camera sensor entity (``'"imx290 2-001a":0``) to the sink pad of the CSI-2 receiver entity (``"csis-32e40000.csi":0``). Links can also be disabled by clearing the ENABLED flag (using ``[0]``), and multiple links can be setup with a single command, although this is discouraged in scripts as the resulting syntax would be hard to read. After running the above command, ``media-ctl -p`` reports the link between the camera sensor and the CSI-2 receiver as enabled: .. code-block:: none :linenos: :lineno-start: 73 :emphasize-lines: 8 - entity 34: imx290 2-001a (1 pad, 1 link) type V4L2 subdev subtype Sensor flags 0 device node name /dev/v4l-subdev3 pad0: Source [fmt:SRGGB10_1X10/1920x1080 field:none crop.bounds:(0,0)/1945x1097 crop:(12,8)/1920x1080] -> "csis-32e40000.csi":0 [ENABLED] This is also displayed in the graphical representation, with the corresponding link now being a thin plain line instead of a dotted line (:numref:`media-graph-nxp-imx8mp-links-enabled`). .. _media-graph-nxp-imx8mp-links-enabled: .. graphviz:: imx8mp-isp-links-enabled.dot :caption: Media graph of the NXP i.MX8MP after enabling links .. note:: It is recommended, before setting up links in a pipeline, to reset all non-immutable links to their disabled state with ``media-ctl --reset``. This helps starting from a known state and, and avoids depending on a previous configuration of the media graph. In this specific example, the links between the CSI-2 receiver and the ISP, and between the ISP and the resizer, would get disabled, and would need to be enabled manually to create a complete pipeline. Let's try to capture frames in NV16 format with a 800x600 resolution again: .. code-block:: none :emphasize-lines: 2 $ v4l2-ctl -d /dev/video0 -v pixelformat=NV16,width=800,height=600 --stream-count 10 --stream-mmap VIDIOC_STREAMON returned -1 (Broken pipe) The kernel still returns the `Broken pipe` error, for an entirely different reason: we haven't taken care of formats in the pipeline. Formats in the Pipeline ----------------------- .. _Media Bus Format: https://linuxtv.org/downloads/v4l-dvb-apis/userspace-api/v4l/subdev-formats.html> It is time to stop ignoring the pad information specific to the entity type that is printed by media-ctl. Let's have a look at the camera sensor entity again: .. code-block:: none :linenos: :lineno-start: 73 :emphasize-lines: 5-7 - entity 34: imx290 2-001a (1 pad, 1 link) type V4L2 subdev subtype Sensor flags 0 device node name /dev/v4l-subdev3 pad0: Source [fmt:SRGGB10_1X10/1920x1080 field:none crop.bounds:(0,0)/1945x1097 crop:(12,8)/1920x1080] -> "csis-32e40000.csi":0 [ENABLED] For entities corresponding to V4L2 subdevices, media-ctl prints the active configuration for each pad (lines 68 to 78). The configuration is made of multiple properties: - The media bus format, expressed as .. code-block:: ebnf media bus format = "fmt:", fcc, "/", size ; size = width, "x", height ; where ``fcc`` is the `media bus format`_ code. Format codes are listed in the ``linux/media-bus-format.h`` header file of the Linux kernel, and expressed here as a string without the ``MEDIA_BUS_FMT_`` prefix. - The field, for interlaced video: .. code-block:: ebnf field = "field:", v4l2-field ; v4l2-field = "any" | "none" | "top" | "bottom" | "interlaced" | "seq-tb" | "seq-bt" | "alternate" | "interlaced-tb" | "interlaced-bt" ; - The color space, expressed as four properties, reporting the V4L2 color space, transfer function, YCbCr encoding and quantization range: .. code-block:: ebnf color space = "colorspace:", v4l2-colorspace, " xfer:", v4l2-xfer-func, " ycbcr:", v4l2-ycbcr-enc, " quantization:", v4l2-quant ; v4l2-colorspace = "default" | "smpte170m" | "smpte240m" | "rec709" | "470m" | "470bg" | "jpeg" | "srgb" | "oprgb" | "bt2020" | "raw" | "dcip3" ; v4l2-xfer-func = "default" | "709" | "srgb" | "oprgb" | "smpte240m" | "smpte2084" | "dcip3" | "none" ; v4l2-ycbcr-enc = "default" | "601" | "709" | "xv601" | "xv709" | "bt2020" | "bt2020c" | "smpte240m" ; v4l2-quant = "default" | "full-range" | "lim-range" ; - The compose and crop rectangles: .. code-block:: ebnf rectangle = "(", left, ",", top, ")", "/", size ; size = width, "x", height ; In order for a pipeline to be valid, the configuration of pads on both sides of every link must match. Unless drivers implement custom validation rules, the default constraint is that the media bus format on both sides must be identical. For interlaced video, the fields must also be identical, or the field on the sink side must be equal to ``none`` (in order to support interlaced sources connected to sinks that only support progressive formats). The crop and compose rectangles are not taken into account when validating pipelines, as they are internal to subdevices. The default validation is performed by the ``v4l2_subdev_link_validate_default()`` function in the kernel. If dynamic debugging is enabled (through the ``CONFIG_DYNAMIC_DEBUG`` kernel configuration option), extra messages can be enabled through debugfs to diagnose the problem more precisely: .. code-block:: none $ mount -t debugfs none /sys/kernel/debug/ $ echo 'func v4l2_subdev_link_validate_default +p' > /sys/kernel/debug/dynamic_debug/control After trying to capture frames with v4l2-ctl and obtaining the `Broken pipe` error, the kernel log (obtained with ``dmesg``) provides useful information: .. code-block:: none rkisp1 32e10000.isp: Validating pad 'rkisp1_isp':0 rkisp1 32e10000.isp: v4l2_subdev_link_validate_default: width does not match (source 640, sink 800) rkisp1 32e10000.isp: v4l2_subdev_link_validate_default: height does not match (source 480, sink 600) rkisp1 32e10000.isp: v4l2_subdev_link_validate_default: media bus code does not match (source 0x0000200f, sink 0x0000300f) rkisp1 32e10000.isp: v4l2_subdev_link_validate_default: link was "csis-32e40000.csi":1 -> "rkisp1_isp":0 rkisp1 32e10000.isp: Link 'csis-32e40000.csi':1 -> 'rkisp1_isp':0 failed validation: -32 This indicates a mismatch between the source pad of the CSI-2 receiver and the sink pad of the ISP, both in the resolution and in the format code. The CSI-2 receiver is configured to output UYVY8_1X16 in 640x480, and the ISP to receive SRGGB10_1X10 in 800x600, leading to a broken pipeline. To fix this, formats must be configuration through the pipeline with a process referred to as `format propagation`. Format Propagation ------------------ Format propagation is a pipeline configuration mechanism where the format is first configured on the source of the pipeline (the camera sensor in this case), and then propagated downstream through all subdevices until the video node at the end of the pipeline. It involves two different types of operations: - To ensure that formats on both ends of a link match, when configuring the format on a source pad of a subdevice, the same format must be set on the connected sink pad at the other end of the link. This is known as `format propagation across links`. Subdevices do not automatically propagate formats across links within the kernel in order to avoid hardcoding use cases in drivers and to simplify driver implementation. This format propagation is the responsibility of userspace. - Within a subdevice, the kernel ensures that pads configuration stays valid at all times. When an application configures a format on a subdevice pad, the driver adjusts parameters that are not valid to the closest supported values. For instance, resolutions are clampedto the supported range, rounded if the device has alignment constraints, and unsupported format codes are replaced with supported values. This is known as `format negotiation` and is similar to how the kernel handles format setting on V4L2 video devices. In addition to the format negotiation at the pad level, the kernel also ensures that the configurations of all pads within a subdevice remain coherent with each other. When an application configures the format on a sink pad, the subdevice driver also adjusts the configuration of the source pads if the new sink pad configuration would otherwise result in an invalid subdevice configuration. For instance, the i.MX8MP resizer can scale images but can't convert between bus format codes, so if an application sets a format code on the resizer sink pad, the driver will automatically set the same format code on the resizer source pad. This is known as `format propagation inside subdevices`, and is the responsibility of the subdevice drivers. Unlike across links where the formats on the source and sink sides of a link must match, formats on the sink and source pads of a subdevice may be different. For instance, the i.MX8MP ISP interpolates the raw Bayer data provided by the camera sensor, and converts it to YUV. The format code on the source pad of the ISP will thus differ from the sink pad. When userspace configures the format on the ISP sink pad, the driver will propagate the sink resolution to the source pad, but will adjust the sink Bayer format code to a YUV format code on the source pad. Format propagation within subdevices ensures that their configuration remains coherent, but doesn't save applications from having to configure formats on source pads. Using the resizer example again, the driver will ensure that the format on the source pad gets adjusted to a value compatible with the format set on the sink pad, but applications will likely want to then configure the format on the source pad explicitly in order to change the resolution and select the scaling ratio. Format propagation within subdevices always operate `from sink to source`. When userspace sets the format on a source pad, the driver will adjust it to values compatible with the current configuration of the sink pads if needed, but will never modify the sink pads configuration by back-propagating the format from source pad to sink pad. Subdevice pad formats can be set with the media-ctl utility, using the ``--set-v4l2`` option (shortened to ``-V``). The syntax is identical to the format printed by media-ctl with the ``-p`` option. The first step in our example is to configure the format on the sensor source pad. We use the default format reported by the media graph (10-bit raw Bayer in 1920x1080), which corresponds to the sensor native resolution: .. code-block:: none $ media-ctl -d /dev/media0 -V '"imx290 2-001a":0 [fmt:SRGGB10_1X10/1920x1080 colorspace:raw xfer:none ycbcr:601 quantization:full-range]' Running ``media-ctl -p`` again, the format on the sensor source pad has been updated with the newly selected color space: .. code-block:: none :linenos: :lineno-start: 73 :emphasize-lines: 5 - entity 34: imx290 2-001a (1 pad, 1 link) type V4L2 subdev subtype Sensor flags 0 device node name /dev/v4l-subdev3 pad0: Source [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range crop.bounds:(0,0)/1945x1097 crop:(12,8)/1920x1080] -> "csis-32e40000.csi":0 [ENABLED] The format is otherwise the same, as we have selected a format code and resolution identical to the previously configured value. However, this is not the only change in the media graph. The configuration of the CSI-2 receiver is now different: .. code-block:: none :linenos: :lineno-start: 64 :emphasize-lines: 5,8 - entity 29: csis-32e40000.csi (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev2 pad0: Sink [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range] <- "imx290 2-001a":0 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range] -> "rkisp1_isp":0 [ENABLED] The change in the sink pad format seems to contradict what we have seen before, as the kernel isn't supposed to propagate formats across links. The reason for this lies in the media-ctl utility. When setting a format on a source pad, media-ctl automatically reads back the format from the subdevice (to take into account possible adjustments applied by the driver) and sets that format on all sink pads connected through enabled links. The source pad format is also modified, due to automatic format propagation by drivers within subdevices. As the CSI-2 receiver can't transcode or scale, the source pad format has thus been set to match the sink pad format. These two effects simplify the format propagation procedure, as formats on sink pads don't need to be set manually. Instead, we can read back the format on the source pad of the next subdevice (the CSI-2 receiver in this case) using ``media-ctl -p``, and set it on the same pad to let media-ctl propagate it to the next subdevice: .. code-block:: none $ media-ctl -d /dev/media0 -V '"csis-32e40000.csi":1 [fmt:SRGGB10_1X10/1920x1080 colorspace:raw xfer:none ycbcr:601 quantization:full-range]' We expect the next subdevice, ´´rkisp1_isp``, to output an image in a processed YUV format with a 1920x1080 resolution. This is not what happens: .. code-block:: none :linenos: :lineno-start: 14 :emphasize-lines: 6-7,13 - entity 1: rkisp1_isp (4 pads, 4 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Sink [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range crop.bounds:(0,0)/1920x1080 crop:(0,0)/800x600] <- "csis-32e40000.csi":1 [ENABLED] pad1: Sink [fmt:unknown/0x0 field:none] <- "rkisp1_params":0 [ENABLED,IMMUTABLE] pad2: Source [fmt:YUYV8_2X8/800x600 field:none colorspace:raw xfer:none ycbcr:601 quantization:lim-range crop.bounds:(0,0)/800x600 crop:(0,0)/800x600] -> "rkisp1_resizer_mainpath":0 [ENABLED] pad3: Source [fmt:unknown/0x0 field:none] -> "rkisp1_stats":0 [ENABLED,IMMUTABLE] The output format code (``YUYV8_2X8``) is fine, but the output resolution has been set to 800x600. This is due to the crop capability of the sink pad. While the crop bounds have been adjusted (line 19), the crop rectangle itself (line 20) hasn't been adjusted as its previous size (``(0,0)/800x600``) is compatible with the new sink format. We must thus set the crop rectangle manually on the sink pad: .. code-block:: none $ media-ctl -d /dev/media0 -V '"rkisp1_isp":0 [crop:(0,0)/1920x1080]' This improves the situation: .. code-block:: none :linenos: :lineno-start: 14 :emphasize-lines: 7,13-15 - entity 1: rkisp1_isp (4 pads, 4 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Sink [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range crop.bounds:(0,0)/1920x1080 crop:(0,0)/1920x1080] <- "csis-32e40000.csi":1 [ENABLED] pad1: Sink [fmt:unknown/0x0 field:none] <- "rkisp1_params":0 [ENABLED,IMMUTABLE] pad2: Source [fmt:YUYV8_2X8/800x600 field:none colorspace:raw xfer:none ycbcr:601 quantization:lim-range crop.bounds:(0,0)/1920x1080 crop:(0,0)/800x600] -> "rkisp1_resizer_mainpath":0 [ENABLED] pad3: Source [fmt:unknown/0x0 field:none] -> "rkisp1_stats":0 [ENABLED,IMMUTABLE] The crop bounds on the source pad (line 27) now cover the full resolution, but the source crop rectangle (line 28) hasn't been adjusted, for the same reason as previously. Fixing this is simple: .. code-block:: none $ media-ctl -d /dev/media0 -V '"rkisp1_isp":2 [crop:(0,0)/1920x1080]' This gives us the configuration we want for the source resolution: .. code-block:: none :linenos: :lineno-start: 14 :emphasize-lines: 13,15 - entity 1: rkisp1_isp (4 pads, 4 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Sink [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range crop.bounds:(0,0)/1920x1080 crop:(0,0)/1920x1080] <- "csis-32e40000.csi":1 [ENABLED] pad1: Sink [fmt:unknown/0x0 field:none] <- "rkisp1_params":0 [ENABLED,IMMUTABLE] pad2: Source [fmt:YUYV8_2X8/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:lim-range crop.bounds:(0,0)/1920x1080 crop:(0,0)/1920x1080] -> "rkisp1_resizer_mainpath":0 [ENABLED] pad3: Source [fmt:unknown/0x0 field:none] -> "rkisp1_stats":0 [ENABLED,IMMUTABLE] Finally, we set the format on the source pad to let media-ctl propagate it to the next subdevice (``rkisp1_resizer_mainpath``): .. code-block:: none $ media-ctl -d /dev/media0 -V '"rkisp1_isp":2 [fmt:YUYV8_2X8/1920x1080]' The resizer sink pad format is configured correctly: .. code-block:: none :linenos: :lineno-start: 34 :emphasize-lines: 5,7,10 - entity 6: rkisp1_resizer_mainpath (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev1 pad0: Sink [fmt:YUYV8_2X8/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:lim-range crop.bounds:(0,0)/1920x1080 crop:(0,0)/800x600] <- "rkisp1_isp":2 [ENABLED] pad1: Source [fmt:YUYV8_2X8/800x600 field:none colorspace:raw xfer:none ycbcr:601 quantization:lim-range] -> "rkisp1_mainpath":0 [ENABLED,IMMUTABLE] The resizer source pad (line 43) matches the resolution we want to capture (``800x600``). However, this is not achieved by scaling, but by cropping the input frame (line 40). This isn't the desired behaviour, so, even if frame capture would work with this configuration, we must adjust the sink pad crop rectangle: .. code-block:: none $ media-ctl -d /dev/media0 -V '"rkisp1_resizer_mainpath":0 [crop:(0,0)/1920x1080]' The source pad format is unchanged and still set to ``YUYV8_2X8/800x600``. If we wanted to scale to a different resolution, we would need to set the format on the resizer source pad at this point. The next entity in the pipeline (``rkisp1_mainpath``) is not a subdevice but a video device. There is no need to configure pad formats, as formats on video devices are set through the classic V4L2 API. Care must be taken to configure the video device with a pixel format compatible with the source pad of the last subdevice in the pipeline, and an identical resolution. Which pixel formats qualify as compatible is device-dependent, as explained in the :ref:`v4l2-subdevice-formats` section. For the i.MX8MP, the DMA engine at the output of the pipeline can write the YUV 4:2:2 data it receives from the resizer in the semi-planar NV16 format: .. code-block:: none $ v4l2-ctl -d /dev/video0 -v pixelformat=NV16,width=800,height=600 --stream-count 10 --stream-mmap <<<<<<<<<< The pipeline is valid, and 10 frames have been captured successfully. Summary ------- Pipelines can be inspected with ``media-ctl -p`` and must configured by userspace before being used. The first configuration step covers link setup. Links should be reset with ``media-ctl -r`` to avoid any influence from a previous configuration, and individual links should then be enabled with ``media-ctl -l``. The second configuration step covers format configuration. Formats are set with ``media-ctl -V`` and must be propagated through the pipeline from source to sink. The combination of automatic format propagation across links in media-ctl with the automatic format propagation inside subdevices in the kernel simplifies format propagation. The generic procedure is as follows: #. Set the format on the source pad of the first subdevice in the pipeline. #. The kernel adjusts the format, and media-ctl automatically propagates it over the link to the connected sink pad of the next subdevice. #. If the next subdevice has crop or compose rectangleon its sink or source pads, sets them as necessary, in the following order: - Sink crop - Sink compose - Source crop #. The kernel automatically propagates the format to the source pad of the subdevice, adjusting it if needed. #. Retrieve the propagated format on the source pad of the subdevice, modify it if required (for instance to configure scaling), and set of on the same source pad to let media-ctl propagate it across the next link. #. Repeat this procedure from subdevice to subdevice, until the last subdevice in the pipeline is reached. After configuring the pipeline, frames can be captured from the video device at the pipeline output with the classic V4L2 API, in a pixel format and resolution that matches the pipeline configuration.