path: root/configuring-pipelines.rst

.. SPDX-License-Identifier: CC-BY-SA-4.0

Configuring Pipelines
=====================

Allowing fine-grained control of formats comes at the cost of increasing
complexity for applications. Where an application used to only have to set the
format on a video device, it must now set formats on all pads in the pipeline,
and ensure that the result forms a coherent configuration.

As previously stated, the examples in this section will be based on the NXP
i.MX8MP ISP device:

.. graphviz:: imx8mp-isp.dot
   :caption: Media graph of the NXP i.MX8MP with default configuration


Anatomy of a Pipeline
---------------------

.. _media-ctl: https://git.linuxtv.org/v4l-utils.git/tree/utils/media-ctl
.. _v4l2-ctl: https://git.linuxtv.org/v4l-utils.git/tree/utils/v4l2-ctl

The `media-ctl`_ utility can be used to display the formats on all pads in a
graph. Let's start by disecting the output of ``media-ctl -d /dev/media0 -p``:

.. code-block:: none
   :linenos:
   :lineno-start: 1

   $ media-ctl -d /dev/media0 -p
   Media controller API version 6.0.0

   Media device information
   ------------------------
   driver          rkisp1
   model           rkisp1
   serial          
   bus info        platform:rkisp1
   hw revision     0xe
   driver version  6.0.0

media-ctl first provides summary information, to quickly identify the hardware
device corresponding to the media controller device (``/dev/media0`` here):

- the driver name
- the device model name
- the device serial number (when available)
- which bus the device sits on
- the device hardware revision (in a device-specific format)
- the driver version (identical to the kernel version)

The ISP integrated in the NXP i.MX8MP is also found in Rockchip SoCs. They are
both handled by the ``rkisp1`` kernel driver, as shown by the driver and device
model names.

media-ctl then prints detailed information for all entities in the graph.
Entities are sorted by their numerical ID, which doesn't necessarily follow any
logical order. To improve readability, the entities below have been reordered
in source to sink order, as shown by the line numbers. This corresponds to the
top to bottom order in the graphical representation of the media graph.

.. code-block:: none
   :linenos:
   :lineno-start: 73

   - entity 34: imx290 2-001a (1 pad, 1 link)
                type V4L2 subdev subtype Sensor flags 0
                device node name /dev/v4l-subdev3
           pad0: Source
                   [fmt:SRGGB10_1X10/1920x1080 field:none
                    crop.bounds:(0,0)/1945x1097
                    crop:(12,8)/1920x1080]
                   -> "csis-32e40000.csi":0 []

All entities have a numerical ID and a name. The first entity in pipeline order
has ID ``34`` and is named ``imx290 2-001a``. Names are chosen by drivers but
should follow a set of naming rules. For devices controlled through an I2C bus,
as is the case here, the entity name is made of the device model (``imx290``)
and bus information (``2-001a``) separated by a space. The I2C bus information
is itself made of the bus number and the device address (in hexadecimal),
separated by a dash.

This particular entity has a total of 1 pad and 1 link (line 73). It is a V4L2
subdevice whose subtype is a camera sensor (line 74). The subdevice is exposed
to userspace through the ``/dev/v4l-subdev3`` device node (line 75).

For each pad of the entity media-ctl then displays the pad index and type, the
available links, as well as pad information specific to the entity type.  Pad
indices start at 0. The IMX290 camera sensor entity has a single pad, thus at
index 0 and named ``pad0``, whose type is a source pad (line 76).

Links are displayed on lines starting with a ``->`` (for outbound links, on
source pads) or ``<-`` (for inbound links, on sink pads) sign, followed by the
link target (``"csis-32e40000.csi":0``) and link flags (``[]``). The link
target is a pad of another entity, expressed as the target entity name
surrounded by double quotes, followed by a colon and the target pad index. The
supported link flags (with their numerical values) are:

ENABLED (0x1)
   The link is enabled. In the absence of this flag, the link is disabled.

IMMUTABLE (0x2)
   The link is immutable, its ``ENABLED`` flag cannot be modified. Immutable
   links in practice always have the ``ENABLED`` flag set.

As the list of link flags is empty, this specific link is thus disabled, but
can be enabled as it isn't immutable. The graphical representation displays
this link as a dotted line.

We will ignore for now the pad information specific to the entity type
displayed on lines 77 to 79, this will be explained further down.

The next entity, in graph order, is the MIPI CSI-2 receiver that the camera
sensor is attached to:

.. code-block:: none
   :linenos:
   :lineno-start: 64

   - entity 29: csis-32e40000.csi (2 pads, 2 links)
                type V4L2 subdev subtype Unknown flags 0
                device node name /dev/v4l-subdev2
           pad0: Sink
                   [fmt:UYVY8_1X16/640x480 field:none colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range]
                   <- "imx290 2-001a":0 []
           pad1: Source
                   [fmt:UYVY8_1X16/640x480 field:none colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range]
                   -> "rkisp1_isp":0 [ENABLED]

The CSI-2 receiver is an IP core internal to the SoC, modelled in the kernel as
a platform device. Entity names from platform devices are more free-formed. In
this case, the driver follows the usual recommendation, which uses the platform
device model name (``csis``) and the platform device name (``32e40000.csi``),
separated by a dash. On machines using Device Tree to describe the system (as
is the case on ARM and ARM64 machines), the platform device name is typically
make of the device's bus address (``32e40000``) and device node name (``csi``).

This entity has two pads, a sink pad and a source pad, with one link each. The
sink pad, ``pad0`` (line 67), is the target of the link originating from the
camera sensor, and is thus without surprise displayed as connected to the
source pad of the camera sensor (line 69). The link flags being a property of
the link, not the pad, they match the link flags displayed in the camera sensor
entity.

The source pad, ``pad1`` (line 70), is connected to pad 0 of the ``rkisp1_isp``
entity (line 72). This link is enabled but not immutable, thus displayed as a
thin plain line in the graphical representation.

The next entity represents the core of the ISP itself:

.. code-block:: none
   :linenos:
   :lineno-start: 14

   - entity 1: rkisp1_isp (4 pads, 4 links)
               type V4L2 subdev subtype Unknown flags 0
               device node name /dev/v4l-subdev0
           pad0: Sink
                   [fmt:SRGGB10_1X10/800x600 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range
                    crop.bounds:(0,0)/800x600
                    crop:(0,0)/800x600]
                   <- "csis-32e40000.csi":1 [ENABLED]
           pad1: Sink
                   [fmt:unknown/0x0 field:none]
                   <- "rkisp1_params":0 [ENABLED,IMMUTABLE]
           pad2: Source
                   [fmt:YUYV8_2X8/800x600 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range
                    crop.bounds:(0,0)/800x600
                    crop:(0,0)/800x600]
                   -> "rkisp1_resizer_mainpath":0 [ENABLED]
           pad3: Source
                   [fmt:unknown/0x0 field:none]
                   -> "rkisp1_stats":0 [ENABLED,IMMUTABLE]

Its name doesn't follow the previously described recommendation for entities
corresponding to platform devices, which shows that applications can't rely on
a particular format for entity names in general. This may be considered as a
kernel bug, but is unlikely to be changed as entity name changes may break
existing applications.

This entity is also a V4L2 subdevice, with four pads, two sink pads (``pad0``
and ``pad1`` on lines 17 and 22 respectively) and two source pads (``pad2`` and
``pad3`` on lines 25 and 30). Pads 1 and 3 are connected to entities
representing the video device nodes through which ISP parameters are supplied
and ISP statistics are captured. We will ignore them in this example as they
are not strictly required to operate the device. It is however worth noting
that the corresponding links are enabled and immutable, and thus displayed in
the graphical representation as thich plain lines.

Pad 2 is a source pad that connects the ISP core to the resizer, which is the
next entity in the pipeline.

.. code-block:: none
   :linenos:
   :lineno-start: 34

   - entity 6: rkisp1_resizer_mainpath (2 pads, 2 links)
               type V4L2 subdev subtype Unknown flags 0
               device node name /dev/v4l-subdev1
           pad0: Sink
                   [fmt:YUYV8_2X8/800x600 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range
                    crop.bounds:(0,0)/800x600
                    crop:(0,0)/800x600]
                   <- "rkisp1_isp":2 [ENABLED]
           pad1: Source
                   [fmt:YUYV8_2X8/800x600 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
                   -> "rkisp1_mainpath":0 [ENABLED,IMMUTABLE]

The resizer is also a V4L2 subdevice, and has nothing noteworthy at this point
compared to the entities previously described. Its source pad (line 42) is
connected to the last entity in the pipeline:

.. code-block:: none
   :linenos:
   :lineno-start: 46

   - entity 9: rkisp1_mainpath (1 pad, 1 link)
               type Node subtype V4L flags 0
               device node name /dev/video0
           pad0: Sink
                   <- "rkisp1_resizer_mainpath":1 [ENABLED,IMMUTABLE]

The ``rkisp1_mainpath`` entity, unlike all previous entities, is not a V4L2
subdevice but a V4L2 video node (line 47). Its device node name is
``/dev/video0`` (line 48), corresponding to the output of the pipeline. This is
the device node from which frames originating from the sensor, processed by the
ISP, and scaled by the resizer are captured.

Let's try to capture frames in NV16 format with a 800x600 resolution:

.. code-block:: none
   :emphasize-lines: 2

   $ v4l2-ctl -d /dev/video0 -v pixelformat=NV16,width=800,height=600 --stream-count 10 --stream-mmap
                   VIDIOC_STREAMON returned -1 (Broken pipe)

The kernel driver returns the `Broken pipe` error code, which indicates that
the pipeline isn't properly configured. The first possibly culprit to
investigate is the links.


Link Setup
----------

.. _EBNF metasyntax: https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form

We have seen that, unless immutable, links can be enabled or disabled. Enabling
or disabling links allows configuring data routing through the graph. For
instance, a device that has multiple camera sensors connected to an ISP, but
can only capture from one of the sensors at a time, could be controlled by
enabling one of the links from the camera sensors to select the source.

In the default configuration of the device, the link from the camera sensor to
the CSI-2 receiver is disabled. The pipeline is thus missing a source, which
causes the capture to fail with the `Broken pipe` error. To fix this, links
must be setup before starting capture. This is done with the media-ctl utility:

.. code-block:: none

   $ media-ctl -d /dev/media0 --link '"imx290 2-001a":0 -> "csis-32e40000.csi":0 [1]'

The ``-link`` option (or its shorter form ``-l``) takes a list of links,
expressed as follows (using a syntax close to the `EBNF metasyntax`_, found in
the output of ``media-ctl --help``):

.. code-block:: ebnf

   links           = link { ',' link } ;
   link            = pad '->' pad '[' flags ']' ;
   pad             = entity ':' pad-number ;
   entity          = entity-number | ( '"' entity-name '"' ) ;

In the above example, we enable (``[1]``) the link from the source pad of the
camera sensor entity (``'"imx290 2-001a":0``) to the sink pad of the CSI-2
receiver entity (``"csis-32e40000.csi":0``). Links can also be disabled by
clearing the ENABLED flag (using ``[0]``), and multiple links can be setup with
a single command, although this is discouraged in scripts as the resulting
syntax would be hard to read.

After running the above command, ``media-ctl -p`` reports the link between the
camera sensor and the CSI-2 receiver as enabled:

.. code-block:: none
   :linenos:
   :lineno-start: 73
   :emphasize-lines: 8

   - entity 34: imx290 2-001a (1 pad, 1 link)
                type V4L2 subdev subtype Sensor flags 0
                device node name /dev/v4l-subdev3
           pad0: Source
                   [fmt:SRGGB10_1X10/1920x1080 field:none
                    crop.bounds:(0,0)/1945x1097
                    crop:(12,8)/1920x1080]
                   -> "csis-32e40000.csi":0 [ENABLED]

This is also displayed in the graphical representation, with the corresponding
link now being a thin plain line instead of a dotted line:

.. graphviz:: imx8mp-isp-links-enabled.dot
   :caption: Media graph of the NXP i.MX8MP after enabling links

.. note::

   It is recommended, before setting up links in a pipeline, to reset all
   non-immutable links to their disabled state with ``media-ctl --reset``. This
   helps starting from a known state and, and avoids depending on a previous
   configuration of the media graph.  In this specific example, the links
   between the CSI-2 receiver and the ISP, and between the ISP and the resizer,
   would get disabled, and would need to be enabled manually to create a
   complete pipeline.

Let's try to capture frames in NV16 format with a 800x600 resolution again:

.. code-block:: none
   :emphasize-lines: 2

   $ v4l2-ctl -d /dev/video0 -v pixelformat=NV16,width=800,height=600 --stream-count 10 --stream-mmap
                   VIDIOC_STREAMON returned -1 (Broken pipe)

The kernel still returns the `Broken pipe` error, for an entirely different
reason: we haven't taken care of formats in the pipeline.


Formats in the Pipeline
-----------------------

.. _Media Bus Format: https://linuxtv.org/downloads/v4l-dvb-apis/userspace-api/v4l/subdev-formats.html>

It is time to stop ignoring the pad information specific to the entity type
that is printed by media-ctl. Let's have a look at the camera sensor entity
again:

.. code-block:: none
   :linenos:
   :lineno-start: 73
   :emphasize-lines: 5-7

   - entity 34: imx290 2-001a (1 pad, 1 link)
                type V4L2 subdev subtype Sensor flags 0
                device node name /dev/v4l-subdev3
           pad0: Source
                   [fmt:SRGGB10_1X10/1920x1080 field:none
                    crop.bounds:(0,0)/1945x1097
                    crop:(12,8)/1920x1080]
                   -> "csis-32e40000.csi":0 [ENABLED]

For entities corresponding to V4L2 subdevices, media-ctl prints the active
configuration for each pad (lines 68 to 78). The configuration is made of
multiple properties:

- The media bus format, expressed as

  .. code-block:: ebnf

     media bus format = "fmt:", fcc, "/", size ;
     size             = width, "x", height ;

  where ``fcc`` is the `media bus format`_ code. Format codes are listed in the
  ``linux/media-bus-format.h`` header file of the Linux kernel, and expressed
  here as a string without the ``MEDIA_BUS_FMT_`` prefix.

- The field, for interlaced video:

  .. code-block:: ebnf

     field      = "field:", v4l2-field ;
     v4l2-field = "any" | "none" | "top" | "bottom" | "interlaced"
                | "seq-tb" | "seq-bt" | "alternate" | "interlaced-tb"
                | "interlaced-bt" ;

- The color space, expressed as four properties, reporting the V4L2 color
  space, transfer function, YCbCr encoding and quantization range:

  .. code-block:: ebnf

     color space    = "colorspace:", v4l2-colorspace, " xfer:", v4l2-xfer-func,
                      " ycbcr:", v4l2-ycbcr-enc, " quantization:", v4l2-quant ;
     v4l2-colorspace = "default" | "smpte170m" | "smpte240m" | "rec709"
                     | "470m" | "470bg" | "jpeg" | "srgb" | "oprgb" | "bt2020"
                     | "raw" | "dcip3" ;
     v4l2-xfer-func  =  "default" | "709" | "srgb" | "oprgb" | "smpte240m"
                     | "smpte2084" | "dcip3" | "none" ;
     v4l2-ycbcr-enc  = "default" | "601" | "709" | "xv601" | "xv709" | "bt2020"
                     | "bt2020c" | "smpte240m" ;
     v4l2-quant      = "default" | "full-range" | "lim-range" ;

- The compose and crop rectangles:

  .. code-block:: ebnf

     rectangle = "(", left, ",", top, ")", "/", size ;
     size      = width, "x", height ;


In order for a pipeline to be valid, the configuration of pads on both sides of
every link must match. Unless drivers implement custom validation rules, the
default constraint is that the media bus format on both sides must be
identical. For interlaced video, the fields must also be identical, or the
field on the sink side must be equal to ``none`` (in order to support
interlaced sources connected to sinks that only support progressive formats).
The crop and compose rectangles are not taken into account when validating
pipelines, as they are internal to subdevices.

The default validation is performed by the ``v4l2_subdev_link_validate_default()``
function in the kernel. If dynamic debugging is enabled (through the
``CONFIG_DYNAMIC_DEBUG`` kernel configuration option), extra messages can be
enabled through debugfs to diagnose the problem more precisely:

.. code-block:: none

   $ mount -t debugfs none /sys/kernel/debug/
   $ echo 'func v4l2_subdev_link_validate_default +p' > /sys/kernel/debug/dynamic_debug/control

After trying to capture frames with v4l2-ctl and obtaining the `Broken pipe`
error, the kernel log (obtained with ``dmesg``) provides useful information:

.. code-block:: none

   rkisp1 32e10000.isp: Validating pad 'rkisp1_isp':0
   rkisp1 32e10000.isp: v4l2_subdev_link_validate_default: width does not match (source 640, sink 800)
   rkisp1 32e10000.isp: v4l2_subdev_link_validate_default: height does not match (source 480, sink 600)
   rkisp1 32e10000.isp: v4l2_subdev_link_validate_default: media bus code does not match (source 0x0000200f, sink 0x0000300f)
   rkisp1 32e10000.isp: v4l2_subdev_link_validate_default: link was "csis-32e40000.csi":1 -> "rkisp1_isp":0
   rkisp1 32e10000.isp: Link 'csis-32e40000.csi':1 -> 'rkisp1_isp':0 failed validation: -32

This indicates a mismatch between the source pad of the CSI-2 receiver and the
sink pad of the ISP, both in the resolution and in the format code. The CSI-2
receiver is configured to output UYVY8_1X16 in 640x480, and the ISP to receive
SRGGB10_1X10 in 800x600, leading to a broken pipeline.

To fix this, formats must be configuration through the pipeline with a process
referred to as `format propagation`.


Format Propagation
------------------

Format propagation is a pipeline configuration mechanism where the format is
first configured on the source of the pipeline (the camera sensor in this
case), and then propagated downstream through all subdevices until the video
node at the end of the pipeline. It involves two different types of operations:

- To ensure that formats on both ends of a link match, when configuring the
  format on a source pad of a subdevice, the same format must be set on the
  connected sink pad at the other end of the link. This is known as `format
  propagation across links`. Subdevices do not automatically propagate formats
  across links within the kernel in order to avoid hardcoding use cases in
  drivers and to simplify driver implementation. This format propagation is the
  responsibility of userspace.

- Within a subdevice, the kernel ensures that pads configuration stays valid at
  all times. When an application configures a format on a subdevice pad, the
  driver adjusts parameters that are not valid to the closest supported values.
  For instance, resolutions are clampedto the supported range, rounded if the
  device has alignment constraints, and unsupported format codes are replaced
  with supported values. This is known as `format negotiation` and is similar
  to how the kernel handles format setting on V4L2 video devices.

  In addition to the format negotiation at the pad level, the kernel also
  ensures that the configurations of all pads within a subdevice remain
  coherent with each other. When an application configures the format on a sink
  pad, the subdevice driver also adjusts the configuration of the source pads
  if the new sink pad configuration would otherwise result in an invalid
  subdevice configuration. For instance, the i.MX8MP resizer can scale images
  but can't convert between bus format codes, so if an application sets a
  format code on the resizer sink pad, the driver will automatically set the
  same format code on the resizer source pad. This is known as `format
  propagation inside subdevices`, and is the responsibility of the subdevice
  drivers.

  Unlike across links where the formats on the source and sink sides of a link
  must match, formats on the sink and source pads of a subdevice may be
  different. For instance, the i.MX8MP ISP interpolates the raw Bayer data
  provided by the camera sensor, and converts it to YUV. The format code on the
  source pad of the ISP will thus differ from the sink pad. When userspace
  configures the format on the ISP sink pad, the driver will propagate the sink
  resolution to the source pad, but will adjust the sink Bayer format code to a
  YUV format code on the source pad.

  Format propagation within subdevices ensures that their configuration remains
  coherent, but doesn't save applications from having to configure formats on
  source pads. Using the resizer example again, the driver will ensure that the
  format on the source pad gets adjusted to a value compatible with the format
  set on the sink pad, but applications will likely want to then configure the
  format on the source pad explicitly in order to change the resolution and
  select the scaling ratio. Format propagation within subdevices always operate
  `from sink to source`. When userspace sets the format on a source pad, the
  driver will adjust it to values compatible with the current configuration of
  the sink pads if needed, but will never modify the sink pads configuration by
  back-propagating the format from source pad to sink pad.

Subdevice pad formats can be set with the media-ctl utility, using the
``--set-v4l2`` option (shortened to ``-V``). The syntax is identical to the
format printed by media-ctl with the ``-p`` option. The first step in our
example is to configure the format on the sensor source pad. We use the default
format reported by the media graph (10-bit raw Bayer in 1920x1080), which
corresponds to the sensor native resolution:

.. code-block:: none

   $ media-ctl -d /dev/media0 -V '"imx290 2-001a":0 [fmt:SRGGB10_1X10/1920x1080 colorspace:raw xfer:none ycbcr:601 quantization:full-range]'

Running ``media-ctl -p`` again, the format on the sensor source pad has been
updated with the newly selected color space:

.. code-block:: none
   :linenos:
   :lineno-start: 73
   :emphasize-lines: 5

   - entity 34: imx290 2-001a (1 pad, 1 link)
                type V4L2 subdev subtype Sensor flags 0
                device node name /dev/v4l-subdev3
           pad0: Source
                   [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range
                    crop.bounds:(0,0)/1945x1097
                    crop:(12,8)/1920x1080]
                   -> "csis-32e40000.csi":0 [ENABLED]

The format is otherwise the same, as we have selected a format code and
resolution identical to the previously configured value. However, this is not
the only change in the media graph. The configuration of the CSI-2 receiver is
now different:

.. code-block:: none
   :linenos:
   :lineno-start: 64
   :emphasize-lines: 5,8

   - entity 29: csis-32e40000.csi (2 pads, 2 links)
                type V4L2 subdev subtype Unknown flags 0
                device node name /dev/v4l-subdev2
           pad0: Sink
                   [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range]
                   <- "imx290 2-001a":0 [ENABLED]
           pad1: Source
                   [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range]
                   -> "rkisp1_isp":0 [ENABLED]

The change in the sink pad format seems to contradict what we have seen before,
as the kernel isn't supposed to propagate formats across links. The reason for
this lies in the media-ctl utility. When setting a format on a source pad,
media-ctl automatically reads back the format from the subdevice (to take into
account possible adjustments applied by the driver) and sets that format on all
sink pads connected through enabled links.

The source pad format is also modified, due to automatic format propagation by
drivers within subdevices. As the CSI-2 receiver can't transcode or scale, the
source pad format has thus been set to match the sink pad format.

These two effects simplify the format propagation procedure, as formats on sink
pads don't need to be set manually. Instead, we can read back the format on the
source pad of the next subdevice (the CSI-2 receiver in this case) using
``media-ctl -p``, and set it on the same pad to let media-ctl propagate it to
the next subdevice:

.. code-block:: none

   $ media-ctl -d /dev/media0 -V '"csis-32e40000.csi":1 [fmt:SRGGB10_1X10/1920x1080 colorspace:raw xfer:none ycbcr:601 quantization:full-range]'

We expect the next subdevice, ´´rkisp1_isp``, to output an image in a processed
YUV format with a 1920x1080 resolution. This is not what happens:

.. code-block:: none
   :linenos:
   :lineno-start: 14
   :emphasize-lines: 6-7,13

   - entity 1: rkisp1_isp (4 pads, 4 links)
               type V4L2 subdev subtype Unknown flags 0
               device node name /dev/v4l-subdev0
           pad0: Sink
                   [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range
                    crop.bounds:(0,0)/1920x1080
                    crop:(0,0)/800x600]
                   <- "csis-32e40000.csi":1 [ENABLED]
           pad1: Sink
                   [fmt:unknown/0x0 field:none]
                   <- "rkisp1_params":0 [ENABLED,IMMUTABLE]
           pad2: Source
                   [fmt:YUYV8_2X8/800x600 field:none colorspace:raw xfer:none ycbcr:601 quantization:lim-range
                    crop.bounds:(0,0)/800x600
                    crop:(0,0)/800x600]
                   -> "rkisp1_resizer_mainpath":0 [ENABLED]
           pad3: Source
                   [fmt:unknown/0x0 field:none]
                   -> "rkisp1_stats":0 [ENABLED,IMMUTABLE]

The output format code (``YUYV8_2X8``) is fine, but the output resolution has
been set to 800x600. This is due to the crop capability of the sink pad. While
the crop bounds have been adjusted (line 19), the crop rectangle itself (line
20) hasn't been adjusted as its previous size (``(0,0)/800x600``) is compatible
with the new sink format. We must thus set the crop rectangle manually on the
sink pad:

.. code-block:: none

   $ media-ctl -d /dev/media0 -V '"rkisp1_isp":0 [crop:(0,0)/1920x1080]'

This improves the situation:

.. code-block:: none
   :linenos:
   :lineno-start: 14
   :emphasize-lines: 7,13-15

   - entity 1: rkisp1_isp (4 pads, 4 links)
               type V4L2 subdev subtype Unknown flags 0
               device node name /dev/v4l-subdev0
           pad0: Sink
                   [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range
                    crop.bounds:(0,0)/1920x1080
                    crop:(0,0)/1920x1080]
                   <- "csis-32e40000.csi":1 [ENABLED]
           pad1: Sink
                   [fmt:unknown/0x0 field:none]
                   <- "rkisp1_params":0 [ENABLED,IMMUTABLE]
           pad2: Source
                   [fmt:YUYV8_2X8/800x600 field:none colorspace:raw xfer:none ycbcr:601 quantization:lim-range
                    crop.bounds:(0,0)/1920x1080
                    crop:(0,0)/800x600]
                   -> "rkisp1_resizer_mainpath":0 [ENABLED]
           pad3: Source
                   [fmt:unknown/0x0 field:none]
                   -> "rkisp1_stats":0 [ENABLED,IMMUTABLE]

The crop bounds on the source pad (line 27) now cover the full resolution, but
the source crop rectangle (line 28) hasn't been adjusted, for the same reason
as previously. Fixing this is simple:

.. code-block:: none

   $ media-ctl -d /dev/media0 -V '"rkisp1_isp":2 [crop:(0,0)/1920x1080]'

This gives us the configuration we want for the source resolution:

.. code-block:: none
   :linenos:
   :lineno-start: 14
   :emphasize-lines: 13,15

   - entity 1: rkisp1_isp (4 pads, 4 links)
               type V4L2 subdev subtype Unknown flags 0
               device node name /dev/v4l-subdev0
           pad0: Sink
                   [fmt:SRGGB10_1X10/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:full-range
                    crop.bounds:(0,0)/1920x1080
                    crop:(0,0)/1920x1080]
                   <- "csis-32e40000.csi":1 [ENABLED]
           pad1: Sink
                   [fmt:unknown/0x0 field:none]
                   <- "rkisp1_params":0 [ENABLED,IMMUTABLE]
           pad2: Source
                   [fmt:YUYV8_2X8/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:lim-range
                    crop.bounds:(0,0)/1920x1080
                    crop:(0,0)/1920x1080]
                   -> "rkisp1_resizer_mainpath":0 [ENABLED]
           pad3: Source
                   [fmt:unknown/0x0 field:none]
                   -> "rkisp1_stats":0 [ENABLED,IMMUTABLE]

Finally, we set the format on the source pad to let media-ctl propagate it to
the next subdevice (``rkisp1_resizer_mainpath``):

.. code-block:: none

   $ media-ctl -d /dev/media0 -V '"rkisp1_isp":2 [fmt:YUYV8_2X8/1920x1080]'

The resizer sink pad format is configured correctly:

.. code-block:: none
   :linenos:
   :lineno-start: 34
   :emphasize-lines: 5,7,10

   - entity 6: rkisp1_resizer_mainpath (2 pads, 2 links)
               type V4L2 subdev subtype Unknown flags 0
               device node name /dev/v4l-subdev1
           pad0: Sink
                   [fmt:YUYV8_2X8/1920x1080 field:none colorspace:raw xfer:none ycbcr:601 quantization:lim-range
                    crop.bounds:(0,0)/1920x1080
                    crop:(0,0)/800x600]
                   <- "rkisp1_isp":2 [ENABLED]
           pad1: Source
                   [fmt:YUYV8_2X8/800x600 field:none colorspace:raw xfer:none ycbcr:601 quantization:lim-range]
                   -> "rkisp1_mainpath":0 [ENABLED,IMMUTABLE]

The resizer source pad (line 43) matches the resolution we want to capture
(``800x600``). However, this is not achieved by scaling, but by cropping the
input frame (line 40). This isn't the desired behaviour, so, even if frame
capture would work with this configuration, we must adjust the sink pad crop
rectangle:

.. code-block:: none

   $ media-ctl -d /dev/media0 -V '"rkisp1_resizer_mainpath":0 [crop:(0,0)/1920x1080]'

The source pad format is unchanged and still set to ``YUYV8_2X8/800x600``. If
we wanted to scale to a different resolution, we would need to set the format
on the resizer source pad at this point.

The next entity in the pipeline (``rkisp1_mainpath``) is not a subdevice but a
video device. There is no need to configure pad formats, as formats on video
devices are set through the classic V4L2 API. Care must be taken to configure
the video device with a pixel format compatible with the source pad of the last
subdevice in the pipeline, and an identical resolution. Which pixel formats
qualify as compatible is device-dependent, as explained in the
:ref:`v4l2-subdevice-formats` section. For the i.MX8MP, the DMA engine at the
output of the pipeline can write the YUV 4:2:2 data it receives from the
resizer in the semi-planar NV16 format:

.. code-block:: none

   $ v4l2-ctl -d /dev/video0 -v pixelformat=NV16,width=800,height=600 --stream-count 10 --stream-mmap
   <<<<<<<<<<

The pipeline is valid, and 10 frames have been captured successfully.


Summary
-------

Pipelines can be inspected with ``media-ctl -p`` and must configured by
userspace before being used.

The first configuration step covers link setup. Links should be reset with
``media-ctl -r`` to avoid any influence from a previous configuration, and
individual links should then be enabled with ``media-ctl -l``.

The second configuration step covers format configuration. Formats are set with
``media-ctl -V`` and must be propagated through the pipeline from source to
sink. The combination of automatic format propagation across links in media-ctl
with the automatic format propagation inside subdevices in the kernel
simplifies format propagation. The generic procedure is as follows:

#. Set the format on the source pad of the first subdevice in the pipeline.
#. The kernel adjusts the format, and media-ctl automatically propagates it
   over the link to the connected sink pad of the next subdevice.
#. If the next subdevice has crop or compose rectangleon its sink or source
   pads, sets them as necessary, in the following order:

   - Sink crop
   - Sink compose
   - Source crop

#. The kernel automatically propagates the format to the source pad of the
   subdevice, adjusting it if needed.
#. Retrieve the propagated format on the source pad of the subdevice, modify it
   if required (for instance to configure scaling), and set of on the same
   source pad to let media-ctl propagate it across the next link.
#. Repeat this procedure from subdevice to subdevice, until the last subdevice
   in the pipeline is reached.

After configuring the pipeline, frames can be captured from the video device at
the pipeline output with the classic V4L2 API, in a pixel format and resolution
that matches the pipeline configuration.