Difference between revisions of "Mainline Hardware Decoding"
Link Mauve (talk | contribs) m (→Supported Codecs By SoC: Oops, AV1 should be in green.) |
|||
(39 intermediate revisions by 7 users not shown) | |||
Line 1: | Line 1: | ||
{{hint|This page is incomplete, you're welcome to improve it.}} | {{hint|This page is incomplete, you're welcome to improve it.}} | ||
'''Mainline Hardware Decoding''' refers to video decoding done using hardware accelerators on the mainline Linux kernel (i.e. what sits in Linus' tree). | |||
On most consumer-oriented SoCs, there is what is referred to as a VPU (Video Processing Unit). The VPU is responsible for power-efficient encoding and decoding of videos. Hardware-accelerated video decoding can be useful to get smoother video playback on your devices, lower power consumption, and lower CPU utilisation. Below is information regarding various hardware PINE64 uses and software that works with it. | |||
In 2018 Bootlin launched a crowdfunding campaign to bring a open source Allwinner VPU driver to mainline Linux, which came to be called Cedrus. The Cedrus media driver (For Allwinner SOCs such as A64) supported by mainline Linux supports H.264 and H.265 video decoding as of Linux 5.10, and with 5.11 came VP8 decoding support and a H.264 stateless video decoder interface. For more information refer to the [https://linux-sunxi.org/Sunxi-Cedrus#Codec_Support Sunxi wiki]. | == Hardware == | ||
Here's a table of the current support for different hardware. "In review" means the patch series to enable support has been posted to the mailing lists but is undergoing review, "linux-next" means it has been accepted and staged for the next Linux merge window. | |||
=== Supported Codecs By SoC === | |||
{| class="wikitable plainrowheaders" border="1" | |||
! scope="col" {{Diagonal split header|Codec|SoC}} | |||
! scope="col" style="width:6em" | A64 | |||
! scope="col" style="width:6em" | RK3328 | |||
! scope="col" style="width:6em" | RK3399 | |||
! scope="col" style="width:6em" | RK3566 | |||
! scope="col" style="width:6em" | RK3588 | |||
|- | |||
! scope="row" | JPEG | |||
| style="background:#F99; text-align:center;"| No | |||
| style="background:#F99; text-align:center;"| No | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
| style="background:#F99; text-align:center;"| No | |||
| style="background:#F99; text-align:center;"| No | |||
|- | |||
! scope="row" | MPEG-2 | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
|- | |||
! scope="row" | MPEG-4 Part 2/H.263 | |||
| style="background:#F99; text-align:center;"| No | |||
| style="background:#F99; text-align:center;"| No | |||
| style="background:#F99; text-align:center;"| No | |||
| style="background:#F99; text-align:center;"| No | |||
| style="background:#F99; text-align:center;"| No | |||
|- | |||
! scope="row" | VP8 | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
|- | |||
! scope="row" | H.264/AVC | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
| style="background:PaleGreen; text-align:center;"| Yes <sup style="font-style:italic;color:green">a</sup> | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
|- | |||
! scope="row" | H.265/HEVC | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
| style="background:LightYellow; text-align:center;"| In review | |||
| style="background:LightYellow; text-align:center;"| Patches Exist | |||
| style="background:#F99; text-align:center;"| No | |||
| style="background:#F99; text-align:center;"| No | |||
|- | |||
! scope="row" | VP9 | |||
| style="text-align:center;"| N/A | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
| style="background:#F99; text-align:center;"| No | |||
| style="background:#F99; text-align:center;"| No | |||
|- | |||
! scope="row" | AVS2 | |||
| style="text-align:center;"| N/A | |||
| style="text-align:center;"| N/A | |||
| style="text-align:center;"| N/A | |||
| style="text-align:center;"| N/A | |||
| style="background:#F99; text-align:center;"| No | |||
|- | |||
! scope="row" | AV1 | |||
| style="text-align:center;"| N/A | |||
| style="text-align:center;"| N/A | |||
| style="text-align:center;"| N/A | |||
| style="text-align:center;"| N/A | |||
| style="background:PaleGreen; text-align:center;"| Yes | |||
|} | |||
==== Notes ==== | |||
<ol style="list-style-type:lower-alpha"> | |||
<li>only Hantro, not rkvdec2, so with a maximum resolution of 1080p for now</li> | |||
</ol> | |||
=== Used SoC By Device === | |||
This table provides a quick reference to check which SoC is used in what Pine devices, for ease of navigating the previous table. | |||
{| class="wikitable plainrowheaders" border="1" | |||
! SoC | |||
! Device(s) | |||
|- | |||
! Allwinner A64 | |||
| PinePhone, PineTab1, Pinebook, PINE A64, PINE A64-LTS, SOPINE | |||
|- | |||
! Rockchip RK3328 | |||
| ROCK64 | |||
|- | |||
! Rockchip RK3399 | |||
| PinePhone Pro, Pinebook Pro, ROCKPro64 | |||
|- | |||
! Rockchip RK3566 | |||
| Quartz64 Model A and B, SOQuartz, PineTab2 | |||
|- | |||
! Rockchip RK3588 | |||
| QuartzPro64 | |||
|} | |||
=== Cedrus === | |||
In 2018, Bootlin launched a crowdfunding campaign to bring a open source Allwinner VPU driver to mainline Linux, which came to be called Cedrus. The Cedrus media driver (For Allwinner SOCs such as A64) supported by mainline Linux supports H.264 and H.265 video decoding as of Linux 5.10, and with 5.11 came VP8 decoding support and a H.264 stateless video decoder interface. For more information refer to the [https://linux-sunxi.org/Sunxi-Cedrus#Codec_Support Sunxi wiki]. | |||
=== Hantro === | |||
The Hantro media driver supports Rockchip and NXP SoCs including the RK3399 used in the Pinebook Pro and RockPro64. In November 2020 it [https://www.cnx-software.com/2020/11/24/hantro-h1-hardware-accelerated-video-encoding-support-in-mainline-linux/ was announced] that Bootlin was working on encoding support for the driver. | The Hantro media driver supports Rockchip and NXP SoCs including the RK3399 used in the Pinebook Pro and RockPro64. In November 2020 it [https://www.cnx-software.com/2020/11/24/hantro-h1-hardware-accelerated-video-encoding-support-in-mainline-linux/ was announced] that Bootlin was working on encoding support for the driver. | ||
==GStreamer== | === rkvdec === | ||
rkvdec is the video decoding hardware that's developed by Rockchip presumably in house. It's what Rockchip uses for decoding 4K H.264/AVC, VP9 and H.265/HEVC content. The driver in mainline linux for the first generation rkvdec (used in RK3328 and RK3399) supports VP9 and H.264, [https://patchwork.kernel.org/project/linux-rockchip/list/?series=659401 patches for HEVC support] exist but require modification and resubmission to mainline. | |||
RK3566, RK3568 and likely RK3588 use a second generation of rkvdec called rkvdec2. No mainline driver for this exists yet. The rkvdec instance on RK3588 additionally supports the [https://en.wikipedia.org/wiki/Audio_Video_Standard AVS2 video codec]. | |||
=== rkdjpeg === | |||
rkdjpeg is Rockchip's in-house hardware accelerated JPEG decoder. It can be found on the RK3566 and RK3568 as well as the RK3588. | |||
No mainline driver exists for it so far. | |||
== Software == | |||
=== GStreamer === | |||
H.264, H.265, MPEG-2, VP8, VP9 and AV1 video decoding is possible when using GStreamer's command line tools, or an application utilizing it such as [https://github.com/Rafostar/clapper Clapper] or [https://flathub.org/apps/details/org.sigxcpu.Livi µPlayer]. µPlayer includes a indicator of when hardware acceleration is properly working and in use. | |||
==== Using gst-launch ==== | |||
To test out hardware decoding from the command line, <code>gst-launch-1.0</code> may be used. An example pipeline to demux, hardware decode and display H.264 video in an MP4 container would be as follows: | |||
gst-launch-1.0 filesrc location=yourfilehere.mp4 ! qtdemux ! v4l2slh264dec ! videoconvert ! autovideosink | |||
To do the same for a VP8 WebM file, one may use: | |||
gst-launch-1.0 filesrc location=yourfilehere.webm ! matroskademux ! v4l2slvp8dec ! videoconvert ! autovideosink | |||
Depending on your GStreamer version and your GStreamer's build configuration, the following may be available as well: <code>v4l2slmpeg2dec</code>, <code>v4l2slav1dec</code>, <code>v4l2slvp9dec</code> and <code>v4l2slh265dec</code>. | |||
On platforms with a 2D accelerator (such as Rockchip's RGA) that is supported by v4l2, one may use <code>v4l2convert</code> instead of <code>videoconvert</code> to also hardware accelerate the pixel format conversion. | |||
=== FFmpeg === | |||
Mainline FFmpeg currently lacks the necessary patches to use the v4l2-requests based API, but [https://github.com/jernejsk/FFmpeg a fork that can utilise it exists]. Fedora users can install precompiled binaries from a [https://dl.kwizart.net/pub/ffmpeg-v4l2-request/ dedicated repository] by one of the Fedora maintainers. | |||
To build it, use something like | |||
git clone -b v4l2-request-n6.0 https://github.com/jernejsk/FFmpeg.git | |||
cd FFmpeg | |||
./configure --enable-v4l2-request --enable-libudev --enable-libdrm --enable-gnutls --prefix=/some/install/prefix/here | |||
make -j$(nproc) | |||
After compiling, you can install it into your specified prefix with | |||
make install | |||
And then modify your PATH to include the prefix's bin directory. | |||
With the patched ffmpeg, you can utilise hardware decoding using the <code>-hwaccel drm</code> parameter, e.g.: | |||
ffmpeg -hwaccel drm -i input.mp4 -f null - -benchmark | |||
to measure how fast it decodes. | |||
=== mpv === | |||
mpv v0.35 or later, built against the aforementioned FFmpeg fork, can be used to play back videos with hardware accelerated decoding. The hardware decoder path features interop with the GPU output, saving an expensive copyback operation. You can utilise the hardware decoding with e.g.: | |||
mpv --hwdec=drm ''video'' | |||
In mpv versions prior to 0.35, you can use the copyback hwdec with: | |||
mpv --hwdec=drm-copy ''video'' | |||
To build an mpv against the FFmpeg in your prefix, you can run | |||
PKG_CONFIG_PATH="/path/to/your/ffmpeg/prefix/lib/pkgconfig:$PKG_CONFIG_PATH" meson setup build | |||
ninja -C build | |||
=== HEVC on Linux for RK3399 === | |||
[https://libreelec.tv/ LibreELEC] maintains [https://github.com/LibreELEC/LibreELEC.tv/tree/master/projects/Rockchip/patches/linux/default patches] applicable to mainline Linux that, almong other things, add HEVC decode to <code>rkvdec</code>. Patches <code>linux-0011-v4l2-from-list.patch</code>, <code>linux-1001-v4l2-rockchip.patch</code>, <code>linux-2000-v4l2-wip-rkvdec-hevc.patch</code> and <code>linux-2001-v4l2-wip-iep-driver.patch</code> from commit [https://github.com/LibreELEC/LibreELEC.tv/tree/269fd4f127ee3acb9306e5ba82e7fb9ca15505cd/projects/Rockchip/patches/linux/default 269fd4f] can be applied to Linux 6.5.2 (tested) and are sufficient to use hardware HEVC decode on the RK3399. | |||
== | == More Resources == | ||
* [https://xnux.eu/devices/feature/cedrus-pp.html Megi's Cedrus Information] | |||
* [https://briandaniels.me/2021/06/27/hardware-accelerated-video-playback-on-the-pinephone.html HW accelerated GStreamer playback on the PinePhone] | |||
* [https://briandaniels.me/2021/07/06/hardware-accelerated-video-playback-on-the-pinephone-with-clapper.html HW accelerated Clapper video player on the PinePhone] | |||
== | == See Also == | ||
[ | * [[Mainline Hardware Encoding]] | ||
[ | [[Category:Quartz64]] [[Category:ROCKPro64]] [[Category:Rockchip RK3566]] [[Category:Rockchip RK3399]] [[Category:Pinebook Pro]] [[Category:PinePhone Pro]] [[Category:PineNote]] [[Category:Rockchip RK3588]] [[Category:Rockchip RK3328]] [[Category:ROCK64]] [[Category:Allwinner A64]] [[Category:PinePhone]] [[Category:PINE A64-LTS]] [[Category:SOPINE]] |
Latest revision as of 15:18, 11 December 2024
Mainline Hardware Decoding refers to video decoding done using hardware accelerators on the mainline Linux kernel (i.e. what sits in Linus' tree).
On most consumer-oriented SoCs, there is what is referred to as a VPU (Video Processing Unit). The VPU is responsible for power-efficient encoding and decoding of videos. Hardware-accelerated video decoding can be useful to get smoother video playback on your devices, lower power consumption, and lower CPU utilisation. Below is information regarding various hardware PINE64 uses and software that works with it.
Hardware
Here's a table of the current support for different hardware. "In review" means the patch series to enable support has been posted to the mailing lists but is undergoing review, "linux-next" means it has been accepted and staged for the next Linux merge window.
Supported Codecs By SoC
SoC Codec
|
A64 | RK3328 | RK3399 | RK3566 | RK3588 |
---|---|---|---|---|---|
JPEG | No | No | Yes | No | No |
MPEG-2 | Yes | Yes | Yes | Yes | Yes |
MPEG-4 Part 2/H.263 | No | No | No | No | No |
VP8 | Yes | Yes | Yes | Yes | Yes |
H.264/AVC | Yes | Yes | Yes | Yes a | Yes |
H.265/HEVC | Yes | In review | Patches Exist | No | No |
VP9 | N/A | Yes | Yes | No | No |
AVS2 | N/A | N/A | N/A | N/A | No |
AV1 | N/A | N/A | N/A | N/A | Yes |
Notes
- only Hantro, not rkvdec2, so with a maximum resolution of 1080p for now
Used SoC By Device
This table provides a quick reference to check which SoC is used in what Pine devices, for ease of navigating the previous table.
SoC | Device(s) |
---|---|
Allwinner A64 | PinePhone, PineTab1, Pinebook, PINE A64, PINE A64-LTS, SOPINE |
Rockchip RK3328 | ROCK64 |
Rockchip RK3399 | PinePhone Pro, Pinebook Pro, ROCKPro64 |
Rockchip RK3566 | Quartz64 Model A and B, SOQuartz, PineTab2 |
Rockchip RK3588 | QuartzPro64 |
Cedrus
In 2018, Bootlin launched a crowdfunding campaign to bring a open source Allwinner VPU driver to mainline Linux, which came to be called Cedrus. The Cedrus media driver (For Allwinner SOCs such as A64) supported by mainline Linux supports H.264 and H.265 video decoding as of Linux 5.10, and with 5.11 came VP8 decoding support and a H.264 stateless video decoder interface. For more information refer to the Sunxi wiki.
Hantro
The Hantro media driver supports Rockchip and NXP SoCs including the RK3399 used in the Pinebook Pro and RockPro64. In November 2020 it was announced that Bootlin was working on encoding support for the driver.
rkvdec
rkvdec is the video decoding hardware that's developed by Rockchip presumably in house. It's what Rockchip uses for decoding 4K H.264/AVC, VP9 and H.265/HEVC content. The driver in mainline linux for the first generation rkvdec (used in RK3328 and RK3399) supports VP9 and H.264, patches for HEVC support exist but require modification and resubmission to mainline.
RK3566, RK3568 and likely RK3588 use a second generation of rkvdec called rkvdec2. No mainline driver for this exists yet. The rkvdec instance on RK3588 additionally supports the AVS2 video codec.
rkdjpeg
rkdjpeg is Rockchip's in-house hardware accelerated JPEG decoder. It can be found on the RK3566 and RK3568 as well as the RK3588.
No mainline driver exists for it so far.
Software
GStreamer
H.264, H.265, MPEG-2, VP8, VP9 and AV1 video decoding is possible when using GStreamer's command line tools, or an application utilizing it such as Clapper or µPlayer. µPlayer includes a indicator of when hardware acceleration is properly working and in use.
Using gst-launch
To test out hardware decoding from the command line, gst-launch-1.0
may be used. An example pipeline to demux, hardware decode and display H.264 video in an MP4 container would be as follows:
gst-launch-1.0 filesrc location=yourfilehere.mp4 ! qtdemux ! v4l2slh264dec ! videoconvert ! autovideosink
To do the same for a VP8 WebM file, one may use:
gst-launch-1.0 filesrc location=yourfilehere.webm ! matroskademux ! v4l2slvp8dec ! videoconvert ! autovideosink
Depending on your GStreamer version and your GStreamer's build configuration, the following may be available as well: v4l2slmpeg2dec
, v4l2slav1dec
, v4l2slvp9dec
and v4l2slh265dec
.
On platforms with a 2D accelerator (such as Rockchip's RGA) that is supported by v4l2, one may use v4l2convert
instead of videoconvert
to also hardware accelerate the pixel format conversion.
FFmpeg
Mainline FFmpeg currently lacks the necessary patches to use the v4l2-requests based API, but a fork that can utilise it exists. Fedora users can install precompiled binaries from a dedicated repository by one of the Fedora maintainers.
To build it, use something like
git clone -b v4l2-request-n6.0 https://github.com/jernejsk/FFmpeg.git cd FFmpeg ./configure --enable-v4l2-request --enable-libudev --enable-libdrm --enable-gnutls --prefix=/some/install/prefix/here make -j$(nproc)
After compiling, you can install it into your specified prefix with
make install
And then modify your PATH to include the prefix's bin directory.
With the patched ffmpeg, you can utilise hardware decoding using the -hwaccel drm
parameter, e.g.:
ffmpeg -hwaccel drm -i input.mp4 -f null - -benchmark
to measure how fast it decodes.
mpv
mpv v0.35 or later, built against the aforementioned FFmpeg fork, can be used to play back videos with hardware accelerated decoding. The hardware decoder path features interop with the GPU output, saving an expensive copyback operation. You can utilise the hardware decoding with e.g.:
mpv --hwdec=drm video
In mpv versions prior to 0.35, you can use the copyback hwdec with:
mpv --hwdec=drm-copy video
To build an mpv against the FFmpeg in your prefix, you can run
PKG_CONFIG_PATH="/path/to/your/ffmpeg/prefix/lib/pkgconfig:$PKG_CONFIG_PATH" meson setup build ninja -C build
HEVC on Linux for RK3399
LibreELEC maintains patches applicable to mainline Linux that, almong other things, add HEVC decode to rkvdec
. Patches linux-0011-v4l2-from-list.patch
, linux-1001-v4l2-rockchip.patch
, linux-2000-v4l2-wip-rkvdec-hevc.patch
and linux-2001-v4l2-wip-iep-driver.patch
from commit 269fd4f can be applied to Linux 6.5.2 (tested) and are sufficient to use hardware HEVC decode on the RK3399.
More Resources
- Megi's Cedrus Information
- HW accelerated GStreamer playback on the PinePhone
- HW accelerated Clapper video player on the PinePhone