Install Jellyfin with hardware transcoding for AMD GPU in Proxmox LXC container

This article is mainly for AMD GPUs. I only tried it for integrated Radeon graphics of AMD Ryzen 7 5825U on AOOSTAR WTR PRO. It may not work on your setup.

For Intel iGPU you might want to check WunderTech's tutorial. Also if you are missing steps - his tutorial is a bit more thorough. I skipped jellyfin setup screens because it is intuitive.

1. Download the container template

To to storage `local` → CT Templates and click “Templates” button. Search for debian and download debian-12-standard template.

2. Create container

Click Create CT on top-right.

General tab - Make sure unprivileged is unchecked. type hostname (ex. jellyfin), type your password and optionally upload ssh public key.

Template - Choose debian-12-standard template.

Disks - choose size. I set mine to 20GiB

Cores - Set cores based on what you have and what you want to use with jellyfin. you can start with 2 cores and hope hardware transcoding does it's job. you can change that later.

Memory - 2048MiB and 512MiB swap

Network - you can choose static and type the ip, or if you have configurable DHCP server you can later set the ip to fixed based on mac address.

After creating go to options and check:

Features:

• NFS - if you want to mount your media via NFS
• SMB/CIFS - if you want to mount your media via SMB (windows shares)

Start on boot - if you want automatic start of jellyfin

First make sure that the devices exist. Click on proxmox node → shell (not the container console) and type:

root@proxmox:~# ls -la /dev/dri
total 0
drwxr-xr-x  3 root root        100 Nov 30 01:16 .
drwxr-xr-x 19 root root       4960 Nov 30 12:27 ..
drwxr-xr-x  2 root root         80 Nov 30 01:16 by-path
crw-rw----  1 root video  226,   0 Nov 30 01:16 card0
crw-rw----  1 root render 226, 128 Nov 30 01:16 renderD128

If you don't have card0 and renderD128 HW transcoding is not going to work. Remember these two (your nubmers at the end could be different).

If you have these - go to your jellyfin container → resources and click on Add → Device Passthrough

Device path: /dev/dri/card0
UID in CT: 0
GID in CT: 44
Access Mode in CT: 0660

Do that again for the second device:

Device path: /dev/dri/renderD128
UID in CT: 0
GID in CT: 104
Access Mode in CT: 0660

configure timezone:

dpkg-reconfigure tzdata

update packages:

apt update
apt upgrade

install vainfo, drivers and other packages:

apt install cifs-utils nfs-common net-tools vainfo mesa-va-drivers -y

confirm drivers are recognized(the output may be a bit different):

root@jf2:~# vainfo
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
error: can't connect to X server!
libva info: VA-API version 1.17.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_17
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.17 (libva 2.12.0)
vainfo: Driver version: Mesa Gallium driver 22.3.6 for AMD Radeon Graphics (renoir, LLVM 15.0.6, DRM 3.57, 6.8.12-4-pve)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSlice
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc

Copy this for later reference.

• VAEntrypointVLD means that your video card can decode this codec
• VAEntrypointEncSlice means it can encode this codec

You can later set these in jellyfin → admin → Playback → transcoding.

Add nfs shares to fstab:

10.0.0.1:/storage/media /media nfs defaults,nofail 0 0

systemctl daemon-reload
mount /media

open

http://<LXC_IP>:8096

and follow the steps. Make sure to check “Allow remote connections” if you are going to use reverse proxy to access jellyfin from internet.

After jellyfin initial setup is done go to Administration → dashboard → Playback → transcoding and select:

• Hardware Acceleration: “Video Acceleration API (VAAPI)”:
• VA-API Device: /dev/dri/renderD128

Select hardware decoding for the codecs that you listed before with vainfo.

Select hardware encoding. If you have integrated AMD graphics you probably don't want to encode HEVC and AV1 formats so keep those unchecked. That's in 2024 - it's possible that later AMD integrated graphics get better in encoding so you can try it.

Remember to turn on options one by one and test encoding. At some point something will break and you want to know what is the last setting that you turned on. Also after you first install jellyfin you can make a snapshot to make sure you can get back to working settings after you mess with the options.