Slow nvenc encodes affect h264 and hevc with certain videos

[nikkoaki]

Summary of the bug:
Been using nvenc for encoding for some time and never had problems. Usually always got 100+ fps converting a 4K video to a lower bit rate 4K one.
Lately i tried the same on a video shot with my latest phone and noticed that i barely get 30fps. First i was trying with handbrake but tried just ffmpeg to rule out the other project and it behaves the same.
I also shot with a lower bit rate to match my old phone but again, it behaves the same.

ffmpeg output from a video from my old phone:

Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2021-05-12T11:49:14.000000Z
    com.android.version: 8.1.0
  Duration: 00:01:13.56, start: 0.000000, bitrate: 35577 kb/s
  Stream #0:0[0x1](eng): Video: hevc (Main) (hvc1 / 0x31637668), yuvj420p(pc, smpte170m), 3840x2160, 35408 kb/s, SAR 1:1 DAR 16:9, 29.97 fps, 30 tbr, 90k tbn (default)
    Metadata:
      creation_time   : 2021-05-12T11:49:14.000000Z
      handler_name    : VideoHandle
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 189 kb/s (default)
    Metadata:
      creation_time   : 2021-05-12T11:49:14.000000Z
      handler_name    : SoundHandle
      vendor_id       : [0][0][0][0]
File 'output.mp4' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> hevc (hevc_nvenc))
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[swscaler @ 0000024f69f37ac0] deprecated pixel format used, make sure you did set range correctly
    Last message repeated 2 times
[swscaler @ 0000024f69f35a80] deprecated pixel format used, make sure you did set range correctly
																								 
Output #0, mp4, to 'output.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    com.android.version: 8.1.0

    encoder         : Lavf60.16.100
  Stream #0:0(eng): Video: hevc (Main) (hev1 / 0x31766568), yuv420p(tv, smpte170m, progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 2000 kb/s, 30 fps, 15360 tbn (default)
    Metadata:
      creation_time   : 2021-05-12T11:49:14.000000Z
      handler_name    : VideoHandle
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.31.102 hevc_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: N/A
  Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      creation_time   : 2021-05-12T11:49:14.000000Z
      handler_name    : SoundHandle
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.31.102 aac
[out#0/mp4 @ 0000024f4f217700] video:22061kB audio:1164kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.343376%
frame= 2200 fps=108 q=47.0 Lsize=   23305kB time=00:01:13.53 bitrate=2596.1kbits/s dup=2 drop=0 speed=3.62x
[aac @ 0000024f4f735c00] Qavg: 576.413

Output from my new phone:

Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2024-03-09T16:47:14.000000Z
    com.android.version: 14
  Duration: 00:00:20.59, start: 0.000000, bitrate: 60203 kb/s
  Stream #0:0[0x1](eng): Video: hevc (Main) (hvc1 / 0x31637668), yuvj420p(pc, bt470bg/bt470bg/smpte170m), 3840x2160, 59950 kb/s, 30.11 fps, 30 tbr, 90k tbn (default)
    Metadata:
      creation_time   : 2024-03-09T16:47:14.000000Z
      handler_name    : VideoHandle
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 96 kb/s (default)
    Metadata:
      creation_time   : 2024-03-09T16:47:14.000000Z
      handler_name    : SoundHandle
      vendor_id       : [0][0][0][0]
File 'output.mp4' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> hevc (hevc_nvenc))
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[swscaler @ 00000244b95befc0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 00000244b9d28040] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 00000244b9b9e040] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 00000244b9630e40] deprecated pixel format used, make sure you did set range correctly
Output #0, mp4, to 'output.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    com.android.version: 14

    encoder         : Lavf60.16.100
  Stream #0:0(eng): Video: hevc (Main) (hev1 / 0x31766568), yuv420p(tv, bt470bg/bt470bg/smpte170m, progressive), 3840x2160, q=2-31, 2000 kb/s, 30 fps, 15360 tbn (default)
    Metadata:
      creation_time   : 2024-03-09T16:47:14.000000Z
      handler_name    : VideoHandle
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.31.102 hevc_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: N/A
  Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      creation_time   : 2024-03-09T16:47:14.000000Z
      handler_name    : SoundHandle
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.31.102 aac
[out#0/mp4 @ 000002449b7a6d80] video:5701kB audio:320kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.392885%
frame=  619 fps= 33 q=39.0 Lsize=    6045kB time=00:00:20.50 bitrate=2415.6kbits/s dup=0 drop=1 speed=1.09x
[aac @ 000002449bc74000] Qavg: 2341.940

How to reproduce:

ffmpeg -i "new video.mp4" -vcodec hevc_nvenc -preset medium -level 5 -qmin 10 -qmax 52 "output.mp4"
ffmpeg version 6.1.1-full_build-www.gyan.dev

[nikkoaki]

Just uploaded a sample file to streams.videolan.org/upload/.
Name is "nothing phone sample 2.mp4"