Seconded, depending on what your goals are with transcoding, you might want to reconsider your strategy.

Hardware encoding (with a GPU) is mostly useful for realtime transcoding applications like streaming video. There are definitely some caviats that come with the realtime performance, and you’ll find that NVENC encoded video is almost always inferior to the slower equivalent software encoded variants.

So let’s talk codecs: While h.265 might seem like the holy grail, it is way more computationally intensive than h.264 is. In some cases the difference in encoding time will even be as high as 3-5x. Not really worth it if all you’re gaining is a slightly lower filesize.

Your results will vary by the media you’re encoding, by your encoder quality settings, tuning and encoding speed. As a rule of thumb: slower encoding speeds equal more efficiently compressed video (a.k.a. relatively higher quality for lower file size).

Handbrake is my choice of software for encoding video. It includes pretty much everything you could ever want if you’re not looking for niche codecs and exotic video formats.

I find myself mostly using x264 because it is relatively fast and still provides awesome results. My encoding speed is always set to “slow” or “superslow” (not much difference for my setup). I usually set the quality by making use of the preview function in handbrake, which transcodes just a short section of the video which I use for pixel peeping and checking for any major artifacts that would ruin the content. The resulting file also provides an estimate for how large the final transcoded file will be. If you’re happy with the quality setting, you can opt to mess with the encoder tuning. There are different presets for film, animated content and such. I usually do use film tuning if transcoding live-action media.

All this generally leaves me with pretty compact file sizes for 1080p media. And transcoding usually happens at a rate of 60-75 fps depending on the resolution. Going up from “slow” to “medium” improves fps by about 25% and increases file size by about 10%. The ideal balance is up to you.

Advanced tips: try using VMAF (objective video quality analysis algorithm developed by Netflix) to score and compare your different encoding settings. VMAF is neatly integrated into FFMetrics, which is a GUI for FFMpeg and a couple of video analysis algorithms. I also use MPV (open source media player) with FFMpeg command line arguments for playing videos synchronized in a 2x1 or 2x2 matrix. This helps compare the results for quality.