It's possibly a container problem, copying it to a new container format with
avconv -i input.ts -c copy output.mp4
may well fix your problem. This will be 100% lossless. If that doesn't work, a crf of 18 is normally considered 'visually lossless'; you can set this is HandBrake (under the 'video' tab), or with avconv:
avconv -i input.ts -c:a copy -c:v libx264 -crf 18 -preset veryfast output.mp4
The presets are: ultrafast, superfast, veryfast, faster, fast, medium, slower, slow, veryslow. The slower the preset, the smaller the file (but with an increase in encode time). In my personal experimentation, I've found the biggest dropoff in terms of filesize is between superfast and veryfast, after that it seems to be much more incremental.
NOTE: If you want, ffmpeg can do everything avconv can do, with identical syntax (simply replace all instances of avconv with ffmpeg).
FOOTNOTE: You may find this ffmpeg x264 encoding guide useful for more information.
Note that for this, you should always use the latest ffmpeg version, and preferably compile it yourself. This gives you access to the most recent libx265 and libfdk-aac for audio encoding.
Also, the data rate savings will be quite drastic if you're going from a ~10 MBit/s DVD to around 1–2 MBit/s for H.264 video and 0.5–1 MBit/s for H.265 video. Changing the quality in the below steps may influence the bitrates, but still the data reduction should be significant.
H.264
For the quality / rate control, you want to use CRF mode in libx264 rather than a constant bitrate. Using CRF ensures that an average quality is preserved, independent of the original video resolution or its complexity. Constant bitrate is only really useful if you're constrained by the transmission medium (e.g. hard drive speed, Internet throughput).
Choosing the CRF value is the tricky part. It requires you to look at the output. The default for libx264 (23) offers a quite good tradeoff between size and quality. But given that your original source is already compressed (and not with a very good quality compared to Blu-rays), you may want to change the CRF to be a little lower, such as 20. This will increase needed bitrate by about a third.
Choose the preset according to how long you want to wait. slow
seems like a good value here.
ffmpeg -i input \
-c:v libx264 -crf 20 -pix_fmt yuv420p \
-x264-params keyint=240:min-keyint=20 \
-preset:v slow -profile:v baseline -level 3.0 \
-c:a libfdk_aac -vbr 4 \
output.mp4
The built-in ffmpeg AAC encoder can be used if libfdk-aac is not available. Use -c:a aac -strict experimental -b:a 128k
instead of -c:a libfdk_aac -vbr 4
.
H.265
Research suggests that using HEVC will lead to up to 74% bitrate saving compared to H.264. This is based on subjective viewing data of Ultra-HD sequences. Of course, it depends on the temporal complexity of the source content, and the amount of data saved will not be as high for hard to code sequences. Either way you can safely say that 50% data reduction is absolutely possible.
The default CRF for libx265 is 28. Using the same source content, it results in about half the bitrate compared to libx264 at CRF 23. This is irrespective of the actual bitrate, i.e., if the H.264 version takes 1.5 MBit/s, then H.265 will use around 750 kBit/s, but it's 750 kBit/s vs. 350 kBit/s for another sequence. I ran it on a couple of sequences at DVD-PAL resolution and was not able to tell the difference in terms of quality.
ffmpeg -i input \
-c:v libx265 -pix_fmt yuv420p \
-x265-params crf=28:keyint=240:min-keyint=20 \
-preset:v slow \
-c:a libfdk_aac -vbr 4 \
output.mp4
For more information, here are the relevant resources:
Best Answer
There is no best settings because quality is subjective, but the usual guidelines of libx265 (assuming you are using this encoder) is similar to libx264:
-crf
value that gives you an acceptable quality. Default value is 28.-preset
you have patience for. Default value is medium.Note that you will be re-encoding the artifacts present in your H.264 inputs which will harm compression efficiency.
I generally wouldn't bother re-encoding H.264 to H.265.
See FFmpeg Wiki: H.265 for some basic info.