Video scaling in ffmpeg causes audio gaps

audioffmpegvideo

I am trying to scale down a video (mp4 format) by using the following process :

  1. Segment the video
ffmpeg -v quiet -i <input-file> -map 0 -c copy -f segment -segment_time <segment_length> -reset_timestamps 1 -avoid_negative_ts make_non_negative output_%03d.mp4
  1. Scale down each segment parallelly
ls output_*.mp4 |  parallel ffmpeg -v quiet -i {} -vf scale=-2:<output-resolution> temp_{#}.mp4
  1. Join segments to get back whole video
for i in temp_*.mp4; do echo file "$i"; done > concat.txt
ffmpeg -v quiet -f concat -i concat.txt -codec copy <output-file>

However, the final output contains small audio gaps. Consider segment length of 15s, then at roughly 15.28s there is a gap in the audio stream as shown in the waveform of audio below.

Audio waveform

The green part highlights the waveform of the audio generated by scaling down the video using this command :

ffmpeg -v quiet -i <input-file> -vf scale=-2:<output-resolution> <output-file>

The yellow part highlights the waveform of the audio scaled down using the segmenting process described earlier.

One approach I could think of is to work on the video stream only while segmenting and copy the audio stream from the input file directly to the output file. But again it might lead to audio sync issues. How do I solve this ?

I am using ffmpeg version git-2020-03-03-60b1f85 in Windows 10.

Please note that I don't want to re-encode the video because this will defeat the purpose of segmenting in the first place (in terms of time taken).

Best Answer

It is not the video scaling, that produces the gaps, but it is the process of splitting and joining while killing the timestamps.

This is quite a usual problem in the broadcast industry, and we typically solve it this way:

  • split audio and video, in the process using the hls segmenter with a segment list file
  • do parallel video processing of the segemnts without changes to the timestamps and audio processing i/a
  • rejoin audio and video by pointing ffmpeg to the segment list file, not a concat file

EXAMPLES

Split:

ffmpeg -y -i ... -dn -sn -an -c:v copy -bsf:v h264_mp4toannexb -copyts -start_at_zero -f segment -segment_list /path/to/output.m3u8 -segment_time 30 /path/to/folder/para%05d.ts -dn -sn -vn -c:a copy /path/to/audiofile

Rejoin:

ffmpeg -y -i /path/to/output.m3u8 -i /path/to/audiofile -map '0:v' -map '1:a' -c:a copy -c:v copy /path/to/output.mp4
Related Question