postproc/postprocess: Check for av_malloc() failure