Return AVERROR(ENOMEM) on memory allocation failure of avcodec_open.