avcodec/magicyuv: check dimensions