avcodec/pictordec: Check plane value before doing value/mask computations