avcodec/vmnc: Check that rectangles are within the picture