avcodec/xsubdec: Check that RLE coded image and colors fit in the buffer