avcodec/rangecoder: Test for invalid corner case