vc1dec: fix invalid memory access for small video dimensions
authorJohn Brooks <john.brooks@bluecherry.net>
Sun, 11 Dec 2011 09:37:21 +0000 (02:37 -0700)
committerKostya Shishkov <kostya.shishkov@gmail.com>
Wed, 4 Jan 2012 09:35:26 +0000 (10:35 +0100)
commitd209c27b09234cc40bbdbd680aa502b493edf595
treec7940a66137507683f77d2e954cef0ae4b67a908
parentaacf6b3a2fd8bc8603e3deaa6e612ea03cf08707
vc1dec: fix invalid memory access for small video dimensions

For small video dimensions, these calculations of the upper bound
for pixel access may have a negative result. Using an unsigned
comparison to bound a potentially negative value only works if
the greater operand is non-negative. Fixed by doing edge emulation
when the upper bound is probably negative, everywhere that this
pattern appears.

Signed-off-by: Kostya Shishkov <kostya.shishkov@gmail.com>
libavcodec/vc1dec.c