avpacket: use ERANGE instead of EOVERFLOW