RTMP: fix a sign warning