rtmp: fix multiple broken overflow checks