rtmp: Return a proper error code instead of -1