rtmp: Return proper error codes in handle_chunk_size