rtmp: Move RTMP digest calculation to a separate file