rtmpdh: Check the output buffer size in the openssl version of dh_compute_key