rtmpdh: Initialize gcrypt before using it
authorRicardo Constantino <wiiaboo@gmail.com>
Tue, 29 Dec 2015 21:40:14 +0000 (21:40 +0000)
committerMichael Niedermayer <michael@niedermayer.cc>
Mon, 11 Jan 2016 23:56:02 +0000 (00:56 +0100)
Either disabling or init'ing secure memory is required after the use
of gcry_check_version. From a look at the functions rtmpdh uses, I
noticed none require the use of secure memory, so we disable it [1][2].

This resolves some errors returned by rtmpdh code with uninitialized
gcrypt, especifically:
Fatal: failed to create the RNG lock: Invalid argument
FATAL: failed to acquire the FSM lock in libgrypt: Invalid argument

Version "1.5.4" was arbitrarily chosen. An older version probably works
as well, but I couldn't compile older versions to test on my machine.

[1]
https://gnupg.org/documentation/manuals/gcrypt/Initializing-the-library.html
[2]
https://www.gnupg.org/documentation/manuals/gcrypt/Controlling-the-library.html

Signed-off-by: Ricardo Constantino <wiiaboo@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
libavformat/rtmpdh.c

index 91b1349..42ad72c 100644 (file)
         mpz_fdiv_r_2exp(bn, bn, num_bits);            \
     } while (0)
 #elif CONFIG_GCRYPT
-#define bn_new(bn)                  bn = gcry_mpi_new(1)
+#define bn_new(bn)                                              \
+    do {                                                        \
+        if (!gcry_control(GCRYCTL_INITIALIZATION_FINISHED_P)) { \
+            if (!gcry_check_version("1.5.4"))                   \
+                return AVERROR(EINVAL);                         \
+            gcry_control(GCRYCTL_DISABLE_SECMEM, 0);            \
+            gcry_control(GCRYCTL_INITIALIZATION_FINISHED, 0);   \
+        }                                                       \
+        bn = gcry_mpi_new(1);                                   \
+    } while (0)
 #define bn_free(bn)                 gcry_mpi_release(bn)
 #define bn_set_word(bn, w)          gcry_mpi_set_ui(bn, w)
 #define bn_cmp(a, b)                gcry_mpi_cmp(a, b)