mxfenc: switch to av_reallocp_array() and check allocation errors