avcodec/dnxhdenc: check that qmax is within the supported range