libopenh264enc: Return a more sensible error code in some init failure paths