lavf/nutenc: provide meaningful error message and error code in case of invalid pts