avformat/nutenc: Fix error handling in write_sm_data()