h264: split reading the ref list modifications and actually building the ref list