Make sure error resilience does not try to use unavailable reference frames.