Simplify constrained_intra_pred code in fill_caches().