Split fill_caches() between filter and decoder.