Make avfilter_ref_pic also handle removal of permissions for the caller.