vlm: Fix destruction by using correctly using refcounting.