Also skip direct/mvd_cache init for skiped blocks.