Extract ignore logic from import visitor

Extract ignore logic from import visitor #1478