This paper proposes a systematic scheme of removing void elements to achieve fast and efficient level set based topology optimization. When performing optimization, unless special treatment is applied to the stiffness matrix, the density of these void elements are usually represented numerically by a small positive value. In level set based topology optimization, since the amount of computational resources required for FEM dwarfs those required for level set evolution, the removal of these elements from the global stiffness matrix can drastically reduce total computation time. The proposed scheme removes the void elements, determined by their nodes' level set values, from the optimization process by use of mapping procedures. The results presented here show time reductions of at least 70%. An additional advantage of the presented scheme is that it can be easily used with any black box FEM routine.