Defect detection for specular surfaces plays a vital role in precision manufacturing. However, traditional defect detection methods are unsuitable for specular surfaces because of their specular reflection property. The defect detection on specular surfaces is usually performed by inspectors, which makes the defect detection a time-consuming and unstable task. Deflectometry has been widely used in defect detection for specular surfaces combined with machine learning. Nevertheless, conventional deflectometry methods use the local curvature deviation map based on the unwrapped phase, which can only detect geometrical defects. Moreover, hand-crafted features need to be defined for each specific task. We present a method based on deflectometry and deep learning. Deflectometry provides the input images for the network, and the deep learning network completes the identification and location of defects. In deflectometry, the proposed method uses the light intensity contrast map to replace the local curvature map, which can detect both geometrical and textural defects. Based on conventional networks, depthwise separable convolution kernel is applied to reduce parameters, and residual convolution block is utilized to alleviate vanishing or exploding gradients. A subnet for feature aggregation is used to obtain multiscale information of defect features. Performance evaluation based on experiment results proved the effectiveness of the proposed method.