To better preserve the global semantic information and edge information of the original image during the multi-focus image fusion process, we propose an encoder–decoder network, which combines pixel loss function, multiscale structural similarity loss function and total variation loss function to further reduce the detail loss in the image reconstruction process. The model introduces dynamic convolution and global context network to improve its feature expression ability and global context modeling ability. In the fusion stage, the fusion strategy based on the edge feature map is used to fuse the two feature maps output by the encoder to obtain the final decision map. Combining the gradient information of the deep feature map and the edge feature map, the fusion strategy can more effectively extract the flat parts of the focused and defocused regions and enhance the edge features of the image. Finally, the proposed algorithm is compared with seven advanced fusion algorithms, and the results are superior to other fusion algorithms in subjective and objective evaluation indexes.