Summary: As for the large differences between the visual and infrared images in gray value caused by different imaging mechanism, inconsistent contour, the low matching probability of traditional matching methods based on gray or feature, the gray information of visual and infrared images was introduced after researching a variety of Hausdorfff Distance (HD) algorithms. Image matching method based on the neighbor grayscale information Hausdorfff distance was proposed. Based on the calculation of the similarity of edge feature points, the calculation of image normalized grayscale variance was added into this method, which effectively solved the low probability problem caused by different edge of visual/infrared image in Hausdorff distance matching algorithms. The simulation results of visual and infrared images matching show that under various conditions, compared with the conventional Hausdorff distance method, the proposed algorithm effectively improves matching effect under different light conditions and anti-jamming of noise.