close
Skip to main content
Springer Nature Link
Log in

Mismatched: Evaluating the Limits of Image Matching Approaches and Benchmarks

  • Conference paper
  • First Online:
Save conference paper
View saved research
Computer Vision – ECCV 2024 Workshops (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15645))

Included in the following conference series:

  • 922 Accesses

  • 3 Citations

Abstract

Three-dimensional (3D) reconstruction from two-dimensional images is an active research field in computer vision, with applications ranging from navigation and object tracking to segmentation and three-dimensional modeling. Traditionally, parametric techniques have been employed for this task. However, recent advancements have seen a shift towards learning-based methods. Given the rapid pace of research and the frequent introduction of new image matching methods, it is essential to evaluate them. In this paper, we present a comprehensive evaluation of various image matching methods using a structure-from-motion pipeline. We assess the performance of these methods on both in-domain and out-of-domain datasets, identifying key limitations in both the methods and benchmarks. We also investigate the impact of edge detection as a pre-processing step. Our analysis reveals that image matching for 3D reconstruction remains an open challenge, necessitating careful selection and tuning of models for specific scenarios, while also highlighting mismatches in how metrics currently represent method performance. Code is available at https://github.com/surgical-vision/colmap-match-converter.

S. Bonilla, C. Di Vece, R. Daher and X. Ju—These authors contributed equally to this work and are co-first authors.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Free shipping worldwide - view details

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

Notes

  1. 1.

    Map-free Relocalization Challenge Evaluation Leaderboard.

  2. 2.

    https://github.com/gmberton/EarthMatch.

  3. 3.

    IMC23 Competition Page.

References

  1. Antoun, M., Asmar, D., Daher, R.: Towards richer 3D reference maps in urban scenes. In: 2020 17th Conference on Computer and Robot Vision (CRV), pp. 39–45. IEEE Computer Society (2020). https://doi.org/10.1109/CRV50864.2020.00014

  2. Arnold, E., et al.: Map-free visual relocalization: metric pose relative to a single image. In: European Conference on Computer Vision, pp. 690–708. Springer (2022)

    Google Scholar 

  3. Bagnolo, V., Argiolas, R., Cuccu, A.: Hbim for archaeological sites: from SFM based survey to algorithmic modeling. Int. Arch. Photogrammetry Remote Sens. Spat. Inf. Sci. 42, 57–63 (2019). https://doi.org/10.5194/isprs-archives-XLII-2-W9-57-2019

  4. Barroso-Laguna, A., Munukutla, S., Prisacariu, V.A., Brachmann, E.: Matching 2D images in 3D: metric relative pose from metric correspondences. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4852–4863 (2024). https://doi.org/10.48550/arXiv.2404.06337

  5. Bay, H., Tuytelaars, T., Van Gool, L.: SURF: speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023_32

  6. Bellavia, F., et al.: Image matching challenge 2024 - hexathlon (2024). https://kaggle.com/competitions/image-matching-challenge-2024

  7. Berton, G., Goletto, G., Trivigno, G., Stoken, A., Caputo, B., Masone, C.: Earthmatch: iterative coregistration for fine-grained localization of astronaut photography. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4264–4274 (2024)

    Google Scholar 

  8. Bökman, G., Edstedt, J., Felsberg, M., Kahl, F.: Steerers: a framework for rotation equivariant keypoint descriptors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4885–4895 (2024)

    Google Scholar 

  9. Borhani, N., Bower, A.J., Boppart, S.A., Psaltis, D.: Digital staining through the application of deep neural networks to multi-modal multi-photon microscopy. Biomed. Optics Express 10(3), 1339–1350 (2019). https://doi.org/10.1364/BOE.10.001339

  10. Calonder, M., Lepetit, V., Strecha, C., Fua, P.: BRIEF: binary robust independent elementary features. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 778–792. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15561-1_56

  11. DeTone, D., Malisiewicz, T., Rabinovich, A.: Superpoint: self-supervised interest point detection and description. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 224–236 (2018). https://doi.org/10.1109/CVPRW.2018.00060

  12. Edstedt, J., Bökman, G., Wadenbäck, M., Felsberg, M.: Dedode: detect, don’t describe - describe, don’t detect for local feature matching. In: 2024 International Conference on 3D Vision (3DV), pp. 148–157 (2023). https://doi.org/10.1109/3DV62453.2024.00035

  13. Edstedt, J., Sun, Q., Bökman, G., Wadenbäck, M., Felsberg, M.: Roma: robust dense feature matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19790–19800 (2024)

    Google Scholar 

  14. Eppstein, D., Erickson, J.: Raising roofs, crashing cycles, and playing pool: applications of a data structure for finding pairwise interactions. Discrete Comput. Geom. 22, 569–592 (1998). https://doi.org/10.1145/276884.276891

  15. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981). https://doi.org/10.1145/358669.358692

    Article  MathSciNet  Google Scholar 

  16. Furukawa, Y., Ponce, J.: Accurate, dense, and robust multiview stereopsis. IEEE Trans. Pattern Anal. Mach. Intell. 32, 1362–1376 (2010). https://doi.org/10.1109/TPAMI.2009.161

  17. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the kitti vision benchmark suite. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361. IEEE (2012). https://doi.org/10.1109/CVPR.2012.6248074

  18. Geiger, A., Ziegler, J., Stiller, C.: Stereoscan: Dense 3D reconstruction in real-time. In: 2011 IEEE Intelligent Vehicles Symposium (IV), pp. 963–968 (2011). https://doi.org/10.1109/IVS.2011.5940405

  19. Häne, C., Zach, C., Lim, J., Ranganathan, A., Pollefeys, M.: Stereo depth map fusion for robot navigation. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1618–1625. IEEE (2011). https://doi.org/10.1109/IROS.2011.6094704

  20. Iheaturu, C., Okolie, C., Ayodele, E., Egogo-Stanley, A., Musa, S., Speranza, C.I.: A simplified structure-from-motion photogrammetry approach for urban development analysis. Remote Sens. Appl. Soc. Environ. 28 (2022). https://doi.org/10.1016/j.rsase.2022.100850

  21. Izquierdo, S., Civera, J.: Optimal transport aggregation for visual place recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17658–17668 (2024)

    Google Scholar 

  22. Jiang, H., Karpur, A., Cao, B., Huang, Q., Araujo, A.: Omniglue: generalizable feature matching with foundation model guidance. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19865–19875 (2024). https://doi.org/10.48550/arXiv.2405.12979

  23. Jin, Y., et al.: Image matching across wide baselines: From paper to practice. Int. J. Comput. Vision 129(2), 517–547 (2021). https://doi.org/10.1007/s11263-020-01385-0

    Article  Google Scholar 

  24. Juan, J.: Programme de classification hiérarchique par l’algorithme de la recherche en chaîne des voisins réciproques. Les cahiers de l’analyse des données 7(2), 219–225 (1982)

    Google Scholar 

  25. Khan, A., Mineo, C., Dobie, G., Macleod, C., Pierce, G.: Vision guided robotic inspection for parts in manufacturing and remanufacturing industry. J. Remanufacturing 11(1), 49–70 (2021). https://doi.org/10.1007/s13243-020-00091-x

    Article  Google Scholar 

  26. Kim, S., Pollefeys, M., Barath, D.: Learning to make keypoints sub-pixel accurate. arXiv:2407.11668 (2024)

  27. Leroy, V., Cabon, Y., Revaud, J.: Grounding image matching in 3D with mast3r. arXiv preprint arXiv:2406.09756 (2024)

  28. Lindenberger, P., Sarlin, P.E., Pollefeys, M.: Lightglue: local feature matching at light speed. In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 17581–17592 (2023). https://doi.org/10.1109/ICCV51070.2023.01616

  29. Lowe, D.G.: Object recognition from local scale-invariant features. In: Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 2, pp. 1150–1157. IEEE (1999). https://doi.org/10.1109/ICCV.1999.790410

  30. Oquab, M., et al.: DINOv2: learning robust visual features without supervision. Trans. Mach. Learn. Res. (2024). https://doi.org/10.48550/arXiv.2304.07193

  31. Potje, G., Cadar, F., Araujo, A., Martins, R., Nascimento, E.R.: Xfeat: accelerated features for lightweight image matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2682–2691 (2024)

    Google Scholar 

  32. Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12179–12188 (2021). https://doi.org/10.1109/ICCV48922.2021.01196

  33. Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 430–443. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023_34

  34. Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: Orb: an efficient alternative to sift or surf. In: 2011 International Conference on Computer Vision, pp. 2564–2571. IEEE (2011). https://doi.org/10.1109/ICCV.2011.6126544

  35. Sarlin, P.E., DeTone, D., Malisiewicz, T., Rabinovich, A.: Superglue: learning feature matching with graph neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4938–4947 (2020). https://doi.org/10.1109/cvpr42600.2020.00499

  36. Sattler, T., et al.: Benchmarking 6dof outdoor visual localization in changing conditions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8601–8610 (2018). https://doi.org/10.1109/CVPR.2018.00897

  37. Schönberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016). https://doi.org/10.1109/CVPR.2016.445

  38. Schönberger, J.L., Zheng, E., Pollefeys, M., Frahm, J.M.: Pixelwise view selection for unstructured multi-view stereo. In: European Conference on Computer Vision (ECCV) (2016). https://doi.org/10.1007/978-3-319-46487-9_31

  39. Shen, X., et al.: Gim: learning generalizable image matcher from internet videos. arXiv preprint arXiv:2402.11095 (2024)

  40. Soria, X., Riba, E., Sappa, A.: Dense extreme inception network: Towards a robust CNN model for edge detection. In: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1912–1921 (2020). https://doi.org/10.1109/WACV45572.2020.9093290

  41. Soria, X., Sappa, A., Humanante, P., Akbarinia, A.: Dense extreme inception network for edge detection. Pattern Recogn. 139, 109461 (2023). https://doi.org/10.1016/j.patcog.2023.109461

  42. Sun, J., Shen, Z., Wang, Y., Bao, H., Zhou, X.: Loftr: detector-free local feature matching with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8922–8931 (2021). https://doi.org/10.1109/CVPR46437.2021.00881

  43. Tyszkiewicz, M., Fua, P., Trulls, E.: Disk: learning local features with policy gradient. In: Advances in Neural Information Processing Systems, vol. 33, pp. 14254–14265 (2020)

    Google Scholar 

  44. Ullman, S.: The interpretation of structure from motion. Proc. R. Soc. London Ser. B Biol. Sci. 203(1153), 405–426 (1979). https://doi.org/10.1109/CVPR.2016.445

  45. Voigtlaender, P., et al.: Mots: multi-object tracking and segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7942–7951 (2019). https://doi.org/10.1109/CVPR.2019.00813

  46. Wang, S., Leroy, V., Cabon, Y., Chidlovskii, B., Revaud, J.: Dust3r: geometric 3D vision made easy. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20697–20709 (2024). https://doi.org/10.48550/arXiv.2312.14132

  47. Wang, Y., He, X., Peng, S., Tan, D., Zhou, X.: Efficient loftr: semi-dense local feature matching with sparse-like speed. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21666–21675 (2024). https://doi.org/10.48550/arXiv.2403.04765

  48. Weinzaepfel, P., et al.: Croco: self-supervised pre-training for 3D vision tasks by cross-view completion. In: Advances in Neural Information Processing Systems, vol. 35, pp. 3502–3516 (2022).

    Google Scholar 

  49. Xiang, Y., Schmidt, T., Narayanan, V., Fox, D.: PoseCNN: a convolutional neural network for 6D object pose estimation in cluttered scenes. ArXiv arXiv:1711.00199 (2017). https://doi.org/10.15607/RSS.2018.XIV.019

  50. Xu, S., Chen, S., Xu, R., Wang, C., Lu, P., Guo, L.: Local feature matching using deep learning: a survey. Inf. Fusion 107 (2024). https://doi.org/10.1016/j.inffus.2024.102344

  51. Yang, Z., Dai, J., Pan, J.: 3D reconstruction from endoscopy images: a survey. Comput. Biol. Med. 108546 (2024). https://doi.org/10.1016/j.compbiomed.2024.108546

  52. Yi, K.M., Trulls, E., Lepetit, V., Fua, P.: LIFT: learned invariant feature transform. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 467–483. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_28

  53. Zhang, Y., et al.: Deep learning in food category recognition. Inf. Fusion 98, 101859 (2023). https://doi.org/10.1016/j.inffus.2023.101859

  54. Zhao, X., Wu, X., Chen, W., Chen, P.C., Xu, Q., Li, Z.: Aliked: a lighter keypoint and descriptor extraction network via deformable transformation. IEEE Trans. Instrum. Measur. 72, 1–16 (2023). https://doi.org/10.1109/TIM.2023.3271000

  55. Zhao, X., Wu, X., Miao, J., Chen, W., Chen, P.C., Li, Z.: Alike: accurate and lightweight keypoint detection and descriptor extraction. IEEE Trans. Multimedia 25, 3101–3112 (2022). https://doi.org/10.1109/TMM.2022.3155927

Download references

Acknowledgements

This work was supported in whole, or in part, by the Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) [203145/Z/16/Z], the Department of Science, Innovation and Technology (DSIT), the Royal Academy of Engineering under the Chair in Emerging Technologies programme, Engineering and Physical Sciences Research Council (EPSRC) [EP/W00805X/1, EP/Y01958X/1, EP/P012841/1]; Horizon 2020 FET [GA863146]. Sierra Bonilla is supported by the UKRI AI Centre for Doctoral Training in Foundational Artificial Intelligence (FAI CDT) [EP/S021566/1]. Rema Daher is supported by the UCL Centre for Digital Innovation through the Amazon Web Services (AWS) Doctoral Scholarship in Digital Innovation 2023/2024. For the purpose of open access, the authors have applied a CC BY public copyright licence to any author accepted manuscript version arising from this submission.

Author information

Authors and Affiliations

  1. Department of Computer Science and UCL Hawkes Institute, University College London, London, WC1E 6BT, UK

    Sierra Bonilla, Chiara Di Vece, Rema Daher, Xinwei Ju, Danail Stoyanov, Francisco Vasconcelos & Sophia Bano

Authors
  1. Sierra Bonilla
    View author publications

    Search author on:PubMed Google Scholar

  2. Chiara Di Vece
    View author publications

    Search author on:PubMed Google Scholar

  3. Rema Daher
    View author publications

    Search author on:PubMed Google Scholar

  4. Xinwei Ju
    View author publications

    Search author on:PubMed Google Scholar

  5. Danail Stoyanov
    View author publications

    Search author on:PubMed Google Scholar

  6. Francisco Vasconcelos
    View author publications

    Search author on:PubMed Google Scholar

  7. Sophia Bano
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Sierra Bonilla.

Editor information

Editors and Affiliations

  1. Istituto Italiano di Tecnologia, Genoa, Italy

    Alessio Del Bue

  2. Meta AI, Barcelona, Spain

    Cristian Canton

  3. Google DeepMind, Zürich, Switzerland

    Jordi Pont-Tuset

  4. Politecnico di Torino, Turin, Italy

    Tatiana Tommasi

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bonilla, S. et al. (2025). Mismatched: Evaluating the Limits of Image Matching Approaches and Benchmarks. In: Del Bue, A., Canton, C., Pont-Tuset, J., Tommasi, T. (eds) Computer Vision – ECCV 2024 Workshops. ECCV 2024. Lecture Notes in Computer Science, vol 15645. Springer, Cham. https://doi.org/10.1007/978-3-031-91989-3_8

Download citation

Keywords

Publish with us

Policies and ethics

Profiles

  1. Chiara Di Vece View author profile
  2. Sophia Bano View author profile