Abstract
Melanoma diagnosis, a particularly aggressive form of skin cancer, remains challenging due to the variability of lesions. To improve classification accuracy, we propose a novel multimodal approach combining dermoscopic images and clinical metadata (sex, age, anatomical location). Our architecture relies on different visual backbones (Vision Transformer, ResNet, EfficientNet) for image feature extraction, coupled with a multilayer perceptron (MLP) dedicated to metadata encoding. The two branches are fused to produce a joint prediction. Results show that the MultimodalViTModel achieves the best overall performance (F1-score of 0.8897, accuracy of 0.9955), closely followed by the MultimodalResNet. The MultimodalEfficientNet model exhibits greater variability on minority classes. In comparison, a baseline model relying solely on images performs worse, confirming the added value of integrating clinical metadata. Finally, the complexity analysis reveals that ResNet represents an excellent trade-off between efficiency and accuracy, making it suitable for constrained clinical settings. Our code will be publicly available at: https://github.com/Aby1diallo/Skin_lesion.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
We’re sorry, something doesn't seem to be working properly.
Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.
Similar content being viewed by others
References
World Health Organization: Ultraviolet (UV) radiation and skin cancer. WHO QA (2023). https://www.who.int/fr/news-room/questions-and-answers/item/ultraviolet-(uv)-radiation-and-skin-cancer
Wu, Y., Chen, B., et al.: Ultraviolet radiation and skin cancer: epidemiology, mechanisms and preventive strategies. Front. Oncol. 12, 893972 (2022). https://www.frontiersin.org/articles/10.3389/fonc.2022.893972/full
Liu, Y., Jain, A., Eng, C., et al.: A deep learning system for differential diagnosis of skin diseases. Sci. Transl. Med. 12(563), eabb3652 (2020). https://www.science.org/doi/10.1126/scitranslmed.abb3652
Jojoa Acosta, M.F., Caballero Tovar, L.Y., Garcia-Zapirain, M.B., Percybrooks, W.S.: Melanoma diagnosis using deep learning techniques on dermatoscopic images. BMC Med. Imaging 20(1), 65 (2021). https://bmcmedimaging.biomedcentral.com/articles/10.1186/s12880-020-00534-8
International Skin Imaging Collaboration (ISIC): ISIC Challenge (2020). https://challenge.isic-archive.com/data/#2020
Wu, Y., Chen, B., Zeng, A., Pan, D., Wang, R., Zhao, S.: Skin cancer classification with deep learning: a systematic review. Front. Oncol. 12, 893972 (2022)
Société canadienne du cancer: Mélanome de la peau : diagnostic (2024). Consulté en mai 2025. https://cancer.ca/fr/cancer-information/cancer-types/melanoma-skin/diagnosis
Dinnes, J., et al.: Dermoscopy, with and without visual inspection, for the diagnosis of melanoma in adults. Cochrane Datab. Syst. Rev. 2018(12), CD011902 (2018). https://doi.org/10.1002/14651858.CD011902.pub2. https://www.researchgate.net/publication/326460219_Dermoscopy_with_and_without_visual_inspection_for_the_diagnosis_of_melanoma_in_adults
Brinker, T.J., et al.: Deep learning outperformed 11 pathologists in the classification of histopathological melanoma images. J. Med. Internet Res. 20(10), e11936 (2018)
Stofa, M.M., Zulkifley, M.A., Zainuri, M.A.A.M.: Skin lesions classification and segmentation: a review. Int. J. Adv. Comput. Sci. Appl. 12(10) (2021). https://doi.org/10.14569/ijacsa.2021.0121060. https://thesai.org/Publications/ViewPaper?Code=IJACSA&Issue=10&SerialNo=60&Volume=12
Efat, A.H., Hasan, S.M.M., Uddin, M.P., Mamun, M.A.: A multi-level ensemble approach for skin lesion classification using customized transfer learning with triple attention. PLoS ONE 19(10), e0309430 (2024)
Viso.ai: Vision Transformer (ViT): How It Works and Why It Matters in Deep Learning (2024). Consulté le 2026/03/28. https://viso.ai/deep-learning/vision-transformer-vit/
Cirrincione, G., et al.: Transformer-based approach to melanoma detection. Sensors 23(12), 5677 (2023)
Yacob, F., et al.: Weakly supervised detection and classification of basal cell carcinoma using graph-transformer on whole slide images. Sci. Rep. 13, 7555 (2023). https://doi.org/10.1038/s41598-023-33863-z. https://www.nature.com/articles/s41598-023-33863-z
Wang, X., Li, Y., Zhang, Z., Chen, Y.: A novel approach for melanoma detection utilizing GAN synthesis and BatchFormer vision transformer model. Comput. Methods Prog. Biomed. (2024). https://www.sciencedirect.com/science/article/abs/pii/S0010482524006577
Roy, V.K., Thakur, V., Baliyan, N., Goyal, N., Nijhawan, R.: A framework for seborrheic keratosis skin disease identification using vision transformer. In: Malik, P., Nautiyal, L., Ram, M. (eds.) Machine Learning for Cyber Security, pp. 117–128. De Gruyter (2023). https://doi.org/10.1515/9783110766745-006
Yang, G., Luo, S., Greer, P.: A novel vision transformer model for skin cancer classification. Neural Process. Lett. 55, 9335–9351 (2023). https://doi.org/10.1007/s11063-023-11204-5
Vachmanus, S., Noraset, T., Piyanonpong, W., Rattananukrom, T., Tuarob, S.: DeepMetaForge: a deep vision-transformer metadata-fusion network for automatic skin lesion classification. IEEE Access 11 (2023). https://doi.org/10.1109/ACCESS.2023.3345225. https://www.researchgate.net/publication/376674813_DeepMetaForge_A_Deep_Vision-Transformer_Metadata-Fusion_Network_for_Automatic_Skin_Lesion_Classification
Khan, S., Khan, A.: SkinViT: a transformer based method for Melanoma and Nonmelanoma classification. PLoS ONE 18(12), e0295151 (2023)
Wang, R., et al.: A novel approach for melanoma detection utilizing GAN synthesis and vision transformer. Comput. Biol. Med. 176, 108572 (2024). https://doi.org/10.1016/j.compbiomed.2024.108572. https://www.sciencedirect.com/science/article/abs/pii/S0010482524006577
Catal Reis, H., Turk, V.: Fusion of transformer attention and CNN features for skin cancer detection. Appl. Soft Comput. 164, 112013 (2024). https://doi.org/10.1016/j.asoc.2024.112013. https://www.sciencedirect.com/science/article/abs/pii/S1568494624007877
Dai, W., Liu, R., Wu, T., Wang, M., Yin, J., Liu, J.: Deeply supervised skin lesions diagnosis with stage and branch attention. IEEE J. Biomed. Health Inf. 28(2), 719–729 (2024)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2026 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Diallo, A., Allaya, M.M., Samb, D., Elbatel, M., Lo, S., Bousso, M. (2026). Multimodal Fusion for Melanoma Classification Using Dermoscopic Images and Clinical Metadata. In: Anazodo, U., Zhang, D., Raymond, C., Kurt, M., Lekadir, K., Crimi, A. (eds) Medical Image Computing in Resource Constrained Settings. MIRASOL 2025. Lecture Notes in Computer Science, vol 16398. Springer, Cham. https://doi.org/10.1007/978-3-032-13654-1_3
Download citation
DOI: https://doi.org/10.1007/978-3-032-13654-1_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-032-13653-4
Online ISBN: 978-3-032-13654-1
eBook Packages: Computer ScienceComputer Science (R0)Springer Nature Proceedings Computer Science
