close
Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 May 5:1-13.
doi: 10.1007/s00371-022-02492-4. Online ahead of print.

A multimodal transformer to fuse images and metadata for skin disease classification

Gan Cai  1 , Yu Zhu  1 , Yue Wu  1 , Xiaoben Jiang  1 , Jiongyao Ye  1 , Dawei Yang  2   3
Affiliations

A multimodal transformer to fuse images and metadata for skin disease classification

Gan Cai et al. Vis Comput. .

Abstract

Skin disease cases are rising in prevalence, and the diagnosis of skin diseases is always a challenging task in the clinic. Utilizing deep learning to diagnose skin diseases could help to meet these challenges. In this study, a novel neural network is proposed for the classification of skin diseases. Since the datasets for the research consist of skin disease images and clinical metadata, we propose a novel multimodal Transformer, which consists of two encoders for both images and metadata and one decoder to fuse the multimodal information. In the proposed network, a suitable Vision Transformer (ViT) model is utilized as the backbone to extract image deep features. As for metadata, they are regarded as labels and a new Soft Label Encoder (SLE) is designed to embed them. Furthermore, in the decoder part, a novel Mutual Attention (MA) block is proposed to better fuse image features and metadata features. To evaluate the model's effectiveness, extensive experiments have been conducted on the private skin disease dataset and the benchmark dataset ISIC 2018. Compared with state-of-the-art methods, the proposed model shows better performance and represents an advancement in skin disease diagnosis.

Keywords: Attention; Deep learning; Multimodal fusion; Skin disease; Transformer.

PubMed Disclaimer

Figures

Fig. 1
Fig. 1
The overall architecture of the model
Fig. 2
Fig. 2
Forward propagation of One-hot encoded vectors in MLP
Fig. 3
Fig. 3
Comparison of Soft Label Encoder with One-hot Encoder
Fig. 4
Fig. 4
Mutual Attention block
Fig. 5
Fig. 5
Multi-head Cross Attention
Fig. 6
Fig. 6
The convergence graphs on the private dataset (a) and public dataset (b)
Fig. 7
Fig. 7
The results of the proposed model on the private dataset: (a) ROC curve, (b) Confusion matrix
Fig. 8
Fig. 8
The results of the proposed model on ISIC 2018: (a) ROC curve, (b) Confusion matrix
See this image and copyright information in PMC

References

    1. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
    1. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
    1. Xiao, H., Ran, Z., Mabu, S., Li, Y., Li, L.: SAUNet++: an automatic segmentation model of COVID-19 lesion from CT slices. Vis. Comput. pp. 1–14 (2022)
    1. Mohamed, E.H., El-Behaidy, W.H.: Enhanced skin lesions classification using deep convolutional networks. In: 2019 Ninth International Conference on Intelligent Computing and Information Systems (ICICIS), IEEE, pp. 180–188 (2019)
    1. Zhang, Y., Wang, C.: SIIM-ISIC melanoma classification with DenseNet. In: 2021 IEEE 2nd international conference on big data, artificial intelligence and internet of things engineering (ICBAIE), IEEE, pp. 14–17 (2021)

LinkOut - more resources