Introduction

With the rapid development of Internet healthcare, electronic health record (EHR) are growing at an unprecedented speed. The Office of the National Coordinator for Health Information Technology in the United States reported that, as of 2021, 78% of office-based physicians were using EHR systems [1]. These EHR data hold significant data value and can provide rich and well-structured health information. EHR based Internet healthcare applications, such as clinical decision support [2,3,4], infectious disease surveillance [5,6,7], EHR management system [8,9,10], are receiving extensive attentions.

Chronic disease risk prediction based on EHR is an important research direction of Internet healthcare. Various machine learning methods have been applied to develop risk prediction models for different types of chronic diseases. At present, related studies mainly focus on developing various supervised machine learning models, especially deep learning models, based on EHR data. With sufficient training samples, deep learning models can achieve the same or even better performance than domain experts in diagnosing certain diseases [11]. Furthermore, the development of chronic diseases is a long-term process. It is obviously difficult to effectively predict disease risk only based on the current health conditions. Therefore, existing studies on chronic disease risk prediction mainly adopted longitudinal sequential methods [12], which have predicted disease risk based on both current health conditions and historical information. These methods need large-scale and high-quality longitudinal EHR data.

However, people often visit different medical institutions for treatments or physical examinations. Therefore, single-visit EHR data tend to far exceed than multiple-visit longitudinal EHR data in a medical institution. Information barriers caused by personal privacy protection make it difficult to collect large-scale and complete longitudinal EHR data from different medical institutions. Furthermore, for specific chronic diseases, there are relatively few records belonging to diseased individuals because of low prevalence. Therefore, the collected longitudinal EHR data is often few-shot and imbalanced in practical scenarios. For example, this study collected a total of 6287 EHR records but only 471 records are involved multiple visits, and only 164 records are involved hypertension. Few-shot data can cause overfitting [13,14,15]. Data imbalance can bring about classification and feature bias [16,17,18]. The few-shot and imbalanced longitudinal EHR data have become an urgent challenge of EHR based chronic disease risk prediction.

Based on the above observation, this study develops a novel deep learning model called CDR-Detector for chronic disease risk prediction based on few-shot and imbalanced longitudinal EHR data. The main contributions are summarized as follows:

  • Firstly, a self-adaptive EHR based pre-training model is proposed to improve the few-shot learning ability. Based on the multi-layer transformer architecture, two self-supervised EHR based pre-training tasks are designed to learn the dependencies between health conditions and chronic diseases for disease risk prediction. Different from existing EHR based pre-training studies, which conducted pre-training and downstream tasks on the same EHR data, this study conducts pre-training on a large number of single-visit data for making up the shortage of longitudinal EHR data. And then, a weight adaptive mechanism is developed to integrate two pre-training tasks and coordinate their contributions.

  • Secondly, a deep reinforcement learning (RL) model with the dual experience replay is developed to integrate the self-adaptive EHR based pre-training model for predicting the chronic disease risk based on few-shot and imbalanced longitudinal EHR data. This model adopts the Q-learning architecture and a new custom reward function. A dual experience replay strategy is realized to help the model select representative data samples and accelerate model convergence on few-shot and imbalanced EHR data.

  • Finally, a group of experiments have been conducted on the personal physical examination data from the regional medical institution. Experimental results show that, compared with the existing state-of-art methods, the proposed CDR-Detector has better accuracy and robustness for predicting chronic disease risk based on the few-shot and imbalanced longitudinal EHR data.

Related work

Chronic disease risk prediction based on EHR

Chronic disease risk prediction is to utilize data sources, such as EHR, for risk prediction of chronic disease development of patients [19]. Early approaches often adopted instance based methods [20,21,22,23], which only trained models based on current health conditions. However, the development of chronic diseases is a long-term process. It is necessary to utilize historical health information for disease risk prediction. Hence, longitudinal sequential methods [24,25,26], which predicted disease risk based on both current health conditions and historical information, become the current research hotspot of EHR based chronic disease risk prediction. Excellent prediction performances in these studies depended on the effective representation of longitudinal EHRs, which needs large-scale longitudinal EHR data. For example, Gao et al. [24] used 2,202,114 samples from the MIMIC-III dataset for disease risk prediction.

However, due to people’s medical habits and information barriers among medical institutions, it is difficult to collect large-scale and complete longitudinal EHR data from different medical institutions. Hence, existing chronic disease risk prediction models are often difficult to apply to practical scenarios. Few-shot learning becomes an urgent challenge for EHR based chronic disease risk prediction. Although some recent studies have focused on this challenge. The few-shot learning studies in the medical field are still in the early stage [27]. Related research mainly adopted data enhancement strategies, which often brought uncertain noise and fell into unstable results [28]. Therefore, it is necessary to develop new few-shot learning methods for chronic disease risk prediction based on EHR.

At present, pre-training is an important approach for realizing few-shot learning. However, existing studies on EHR based pre-training mainly focused on designing well-defined pre-training tasks to mine domain-specific characteristics in EHR data for improving downstream tasks, such as disease risk prediction. For example, Rasmy et al. [29] developed an EHR based pre-training model Med-BERT and proposed two pre-training tasks, namely Masked LM and Prolonged LOS, to capture contextual semantics of structured EHR. Fine-tuning experiments on two clinical databases showed that Med-BERT could effectively improve disease prediction on local training datasets. Ren et al. [30] designed three EHR based pre-training tasks, namely Similarity Prediction, Masked Prediction and Reasonability Check, to capture deep-level EHR characteristics for dealing with insufficient data, incomplete data, and short sequence problems. Experimental results of various downstream tasks have shown their effectiveness. Meng et al. [31] adopted a transformer architecture and the Masked LM task to learn bidirectional representation learning of EHR sequences. These EHR based pre-training models still needed large-scale longitudinal EHR data as pre-training data. For example, Rasmy et al. [29] used 28,490,650 patients’ EHR data for pre-training. Their pre-training tasks were mainly designed to learn potential dependencies in EHR sequences. Therefore, it is necessary to develop an EHR based pre-training model with new pre-training tasks for realizing few-shot learning of EHR based chronic disease risk prediction.

Fig. 1
Fig. 1
Full size image

The overall structure of CDR-Detector

Deep reinforcement learning based on EHR

RL aims at optimizing decision making by using interaction samples of an agent with its environment and the potentially delayed feedbacks [32,33,34]. It can simultaneously process the sequential decision-making problem of sampling, evaluation, and delayed feedback, which is applicable to the continuous and delayed feedback processes in the medical field. At present, RL, especially deep RL, has become an important approach to solve sequential decision-making problems based on EHR. For example, Li et al. [35] proposed an deep RL model based on EHR to optimize the sequential treatment strategy for sepsis, diabetes and their complications. Oh et al. [36] proposed a deep RL model to recommend drugs for improving long-term treatments of type 2 diabetes. Wang et al. [27] proposed a graph convolutional RL model for medicine prediction. These studies mainly focused on improving the accuracy of the model by enhancing the expression ability of EHR features, which still depended on large-scale longitudinal EHR data. Therefore, the few-shot learning is also an urgent challenge for EHR based deep RL studies.

Additionally, for a specific chronic disease, low prevalence often lead to an imbalance between between positive and negative samples. This is another important challenge for chronic disease risk prediction based on longitudinal EHR data. At present, an increasing number of studies are utilizing RL to solve the challenge of data imbalance. Related studies focused on the optimization of reward functions. By matching the scenario and assigning higher rewards, a few categories can obtain better learning weights for resolving the negative impact of data imbalance on model training. For examples, Yang et al. [16] proposed an imbalance classification framework based on deep RL for the diagnosis of COVID-19. By developing a customized reward function, the framework is able to effectively learn on extremely imbalanced datasets. Lin et al. [37] proposed a general imbalanced classification model based on deep RL. They designed a specific reward function to guide agents in finding the optimal classification strategy for imbalanced data.

However, the methods of optimizing the reward function may produce some problems, such as low sample utilization, slow convergence speed, and unrelated to past samples. Low sample utilization and slow convergence speed are not suitable for model learning on few-shot data. Unrelated to past samples makes it difficult to learn timing characteristics from longitudinal data. Hence, predicting chronic disease risks based on few-shot longitudinal EHR data needs to develop new RL based methods for handling data imbalance.

Materials and methods

Problem definition

Definition 1

Single-visit EHR records. In EHR data, each record is represented as \(P^{(n)}=\{A^{(n)},Tar^{(n)}\}\) where n represents the n-th physical examinee, \(A=\{Age,LSBP,Height,\ldots Exercise-Freq-Code\} \) is the attribute set, \(A^{(n)}=\{Age^{(n)},LSBP^{(n) },Height^{(n)},\ldots ,Exercise-Frea-Code^{(n)}\}\) is the attribute value set of \(P^{(n)}\), \(Tar^{(n)}\) is the sample tag of \(P^{(n)}\) and used to identify whether it is a positive sample or negative sample.

Definition 2

Multiple-visit EHR records. In longitudinal EHR data, each physical examinee’s record can be represented as a sequence of multivariate observations: \(M\_P_{T^{(n)}}^{(n)}= \langle P_1^{(n)},P_2^{(n)},P_3^{(n)},\ldots ,P_{T^{(n)}}^{(n)} \rangle \) where \(T^{(n)}\) represents the number of the n-th physical examinee’s examinations. Each EHR record is represented as \(P_t^{(n)}=\{A_t^{(n)},Tar_t^{(n)}\}\) where t represents the t-th visit of the n-th physical examinee, \(A_t^{(n)}=\{Age_t^{(n)},LSBP_t^{(n)},Height_t^{(n)},\ldots ,Exercise-Frea-Code_t^{(n)}\}\) is the attribute value set of the t-th visit of the n-th physical examinee, \(Tar_t^{(n)}\epsilon \)[0,1] is the sample tag of the t-th visit of the n-th physical examinee. \(Tar_t^{(n)}=0\) represents no illness and \(Tar_t^{(n)}=1\) represents illness.

Definition 3

Chronic disease risk prediction task. Give a physical examinee’s history EHR record \(M\_P_{t-1}^{(n)}= \langle P_1^{(n)},P_2^{(n)},P_3^{(n)},\ldots ,P_{t-1}^{(n)}\rangle \) and the attribute value set \(A_t^{(n)}\) at the t-th visit record \(P_t^n\), chronic disease risk prediction is to predict the sample tag \(Tar_t^{(n)}\) at the t-th visit by generating the output \(y_t\epsilon \{0,1\}\). It is a binary classification problem.

Overall structure of the model

This study proposes a novel chronic disease risk prediction model based on few-shot and imbalanced EHR data, called CDR-Detector, by combining EHR based pre-training and the deep RL. As shown in Fig. 1, this model consists of two modules: the pre-training module and the disease prediction module. Their details will be introduced in subsequent chapters.

The pre-training module based on single-visit EHR data

The pre-training module mines potential features from single-visit EHR data for the downstream prediction task. It is composed of an input representation layer and a pre-training learning layer.

The input representation layer

The input representation layer converts each EHR record into a multidimensional input embedding. For each single-visit EHR record \(P^n\), its input embedding can be defined as:

$$\begin{aligned} hea^n = \Vert ^{|M|}_{i}attr^{(n)}[i],\hspace{3em} 0<i\le |M| \end{aligned}$$
(1)

where n represents the n-th physical examinee, |M| represents the number of attributes in \(P^n\), i represents the i-th attribute, \(attr^{(n)} [i]\) represents the one-hot coding of the i-th attribute, and \(\Vert \) represents the connection of \(attr^{(n)} [0],attr^{(n)} [1]\ldots attr^{(n)} [|M|]\).

The pre-training learning layer based on transformer

The pre-training learning layer mines potential features useful for disease prediction from the input embedding sequence of single-visit EHR data. This study adopts the multi-layer transformer [38] architecture to construct the pre-training learning layer. We randomly mask a one-hot coding of attribute \(attr^{(n)} [i]\) as follows:

$$\begin{aligned} pre\_attr^{(n)}[i] & = Transformer(\varphi :[CLS]\nonumber \\ & \cup \quad random\_mask(attr^{(n)}[i])) \end{aligned}$$
(2)

where \(random\_mask\) is a random attribute masking method and can be expressed as follows:

$$\begin{aligned} random\_mask_{(attr^{(n))}[i]} = {\left\{ \begin{array}{ll} |mask|, & 0\le prob\le 0.8 \\ attr^{(n)}[i], & 0.8\le prob \\ p|p\epsilon VOC_i, & 0.9\le prob \end{array}\right. } \end{aligned}$$
(3)

where prob is a random number in the range of \(0\sim 1\), |mask| represents masking examination attribute value, \(VOC_i\) represents the dictionary corresponding to the i-th attribute, and p represents an attribute value in \(VOC_i\). As shown in this formula, each attribute has 80% probability to use the |mask| field for masking, 10% probability remains unchanged, and 10% probability is replaced with other fields in the same attribute. Therefore, a complete pre-training single-visit record can be expressed as follows:

$$\begin{aligned} pre\_hea^n = ||^{|p|}_{i}pre\_attr^{(n)}[i] \end{aligned}$$
(4)

where || represents the connection of \(pre\_attr^{(n)} [0]\), \(pre\_attr^{(n)} [1]\ldots pre\_attr^{(n)} [|M|]\).

Because of the difference between texts and EHR data, we do not use position embeddings of BERT to mark each EHR attribute because there is no clear positional relationship between EHR attributes. Similarly, we do not use segment embeddings of BERT due to the lack of relevance between single-visit records. Attribute values in EHR data are often incomplete because of incomplete physical examination items or data loss during the recording process. Although we filled in missing items using the neighbor value from existing data, the absence of some important attribute values can affect the downstream prediction task. Therefore, our pre-training needs to mine the features of attributes themselves. Furthermore, due to the use of data different from the downstream task, domain knowledge useful for disease risk prediction should be extracted from single-visit data. Therefore, our pre-training also needs to mine association between attributes and sample tags, namely the dependencies between health conditions and chronic diseases. Based on these mining requirements, this study designs the following two kinds of EHR based pre-training tasks.

Self-composition prediction task: This task masks each attribute and predicts it to improve the self-prediction ability of the model. Its loss function is as follows:

$$\begin{aligned} \begin{aligned}&L_{self}(\varphi ;pre\_attr^{(n)},attr^{(n)})\\&\quad = -\sum ^{|M|}_{i=0}L_{s[i]}(pre\_attr^{(n)},attr^{(n)})\\&\quad =logP(pre\_attr^{(n)}[i]|attr^{(n)}[i]) \\&\qquad +\sum \limits _{p_i\epsilon \{VOC_i \backslash attr^{(n)}[i]\}} \end{aligned} \end{aligned}$$
(5)

where \(p_i\epsilon \{VOC_i \backslash attr^{(n)}[i]\}\) indicates \(p_i\) is any value in the i-th attribute dictionary except for \(attr^{(n)} [i]\). This study minimizes \(L_{self}\) to achieve the stronger self-prediction ability.

Objective prediction task:This task is used to mine association between attributes and sample targets. Its loss function is as follows:

$$\begin{aligned}&L_{target}(\varphi ;pre\_attr^{(n)},attr^{(n)}) \nonumber \\&\quad = -\sum ^{|M|}_{i=0}(attr^{(n)}[i]| pre\_attr^{(n)}[i])\nonumber \\&\qquad -\sum ^{|M|}_{i=0}(tar^{(n)}[i]|pre\_attr^{(n)}[i]) \end{aligned}$$
(6)

where \(pre\_tar^{(n)} \)is obtained by conducting \(random\_mask\) on the one-hot sample tag encoding of the n-th physical examinee’s record.

The self-composed prediction task and the objective prediction task jointly constitute the final optimization objective. The dynamic adjustment of the weight of the total loss function aims to optimize the two pre-training tasks as much as possible without interference between each other, so as to avoid the dominance of some tasks and the inability of others to fully optimize.

The total loss function is expressed as follows:

$$\begin{aligned}&L_{total}(\varphi ;pre\_attr^{(n)},attr^{(n)}) \nonumber \\&\quad = a*L_{self}(\varphi ;pre\_attr^{(n)},attr^{(n)}) \nonumber \\&\quad \quad + b*L_{target}(\varphi ;pre\_attr^{(n)},attr^{(n)}) \end{aligned}$$
(7)

The disease risk prediction module based on deep RL

This study transforms chronic disease prediction into a sequential decision making problem and develops the disease risk prediction module based on deep RL.

Key components of RL

Chronic disease prediction can be formulated as a Markov Decision Process problem and formalized by the tuple \(\{S,A,T,R,\gamma \}\). S is the state of environment, A is the action space of the agent, T is the transition function, R is the reward function and \(\gamma \) is a factor for delayed reward. These key components of RL can be defined as follows.

Agent: The disease risk prediction module is considered as an agent. It can provide personalized disease risk prediction based on current environment status. Its action space is defined as follows:

$$\begin{aligned} a_t= {\left\{ \begin{array}{ll} 1, & be \ ill \\ 0, & no \ ill \\ \end{array}\right. } \end{aligned}$$
(8)

Environment: The environment is defined as health conditions of all physical examinees. It involves not only the current health statuses of physical examinees but also their historical health statuses.

State: The state is the physical examinee’s health condition. For the n-th physical examinee, his state in the time step t, i.e., the t-th visit, is represented as follows:

$$\begin{aligned} s^{(n)}_t = [cur\_s_t^{(n)},his\_s^{(n)}_{< t}] \end{aligned}$$
(9)

where \(s_t^{(n)}\) is the health status of the n-th physical examinee in the t-th visit, and \(cur\_s_t^{(n)}\) is the health status of the t-th visit of the n-th physical examinee, and \(his\_s_{<t}^{(n)}\) is the historical health statuses of the n-th physical examinee before the t-th visit. They can be obtained based on multi-visit records \(M\_P_t^{(n)}\) of the n-th physical examinee and the pre-training model:

$$\begin{aligned} cur\_s^{(n)}_t= & \big [vem[1]^{(n)}_t,vem[2]^{(n)}_t,\ldots ,\nonumber \\ & \quad vem[i]^{(n)}_t,\ldots ,vem[|A|]^{(n)}_t\big ] \end{aligned}$$
(10)
$$\begin{aligned} his\_s^{(n)}_{< t}= & \frac{1}{t-1}\sum ^{t-1}_{i} cur\_s^{(n)}_i \end{aligned}$$
(11)

where \(vem[i]_t^{(n)}\) is the visit embedding of the i-th attribute of the t-th visit record obtained by the pre-training model, and |A| is the number of the attribute set A. As shown in the formula (11), we average the visit embeddings before the t-th visit to provide a general expression for the historical health statuses of the n-th physical examinee.

Reward: The reward is the feedback about chronic disease risk prediction and defined as the sum of (i) \(r1_t\), whether the predicted results are consistent with real results; (ii) \(r2_t\), the reciprocal of the distance between the predicted and real results. The former is a stricter punishment for obtaining a more accurate choice. The latter is a looser punishment for increasing the robustness of action judgment. Then, the reward can be defined as follows:

$$\begin{aligned} & r_t = r1_t+r2_t \end{aligned}$$
(12)
$$\begin{aligned} & r1_t= {\left\{ \begin{array}{ll} -1, & if \ \hat{Tar^{(n)}_t} != Tar^{(n)}_t \\ 1, & if \ \hat{Tar^{(n)}_t} = Tar^{(n)}_t \\ \end{array}\right. } \end{aligned}$$
(13)
$$\begin{aligned} & r2_t= Sigmoid \nonumber \\ & \left( \frac{1}{\root 2 \of {(no\_illness+1-Tar^{(n)}_{t})^2+(illness -Tar^{(n)}_{t})^2}}\right) \nonumber \\ \end{aligned}$$
(14)

where \(\hat{{Tar}_t^{(n)}} \) is the predicted sample tag of t-th visit of n-th physical examinee, and \(no\_illness\) is the predicted probability of no getting ill, and illness is the predicted probability of getting ill.

Based on the key components mentioned above, this study adopts Deep Q Network (DQN) [39] to construct the chronic disease prediction model based on deep RL. The current state is taken as the input of the Q main network to generate the action and select the subsequent action with the maximum state action value \(argmaxQ(s_t,a_t,\omega )\). Then, the agent performs the action to obtain the corresponding reward \(r_t\), and stores the current state \(s_t\), action \(a_t\), next state \(s_{t+1}\), and reward \(r_t\) as historical experience information in the experience playback memory. Finally, the loss function is calculated by combining the Q values of the main network and the target network.

Dual experience replay strategy

Low prevalence of diseases brings out imbalanced longitudinal EHR data. Due to data imbalance, the buffer of DQN tends to select data samples from most categories, resulting in poor prediction performance. Therefore, a new replay buffer is added to realize the dual experience replay strategy for solving the challenge of data imbalance.

Algorithm 1 describes the dual experience replay strategy. Two experience replay buffers are set to store data samples. The replay buffer \(m_1\) is used to store unchanged data samples which have the same sample tag in the i-1th and i-th visits, i.e., \(Tar_{t-1}^{(n)} = Tar_t^{(n)}\), and the replay buffer \(m_2\) is used to store changed data samples which have the different sample tag in the i-1th and i-th visits, i.e., \(Tar_{t-1}^{(n)} != Tar_t^{(n)}\). After the beginning of each epoch, data samples are stored into the corresponding replay buffers. If the replay buffer is full, the sampling strategy will be used to choose a redundant sample from this buffer. And then, the redundant sample will be removed and input into the neural network for calculating the loss.

Algorithm 1
Algorithm 1
Full size image

Dual experience replay

Loss function

After providing experience samples to the agent, the key is how to train the model to achieve better performance. For the Q-learning architecture, a priority allocation strategy based on time difference is designed as follows:

$$\begin{aligned} \delta ^{DQN}_t = r_{t+1}+\gamma \mathop {max}\limits _{a_t}Q(s_{t+1},a_{t+1},\omega ^{-}) -Q(s_t,a_t,\omega ) \end{aligned}$$
(15)

where \(\gamma \) is the discount factor, \(\omega ^{-}\) represents the Q target network, \(\omega \) represents the Q main network, \(r_{t+1}+\gamma \mathop {max}\limits _{a_t}Q(s_{t+1},a_{t+1},\omega ^{-})\) represents the target value, and \(Q(s_t,a_t,\omega )\) represents the current value. \(\omega \) and \(\omega ^{-}\) have the same network structure, and the parameters of \(\omega \) are assigned to \(\omega ^{-}\) at each step.

Table 1 Statistics of the data set
Full size table

After calculating the priority allocation score \(\delta ^{DQN}_t\), we need to calculate the selected probability of each data sample according to the priority allocation score \(\delta ^{DQN}_t\). This study uses the softmax function to map the priority score of each data sample. The sampling probability formula is as follows:

$$\begin{aligned} P(k) = \frac{e^{x^{(k)}}}{\sum ^K_{i=1}e^{x^{(i)}}} \end{aligned}$$
(16)

where P(k) represents the probability that the k-th data sample will be selected under the \(x^{(k)}\) priority score.

The objective function can be optimized as follows:

$$\begin{aligned} L(w)&= \mathbb {E}\left[ \delta ^{DQN^2}_t\right] \nonumber \\&= \mathbb {E}\Big [(r_{t+1}+\gamma \mathop {max}\limits _{a_t}Q(s_{t+1},a_{t+1},\omega ^{-})\nonumber \\&\quad \ -Q(s_t,a_t,\omega ))^2\Big ] \end{aligned}$$
(17)

where \(\mathbb {E}\) represents expectations of \(\delta ^{DQN^2}_t\). Then, the loss function for the single buffer experience replay strategy can be defined as:

$$\begin{aligned} L = \sum \limits _{t \epsilon P_{all}} \delta ^{DQN^2}_t \end{aligned}$$
(18)

where \(P_{all}\) represents the experience replay strategy based on single buffer. Finally, the loss function for the above dual experience replay strategy can be defined as follows:

$$\begin{aligned} L & = \lambda _1 \sum \limits _{t \epsilon P_{sample\_is\_change==0}} \delta ^{DQN^2}_t \nonumber \\ & \quad \ + \lambda _2 \sum \limits _{t \epsilon P_{sample\_is\_change==1}} \delta ^{DQN^2}_t \end{aligned}$$
(19)

where the hyperparameters \(\lambda _1\) and \(\lambda _2\) are used to choose data samples from the replay buffer. When selecting samples from \(m_1\), we set \(\lambda _1=1\) and \(\lambda _2=0\). Conversely, when retrieving samples from the experience replay buffer \(m_2\), we set \(\lambda _1=0\) and \(\lambda _2=1\).

Experiment

Experimental setup

Data set

This study uses the personal physical examination records obtained from a regional medical institution to construct the experiment dataset. After desensitization, 20 physical examination attributes have been extracted, including Age, Body Temperature, Pulse Rate, Left Systolic Blood Pressure (LSBP), Left Diastolic Blood Pressure (LDBP), Right Systolic Blood Pressure (RSBP), Right Diastolic Blood Pressure (RDBP), Height, Weight, Waist, Body Mass Index (BMI), Heart Rate, Total Serum Cholesterol (TCHO), Triglyceride (TG), Low Density Lipoprotein Cholesterol (LDLC), High Density Lipoprotein Cholesterol (HDLC), Exercise Frequency Code, Smoking Status Code, and Drinking Frequency Code. In this study, hypertension and diabetes are chosen as target chronic diseases. The corresponding hypertension dataset and diabetes dataset are constructed respectively. Three-year physical examination records of individuals before falling ill are taken as positive samples. Other physical examination records of healthy individuals are taken as negative samples. Table 1 provides the statistics of average value of attributes in the hypertension and diabetes datasets.

Each dataset has been further divided into the single-visit dataset for pre-training and the multi-visit dataset for disease risk prediction. Table 2 provides the numbers of physical examinees and records in the hypertension dataset and the diabetes dataset. As shown in this table, the number of physical examinees in the single-visit dataset is significantly higher than that in the multi-visit dataset. This proves the existence of few-shot challenge. Their positive and negative sample ratios are shown in Fig. 2. As shown in this figure, the numbers of negative samples in both the single-visit dataset and the multi-visit dataset are significantly larger than those of positive samples. This proves the existence of data imbalance challenge. For comparative analysis, negative samples are also randomly removed from imbalanced datasets to construct balanced datasets, which have the same number of positive and negative samples.

Table 2 Statistics of data sets
Full size table
Fig. 2
Fig. 2
Full size image

The positive and negative sample ratios in different datasets. a The single-visit hypertension dataset. b The multi-visit hypertension dataset. c The single-visit diabetes dataset. d The multi-visit diabetes dataset

Parameter setting

This study randomly divided all datasets into three parts in a 3:1:1 ratio, namely the training set, the validation set, and the testing set. The 5-fold cross-validation were conducted to obtain the average value of results. In the pre-training module, the epoch number is set to 20. In the disease prediction module, the hidden layer dimension of the DQN network is set to 300, and the state vector dimension is set to 12,000, the epoch number is set to 100, and the learning rate is set to 1e\({-}\)3, respectively.

Evaluation indicators

In the experiment, Jaccard similarity score (Jaccard) and F1 were used to measure the experimental results. Jaccard can be calculated as follows:

$$\begin{aligned} Jaccard = \frac{1}{\sum ^N_k\sum ^{T_k}_t 1} \sum ^N_k \sum ^{T_k}_t \frac{|Y^{(k)}_t \cap \breve{Y}_t^{(k)}|}{|Y^{(k)}_t \cup \breve{Y}_t^{(k)}|} \end{aligned}$$
(20)

where N represents the number of physical examinees, and \(T_k\) represents the visit number of the k-th physical examinees. \(Y_t^{(k)}\) is the prediction set, and \(\breve{Y}_t^{(k)}\) is the ground truth set.

F1 is the harmonic average of accuracy rate and recall rate. Its calculation formula is as follows:

$$\begin{aligned} F_1 = 2*\frac{P*R}{P+R} \end{aligned}$$
(21)

where the accuracy rate P represents the precision rate and R is the recall rate.

In addition to Jaccard and F1, precision recall AUC (PR-AUC) was also adopted as the evaluation indicator.

Baseline methods

The proposed CDR-Detector has been compared with the following baseline methods.

  • Logistic Regression (LR): This is a generalized linear regression analysis model and has been adopted to predict type 2 diabetes in Indian women based on single-visit EHR data [19]. In the experiment, \(max\_iter\) is set to 100 and tol is set to 1e\({-}\)4.

  • Random Forest (RF): This is a typical ensemble learning model and has been adopted to predict hypertension and hyperlipidemia based on single-visit EHR data [24]. In the experiment, the number of decision trees is set to 100, the minimum required samples at an internal node is set to 2, and the minimum required samples at a leaf node is set to 1.

  • KNN-LightGBM (KNN-L) [22]: This is a disease risk prediction model based on deep learning. A hybrid deep neural network combining KNN and LightGBM is adopted to predict the hypertension based on single-visit EHR data. In the experiment, the number of epoch is set to 100 and the learn rate is set to 1e\({-}\)3.

  • Retain [25]: This is also a disease risk prediction model based on deep learning. Different from KNN-L, Retain is a longitudinal sequential method. A reverse time attention mechanism is adopted to capture time series factors for predicting the heart failure based on longitudinal EHR data. In the experiment the number of epoch is set to 100 and the learn rate is set to 1e\({-}\)3.

  • ConCare [40]: It is a longitudinal sequential method based on deep learning. An improved multi-head self-attention mechanism is developed to capture inter-dependencies between dynamic features and static baseline information, in order to form the personal health context for disease risk prediction. In the experiment, the number of epoch is set to 100 and the learn rate is set to 1e-3.

Results

Experimental results on the hypertension dataset

Experimental results on the hypertension dataset are presented in Table 3. As shown in this table, the results of all models on the original imbalanced dataset are significantly better than those on the balanced dataset. The reason may be that undersampling changes the distribution of the original data and leads to biased results. This proves that simple data-level methods for handling data imbalance are not suitable for chronic disease risk prediction based on longitudinal EHR data. It is necessary and valuable to innovate data imbalance solutions based on deep RL in this study.

Table 3 Experimental results on the hypertension dataset
Full size table

The first three rows of Table 3 show the results of three instance based methods, namely LR, RF and KNN-L. Though the first two use traditional shallow machine learning models and the third uses the deep learning model, all of them achieve similar poor results on the imbalanced dataset. The reason is that these methods only focus on current health conditions and ignore the time series factors of historical EHR data. This proves that it is necessary for this study to predict chronic disease risk based on longitudinal EHR data.

Two longitudinal sequential methods, namely Retain and ConCare, cannot achieve significant advantages than instance based methods. Retain has the highest Jaccard value, but its F1 value is quite low because of the extremely low recall rate. Three evaluation indicators of ConCare are also similar to three instance based methods. This indicates that complex deep networks are difficult to play their advantages on the few-shot and imbalanced EHR data. The reason is that the EHR sequences are short and sparse. As shown in Table 1, the average visit number of each patient is only 3.5 in the multi-visit dataset of hypertension. Neither the reverse time attention mechanism of Retain nor the complex multi-head self-attention mechanism of ConCare can effectively extract EHR features from such short and sparse EHR sequences.

Compared with all baseline methods, the proposed CDR-Detector achieves the best results in almost all three indicators. All these comparisons proves the necessity and effectiveness of this study.

Experimental results on the diabetes dataset

Experimental results on the diabetes dataset are presented in Table 4. All results are from the imbalanced dataset. In three instance based methods, the deep learning method KNN-L is not significantly superior to those two shallow machine learning methods. This indicates that, the current popular deep learning methods are difficult to play their advantages on the few-shot and imbalanced EHR data. Two longitudinal sequential baseline methods also haven’t achieved significant advantages than three instance based methods. This once again confirms the above assertion that complex deep networks are difficult to extract enough EHR features from the short and sparse EHR sequence for improving the prediction ability.

The proposed CDR-Detector still achieves the best results in almost all three indicators compared with all baseline methods. However, by comparing Tables 3 and 4, it can be found that the improvement of CDR-Detector on the diabetes dataset is far less than that on the hypertension dataset. Its result is very similar to the optimal baseline method RF. As shown in Table 2, the number of patients in the diabetes multi-visit dataset is much less than that in the hypertension multi-visit dataset. Therefore, the relatively poor performance of CDR-Detector on the diabetes dataset indicates that the proposed method can effectively improve the prediction ability of the model on the few-shot and imbalanced datasets. However, its improvement ability will also rapidly decrease if the sample size is too small (less than 100).

Table 4 Experimental results on the imbalanced diabetes dataset
Full size table

Ablation experiments

The two core optimizations of this study are pre-training based on single-visit data and the dual experience replay strategy. This section will discuss their effectiveness by a group of ablation experiments.

Ablation experiments setting

Three CDR-Detector variants are designed as follows:

  • \(CDR{\text {-}}Detector_{pre{\text {-}}rep{\text {-}}}\): This variant is constructed by removing both the pre-training module and the dual experience replay strategy from CDR-Detector. It directly uses the deep RL module for chronic disease risk prediction and serves as the baseline of this ablation study.

  • \(CDR{\text {-}}Detector_{rep{\text {-}}}\): This variant is constructed by removing the dual experience replay strategy from the deep RL module of CDR-Detector. It retains pre-training module and can be used to verify the effectiveness of pre-training by comparing with \(CDR{\text {-}}Detector_{pre{\text {-}}rep{\text {-}}}\).

  • \(CDR{\text {-}}Detector_{pre{\text {-}}}\): This variant is constructed by removing the pre-training module from CDR-Detector. It can be used to verify the effectiveness of the dual experience replay strategy by comparing with \(CDR{\text {-}}Detector_{pre{\text {-}}rep{\text {-}}}\).

Results of ablation experiments

Results of ablation experiments on the hypertension dataset are presented in Table 5. As shown in this table, \(CDR{\text {-}}Detector_{pre{\text {-}}rep{\text {-}}}\) has the worst performance, indicating that both pre-training based on single-visit data and the dual experience replay strategy are necessary and effective.

Table 5 Results of ablation experiments on the hypertension dataset
Full size table

Pre-training based on single-visit data can effectively improve model performance on the few-shot dataset. Therefore, the result of \(CDR{\text {-}}Detector_{rep{\text {-}}}\) on the balanced dataset is significantly better than that of \(CDR{\text {-}}Detector_{pre{\text {-}}rep{\text {-}}}\). However, the result of \(CDR{\text {-}}Detector_{rep{\text {-}}}\) on the imbalanced dataset is poor and similar to that of \(CDR{\text {-}}Detector_{pre{\text {-}}rep{\text {-}}}\). This indicates that pre-training cannot solve the challenge of data imbalance. It is necessary to develop new solutions for handling data imbalance.

Though this study adopts the dual experience replay strategy to solve the challenge of data imbalance, it also helps to improve the few-shot learning ability of the model. The reason is that the dual experience replay strategy makes the model tend to learn more valuable data samples. Therefore, the result of \(CDR{\text {-}}Detector_{pre{\text {-}}}\) on the balanced dataset has significantly improved compared with \(CDR{\text {-}}Detector_{pre{\text {-}}rep{\text {-}}}\). However, the improvement of \(CDR{\text {-}}Detector_{pre{\text {-}}}\) is weaker than that of \(CDR{\text {-}}Detector_{rep{\text {-}}}\), indicating its effect in improving the few-shot learning ability may not be as good as the pre-training. Compared with \(CDR{\text {-}}Detector_{pre{\text {-}}rep{\text {-}}}\), the result of \(CDR{\text {-}}Detector_{pre{\text {-}}}\) on the imbalanced dataset has also significantly improved, indicating that the dual experience replay strategy is very effective for solving the challenge of data imbalance.

The proposed \(CDR{\text {-}}Detector\) achieves the best result, indicating that pre-training based on single-visit data and the dual experience replay strategy are compatible, and their combination can bring out the best performance.

Table 6 gives the results of ablation experiments on the imbalanced diabetes dataset. Similarly, \(CDR{\text {-}}Detector_{pre{\text {-}}rep{\text {-}}}\) achieves the worst performance. The dual experience replay strategy is the most important optimization in this study because it can help solve the challenges of few-shot and data imbalance. Therefore, \(CDR{\text {-}}Detector_{pre{\text {-}}}\) achieves the second best result. Compared with \(CDR{\text {-}}Detector_{rep{\text {-}}}\), the improvement of \(CDR{\text {-}}Detector_{pre{\text {-}}}\) is significant.

Table 6 Results of ablation experiments on the imbalanced diabetes dataset
Full size table

The result of \(CDR{\text {-}}Detector_{rep{\text {-}}}\) is slightly better than \(CDR{\text {-}}Detector_{pre{\text {-}}rep{\text {-}}}\), indicating that pre-training can play a role on the few-shot and imbalanced dataset, but its effectiveness is relatively limited. However, the role of pre-training will be highlighted when faced with an ultra-small dataset (less than 100 data samples). Compared with the results on the hypertension dataset shown in Table 5, the result of \(CDR{\text {-}}Detector_{pre{\text {-}}}\) is dropped more than that of \(CDR{\text {-}}Detector\) on the imbalanced diabetes dataset. Therefore, pre-training based on single-visit data is also indispensable for chronic disease risk prediction based on few-shot and imbalanced EHR data.

The proposed CDR-Detector achieves the best result, indicating once again that pre-training based on single-visit data and the dual experience replay strategy are effective and compatible for chronic disease risk prediction based on few-shot and imbalanced EHR data.

Fig. 3
Fig. 3
Full size image

Loss curves on the imbalanced hypertension dataset

Fig. 4
Fig. 4
Full size image

Loss curves on the imbalanced diabetes dataset

Figures 3 and 4 give the loss curves of the proposed CDR-Detector and three variants when they learned on the imbalanced hypertension and diabetes datasets. It can be seen from Figs. 3 and 4, the dual experience replay strategy can help the model select representative samples and accelerate model convergence on the few-shot and imbalanced EHR data. The loss curves of both \(CDR{\text {-}}Detector_{pre{\text {-}}rep{\text {-}}}\) and \(CDR{\text {-}}Detector_{rep{\text {-}}}\) are stable, indicating that these two variants are difficult to converge on the few-shot and imbalanced EHR data. Therefore, their results are the worst as show in Tables 5 and 6. The loss curves of both \(CDR{\text {-}}Detector_{pre{\text {-}}}\) and CDR-Detector effectively converge on two datasets, so their results are similar and significantly better than those of \(CDR{\text {-}}Detector_{pre{\text {-}}rep{\text {-}}}\) and \(CDR{\text {-}}Detector_{rep{\text {-}}}\). This once again indicates that the dual experience replay strategy is the most important optimization in this study.

Pre-training based on single-visit data can help the model alleviate overfitting. As shown in Fig. 3, \(CDR{\text {-}}Detector_{pre{\text {-}}}\) appears to converge better than CDR-Detector on the hypertension dataset. Its loss curve tends to flatten out after a rapid decline. But the overly gentle loss curve poses a risk of overfitting. Therefore, the result of \(CDR{\text {-}}Detector_{pre{\text {-}}}\) is slightly worse than that of CDR-Detector, as shown in Table 5. Figure 4 shows that, CDR-Detector can converge best on the diabetes dataset. Therefore, the result of CDR-Detector is the best, as shown in Table 6.

Conclusion and future work

In real-world scenarios, people’s medical habits and low prevalence of diseases often lead to few-shot and imbalanced longitudinal EHR data, which cannot meet the modeling needs of existing chronic disease risk prediction models. Few-shot learning and data imbalance have become urgent challenges in current studies on chronic disease risk prediction based on EHR data. To solve these challenges, this study combines EHR based pre-training with deep RL to develop a novel chronic disease risk prediction model called \(CDR{\text {-}}Detector\). We expand the application of EHR based pre-training and propose a self-adaptive EHR based pre-training model with two new pre-training tasks to solve the challenge of few-shot learning. We also apply EHR based deep RL to chronic disease risk prediction and develop a dual experience replay mechanism to innovate data imbalance solutions based on deep RL. The experiment results prove that the proposed \(CDR{\text {-}}Detector\) can effectively realize chronic disease risk prediction based on few-shot and imbalanced longitudinal EHR data.

In the future, we will further optimize the model to improve its accuracy and robustness. We will also conduct research on incremental learning of the model. All of these efforts will make the proposed model practical and help to promote the widespread application of EHR based disease risk prediction.