1 Introduction

The widespread use of smart mobile devices today generates a large amount of data, and it is difficult for traditional machine learning techniques to effectively perform centralized data processing from the distributed data of individual mobile devices, while distributed machine learning requires data exchange during training, which poses a great risk of data privacy leakage. In response, Google introduced a new distributed machine learning framework called FL to improve device performance by training data on individual devices [1] and has been used by many large companies and organizations for distributed machine learning tasks on thousands to millions of user devices [2].

The advantage of federated learning is that the raw data on the local devices are independent throughout the process, which not only effectively protects the data privacy of each device, but also reduces the communication cost of data transmission [3]. However, the process of federated learning also faces many problems; the data between multiple devices are usually unbalanced and non-independently homogeneously distributed, the size of the data has different specifications, and there are differences between devices in terms of computational power and resource availability, which can affect the local model trained by each device and thus the efficiency of global model convergence.

Despite the significant advantages of federated learning from a privacy-preserving perspective, some pressing issues in federated learning still need to be addressed. The most important issue to be addressed is the problem of data heterogeneity and resource constraints on edge devices. Since the training data are distributed in a highly heterogeneous manner across multiple devices, and each training device has limited resources such as arithmetic power and network bandwidth, the two aspects of Systems heterogeneity and Statistical heterogeneity [4] not only negatively impact the training performance of the model in terms of accuracy and convergence, but also lower resource utilization affects the training speed of the model. In addition to training performance issues, another equally important issue is fairness. Although federated learning can play a role in protecting data privacy, there will be failures or malicious damage to the trained nodes during the training process, so it is also necessary to ensure normal training of federated learning.

In order to accomplish efficient and fair federated learning, numerous methods have been proposed and studied to enhance efficiency and diminish costs. Among these methods, q-FedAvg has shown promise as a lightweight and scalable distributed algorithm that considers important features in federated learning, such as communication efficiency and the low participation of devices, which can ensure fairness based on device contribution [5]. This algorithm can also reduce communication costs by condensing local datasets [6]. In addition, an effective and fair scheduling algorithm has been proposed to determine device contribution based on its dataset and resource utilization [7]. The decentralized and privacy-protecting nature of blockchain [8] coincides with the original intention of federated learning, so many studies have introduced third-party blockchain networks into FL systems to improve the security of the model at the expense of additional mining latency consumption [9,10,11,12,13]. Specifically, a blockchain-assisted federated learning framework has been proposed to optimize the times required between learning and mining in order to achieve efficient learning performance [13]. Although some of the approaches above have been effective in improving training efficiency, resource utilization, and security, efficient, fair, and secure federated learning in resource-constrained situations has not been thoroughly investigated.

In this study, we aim to develop an effective selection scheme for multi-attribute decision-making in federated learning. We achieve this by combining the TOPSIS [14] and VIKOR [15] methods. While TOPSIS serves as an objective evaluation model, VIKOR focuses on ranking alternatives closer to the ideal solution. We also incorporate an incentivization mechanism that encourages customer participation and enhances training outcomes to ensure optimal results [16]. Integrating the collective interests and individual regret values from the VIKOR framework into the TOPSIS method, we enhance its efficacy and applicability in evaluating alternative solutions. Our primary objective is to achieve equitable and efficient model training while promoting rational resource allocation. In this scheme, we aim to enhance the overall performance of federated learning by considering multiple attributes and making well-informed decisions. By utilizing the synergistic effect of VIKOR and TOPSIS, we optimized the selection process and comprehensively evaluated the equipment based on factors such as training results, available resources, dataset size, and historical contribution. By integrating incentive mechanisms, we have determined the best option for devices to participate in global model updates. Then, after each round of global updates, resources are reassigned to each device based on the training results of the previous round in order to determine a model that exhibits excellent accuracy and resource utilization and maintains fairness throughout the training process.

The main contributions can be summarized as follows:

  • In this paper, we propose a dynamic global aggregation mechanism based on multi-criteria decision making, where the dataset size, arithmetic power, historical contribution to participation in aggregation and loss value per round of a device are used as evaluation metrics. Then, the model is used to make multi-criteria decisions for all devices as a basis for selecting devices to participate in the blockchain-assisted global model of aggregation.

  • In this paper, we design an improved scoring model based on VIKOR and TOPSIS. The distance formula in our model considers the relative importance of each factor’s positive and negative ideal solutions, which allows more flexibility to derive the optimal solution compared to the Euclidean distance.

  • In this paper, we propose an adaptive resource allocation scheme based on the proportional scores derived from comprehensive evaluations. By combining incentive mechanisms, we redistribute resources, including computing power and bandwidth, based on the scores obtained in each round. The flexible resource allocation adjustment for computing power and bandwidth can accelerate model training and improve communication efficiency during aggregation.

  • By comparing the simulation experimental results with existing studies on federated learning algorithms, multi-criteria evaluation methods, and resource allocation methods, the performance of the proposed method is verified, and it has significant advantages in terms of both fairness and efficiency.

The rest of the paper is organized as follows, Sect. 2 details the work related to federated learning and resource allocation. Then, we model the integrated scoring and resource allocation and describe it in detail in Sects. 3, and 4 gives the results of our experimental setup and comparison experiments. Section 5 concludes the paper.

2 Related Work

2.1 Federated Learning

Numerous methods have been proposed in comparable literature for improving the learning efficiency of Federated Learning in the presence of heterogeneous client data distribution. The authors recommend a self-balancing FL framework that addresses class imbalance through the utilization of data augmentation and downsampling of local data [17]. After the completion of the FL process, each client conducts further personalization to a locally enhanced class-balanced dataset [18]. The authors suggest applying a client selection mechanism to choose a subset of participating devices for each training round, thus reducing bias introduced by non-independent identically distributed data to achieve high accuracy while minimizing the number of required communication rounds [19]. An investigation conducted by recent studies estimates the local class distribution by comparing the similarity between the local gradient updates that are submitted to the FL server and gradients inferred from a balanced proxy dataset that resides on the server. This method is useful in selecting the subset of devices with minimal class imbalance [20]. A flexible social scene model has recently been developed based on federated learning to enhance its performance while providing robust privacy protection. The algorithm is specifically designed to improve the accuracy of social scene recommendations and protect user data privacy [21]. To address drift caused by data heterogeneity, regularization is implemented between global and local models [22,23,24].

The problem of poor convergence performance caused by heterogeneous data with limited resources needs to be addressed in the context of Federated Learning since not all devices participating in model training are available for model aggregation. A selection algorithm is recommended to randomly select maximum possible user devices without violating resource constraints [25]. Amiri et al. [26] utilize scheduling of devices for model training based on channel conditions and the importance of local model updates. The utilization of both adaptive update frequency and selection of user devices per round is incorporated into the Federated Learning process through resource utilization during device training to achieve optimal utilization of limited resources and ensure fairness in learning models among devices with different data distribution and computational power [7]. The authors suggest a decentralized optimization scheme for partial computation offloading and resource allocation that is based on deep reinforcement learning. This scheme optimizes latency, energy consumption, bandwidth, privacy, and security cost [27]. The authors proposed personalized Federated Learning, which introduces a novel two-tier optimization framework for optimization of personalized models. And the process of optimizing personalized models is separated from learning global models to optimize personalized models in parallel with low complexity [28]. The authors introduce blockchain technology and suggest a blockchain-assisted Federated Learning framework that substitutes the role of the central server, responsible for aggregation in conventional Federated Learning, by optimizing the number of times between learning and mining to efficiently achieve learning performance in a limited time [13].

2.2 Multi-criteria Decision Making

The authors proposed an IoT-oriented multi-objective resource allocation method that utilizes multi-criteria decision-making and sequential preference techniques with similarities to ideal solutions to optimize multi-objective resource allocation strategies [29]. The present paper depicts the development of a novel risk assessment system, known as FMRES. FMRES utilizes fuzzy logic techniques and multi-criteria decision-making methods to evaluate cold chain risks in supply chain firms. A best-wrong method is incorporated into the system to provide a smart and adaptive approach to risk quantification [30]. The authors developed a Multi-Criteria Decision-Making framework utilizing the Fuzzy Delphi Method to standardize the evaluation criteria of machine learning-based Intrusion Detection Systems. The proposed framework includes the development of an evaluation decision matrix based on the intersection of standardized evaluation criteria and a list of ML-based IDS’s utilized in Federated Learning architecture for standardizing and benchmarking Internet of Medical Things applications [31]. Liu et al. proposed using Vague set and improved DPoS negotiation mechanism to enhance the realism and reasonableness of node voting [32]. They also proposed a replica placement algorithm based on the entropy weight TOPSIS method, called TS-REPLICA [33]. In order to evaluate the safety of urban water supply, the authors proposed a comprehensive evaluation model based on VIKOR and TOPSIS models. The model involved performing grey correlation analysis based on traditional TOPSIS, correcting TOPSIS evaluation results by comparing the geometric similarity of positive and negative ideal solutions with actual evaluation objects, and combining the improved TOPSIS method with the VIKOR method [34]. Some studies have employed TOPSIS in the PBFT consensus mechanism and introduced a technique to elect master nodes through a probabilistic linguistic term set [35].

3 Improved Scoring and Resource Allocation Model Design Based on TOPSIS and VIKOR

The proposed model and its theoretical analysis are introduced in this paper, with the primary symbols used in the model listed in Table 1.

Table 1 Summary of main symbols
Full size table

3.1 Model Design

The TV-FedAvg system comprises a set of N devices, each initially endowed with a unique computational power and measured via CPU cycles per second. In this distributed system, individual devices are responsible for local training and participation in consensus for blockouts, with role transition being executed as follows. First, each device trains a local model then disseminates the resulting model to the entire blockchain network via broadcast. Second, network devices perform consensus blockouts headed by the leader nodes that have already been elected. Once most devices verify a newly generated block, the verified model remains immutable. Without central servers, each device affects global aggregation by employing data from the validated block to update its local model and continue the training process. The overall model flow is shown in Fig. 1.

Fig. 1
Fig. 1
Full size image

Our model involves preloading each device with the model, with subsequent blocking out and uploading onto the chain following consensus

In particular, we denote the n-th device’s local dataset as \({D_n}\) with size \({|D_i|}\). For the dataset \({|D_n|}\) of device n, the loss function associated with that device can be expressed as:

$$\begin{aligned} \begin{aligned} F_n(\omega )\triangleq \frac{1}{|D_n|}\underset{i\in {D_n}}{\sum }F_i(\omega ) \end{aligned} \end{aligned}$$

\({f_i(\omega )}\) is the loss function defined on the parameter vector \({\omega }\) for each data point i in the dataset. We denote the entire set of data sets by \(D\) \(\triangleq \) \(\sum ^N_{n=1}|D_n|\).

The aggregation process for TV-FedAvg comprises the selection of a subset of devices to participate in global aggregation in each round, followed by the reallocation of resources for the subsequent round. To make well-informed decisions on device selection, a combination of locally trained loss functions linked to each device, dataset size, computational power, and the time required to participate in global aggregation is considered. Subsequently, TV-FedAvg utilizes a scoring model based on VIKOR and TOPSIS, thereby enhancing the prioritization and selection of each device for global aggregation.

In particular, the following factors will be considered for the i-th device: the value of the loss function \(F_i(\omega )\) obtained through local training, the size of the local data \(|D_n|\), The current amount of computing power possessed \(f_i\), and the age of update (AoU) \(t_i\), which is the number of rounds from the last time the device participated in the global aggregation to the present.

For n devices to be evaluated based on four evaluation indicators, a forward-looking matrix can be constructed as follows:

$$\begin{aligned} X=\left[ \begin{matrix} F_{\max } (\omega )-F_1 (\omega ) &{}\quad D_1 &{}\quad f_1 &{}\quad t_1 \\ F_{\max } (\omega )-F_2 (\omega ) &{}\quad D_2 &{}\quad f_2 &{}\quad t_2 \\ \vdots &{}\quad \vdots &{}\quad \ddots &{}\quad \vdots \\ F_{\max } (\omega )-F_n (\omega ) &{}\quad D_n &{}\quad f_n &{}\quad t_n \\ \end{matrix} \right] =\left[ \begin{matrix} x_{11} &{}\quad x_{12} &{}\quad x_{13} &{}\quad x_{14} \\ x_{21} &{}\quad x_{22} &{}\quad x_{23} &{}\quad x_{24} \\ \vdots &{}\quad \vdots &{}\quad \ddots &{}\quad \vdots \\ x_{n1} &{}\quad x_{n2} &{}\quad x_{n3} &{}\quad x_{n4} \\ \end{matrix} \right] \end{aligned}$$

As \(F_i (\omega )\) is a minimal type index, it needs to be converted into a maximal type index:

$$\begin{aligned} \begin{aligned} F_{\max }(\omega )-F_i(\omega ) \end{aligned} \end{aligned}$$

Then, the matrix normalized to it is denoted Z. Each element in Z:

$$\begin{aligned} \begin{aligned} z_{ij}=\frac{x_{ij}}{\sqrt{\sum ^n_{i=1}x^2_{ij}}} \end{aligned} \end{aligned}$$

Standardized matrix:

$$\begin{aligned} { Z=\left[ \begin{matrix} z_{11} &{}\quad z_{12} &{}\quad z_{13} &{}\quad z_{14} \\ z_{21} &{}\quad z_{22} &{}\quad z_{23} &{}\quad z_{24} \\ \vdots &{}\quad \vdots &{}\quad \ddots &{}\quad \vdots \\ z_{n1} &{}\quad z_{n2} &{}\quad z_{n3} &{}\quad z_{n4} \\ \end{matrix} \right] } \end{aligned}$$

Next, we determine the ideal solution, group utility, and individual regret valueswe [34]. We use the absolute value method to determine the ideal solutions:

$$\begin{aligned}{} & {} r_j^+=1, j=1,2,\dots ,n\\{} & {} r_j^-=0,j=1,2,\dots ,n \end{aligned}$$

where \(r_j^+\) is the positive ideal solution for the j-th evaluation index and \(r_j^-\) the negative ideal solution for the j-th evaluation index.

Calculate the group utility \(S_i\) and the individual regret value \(R_i\) using the positive and negative ideal solutions as references, respectively:

$$\begin{aligned}{} & {} S_i^+=\sum \limits _{j=1}^{4}w_j\frac{r_j^+-z_{ij}}{r_j^+-r_j^-}, i=1,2,\dots ,n\\{} & {} R_i^+={\max } w_j \frac{r_j^+-z_{ij}}{r_j^+-r_j^-}, i=1,2,\dots ,n\\{} & {} S_i^-=\sum \limits _{j=1}^{4}{w_j \frac{z_{ij}-r_j^-}{r_j^+-r_j^-}}, i=1,2,\dots ,n\\{} & {} R_i^-={\min } w_j \frac{z_{ij}-r_j^-}{r_j^+-r_j^-}, i=1,2,\dots ,n \end{aligned}$$

The weight corresponding to the j-th evaluation indicator is indicated by \(w_j\). A more considerable weight value suggests a higher priority for that evaluation metric. A lower value of loss function \(F_i (\omega )\) indicates higher accuracy, which makes the device more suitable to participate in and facilitate global model aggregation for faster convergence, thus assigning it a higher weight. Similarly, if a device has more local datasets, models developed by it can be more comprehensive; hence, it is preferred for selection in global aggregation. The third metric, \(f_i\), represents arithmetic resources of the device. Higher arithmetic power makes the device more competitive for selection by accelerating the training time. Finally, the value \(t_i\) prevents some devices from being excluded from global aggregation for a long time. Local devices with more training data, higher computational power, more extended parameter sending waiting time, and fewer resource demands should have a higher probability of being picked. Therefore, different weights must be assigned to these four evaluation metrics based on their varying degrees of influence.

Calculate the distance between the i-th device \((i=1,2,\dots ,n)\) and the minimum value:

$$\begin{aligned} \begin{aligned} D_i^+=\sqrt{\sum ^4_{j=1}w_j[a_1(S_i^+-z_{ij})^2+a_2(R_i^+-z_{ij})^2]} \end{aligned} \end{aligned}$$

Calculate the distance between the i-th device \((i=1,2,\dots ,n)\) and the maximum value:

$$\begin{aligned} \begin{aligned} D_i^-=\sqrt{\sum ^4_{j=1}w_j[a_1(S_i^--z_{ij})^2+a_2(R_i^--z_{ij})^2]} \end{aligned} \end{aligned}$$

where \(a_1\) and\( a_2\) denote the weight assignments of group benefits and individual regret values, \(a_1+a_2=1\).

The scoring formula for the TOPSIS scoring model:

$$\begin{aligned} \begin{aligned} s=\frac{(x-\min )}{(\max -\min )} \end{aligned} \end{aligned}$$

We modified the formula slightly as follows:

$$\begin{aligned} \begin{aligned} s=\frac{(x-\min )}{(\max -x)+(x-\min )} \end{aligned} \end{aligned}$$

Then, the score of the i-th \((i=1,2,\dots ,n)\) device without normalization can be calculated as follows:

$$\begin{aligned} \begin{aligned} S_i=\frac{D_i^-}{D_i^++D_i^-} \end{aligned} \end{aligned}$$

It is obvious that \({0 \le S}_i \le 1\), And the smaller \(D_i^+\) is, the larger \( S_i\) is, i.e., the closer it is to the maximum value.

Next, the unnormalized score of the i-th\( (i=1,2,\dots ,n)\) device is normalized:

$$\begin{aligned} \begin{aligned} m_i=\frac{S_i}{\sum ^n_{j=1}S_j} \end{aligned} \end{aligned}$$

Based on the normalized scores of each device, we can rank the devices and utilize these scores to reward or penalize them. This serves as a motivation for devices to actively participate in training. Additionally, after reaching consensus during the block validation process in a blockchain, we can optimize the allocation of cloud resources to devices based on their scores. The algorithm for local training and global aggregation is presented in Algorithm 1 and Algorithm 2, respectively, which is a fundamental framework for dynamically selecting devices and allocating resources based on the TOPSIS-VIKOR scoring model for the next round. Initially, all devices commence the learning process by initializing the model parameters \(\omega \) along with the number of iterations \(\tau \) for each round of local training (lines 7–8). Subsequently, the devices perform \(\tau \) iterations for local training and personalization after receiving the data. Following that, every device disseminates a report on local updates and resources utilized (as embodied in Algorithm 2) for TOPSIS-VIKOR scoring (line 13). After the TOPSIS-VIKOR scoring model computes a composite score for each device at the conclusion of that round of local training, the \(\delta \)% top-scoring devices are selected to participate in global aggregation, and the global model is updated based on the average value of these scores (rows 14–17). The devices picked for global aggregation have their AoU value reassigned to 1, whereas for the non-participating devices, the value of their AoU is incremented by one (line 22). In the next round, the proportion of arithmetic resources allocated to each device is determined by normalizing the score(line 25).

Algorithm 1
Algorithm 1
Full size image

The process of global aggregation

Algorithm 2
Algorithm 2
Full size image

Local personalized training for each device

3.2 Model Resource Consumption Analysis

In our federated Learning model, training aims to minimize the global loss function \(F(\omega )\) and discover \(\omega ^{*} ={\textrm{argmin}} F(\omega )\). Therefore, model convergence is attained when the upper limit of the following equation is found:

$$\begin{aligned} \begin{aligned} F(\omega ^f )-F(\omega ^*) \end{aligned} \end{aligned}$$

The function for global aggregation, derived from the selection scheme based on scoring can be expressed as follows:

$$\begin{aligned} \begin{aligned} F(\omega )\triangleq \frac{\sum ^p_{n=1}D_n F_n(\omega )}{\sum ^p_{n=1}D_n} \end{aligned} \end{aligned}$$

where P is the number of device sets selected in each round after scoring by an improved scoring model based on VIKOR and TOPSIS.

The resource consumption of the model can be divided into two parts: resource consumption for model training and consensus out blocks. In this context, \(\tau \) represents the number of local model iterations per round. In contrast, T represents the total number of local iterations required for the device to complete its learning process. Additionally, we use K to denote the total number of global aggregations performed throughout the learning process. Assuming that T is an integer multiple of \(\tau \), we can compute the value of \(K\tau =T\).

Local training: local training for each device consists of \(\tau \) iterations, and the training time consumed for each training iteration on the i-th device [36]:

$$\begin{aligned} \begin{aligned} \alpha _i\triangleq \frac{|D_i|\rho }{f_{ci}} \end{aligned} \end{aligned}$$

where \(\rho \) denotes the CPU cycles required to train a sample, and \(f_{ci}\) denotes the CPU cycles per second per device.

Block generation: The generation rate of PBFT consensus-based blocks is determined by the blockchain network’s total computing power and bandwidth resources. The time of block generation in a blockchain network [37]:

$$\begin{aligned} \begin{aligned} \beta \triangleq \frac{P}{\sum ^K_{k=1}[w_kp_k]^{\mu }} \end{aligned} \end{aligned}$$

where \(\mu \in [0,1]\) is the normalized parameter associated with the consistent speed of the PBFT protocol, and \(p_k\) denotes the total computing power and bandwidth in the corresponding blockchain network.

Since the dataset size of each device is different and the arithmetic power is reallocated to each device in each round in our design, the time for each device to iterate locally \(\tau \) times per round is \(\alpha _i\tau \), since the time consumed by consensus out of blocks per round is \(\beta \), we use the sum of the time consumed by training and consensus in all rounds as the performance evaluation metric:

$$\begin{aligned} \begin{aligned} t_{sum}=\sum _{k=1}^K{\alpha _{i,k,max}\tau +K\beta } \end{aligned} \end{aligned}$$

where \(\alpha _{i,k,max}\) refers to the device’s speed with the fastest local training speed in k rounds.

4 Experimental Analysis

In this section, we present the evaluation of our model through a series of experiments analyzing the machine learning model’s performance, resource efficiency, and fairness, among others. We demonstrate that, in conjunction, all these metrics highlight how our model achieves superior fairness and efficiency. Specifically, by utilizing device selection and resource reallocation in TV-FedAvg, our experiments show that we improved the performance of our model while increasing resource utilization.

4.1 Experimental Setup

Our experiments use a prototype system and a simulated large-scale environment. We allocate computational power to each device as a cloud resource when conducting experiments in a resource-constrained environment. We simulate a realistic scenario of non-independent and homogeneous data distribution among devices by randomly assigning the complete dataset to 50 devices (N=50), and all devices train the model based on the local dataset. The experimental architecture diagram is shown in the following Fig. 2.

Fig. 2
Fig. 2
Full size image

The experimental architecture diagram

(1) Datasets: In our experiments, we use three datasets for non-IID settings to assign to devices for experiments

  • MNIST. Standard MNIST handwritten digit recognition dataset containing 10 labels and 70,000 instances, each of which is a handwritten digit of \(28\times 28\) size, labeled from 0 to 9.

  • Cifar-10: Cifar-10 is a dataset for recognizing universal objects and contains RGB color images of 10 kinds: airplanes, cars, birds, cats, deer, dogs, frogs, horses, boats, and trucks. The dataset contains 50,000 training examples and 10,000 images of \(32\times 32\) test examples.

  • Fashion MNIST: Fashion MNIST has 10 different types such as T-shirts, pants, pullovers, dresses, jackets, sandals, shirts, sneakers, bags and ankle boots.

(2) Learning models: We use convolutional neural networks (CNNs) and deep neural networks (DNNs) as learning models.

(3) Parameter settings. In our experiments, in order to simulate the real training situation as much as possible under limited objective conditions, we set the number of devices N=50, the total arithmetic power \(p_1=100\), Total bandwidth \(M=200\), the learning rate \(\eta =0.005\), the global aggregation number \(K=50\), the device selection interval \(\alpha \) is set to 80, the PBFT consensus parameter \(\mu \) is set to 0.5, and the two resource consumption weights of arithmetic power and broadband at consensus are set to 6 and 4, respectively.

(4) Baselines. We compare TV-FedAvg with the following baselines to evaluate their model accuracy performance.

  • FedAvg, in which each participating device uses resources such as fixed computing power and bandwidth for model training.

  • pFedMe [28], where the device uses Moreau envelopes as a regularized loss function to optimize model training in the case of heterogeneous dataset distributions.

  • Per-FedAvg [38], a personalized federated learning algorithm based on meta-learning.

  • TOPSIS [14], a method for ranking a finite number of evaluation objects according to their proximity to an idealized target.

  • VIKOR [15], a multi-attribute decision making method

  • VIKOR-TOPSIS [34], a method for the integrated evaluation of VIKOR and TOPSIS combined.

4.2 Results and Analysis

4.2.1 Model Training Efficiency Analysis

Two sets of experiments were conducted to evaluate the model performance of TV- FedAvg and compared it with other methods such as pFedMe [28], Per-FedAvg [38], VIKOR-TOPSIS [35], etc. Table 2 shows the results of CNN and DNN models trained on the MNIST dataset involving 50 simulation devices with different computing power and non-IID data distributions. We unevenly distribute the MNIST dataset to 50 devices, and distribute computing power equally unevenly to various devices as a cloud resource. Considering the resource-constrained conditions, we need to control the computational cost of training and the communication cost consumed by model aggregation under limited conditions again, and through our experiments, we set the number of global aggregations to 50, with 100 local training iterations per device without return. In particular, we show the loss functions and accuracies obtained by training under different scenarios.

Table 2 Different metric weights trained with DNN and CNN on MNIST
Full size table
Fig. 3
Fig. 3
Full size image

Loss and accuracy of different metric weights trained with DNN and CNN on MNIST

As shown in Table 2, in order to achieve the best training effect of the model, we investigated the influence of different evaluation metrics on the model training in the TV-FedAvg model. Considering that among the four influencing factors of loss function, dataset, arithmetic power, and participation aggregation time, the loss function and dataset size, which directly represent the training effect, have a relatively obvious influence on the training effect of the model and the arithmetic power and participation aggregation time have less influence, we set the proportional weights as [5,3,1,1], [4,4,1,1], [6,2,1,1], respectively, and use these three proportions to perform the experiments. The following are the training results of the MNIST dataset trained with DNN learning model and CNN learning model for the TV-FedAvg three scale weights, respectively. The three colored dashes in Fig. 3 reflect the training results when the weights are [5,3,1,1], [4,4,1,1], [6,2,1,1], respectively. Whether the training is performed with CNN or DNN, the convergence speed and convergence effect of both accuracy and loss function from the beginning to the global aggregation 50 times clearly show the convergence of the loss function, dataset, arithmetic power, and The weights of the four influencing factors, age, are set to [5,3,1,1] to make the training of the model optimal. Thus, it can be concluded that during the evaluation of training effectiveness, particular emphasis should be placed on the post-iteration loss function, followed by consideration of the dataset characteristics. Nonetheless, it is essential to avoid assigning an excessively low weight to the dataset, as there is a possibility that some iterations with promising performance are a result of overfitting and may not be applicable to the entire training process.

Apart from that, we conducted four additional experiments with different client quantities: 20, 30, 50, and 100. These experiments were performed using the optimal weight allocation scheme mentioned earlier. Table 3 displays the results of training DNN and CNN on both the MNIST datasets for each client quantity. As shown in the table, overall, as more and more devices participate in federated learning, the effectiveness of joint training also improves. However, there is minimal variation in the results between experiments with 100 clients and those with 50 clients. This is likely due to the fact that, in our simulated experiments, as the number of clients increased, the overall dataset did not increase proportionally. Consequently, each client received a smaller portion of the dataset, which did not lead to significant improvements in training results. It is worth noting that as the number of clients increased, the gap between the losses from poorly performing devices and the average and best devices diminished. So we set the experimental simulation to 50 clients to analyze the training performance of the model.

Table 3 Different metric weights trained with DNN and CNN on MNIST
Full size table
Table 4 Comparison of training results of MNIST dataset in CNN and DNN using TV-FedAvg with FedAvg, pFedMe, Per-FedAvg, TOPSIS, VIKOR and VIKOR-TOPSIS
Full size table

As mentioned earlier, we also compared the strategies with weights [5,3,1,1] with the six baselines in terms of model loss and accuracy, and the loss and accuracy values of the CNN and DNN models are shown in Table 4 for 100 times per round of local training and 50 times per round of global training, respectively. The variation of loss and accuracy values for TV-FedAvg and global training up to 50 times for each baseline are shown in Fig. 4, respectively. Through the experiments, the convergence speed of TV-FedAvg is improved by 20\(\%\) compared to the other baselines, and the convergence effect is improved by nearly 5\(\%\). From the figure, we can intuitively see that the accuracy of TV-FedAvg is significantly higher than that of FedAvg, pFedMe and Per-FedAvg, and TV-FedAvg also presents a clear advantage compared with several other decisions of comprehensive evaluation. In general, the training convergence speed of TV-FedAvg is significantly higher than the convergence speed of the baseline, and its training results are also significantly better than the training results of each baseline. We also conducted experiments using Fashion MNIST dataset and Cifar-10 dataset using the CNN model, also compared with the six baselines, and the experimental results are shown in Table 5. It can be seen from both the training loss values and accuracy rates that the results obtained by training with the TV-FedAvg method are better than the results obtained by training with the other six methods.

4.2.2 Resource Efficiency Analysis

In this paper, our proposed method aims to enhance the training efficiency of models in resource-constrained scenarios. We aim to leverage limited computing power and bandwidth as valuable resources on the cloud and dynamically allocate them to each participating device during the training process. By considering the training performance of each device and incorporating various factors, we strive to determine the optimal resource allocation scheme. This approach addresses the limitations of resources and seeks to effectively utilize the available resources, ultimately improving overall training efficiency. Resource efficiency can be intuitively interpreted as increasing the speed of learning models by coordinating the allocation of resources across multiple devices across time. These resources are usually related to computation and communication. Therefore, after a given number of local iterations, we can evaluate the resource efficiency by the performance of the local and global models. According to the previous description, the time \(\alpha \) for local training and the time \(\beta \) for consensus generation of blocks, the time consumed for training: \(t_{sum}=\sum _{k=1}^K{\alpha _{i,k,max}\tau +K\beta }\)

Fig. 4
Fig. 4
Full size image

Comparison of TV-FedAvg with FedAvg, pFedMe, Per-FedAvg, TOPSIS, VIKOR and VIKOR-TOPSIS trained with CNN and DNN using MNIST dataset

Table 5 Comparison of Fashion MNIST and Cifar-10 dataset trained with CNN model with other baselines
Full size table

We conducted experiments for each of the three weighting schemes of TV-FedAvg using two datasets and compared the experimental results with the following two baseline schemes.

  • Top-S Res.Allocation [39].This approach also means that the resources are allocated according to the dataset size, and devices with larger datasets will be provided with more resources for training the model.

  • Average Res. Distributes resources evenly to individual devices based on the number of devices.

Figure 5 below illustrates the advantage of our scheme concerning reasonable resource allocation over the course of progressively increasing the number of global training iterations in comparison to the other two baselines. While the training time for the two baselines remains constant due to fixed allocation of resources, our scheme demonstrates a measurable decrease in training time over the course of global training. The Average Res. scheme, in which devices with larger datasets struggle to obtain ample resources for training, results in increased training time consumption. Conversely, the Top-S Res. scheme allocates resources based solely on the size of a device’s dataset, leading to instances where insufficient resources are allocated to devices that have demonstrated better training performance.

Fig. 5
Fig. 5
Full size image

Comparison of training time consumption of TV-FedAvg and other resource allocation schemes, where a is the training of MNIST dataset and b is the training of Cifar dataset

4.2.3 Fairness Analysis

We conducted three sets of experiments to evaluate the fairness achieved by TV-FedAvg. In each experiment, we varied the number of participating clients to 20, 30, 50, and 100, respectively, and implemented the optimal weight allocation scheme mentioned earlier. For example, Fig. 6a, b illustrates the uneven distribution of the mnist and Cifar-10 datasets into 20 subsets, simulating the scenario where different devices possess non-independent and non-identical datasets. Table 6 presents the results of training CNN on the MNIST dataset for the three different client quantities. It includes the average loss function of all devices, the average loss of the top 10\(\%\) devices with the most data, the average loss of the worst 10\(\%\) devices, and the variance in loss across all user equipment (UE). As shown in Table 6, as the number of clients increases, devices with poorer data distribution exhibit less disparity in training loss compared to the average loss and the loss of the best-performing device. Additionally, from a variance perspective, the training results become more stable.

Fig. 6
Fig. 6
Full size image

Data set assignment, where plot a is the MNIST data set and plot b is the Cifar-10 data set

Table 6 Training results for different client quantities
Full size table
Fig. 7
Fig. 7
Full size image

Number of experimental aggregation engagements for the Cifar-10. a and b show the results of the TV-FedAvg scheme compared with the other two baselines trained using CNN and DNN

In addition, we conducted a comparative analysis between our TV-FedAvg model and two baseline models: a TOPSIS-based scoring model and the original FedAvg prototype experiment. As shown in Fig. 7, when considering 20 clients as an example, the TV-FedAvg model exhibited a significant discrepancy in dataset sizes between devices ranging from the eighth to the twentieth. These devices had considerably smaller datasets compared to earlier devices. However, despite this imbalance, the devices with smaller datasets still achieved outstanding performance through local dataset training. Consequently, they had increased opportunities to contribute to the global aggregation during the aggregation process. In contrast, in the other two baselines, devices with smaller datasets demonstrated limited participation in global aggregation. Remarkably, the TV-FedAvg model provided more chances for devices with smaller yet high-quality datasets to actively participate in the global aggregation process, resulting in a substantial improvement in the fairness of the model.

5 Conclusion

In this study, we propose the TV-FedAvg method, a secure, fair, and efficient model for Federated Learning in resource-constrained environments that integrates the training and resource reallocation process in each device. This approach overcomes the problem of poor training performance with limited resources and improves the Federated Learning process through PBFT consensus algorithm at the time of global model aggregation privacy-preserving capabilities. To assess the learning performance of TV-FedAvg, we not only analyze the model convergence theoretically, but also perform experiments to simulate real-world data distribution. Results demonstrate that locally trained models that participate in convergence and resource reallocation can improve model performance while increasing resource utilization in the case of resource constraints. We achieve this by combining evaluation metrics, including the effect of training per round and the resources owned by each device.

Tv-FedAvg coordinates model learning based on multiple relevant factors, including data sets, arithmetic power, and bandwidth. Key challenges addressed and solved by this framework have potential applications in various federated learning scenarios in current IoT infrastructure. Due to the utilization of PBFT consensus for decentralized security assurance during global aggregation, nodes in the network are required to expend computational power and bandwidth for consensus during each global aggregation. This leads to a considerable increase in resource consumption costs for our solution, despite its enhancement of security and training effectiveness. While we employ a multi-attribute decision-making approach to reduce resource consumption costs during training through resource allocation and dynamic selection, we still sacrifice a substantial amount of resources in exchange for security, we hope to explore further model selection and algorithmic design, in the more realistic and demanding real-world conditions, including a more complex distribution of mobile devices, the emergence of faults in the accidental situation, the more constrained availability of resources and a highly heterogeneous distribution of data, and still be able to take account of the performance of model training and resource utilization.