Introduction

Citations have been extensively studied since the very beginning of scientometrics. It has made possible not only to analyze science through scientific literature but also to know the particular characteristics of this metric, such as its skewed distribution (Price, 1965; Seglen, 1992) or life cycles (Burton & Kebler, 1960). Furthermore, there are also several metrics derived from citations and directly oriented to the evaluation of scientific performance, such as the Journal Impact Factor. However, the advent of altmetrics has altered this citation-dominated paradigm, with the emergence of a wide range of social media metrics (Priem et al., 2010). Since the first moment, attempts have been made to find a relationship between altmetrics and citations without success (Thelwall et al., 2013). This is a completely different phenomenon, marked by a wide variety of sources, each one with a different nature, and above all the evanescence of these mentions, which is not the case with citations.

Altmetrics englobe a broad set of metrics derived from the multiple interactions produced around science in social media (Díaz-Faes et al., 2019; Wouters et al., 2019). This makes it possible to capture science-related interactions outside of the scientific realm, but at the same time they are completely platform-dependent. That is why this activity can be faded out with the cessation of the activity or the discontinuation of the platform itself, e.g., Google+. In addition to the variety of sources is the fact that altmetrics are extracted from each of their platforms, with multiple strategies in place. Altmetrics data aggregators such as Altmetric.com or Crossref Event Data differ greatly for this reason, offering different levels of coverage for the same source (Ortega, 2018; Zahedi & Costas, 2018). There are also differences between countries (Torres-Salinas et al., 2022). In addition to all these limitations, there is also the volatility of the mentions of some of them. On Twitter, not only the stability of tweets mentioning scholarly outputs has been studied (Fang et al., 2020), understood as their unavailability over time, but also how such tweets become available again (Fang et al., 2022), similar phenomenon has also been identified with Wikipedia references (Arroyo-Machado et al., 2020).

Altmetrics is a complex and liquid phenomenon that is further complicated by the creation of proposals such as the Altmetric Attention Score (AAS) (Altmetric Support, 2021), a weighted metric that aggregates mentions from different sources and whose reading and interpretation is confusing (Thelwall, 2020a). This metric has not only been widely criticized for the lack of empirical criteria in the weighting assignment (Gumpenberger et al., 2016; Mukherjee et al., 2018), but the reproducibility of its calculation has also been questioned (Ortega, 2019). Nevertheless, this metric has a strong presence through the Altmetric.com badges included in journals, databases and repositories, and is also widely used as a social attention proxy, especially in Health Sciences (Kolahi et al., 2021). That is why, without disregarding its limitations, it cannot be ignored. Despite all the research done on the Altmetric Attention Score, there are no studies that have analyzed the fluctuation of its value. There are only proposals that analyze the accumulation of mentions from different sources (Fang & Costas, 2020), but without considering the volatility of these and their impact on the Altmetric Attention Score.

This potential problem with altmetrics fluctuations must be taken into consideration within the new contexts of evaluative bibliometrics that have emerged following manifestos such as DORA (2012). Thus, there is a call for metric diversity to cover different dimensions of the impact of research results (Aubert Bonn & Bouter, 2023). In this new scenario, altmetrics play a pivotal role, even entering international agendas for this new generation of research evaluation (European Commission et al., 2017). Although their limitations are well known (Thelwall, 2020b), and can be resolved by adhering to basic principles (Thelwall, 2020a), there are few proposals and practical applications. Therefore, the emergence of the so-called ‘evaluative altmetrics’ is promising in terms of their ability to contextualize the attention research receives, potentially going beyond mere mention counts and paying attention to issues like audiences or engagement (Arroyo-Machado & Torres-Salinas, 2023). However, the starting point for these analyses lies in the data, with commercial aggregators like Altmetric.com and PlumX leading the way (Karmakar et al., 2021). Therefore, it is more necessary than ever to understand aspects such as the reliability of their data and derived indicators for responsible use.

The main objective of this research is to investigate, over time, the performance of the Altmetric Attention Score (AAS), focusing on tracking its fluctuations and the vanishing of altmetric mentions that may be causing them. To achieve this, we will examine the variations of this value over a year for a set of papers in Information Science & Library Sciences (ISLS) and will also analyze the fluctuations of each of the altmetrics sources involved in its calculation to verify the stability of each and their impact on the AAS. The specific objectives are the following:

  • Objective 1 To assess the impact of fluctuation in the AAS on ISLS publications, specifically determining the number and extent of publications affected.

  • Objective 2 To investigate the contributions of individual altmetrics sources to these fluctuations, aiming to pinpoint the root causes influencing the overall AAS.

  • Objective 3 To conduct a detailed analysis of the stability of altmetrics sources, focusing particularly on specific mentions as opposed to overall metric values, in order to acquire a more nuanced picture of the underlying phenomenon.

The paper is structured as follows: (1) in the methodology section, the process of collecting and analyzing altmetrics data from ISLS papers over a year is described. (2) In the results, we analyze the (i) fluctuation of the AAS at both individual and aggregate levels, (ii) as well as the fluctuations of the various altmetrics sources that contribute to this aggregate metric, and (iii) we delve into the vanishing of metrics to better understand this phenomenon and the stability of altmetrics. (3) We then discuss the results and their implications. (4) Finally, we offer conclusions of the study.

Methodology

In order to overcome the above commented limitations in the study of an altmetric indicator, especially the Altmetric Attention Score (Thelwall, 2020a, 2020b), the sample of publications used is related to the same scientific category and takes into account their different publication dates. On 1st February 2022, bibliographic records of publications indexed in the Web of Science Information Science & Library Science (ISLS) category were retrieved from Clarivate's Incites. Publications in the Emerging Sources Citation Index (ESCI) were included, and the query was limited to the period from 2012 to 2021, focusing on the citable document typologies of articles, letters, and reviews. A total of 73,367 documents were collected. This dataset of ISLS publications was used as the basis for an iterative altmetrics data retrieval process. From February 2022 to February 2023, every 2 weeks, the DOIs of these 73,367 ISLS publications were queried on Altmetric.com to retrieve their Altmetric Attention Score values and altmetrics mentions. Thus, for this dataset of ISLS publications, 25 biweekly snapshots were generated with data downloaded from Altmetric.com.

Table 1 provides an overview of the data used in this paper. In total we have identified 32,581 ISLS publications indexed in Altmetric.com, but only those with an Altmetric Attention Score higher than 0 in the first download of February 2022 and that were present in all 25 snapshots were used, giving a total of 26,474 (36% of the total). The sample we used only considers papers with an AAS of at least 1 in order to be able to observe not only increases but, above all, decreases in this metric over time. Table 1 shows the number of ISLS publications retrieved from Web of Science and their distribution by year of publication, as well as how many of them are indexed in Altmetric.com and those with mentions.

Table 1 Number of papers analyzed per year published in Information Science & Library Science (Web of Science category) with the detail of papers with Altmetric Attention Score
Full size table

Once the data were collected to study possible changes in AAS values, three analyses were conducted. Firstly, the fluctuations of the AAS values have been analyzed. To achieve this, variations of this metric at the level of individual articles over a year and their aggregated evolution during the studied period were explored. The aim was to determine how many publications are affected by these fluctuations, the severity of these variations, and to pinpoint specific moments of interest for identifying the vanishing of mentions. In the last analysis, the percentage increase of the AAS in a specific period compared to the previous 2 weeks was calculated, naming this value the AAS Biweekly Variability Rate (AASBVR). Secondly, the fluctuations of the altmetrics sources influencing the AAS value have been studied, concentrating solely on the active ones,Footnote 1 to identify not only the trends but also the main responsible ones. For this purpose, the previous analysis was replicated at the level of each altmetric source, and the Altmetrics Biweekly Variability Rate (ABVR) was calculated, identical to AASBVR but applied to each altmetric source. Thirdly, the vanishing of specific mentions has been analyzed, thus identifying the origin of the problem for the fluctuations and being able to estimate the extent of this problem by calculating the total of vanished mentions and compromised papers.

This analysis was conducted using R and RStudio. All scripts used for data processing, analysis, and generating visualizations are available on GitHub: https://github.com/Wences91/altmetrics_fluctuations

Results

Estimating the impact of AAS fluctuations on ISLS publications

To begin delving into and understanding this phenomenon, the impact of fluctuations in the ISLS after one year has been estimated. Figure 1 shows the fluctuation in AAS between the first altmetric data collection in February 2022 (x-axis) and the last one in February 2023 (y-axis) for the 26,474 Web of Science documents analyzed. Three possible statuses of the items have been depicted in the graph: (1) on the diagonal (yellow points) are the publications that at both times have the same AAS value; (2) above it are those that have increased (blue points); and (3) below it is those that have decreased (red points). Here we find that only 2309 publications (8.72% of the total) have increased their AAS in February 2023 with respect to February 2022, 4538 (17.14%) have reduced it, and 19,627 have kept it the same (74.14%). It is notable that twice as many publications experience a reduction in their AAS compared to those that see an increase, though it should be considered that this includes works up to 10 years old at the time of data collection. Furthermore, it is important to note that these losses of the AAS do not affect all the impacted papers equally. A total of 1603 publications (6.05% of the total) have experienced a moderate or significant reduction in their value, with changes exceeding 25%, while 495 publications (1.87%) have completely lost their value, dropping to zero. We are therefore faced with an important metric anomaly.

Fig. 1
Fig. 1
Full size image

Plot of Altmetric Attention Score (AAS) values in February 2022 and February 2023 (log–log). All values have been increased by 1 to represent the 495 cases with an AAS of zero

However, the described reduction is not limited to year-over-year differences. Upon reviewing the fluctuations over the study period, it has been observed that up to 6275 papers (23.7% of the total) experienced a decrease in their AAS at some point. Additionally, it was noted that on average, 486 works (1.84% of the total) show a biweekly reduction in their AAS. Yet, this average is not constant; there are two distinct times when a greater number of works are impacted. The first and most pronounced is in mid-June 2022, when a total of 1495 papers (5.65%) saw their AAS values decrease. The next notable period begins in early December of 2022, with 1189 papers (4.49%) affected. Therefore, ISLS papers are susceptible to both short-term and long-term effects, and it is possible to identify specific moments when events may have triggered such reductions. In view of these findings, a more detailed examination of the fluctuations throughout the year is warranted, with an additional focus on the age of the publications to determine whether it constitutes a determining factor.

An examination of this phenomenon and its intensity in relation to the year of publication is enlightening for identifying possible patterns (Fig. 2). When examining the publications impacted by annual fluctuations (Fig. 2A), it becomes apparent that the general distribution of papers with a decreased AAS is quite like that observed when categorized by year of publication (Fig. 2B). For all years of publication, there is a significant majority of works that do not experience a change in their AAS, and this trend remains relatively stable over the years. The proportion of publications experiencing a decrease is equally consistent regardless of the year of publication, varying between 15 and 20%, suggesting that the reduction of the AAS does not depend on the age of the publication. However, the increase in the AAS appears to vary more significantly between years and seems to be closely linked to it. In any case, the presence of a constant percentage of publications affected by a decrease in AAS does not mean that all experience it with the same intensity.

Fig. 2
Fig. 2
Full size image

A Percentage distribution of ISLS publications by type of change in their AAS value between 2022 and 2023 (B) and year of publication; C AAS Biweekly Variability Rate (AASBVR) of Information Science & Library Science publications between February 2022 and February 2023 (D) and by year of publication

When considering the variations in the aggregated AAS value by year of publication, notable differences can be discerned. Figure 2C shows the AAS Biweekly Variability Rate (ABVR) of the publications studied. The fluctuations are stable in that there are no abrupt changes, although an anomalous reduction in mid-June 2022 is noteworthy, and other moments of negative variability are apparent towards the end of 2022. This reflects a vanishing of mentions that directly impact the values of the AAS. However, this fluctuation pattern is not exactly the same for all years (Fig. 2D). It can be seen that the oldest publications show a very low growth and even decreases at certain times. The widespread decrease in the number of AAS in July 2022 is particularly pronounced in the older publications. For example, in 2014 the aggregate value of AAS is reduced by up to 5%. This effect is smaller in more recent years. In contrast, it is the publications from 2020 and 2021 that exhibit a greater positive variability, although even these works also include some moments of decrease towards the end of 2022.

Dissecting fluctuations: a detailed look at individual altmetrics sources

In the analysis of the influence of various altmetric sources on the Altmetric Attention Score (AAS) for ISLS papers, a pattern similar to that observed in other studies emerges, where certain sources disproportionately generate mentions (Table 2). Notably, Twitter mentions, despite its lower weighting in the AAS formula, dominates the landscape with 90.42% of mentions and affects 90.32% of the papers. This high prevalence, despite its minor weight in the AAS, endows it with a crucial role in determining the AAS. In comparison, news mentions, which carries a higher weighting, contributes to 2.95% of mentions and influences 6.50% of the papers. In contrast, Facebook mentions, assigned a much lower weighting, exhibits a substantial impact, affecting 15.08% of papers. Similarly, blog mentions, bearing a moderate weighting, significantly impact 14.50% of the papers. Collectively, these findings indicate that the impact of different altmetric sources on ISLS research transcends their designated weightings in the AAS calculation, being significantly shaped by the volume of mentions and the range of papers they influence.

Table 2 Weight of each of the active altmetric sources contributing to the AAS of ISLS papers and the number of mentions and papers mentioned
Full size table

Considering the difference in paper mentions between February 2022 and February 2023 for each active altmetric source contributing to the AAS, there are varying scenarios, and not all are equally affected (Fig. 3). The most notable observation comes from Twitter mentions, where a significant decrease in mentions over a year is evident, affecting 4498 papers (18.81% of the works mentioned by this source). Following Twitter, policy documents also show a decrease in the number of mentions, specifically impacting 739 papers (37.47%). Therefore, while the number of papers affected by policy document mentions is fewer than those by Twitter, they represent a more significant issue since they account for a higher percentage of this source's mentions, and each policy document mention has 12 times more weight in the AAS calculation than a mention on Twitter. Excluding YouTube mentions, with 5.38% of the mentioned papers decreasing in that metric, the other sources only see a very small percentage of mentions decrease, below 1%. Mentions from peer review, Reddit, Q&A, and F1000 are the only ones that do not experience decreases.

Fig. 3
Fig. 3
Full size image

Plot of altmetric mentions values in February 2022 and February 2023 (log–log). All values have been increased by 1 to represent the cases with zero mentions

Reviewing this phenomenon throughout the entire period, we once again confirm that these decreases occur both short-term and long-term. Specifically examining the two primary sources accounting for most decreases, it is noted that 6015 papers see a decrease in their Twitter mentions at some point, with always some work affected in each period studied. In this sense, there is an average of 518 papers experiencing a decrease in mentions biweekly. The two periods of greatest decrease align with those identified in the AAS analysis, with Twitter mentions for 769 papers declining in mid-June 2022, and for 1480 papers in early December 2022. The fluctuations in policy document mentions reveal a distinct pattern, with 763 papers affected, all during mid-June 2022. These findings suggest that tweets, and particularly policy documents, are the main contributors to the substantial AAS decreases in June, while tweets are the main cause of the December decreases.

The analysis of the variability rate of aggregated mentions is enlightening in this regard. In Fig. 4, the Altmetrics Biweekly Variability Rate (ABVR) for each altmetric source contributing to the AAS of ISLS papers is detailed. This analysis not only identify the moments where the most drastic fluctuations occur that have caused the metrics to decrease but also allows aligning these variations with the fluctuations of the AAS previously revealed. Among the altmetric sources, Twitter and policy documents are especially significant. Twitter mentions shows ten occurrences of negative variability within the series, ranging from November 2022 through the end of the period in February 2023, culminating in a notable decrease in the ABVR in December 2022 (− 1.06%). In contrast, policy documents mentions generally show stable trends except for an anomalous negative fluctuation in June 2022 (− 31.99%), indicating that nearly a third of the mentions from this source vanished at this point. This substantial drop coincides with the most severe decrease in AAS. Consequently, these two altmetric sources, with the most prominent shifts, are likely the driving forces behind these variations in AAS. Regarding the other altmetric sources, only Facebook and YouTube have shown a negative ABVR at certain times within the analyzed timeframe.

Fig. 4
Fig. 4
Full size image

Altmetric Biweekly Variability Rate (ABVR) of Information Science & Library Science publications between February 2022 and February 2023 by altmetric source

However, this analysis is limited when it comes to capturing the vanishing of mentions and providing a clear overview of the stability of each source. When analyzing variations in counts and aggregated values, there may be a vanishing of mentions, but this is offset by a greater increase in mentions. A fact that influences both altmetric mentions and the derived AAS.

Stability analysis of altmetrics sources with a focus on specific mentions

It is ultimately necessary not only to attend to the negative variations in the values of Altmetric Attention Score (AAS) and altmetric mentions, but also to the mentions themselves. Therefore, Table 3 includes for each source the total number of periods in which at least one mention has vanished, along with the average number of vanishings across the entire series, as well as figures on the total number of vanished mentions and papers that have been affected. This provides a picture of this issue and allows for the contemplation of which altmetric sources are the most unstable, understanding as such those in which not only is a high percentage of mentions and papers compromised, but also where the phenomenon is habitual.

Table 3 Statistics of vanished mentions in ISLS papers between February 2022 and February 2023 and papers affected by altmetric source
Full size table

An exploration of the specific mentions that vanish allows for a better understanding of the nature of this phenomenon. The most particular case is found in the mentions of policy documents, as these vanishings have a clear justification, since 1116 of the removed mentions (97.2%) are due to the suppression of the Analysis & Policy Observatory (APO) as a policy source on Altmetric.com.Footnote 2 This decision comes from Altmetric.com and results in nearly halving the mentions of this source in ISLS papers. Meanwhile, the vanishing of Twitter mentions is a common occurrence in this social media, representing a well-known and complex phenomenon, as mentions not only disappear but can also reappear over time (Fang et al., 2020). In this case, of the 26,862 vanishings that occurred, 9389 reappeared (34.95%). The reasons for these changes on Twitter are varied within the media itself, whether it is users deleting tweets, temporarily or permanently closing their profiles, or even being blocked. However, in this case, these vanishings occur precisely at the time of the greatest exodus of users from this social media to others like Mastodon following Elon Musk's purchase of the social media platform (Kupferschmidt, 2022).

In the rest of the sources, although the vanishing is much less significant, they have different motivations. Wikipedia is one of the most particular cases because its mentions, unlike other sources, are alive (Arroyo-Machado et al., 2022). This implies that at any moment an editor can revoke a change that includes a reference to a paper or even recover it. In the case of news and blogs, we find the disappearance of the original source and/or removal from Altmetric.com's track lists. This is precisely the case with news outlets like Environmental Health Perspectives or blogs like Hack Library School. On Facebook, although some of the losses are due to changes in privacy or deletion of the original posts (Yu et al., 2021), some mentions remain active but are no longer included.Footnote 3 Meanwhile, on YouTube, the losses are due to the deletion of the videos themselves.

We can thereby differentiate between four types of vanishing:

  1. 1.

    Administrative vanishing: This occurs when an altmetric data aggregator decides to remove certain sources from their tracking system. This type of vanishing is a direct result of editorial or strategic decisions made by the aggregators. For example, the suppression of the Analysis & Policy Observatory (APO) as a policy source.

  2. 2.

    Technological vanishing: In this case, the loss of mentions occurs due to the disappearance of a source or the breaking of a link that, ultimately, prevents its tracking. It is a type of vanishing that is beyond the user's control and is linked to technological infrastructure. For example, the downfall or removal of a blog website.

  3. 3.

    Digital vanishing: In this case, the disappearance is due to the user or activity in which the mention is included becoming unavailable or no longer openly accessible on the social media. It focuses on the actions of users within digital platforms. For example, a user being banned on Twitter.

  4. 4.

    Natural vanishing: This corresponds to the natural activity of the social media. This type of vanishing reflects the organic and ever-changing nature of the content in some media. For example, on Wikipedia, where edits and revisions are part of the constant and living process of maintaining information.

Discussion

There is no doubt about the limitations of the Altmetric Attention Score (AAS) at the moment. This aggregated metric does not take into account the nature and specificities of each social media (Thelwall, 2020a), the weight assigned to mentions is not justified (Gumpenberger et al., 2016; Mukherjee et al., 2018) and its replication is not entirely possible (Ortega, 2019). However, as with other metrics, the fact that it is not free of limitations does not make it invisible to the community, nor does it mean that it is not used. Something that becomes especially important as this metric is integrated through badges in databases, repositories, and journals (Elmore, 2018), offering live tracking of the overall impact of an academic object.

Our results highlight the existence of fluctuations in the AAS, something that has been evidenced in 6275 papers (23.7% of the total analyzed) during the observation year. The volatility of this metric also originates from the different metrics that compose it, although fluctuations in each of them suggest different situations. Twitter, a metric with a well-known volatility rate, leads in this respect by having constant vanishings throughout the period and compromising 10% of the mentions from this source, which affects nearly 30% of the papers receiving attention from this social media (Fang et al., 2020). Policy documents are another source where the origin of AAS fluctuations has been found. However, in their case, it is due to a specific change in the Altmetric.com tracking system when removing the Analysis & Policy Observatory (APO) as a policy source. Nevertheless, these decisions are also relevant and can have a significant impact. In the rest of the metrics, the disappearance of mentions are aspects that do not pose great risks and mostly affect a very small percentage of the publications. The reasons for these vanishings are equally varied. Therefore, four types of vanishings have been identified, involving those driven by administrative, technological, user, or media issues.

This fluctuation phenomenon carries significant implications for the use of altmetrics in evaluative contexts. Despite the demonstrated potential and utility of these metrics for research assessment (Arroyo-Machado & Torres-Salinas, 2023) and the call for their use to ensure a wider diversity of metrics (Aubert Bonn & Bouter, 2023), these instabilities represent a previously unconsidered limitation of altmetrics. Unlike other well-known issues, this problem is intrinsically linked to the very nature of social media and their underlying technology. Thus, while their application in evaluation is recommended, this challenge does not deem their usage impractical but instead demands a more comprehensive approach. This firstly involves foregoing aggregated metrics, such as the AAS, as the vanishing of mentions across different social media platforms renders it sensitive depending on the type of source, potentially leading to a complete loss of value in some cases. Similarly, it is necessary to move beyond mere “bean counting” and examine each altmetric source individually to understand the type of impact, covering aspects such as audiences or engagement. This usage, alongside responsible and transparent practices, for instance, making the process and consultation dates visible, and acknowledging this potential risk of vanishing, ensures that its possible effects minimize the risks within the evaluation. Thus, it is essential for evaluators and policy makers to be aware of this phenomenon, as their insight is crucial for conducting well-informed assessments. This knowledge enables them to address the complexities of altmetrics, shape effective evaluation strategies, and ensure thorough research assessments.

Therefore, this paper contributes to a better understanding of the nature of altmetrics, and its findings highlight the sensitivity of the AAS to changes in the various contributing altmetric sources and their short and long-term impact, as well as the different reasons explaining these changes. With these results, we do not intend to invalidate the use of AAS or altmetrics, but rather to call on data managements to consider this issue and offer practical solutions to safeguard them. Far from considering these fluctuations a risk, this phenomenon should be seen as an opportunity. An opportunity to improve the way these metrics are stored and displayed. It is necessary to have not only an image of the prevalence or current situation of the attention but a complete view of the entire historical mentions activity, for example, as is done with disappeared sources like Google+, or by applying a blockchain system where attention is reflected transparently. The existence of services like Crossref Event Data, which provides comprehensive logging of deleted mentions on dynamic platforms like Wikipedia, illustrates the potential for maintaining a robust historical record of altmetric activities. This method, applied to all types of vanishing mentions, is essential for improving the accuracy and utility of altmetrics, ensuring a comprehensive view of social attention across time. Such a strategy offers a practical and effective way to address challenges in evaluation contexts.

This paper is not without its limitations. Firstly, for this analysis, only one year has been analyzed and it has focused on a single discipline and type of document, therefore it is possible that these results may vary with different selections. Secondly, the use of biweekly periods, while allowing to pinpoint the moments of fluctuation, does not enable the determination of the specific events where these fluctuations occur most. Thirdly, the fluctuations of other altmetric data aggregators have not been considered, which could provide a better discernment of the possible advantages and disadvantages of each.