RIA-Net:Realistic Image Animation through Semantic-aware Feature Learning Network

Abstract

This paper presents the Realistic Image Animation (RIA-Net), a novel framework that leverages semantic-aware feature learning and adversarial training to generate high-quality animations from a single static image and a driving video. Unlike traditional keypoint-based methods that often suffer from local distortions and temporal instability, RIA-Net introduces a transformer-based architecture integrated with landmark and keypoint detection to preserve semantic details and capture long-range motion dynamics. The proposed semantic-aware transformer explicitly models global dependencies and predictive spatiotemporal relationships, enabling smooth and temporally consistent animations. Extensive experiments on diverse datasets, including VoxCeleb, TaiChiHD, and TED-Talks, demonstrate that RIA-Net consistently outperforms state-of-the-art methods in terms of animation quality, temporal coherence, and visual fidelity. This work opens new opportunities for realistic image animation in applications such as entertainment, virtual reality, and digital content creation.

1 Introduction

Image animation, the task of synthesizing realistic video sequences by transferring motion from a driving video to a static source image, has become a key research area in computer vision and graphics. Human image animation, in particular, is a compelling yet challenging task with applications spanning online retail, entertainment, virtual reality, gaming, video conferencing, healthcare, sports, analysis, and human–computer interaction (Zablotskaia et al. 2019; Wang et al. 2021; Zhang et al. 2024). The advent of Generative Adversarial Network (GAN) (Arjovsky et al. 2017; Goodfellow et al. 2014) has driven substantial advancements in image animation, with seminal works exploring motion transfer and video generation (Chan et al. 2019; Siarohin et al. 2019a; Siarohin et al. 2021; Zhao and Zhang 2022).

Recent progress in generative models and video synthesis techniques (Ma et al. 2312; Xu et al. 2024a; Wang et al. 2024a, 2024b) has further improved animation realism, motion expressiveness, and visual diversity. These developments have significantly expanded the scope of digital content creation and interactive media. Despite these advancements, generating smooth, coherent, and visually appealing animations remains a significant challenge. Problems like temporal instability, semantic inconsistency, local distortion, and loss of fine details, particularly in scenarios involving long-term, complex, or highly dynamic motion patterns, continue to limit the broader adoption of image animation technologies. Addressing these challenges is critical for advancing realistic and reliable image animation in practical applications.

Traditional motion-transfer approaches, particularly those relying on pose-guided representations (Zablotskaia et al. 2019; Yu et al. 2023; Zhang et al. 2022), suffer from long-range semantic context loss, training instability, and limited generalization, resulting in flickering and inconsistent motion, especially in complex scenarios such as dance videos (Chan et al. 2019; Wang et al. 2018; Zhou et al. 2019). Among these, keypoint based motion transfer methods such as the First Order Motion Model (FOMM) (Siarohin et al 2019a) and Motion Representations for Articulated Animation (MRAA) (Siarohin et al. 2021) have achieved notable success by regressing sparse keypoints and local affine transformations to model part-wise deformations. However, these approaches remain fundamentally local and geometric in nature. They rely on convolutional operations with limited receptive fields and first-order motion representations that capture only short-range, independent part movements, often leading to temporal flickering, loss of long-range dependencies, and semantic inconsistencies in complex, full-body, or expressive motions.

Recently, diffusion-based methods (Xu et al. 2024a; Hu 2024; Chang et al. 2023; Karras et al. 2023; Ma et al. 2024a; Wang et al. 2024; Zhu et al. 2025) have shown promise by leveraging the strong generative capabilities of models such as Stable Diffusion (Rombach et al. 2022) and ControlNet (Zhang et al. 2023), enabling improved control over human pose and background features. However, these methods often rely on coarse-grained text prompts (Karras et al. 2023), which limits their ability to preserve rich human semantics and background consistency in real-world settings (Wang et al. 2024c). Moreover, limited training data leads to poor zero-shot generalization to unseen scenarios, making it difficult to capture nuanced human attributes or complex poses (Li et al. 2021a; Liu et al. 2019). Recent approaches such as MagicAnimate and Animate Anyone (Xu et al. 2024a) adopt 3D-UNet architectures for video denoising but lack explicit temporal transformer modules, restricting their ability to model long-term temporal dependencies. These limitations collectively motivate the development of a new image animation framework that explicitly enhances long-range temporal consistency, preserves semantic details, and improves generalization across diverse and complex motion patterns, thereby advancing the realism and practical applicability of image animation.

To overcome these limitations, we propose RIA-Net, which leverages a semantic aware transformer to capture long-range dependencies and semantic context from the driving video, producing smooth and temporally consistent animations from a single source image. The framework takes a source image and a driving video as input, extracts video frames, and applies standard preprocessing steps such as resizing and normalization. Unlike existing methods, we employ landmark and keypoint detection to identify salient feature points in each frame, storing their coordinates for detailed motion analysis. The proposed module adopts a transformer encoder-decoder architecture to capture semantic context and long-range dependencies. Through self attention mechanisms, the model enables coherent animations with consistent motion across frames while enhancing semantic understanding for narrative-rich results. The resulting contextual embeddings improve robustness to variations in pose and scene changes, leading to more realistic animations. Our approach incorporates a symmetrical structure designed to capture the spatial features of the reference image. At each corresponding layer, we integrate features using spatial attention mechanisms, allowing the model to effectively associate appearance details with the reference image within a consistent feature space. In addition, we introduce an explicit temporal modeling layer to capture relationships across multiple frames, ensuring smooth and continuous temporal motion while preserving fine-grained details. This design ultimately results in more realistic and temporally stable animations.

In this paper, we present Realistic Image Animation through a semantic-aware transformer integrated with an adversarial network to overcome the limitations of existing image-to-video synthesis methods. As shown in Fig. 1, our approach aims to capture long-range semantic context and generate smooth and temporally consistent animations from a single input image and a driving video, particularly for complex motion dynamics that traditional techniques struggle to model. Leveraging transformers, our method learns robust motion patterns by encoding spatiotemporal features and decoding semantic-aware representations. We detect landmarks and keypoints from the source image and driving video, extract features, and feed them into a transformer encoder-decoder to obtain semantic context-aware representations. Based on these representations, we estimate masks and affine transformations, which are combined with optical flow and integrated into an adversarial training framework. The adversarial component further enhances temporal consistency by minimizing frame to frame discrepancies, resulting in smoother and more realistic motion. Our method is efficient, scalable, and achieves high-quality results with limited training data. The main contributions of this paper are summarized as follows:

• We introduce adversarial animation, which combines transformer-based motion dynamics with an adversarial network. By extracting features from keypoint and landmark detectors, our method employs semantic-aware attention that effectively addresses long-range dependencies and semantic context loss, enabling the generation of realistic, long-range, temporally consistent animations from static images.

• We propose semantic-aware feature learning that focuses on transformer-based motion dynamics to create smooth, temporally consistent animations from a single input image and driving video. Furthermore, the proposed module captures long- range semantic context by modeling the entire object and deriving motion from the driving video, ensuring temporal consistency and fluid, contextually rich character movements.

• We propose a spatiotemporal enhancement module that employs predictive modeling to anticipate feature movements in the driving video, enhancing the animation process and maintaining temporal coherence throughout the sequence. Furthermore, this module adapts the source image to align with predicted motion patterns, resulting in more natural and visually appealing animations.

• We conduct extensive experiments to evaluate the effectiveness of our approach on the VoxCeleb (Nagrani et al. 2017), TaiChiHD, and TED-talks (Siarohin et al. 2021) datasets

Fig. 1

The overall structure of our proposed approach. The proposed approach comprises two essential modules: Realistic Image Animation generation through landmark and semantic-aware feature learning. The proposed approach focuses solely on transformer-based motion dynamics to create smooth, temporally consistent animations from a single input image

This is how the rest of the paper is organized: related works are compiled in Section II. Section III outlines the main approach we use, and Section IV provides the experiment setup and metrics, followed by a comparison with state-of-the-art approaches. Finally, in Section VI, we conclude our work.

2 Related work

2.1 Techniques in human image animation

Image animation has gained significant attention in recent years, particularly focusing on specific object types. Different techniques have emerged as the most popular for animating the human body and face. These methods utilize object-specific, keypoint, landmark detectors, or 3D models, or are pre-trained on extensive labeled datasets. While they provide a structured approach to learning motion dynamics, they also come with inherent limitations, primarily their applicability to specific object types.

These approaches typically exploit the vast amounts of unlabeled web videos available, employing image reconstruction losses to derive depictions of intermediate motion such as affine matrices and keypoints. The flexibility this approach allows them to animate a broader spectrum of objects, including human bodies, faces, and even animals, without the need for predefined structural representations. A prominent example is Monkey-Net (Siarohin et al 2019b), which learns keypoints in the intermediate part as sparse motions through an image reconstruction task. While this approach represents a significant advancement, it is limited by its coarsely defined motion model.

Following this, FOMM (Siarohin et al 2019a) improved upon Monkey-Net by regressing local affine matrices concurrently alongside the keypoints of object parts. This dual regression allows for more nuanced motion representations, although FOMM still grapples with the challenge of stabilizing the affine matrices. Building on these developments, MRAA (Siarohin et al. 2021) takes a further step by integrating local affine matrices and part keypoints into a single heat-map estimate process component. This innovation enhances the model’s capability to capture more complex motions. However, MRAA faces challenges when modeling relatively rigid motions, such as those found in human faces, and it often neglects the cooperative movements of different parts. Human image animation involves creating videos from one or more input images.

Most prior unsupervised image animation works particularly keypoint-driven methods like FOMM(Siarohin et al 2019a) and MRAA (Siarohin et al. 2021) rely on CNN-based local motion estimation without explicit long-range modelling. These methods excel at capturing fine-grained local deformations but inherently suffer from limited receptive fields, independent part modelling, and lack of global semantic reasoning, which can lead to inconsistencies in long sequences, failure to preserve holistic object context, and poor handling of cooperative or non-rigid motions across distant body parts.

Traditional non-diffusion methods typically utilize multiple separate networks for tasks such as background prediction (Siarohin et al. 2021; Zhao and Zhang 2022; Weng et al. 2019), motion representation (Siarohin et al 2019a, 2019b; Siarohin et al. 2021; Zhao and Zhang 2022), occlusion mapping (Siarohin et al 2019a; Siarohin et al. 2021; Zhao and Zhang 2022; Weng et al. 2019), and depth estimation (Bao et al. 2019). This approach necessitates independent training for each component and often relies on potentially flawed or unavailable ground-truth data for intermediate stages, like motion or depth. As motion becomes more extensive and complex, deriving accurate ground-truth estimates becomes increasingly challenging and prone to errors. In response, recent studies have begun to explore end-to-end single-network solutions, which integrate various functions, such as merging optical flow with warping (Reda et al. 2022).

2.2 Generative models

The quick development of generative networks has led to various innovative methods in this field. Historically, many approaches have concentrated on utilizing Generative Adversarial Network (GAN) to achieve these animations (Siarohin et al 2019a; Siarohin et al. 2021; Zhao and Zhang 2022; Li et al. 2019; Meftah et al. 2024). These methods typically involve a motion network that predicts dense appearance flows and employs feature warping on reference images to generate realistic visuals that align with target frames. GANs have played a prominent and successful role in a variety of activities, including image-to-image generation (Zhao and Zhang 2022; Zhu et al. 2017; Yang et al. 2018, 2018) and video-to-video generation (Wang et al. 2018; Mathieu et al. 2015).

In the context of video generation, GANs involve a generator and a discriminator, with the generator aiming to produce artificial data that closely mimics actual data, challenging the discriminator’s ability to differentiate between the two. However, such techniques often face significant obstacles, including instability during training and issues related to mode collapse (Hu 2024). As a result, they struggle to accurately control the generated human motions(Xu et al. 2024a, 2024b; Hu 2024; Chang et al. 2023; Wang et al. 2307; Zhu et al. 2025, 2404), leading to less than optimal synthesis quality.

To address these challenges, researchers have increasingly turned to diffusion models (Wang et al. 2024a; Xing et al. 2025), which have demonstrated superior training stability and the ability to produce high-fidelity results through an iterative refinement process. Diffusion-based methods have emerged as a leading approach to research, (Balaji et al. 2022; Huang et al. 2023; Ramesh et al. 2022; Saharia et al. 2022) significantly enhancing the quality of generated images. The Latent Diffusion Model (LDM)(Rombach et al. 2022) has been particularly impactful, as it addresses computational complexity by performing denoising in a latent space. For example, Disco (Wang et al. 2024) introduces a groundbreaking approach to generating realistic human dance videos, particularly for social media, by emphasizing two vital properties: generalizability and compositionality. The novel DisCo architecture utilizes disentangled control mechanisms to separately manage the human subject, background, and pose, enhancing the accuracy of dance synthesis. It incorporates human attribute pre-training to improve generalizability to unseen subjects and employs advanced control mechanisms leveraging local and global embeddings from the CLIP model.

Further advancements are seen in models like MagicAnimate(Xu et al. 2024a), which introduces a groundbreaking framework for human image animation that leverages a diffusion based approach to tackle challenges of temporal consistency and identity preservation. This MagicAnimate model, which employs a video diffusion model with temporal attention blocks to maintain long-range temporal coherence across frames, effectively addresses issues like flickering prevalent in previous methods. Additionally, the introduction of an appearance encoder captures intricate details from reference images, ensuring that identity features, such as clothing and background, are consistently preserved throughout the animation. To facilitate smooth transitions in longer video sequences, the framework incorporates a video fusion technique that utilizes overlapping segments, enhancing continuity and overall animation quality. Furthermore, the authors implement an image-video joint training strategy to bolster the model’s generalization capabilities, enabling it to produce high-fidelity, temporally coherent animations across diverse backgrounds and poses.

Animate Anyone(Hu 2024) also introduces an innovative approach for generating realistic videos from still images using pose sequences, leveraging the strengths of diffusion models to tackle key challenges in character animation. Central to this method are several contributions, including ReferenceNet, which merges detailed features from reference images through spatial attention to maintain fine-grained appearance consistency; a Pose Guider that ensures controllable character movements; and a Temporal Layer that models inter-frame relationships for smooth transitions and stability. By enhancing the Stable Diffusion architecture with multi-frame inputs and detailed feature extraction, the authors claim significant improvements over existing techniques, reaching state-of-the-art outcomes on benchmarks such as fashion video synthesis and human dance generation. The model demonstrates impressive generalization capabilities, allowing it to animate arbitrary characters while addressing common issues like temporal jitter and flickering.

Both the above papers utilize video diffusion techniques to enhance temporal consistency in animations. These models adopt a two-phase learning process that distinguishes between motion guidance and appearance alignment, improving the overall quality of the generated animations. A common framework among these diffusion-based approaches is the use of a ControlNet (Zhang et al. 2023), like a 3D-UNet architecture (Hu 2024; Zhang et al. 2023; Guo et al. 2023). This model is designed to maintain temporal coherence throughout the animation process. Additionally, a reference encoder is often employed, which mirrors the 3D-UNet architecture but omits the temporal Transformer layers. This design choice helps preserve the fine details of the reference images, ensuring that the generated animations retain a high level of visual fidelity.

2.3 Integrating transformers in animation frameworks

Transformers in sequence generation have revolutionized sequential data processing because of their capacity to model long-range dependencies. Initially introduced in the field of natural language processing(Vaswani 2017), transformers have since been applied to various domains, including video synthesis. The success of models like GPT (Radford 2018) and BERT(Devlin 2018) in understanding and generating sequential data inspired researchers to explore transformers for video generation. Transformer-based models, such as VideoGPT (Yan et al. 2021), have been used to model the temporal dynamics in videos by addressing the process of generation as a sequence prediction problem. These models leverage self-attention mechanisms to capture dependencies across frames, enabling them to learn complex motion patterns.

Based on the above Transformers(Vaswani 2017), originally developed for natural language processing (NLP), have recently made significant strides in the field of computer vision. Among the pioneering models in this domain are DETR(Carion et al. 2020) (Detection TRansformer) and ViT(Dosovitskiy 2010) (Vision Transformer), which have set the foundation for a series of innovative approaches leveraging transformer architectures for visual tasks. DETR was the first model to successfully incorporate transformers into object detection. It employs an encoder-decoder framework similar to those used in NLP. In this architecture, object queries are treated as learnable parameters, allowing the model to effectively identify and classify objects within an image. By utilizing this method, DETR revolutionized how object detection tasks are approached, moving away from traditional convolutional neural networks (CNNs) and establishing a new paradigm based on transformer principles.

ViT, on the other hand, introduced a transformer-based encoder specifically designed for image classification. This model operates by dividing images into smaller patches, treating these patches similarly to tokens in NLP. A unique feature of ViT is the introduction of a learnable classification token, which enhances the model’s ability to perform classification tasks. This innovative approach demonstrated that transformers could effectively handle image data, leading to significant improvements in classification accuracy. Following the successes of DETR and ViT, many researchers have explored various adaptations of transformer architectures for diverse computer vision challenges (Yang et al. 2021; Wan et al. 2021). Some of these adaptations focus on specific tasks, such as landmark detection, where the objective is to identify and locate key points in images. Numerous methods have emerged that build upon the foundational principles established by DETR (Carion et al. 2020) and ViT (Dosovitskiy 2010), enhancing their capabilities and expanding their applicability in computer vision.

Recent advancements in this space have inspired new methodologies that aim to integrate the strengths of transformer architectures with specific task requirements. For instance, our approach is influenced by these developments, as we focus on regressing object keypoints (Li et al. 2021b; Watchareeruetai et al. 2022) in addition to affine matrices for the purpose of image animation. This dual focus allows for more nuanced interactions and transformations within images, facilitating more dynamic and engaging visual outputs.

Our work directly addresses these limitations by integrating a semantic-aware transformer into the motion modeling pipeline. Unlike FOMM (Siarohin et al 2019a) and MRAA (Siarohin et al. 2021), which perform localized regression of motion parameters, our approach introduces a semantic-aware transformer that fundamentally shifts the paradigm toward global, long-range dependency modeling and semantic-contextual motion understanding. By employing self-attention over unified motion and image tokens, the transformer enables each motion descriptor to attend globally to all object parts and the entire source image, capturing holistic object semantics and inter-part relationships across extended temporal and spatial ranges. Furthermore, cross-attention mechanisms fuse motion context derived from the driving video into source-image features, enriching static appearance with dynamic, semantically meaningful guidance rather than purely geometric warping. These design choices result in significantly improved temporal coherence, semantic fidelity, and robustness to complex motion dynamics, going well beyond simple feature aggregation or enhancement of existing keypoint-based pipelines.

The exploration of transformers in computer vision is not just limited to object detection and classification. Researchers are continuously pushing the boundaries to address a variety of visual tasks, including segmentation, image synthesis, and even generative modeling. The flexibility and scalability of transformer architectures make them particularly well-suited for these applications, as they can process large amounts of data and learn complex relationships within that data.

The future of computer vision seems promising with the continued evolution of transformer models. As more sophisticated architectures are developed, we can expect enhancements in performance across a wide range of tasks. This will likely lead to more efficient models that require less computational power while maintaining high accuracy levels. Additionally, the integration of transformers with other machine learning techniques may yield synergistic effects, further improving the capabilities of computer vision systems. However, while transformers excel at modeling temporal dependencies, they often fall short in generating high-quality, visually detailed frames when used alone. This has led to the exploration of hybrid models that integrate the temporal modeling capabilities of transformers with other generative techniques.

GANs to enhance frame quality (Zhou et al. 2021). Similarly, combining transformers with diffusion models has been investigated to produce both high-quality and temporally consistent animations. While these hybrid approaches represent a step forward, they often require extensive computational resources and large datasets to train effectively. Moreover, issues such as flickering and inconsistency across frames persist, especially in scenarios involving complex motion dynamics. Thus, there is a clear need for a framework that can integrate these advanced techniques in a way that balances visual fidelity, temporal coherence, and computational efficiency.

These methods are limited by their dependence on precise motion vectors and keypoints, which makes them less effective for complex or non-rigid motions. GANbased approaches, despite their success in generating realistic frames, often struggle with training stability and maintaining temporal consistency across longer sequences. Transformers have proven effective in modeling temporal dependencies but lack the ability to generate high-quality frames on their own.

In contrast, our proposed RIA-Net introduces a semantic-aware transformer that fundamentally shifts the paradigm toward global, long-range dependency modeling and semantic-contextual motion understanding. By employing self-attention across unified motion and image tokens, the transformer enables each motion descriptor to attend to all other parts and the entire source image globally, capturing holistic object semantics and inter-part relationships over extended temporal and spatial ranges. Furthermore, cross-attention mechanisms fuse driving video derived motion context into source-image features, enriching static appearance with dynamic, semantically meaningful guidance rather than purely geometric warping. This results in significantly improved temporal coherence, semantic fidelity, and robustness to complex dynamics—capabilities that go well beyond mere feature aggregation or enhancement of existing keypoint pipelines.

Our proposed Adversarial animation framework seeks to address these gaps by a semantic-aware transformer-based motion dynamics with adversarial network, combined with an adversarial training approach to refine temporal coherence. By leveraging the strengths of each generative technique, we aim to create a robust, scalable solution for generating smooth, realistic, and temporally consistent animations from static images, even in the presence of complex motion dynamics.

3 Proposed approach

To achieve realistic image animation, the Adversarial animation framework generates smooth, lifelike animations from static images by integrating semantic-aware motion extraction, adversarial module frame generation, and adversarial temporal refinement. The semantic-aware module identifies and tracks meaningful features like facial landmarks or object positions to transfer motion from driving videos to source images. Our approach includes 1) keypoint and landmark feature learning, 2) semantic-aware transformer feature learning for motion dynamics with spatiotemporal enhancement, and 3) a video generation phase. These modules work together to create dynamic animations, even with complex motions.

4 Problem formulation

In the context of video synthesis, our goal is to generate a reconstructed driving video that combines the visual identity of a given source image with the motion patterns observed in a driving video. More specifically, we are provided with a source image I ∈ R^H×W×C and a driving video $V={\left(v\right)}_{i=1}^{t}\in {\mathbb{R}}^{T\times H\times W\times C}$ (where T,H,W and C refer to the clip length, height, width, and the number of channels, respectively). Our proposed method aims to reconstruct driving video V^ˆ = (v)^t_i=1 ∈ R^T×H×W×C. The rebuilt video should imitate the motion seen in the driving video in addition to maintaining the original image’s visual identity. By doing this, we are able to produce a video that blends the dynamic qualities recorded in the driving video with the appearance of the source image.

4.1 Landmark and keypoint feature learning

The initial stage involves the extraction and feature learning of key points and landmarks. A key point detector identifies salient features in the source image, such as facial landmarks or joint positions, which serve as critical reference points governing the animation process. Concurrently, a landmark detector locates additional specific landmarks on the source image, offering further structural context. This dual extraction mechanism guarantees the precise recording of the subject’s fine-grained details and global structure, which is indispensable for producing authentic and high-fidelity animations.

4.2 Landmark feature extraction

The module takes both the source image I and the driving frame as inputs, enabling it to understand the context and dynamics of the input features. The Landmark module contains locating specific predefined points on objects within an image and video frame. The process begins by encoding the source image I and driving video V, denoted as I_lm and V_lm, respectively. This encoding step allows us to extract landmarks, which are essential for capturing the distinctive features and characteristics of the objects in the image and video (Newell et al. 2016). The detector produces a total of 68 landmark points (Asebe Teka et al. 2025). By taking the mean value, the landmark detector network effectively captures the spatial information of the input video and image points. Additionally, keypoints are incorporated into the landmark detection process. The landmark point covariance provides additional information that encourages the network to consider not only the location but also the orientation of each landmark point (Siarohin et al 2019b). The covariance matrices capture the relationship between the landmark points in terms of their position and orientation, ensuring that the module captures both aspects accurately. By effectively encoding and extracting landmarks, our framework lays the foundation for further analysis and animation of the source image from the driving video using motion data.

4.3 Key-point feature extraction

We utilize key point detection to identify distinctive and repeatable points in image and video frames, providing a more comprehensive representation of salient features in the source image I and driving video V. Our objective is to jointly learn a detector that captures both the image structure and its motion. We employ a key point detector, denoted as I_kp and V_kp, to predict the locations of key points in consecutive frames v_t and v_t+1, extracted from a derived video. By employing this approach, we ensure that the learned key points are accurately positioned on the regions of the object that are more likely to exhibit noticeable motion.

Based on the references (Siarohin et al 2019a; Siarohin et al. 2021), In order to estimate the transition from the source image to the driving video and execute image animation of the source image, we make use of learned landmarks and key points. To achieve accurate transformation estimation, to characterize the relationship around landmarks and key points, we use affine transformation matrices. In the motion estimation phase of object manipulation, we aim to determine a motion transformation for each of the K parts of an object. For the k-th part, this transformation consists of three components: t^k which is a keypoint represented as a 2D vector in R^2×1, lm^k which serves as a landmark also in R^2×1, and A^k, a 2D affine transformation matrix in R^2×2. The index k ranges from 1 to K, indicating the various parts of the object.

An important consideration is that the affine transformation A^k should only affect a specific mask associated with the k-th part, rather than the entire image. To achieve this, we introduce a mask M^k that delineates the area impacted by A^k. In existing research, particularly in works (Siarohin et al 2019a; Siarohin et al. 2021), the method for estimating this mask typically employs a convolutional neural network structured in an encoder-decoder fashion. This architecture processes the warped source image to output a set of masks {M^k|k = 1,…,K} corresponding to each of the K parts of the object.

Additionally, to improve the generated images’ visual quality, we can learn an occlusion map. This map plays a crucial role in guiding the image generator to effectively fill in any occluded areas that might arise during the transformation process. By accounting for occlusions, the generator can better reconstruct areas of the image that are not directly visible, leading to more coherent and realistic renderings of the transformed object.

The integration of these components, the motion transformation parameters, the specific masks for each part, and the occlusion map, facilitates a more nuanced and accurate representation of how an object can be manipulated and transformed in a visual context. This involves a systematic learning of motion transformations for different parts of an object, constrained by masks that specify areas of influence. This is achieved through a CNN-based encoder-decoder model that generates the necessary masks and occlusion maps. By leveraging these, we can enhance the visual quality and realism of transformed images, addressing challenges related to dynamic object manipulation and occlusion handling effectively.

Based on the first-order motion model (Siarohin et al 2019a; Siarohin et al. 2021) we may drive motion flow from the driving frame V to the source image I for the k-th object part after learning $\left({t}_{I}^{k},{lm}_{I}^{k},{A}_{I}^{k}\right)$

And $\left({t}_{V}^{k},{lm}_{V}^{k},{A}_{V}^{k}\right)$ as follows:

$${T}_{I\leftarrow V}\left(\chi \right)={t}_{I}^{k}+{lm}_{I}^{k}+{lm}_{I}^{k}+{A}_{I}^{k}{\left({A}_{V}^{k}\right)}^{-1}\left(x-\left({t}_{V}^{k}+{lm}_{V}^{K}\right)\right)$$

(1)

where x denotes any image coordinate in the driving frame. We next obtain the dense motion flow T_I←V (x) by combining ${T}_{I\leftarrow V}^{k}\left(x\right)$ with masks M^k(x) as linear weights:

$$\begin{array}{c}K\\ {{T}_{I\leftarrow V}\left(x\right)=}^{X}{M}^{k}\left(x\right)\cdot {{T}_{I}}^{k}\leftarrow v\left(x\right)\\ k=1\end{array}$$

(2)

where M.^k(x) is the mask of the k-th object part at the coordinate x and ${\sum }_{k=1}^{K}{M}^{k}\left(x\right)=1$

. Moreover, the dense motion flow T_I←V (x) represents that the pixel value at coordinate x of the generated image V^˜ is warped and obtained based on the pixel value at coordinate T_I←V (x) of the source image.

4.4 Semantic-ware feature learning

Capturing complex motion patterns is essential for generating realistic image animations. Conventional approaches often depend on rigid motion vectors, which lack the flexibility to generalize effectively across diverse scenarios. To overcome these limitations, our framework employs transformers, renowned for their capability to model long-range dependencies and temporal dynamics within sequences. Central to this approach are two primary inputs: 1) a dynamic driving video and 2) a static source image. The source image represents the character or object to be animated, while the driving video provides the requisite motion information. This dual-input design facilitates the seamless integration of motion into a static image, thereby enabling the generation of lifelike animations.

In our approach, each motion token is a learnable embedding $\left.{{M}_{t0}}^{k}\right|k=1,\cdots ,\left.K\right\}$, where ${M}_{{t}_{0}}^{k}\in {\mathbb{R}}^{d}$ represents the k-th object part and d is the embedding dimension. Image tokens $\left\{\left.{I}_{{t}_{0}}^{n}\in {\mathbb{R}}^{d}\right|n=1,\cdots ,N\right\}$ are derived by extracting low-level features from image patches using a CNN, flattening them, projecting to dimension d, and adding absolute position encodings. After accurately extracting key-points and landmarks, the semantic-aware feature learning processes this information to learn robust long-range feature representations essential for realistic image animation. Thus, the proposed module consists of two main components: spatial–temporal feature enhancement and a semantic-aware feature learning, each designed to model spatial and semantic including with capturing long-range dependence relationships within the input features.

4.5 Spatiotemporal feature enhancement

The spatial–temporal feature enhancement transforms the extracted key points and landmarks into high-dimensional embeddings that capture their spatial and semantic dependencies. As shown in Fig. 2, utilizing self-attention mechanisms, the encoder models intricate relationships among input tokens, generating contextual embeddings that represent both individual features and their interactions. These embeddings effectively enhance the spatial–temporal dynamics of object parts, referred to as motion tokens, which represent dynamic changes in position and orientation over time. Furthermore, the positional encodings are incorporated to preserve spatial arrangements throughout the encoding process. Thus, the proposed temporal enhancement self-attention is formulated as follows, similar to Vaswani (2017):

Fig. 2

Overview of spatiotemporal feature enhancement. The proposed module captures their spatial and long-range temporal dependencies

$${MI}^{Att}=softmax\left(\frac{{{Q}_{MI\left({K}_{MI}\right)}}^{T}}{\sqrt{d}}\right){V}_{MI}$$

(3)

To improve prediction accuracy, a linear projection head is attached to the final transformer layer, directly regressing key points and affine transformation matrices that characterize the motions of object parts. This direct mapping ensures that generated animations are visually coherent and contextually consistent.

4.6 Semantic-aware feature learning

Through cross-attention, key points and landmarks are enriched with contextual information derived from dynamic motion data, enabling a more comprehensive understanding of how these features evolve over time. This enhances the key point and landmark features extracted from the source image by allowing the model to integrate motion information from the driving video with the static features of the source image. The key point and landmark detectors play a vital part by offering essential spatial cues that facilitate this enhancement. Such integration is critical for accurately aligning the movements captured in the driving video with the corresponding object features represented in the static image. From Eq. 3, we obtain two enhanced action features: F_I and F_M, corresponding to the image and motion features, respectively. These features are enhanced with residual connections, defined as

$$\begin{array}{cc}FI = MIAtt + It0n& FM = MIAtt + Mt0k\end{array}$$

(4)

Here, it F_I serves as the set of queries, while it F_M functions as the memory (keys and values) within the semantic-aware feature learning. As shown in Fig. 3, three matrices with learnable weight parameters, ω_I^Q, ω_M^K,and ω_M^V are created by projecting both F_I and F_M. The resulting intermediate tensors Q_I, K_M, and, V_M are obtained through linear operations as formulated in Eq. 5:

$$QI = FI\omega IQ, KM = FM\omega MK , VM = FM\omega MV$$

(5)

where F_I and F_M represent the input features for queries and keys/values respectively. Subsequently, the attention matrix Atten is computed using Eq. 6:

$$Atten=Multihead\left(LN\left(\frac{{Q}_{I}{K}_{M}^{T}}{\sqrt{{d}_{k}}}\right)VN\right)+{F}_{I}$$

(6)

where LN(·) denotes layer normalization and d_k is the dimensionality of the key vectors. By leveraging these semantic-aware feature attention mechanisms, the model effectively fuses spatial and dynamic inputs, ensuring that the generated animations are both realistic and contextually accurate. Thus, the motion token update combines temporal enhanced and semantic-aware feature attention as follows:

$${M}_{{t}_{l}}^{i}=FFN\cdot LN\left(M{I}^{Att}+Atten\right)$$

(7)

followed by layer normalization and a Feed-Forward Network (FFN). This formulation allows the model to capture both intra-motion dependencies and interactions with image features effectively. Furthermore, the computational efficiency, motion, and image tokens are concatenated as

$${F}_{0}=\left[{M}_{{t}_{0}}^{1},\cdots ,{M}_{{t}_{0}}^{K},{I}_{{t}_{0}}^{1}\right]\epsilon {\mathbb{R}}^{\left(K+N\right)\times d}$$

(8)

and updated jointly using a single self-attention mechanism:

Fig. 3

Overview of semantic-aware feature learning module. The proposed module captures long range, semantic context by considering the entire object and deriving motion from a driving video

$$\begin{array}{c}{{F}_{l}^{i}=}^{{X}_{MSA}}\left({Q}_{{F}_{l-1}^{i}},{K}_{{F}_{l-1}^{j}},{V}_{{F}_{l-1}^{j}}\right)\\ j\end{array}$$

(9)

$${F}_{l}^{i}=FF{N}^{\left.LN\left({F}_{l}^{i}\right)+{F}_{l}^{i}\right)}$$

(10)

Here, image tokens are also updated with attention to all tokens, and effective end-to-end training is supported by the unified self-attention mechanism.

After processing the feature representations through the transformer layers, the model performs a final step involving affine transformations and mask estimation. At the last transformer layer L, motion tokens ${M}_{tL}^{k}$ are extracted from F_L and passed through a linear head W_h ∈ R^d×6 to regress affine matrices and translation vectors:

$$\left[{A}^{k},{t}^{k},l{m}^{k}\right]={M}_{tL}^{k}{W}_{h}$$

(11)

Here, A^k, t^k and lm^k represent the affine transformation parameters for the k-th object part. This estimation process ensures precise and context-aware motion modeling, culminating in the generation of realistic animations. The estimated masks play a critical role in defining the regions of interest for applying the transformations, further enhancing the fidelity of the final output. This structured approach to feature learning, combined with advancements in self-attention and cross-attention mechanisms, represents a significant step forward in the realm of realistic image animation.

4.7 Optimization

To train the whole network effectively, a combination of loss functions is utilized. The entire training procedure is conducted in an end-to-end fashion. To create our method’s objective function, we take into account the following losses.

• 1)1)1)1)1) Perceptual loss: Following FOMM and MRAA, we adopt the multi-resolution perceptual loss (Johnson et al. 2016) defined with a pre-trained VGG-19 (Simonyan and Zisserman 2014) network. Given the driving frame V with resolution index i, the generated image V^˜, and the feature extractor ϕ with layer index l, the perceptual loss can be written as follows:

  $${L}_{per}=\sum_{i}\sum_{l}{\Vert \phi l\left({V}_{i}\right)-\phi l\left({\widetilde{V}}_{i}\right)\Vert }_{1}$$

  (12)

• 2)2)2)2)2) Equivariance loss: Following FOMM and MRAA, the equivariance loss is adopted here. Given a random geometric transformation T and a driving image V, this loss can be written as follows:

  $${L}_{equiv}=\sum_{k}{\Vert T\left({t}^{k}Z\right)-{t}_{T}^{k}\left(V\right)\Vert }_{1}+\sum {\Vert T\left(l{m}^{k}Z\right)-{lm}_{T}^{k}\left(V\right)\Vert }_{1}$$

  (13)

• 3)3)3)3)3) Background losses: We use a background predictor network in accordance with the MRAA to forecast the background motion flow in order to deal with scenarios where the background is not static. Motivated by Gao et al. (2021), we use the following losses to make it easier to separate the motion learning for the background and foreground to measures how well the generated image matches the original image I:

  $$\begin{array}{c}Lreconstruction = EI,z h||I - V||2i\\ {L}_{mask}={\Vert {M}^{0}-1\Vert }_{1}+\sum_{k\ne 0}{\Vert {M}^{k}\Vert }_{1}\end{array}$$

  (14)
  $${L}_{con}=\sum_{k\ne 0}\sum_{x}\frac{{M}^{k}\left(x\right)\cdot {\Vert x-{\mu }_{k}\Vert }^{2}}{\sum_{M}k}+{\mathbb{E}}_{I,z}\left[{\Vert I-\widetilde{V}\Vert }^{2}\right]$$

  (15)

where ${u}_{k}=\frac{{\sum }_{c}{M}_{k}\left(x\right)\cdot x}{{\sum }_{M}k}$. Intuitively, the mask loss L_mask is used to constrain the portion of the foreground and background area in an image, and the foreground motion mask is considered to be more focused under the constraint of the concentration loss.

4.8 Lcon.

To create the overall objective function of our suggested vision transformer, the total loss adds up all of the aforementioned losses as follows:

$$L = \lambda 1(Lper) + \lambda 2(Lequiv) + \lambda 3(Lmask) + \lambda 4Lcon)$$

(16)

where hyperparameters λ₁,λ₂,λ₃,λ₄) balance each loss function’s contribution.

5 Experiments and result analysis

5.1 Datasets

In the proposed study, we employed a diverse range of datasets that encompassed various objects, specifically including talking faces, human bodies, and pixel animations. The specific dataset names utilized in our research are VoxCeleb, TaiChiHD and TED-talks.

• VoxCeleb (Nagrani et al. 2017): The VoxCeleb is a large-scale audio-visual dataset designed for speaker recognition and verification research. It contains audio and visual recordings of celebrities and public figures, showcasing their diverse backgrounds and professions, and includes approximately 100,000 utterances from 1,251 individuals. The videos in VoxCeleb are cropped to a resolution of 256 × 256 pixels and are extracted from YouTube. The video lengths range from 64 to 1024 frames.

• TaiChiHD (Siarohin et al 2019a): The TaiChiHD dataset consists of cropped videos showcasing TaiChi movements, which were sourced from YouTube. This dataset’s videos have been reduced to 256 by 256 pixels in resolution, and they primarily focus on complete human bodies performing TaiChi. The TaiChiHD dataset is particularly suitable for research purposes related to TaiChi.

• TED-talks (Siarohin et al 2019b): The MRAA dataset provides TED Talk videos that have been downloaded from YouTube. These videos have been resized to a resolution of 384 × 384 pixels, with a specific emphasis on the upper body region. The dataset contains TED Talk videos of varied lengths, between 64 and 1024 frames.

5.2 Evaluation Metrics

In order to maintain consistency with the existing approach, we take up the three measurements suggested in Monkey-Net (Siarohin et al 2019b). These metrics include L1, Average Euclidean Distance (AED), and Average Key Point Distance (AKD).

• 1) The L₁ metrics: To compute the MAE of the distance between the generated video (V^ˆ) and the actual video (V), we take the absolute difference between each predicted video value (v_i) and its corresponding actual value (V_i), and then calculate the mean of these absolute differences. Mathematically, if V = (V₁,···,V_N) ∈ R^H×W×3 represents the actual video and V^ˆ = (v₁,···,v_N) ∈ R^H×W×3 represents the generated video. This N represents the total number of frames in the video, and H and W represent the height and width dimensions of each frame, respectively.

• 2) Average Euclidean Distance (AED). We assess the performance of the proposed approach by measuring the AED between the ground truth and the generated frame representation. This allows us to quantify the similarity or discrepancy between the two representations and evaluate the accuracy of our approach using key points (x,y) and their generated counterparts (x,ˆ yˆ).

• 3) Average Key Point Distance (AKD). We utilized the AKD metric to evaluate the accuracy of estimating key point locations in an image or frame in our proposed approach. By assessing whether the motion of the input video is preserved, AKD verifies the contribution of our approach and validates its effectiveness.

• 4) Missing Key Point Rate (MKR). We also utilize the MKR to evaluate the appearance quality of our proposed approach by assessing the presence or absence of key points in the generated frames. MKR involves assigning a binary label to each key point, indicating successful detection or whether it was missed.

• 5) Frechet Video Distance (FVD). We utilize this metric to evaluate the quality of generated video sequences by measuring the distance between the distributions of real and generated videos within a high-dimensional feature space, providing a comprehensive understanding of how closely the generated content matches the original.

5.3 Implementation details

To implement the proposed approach for image generation using landmarks and keypoints, we leverage transformer architectures combined with an adversarial network to create a robust framework. This serves as the mask predictor, using skip connections, which are essential for accurate mask generation since they help retain spatial information across scales. For estimating motion, we utilize the first three stages of the HRNet-W32 (Sun et al. 2019) encoder, pretrained on ImageNet, to effectively extract high-resolution features while down sampling them by a factor of 4 to optimize computational efficiency.

These image features are then processed by a 12-layer transformer encoder that captures complex relationships within the data. We implement sine functions for position encoding and set the image patch size to 4 × 4, generating 256 image tokens for a resolution of 256 × 256 and 576 tokens for 384 × 384, with a token dimension of 192. Additionally, we integrate 10 motion tokens to account for dynamic elements from the input video, allowing the model to learn motion patterns effectively.

Using affine transformation techniques, we apply the estimated masks to the images, enhancing the realism of the generated output based on motion cues. To further improve the model’s ability to understand temporal changes, we incorporate optical flow methods integrated into the transformer architecture to consider both spatial and temporal information.

After the encoder has processed the original image, feature maps are produced, which are subsequently distorted in accordance with the dense motion flow. Lastly, using the warped feature map, the decoder learns the synthesized picture.

This comprehensive strategy aims to advance the field of image synthesis by generating semantically rich and context-aware images from dynamic video inputs.

5.4 Ablation study

This section presents an ablation study that we conducted to evaluate the performance of the various components our proposed modules. It is helpful to understand the contribution of each component toward generating coherent and realistic animated frames or each module. By systematically removing or modifying specific modules and analyzing their impact on overall performance, we aim to gain insights into the importance of individual components and our design choices.

5.5 Impact of semantic awareness in transformer-based motion dynamics

The integration of semantic awareness in transformer-based motion dynamics represents a significant advancement in our proposed method, RIA-Net (OURS). This approach leverages the self-attention mechanism inherent to transformers, enabling model to capture long-range dependencies in motion sequences while maintaining a focus on semantic context. To elucidate its significance, we conducted an ablation study by substituting the transformer architecture with a conventional convolutional neural network (CNN). In this experimental framework, both RIA-Net and the CNN-based model were trained on identical datasets, enabling a direct comparison of their performance metrics. The results in Table 1 indicated a substantial decline in performance when utilizing the CNN architecture.

Table 1 Evaluation of semantic awareness in transformer-based motion dynamics

For example, the L1 loss increased from 0.021 in RIA-Net to 0.045 in the CNN model. This change demonstrates a pronounced discrepancy between generated outputs and actual video sequences. It suggests that the CNN architecture is less capable of preserving fidelity than its transformer-based counterpart. Furthermore, the Average Key Distance (AKD) rose from 3.001 to 4.200, indicating a marked cline key point accuracy in the CNN. The Average Euclidean Distance (AED) also increased from 0.111 to 0.200. This results reinforce that the transformer architecture is better suited for capturing complex motion dynamics.

These findings underscore the effectiveness of the transformer architecture in facilitating motion representation, particularly through its incorporation of semantic awareness. The self-attention mechanism allows the model to evaluate all input positions concurrently, effectively mitigating the limitations associated with CNNs, which often struggle to model long-range dependencies and contextual relationships. This capability is critical for generating coherent animations that necessitate an understanding of the temporal and semantic relationships among frames. By capturing intricate motion patterns and contextual semantics, transformers significantly enhance the overall quality of the generated animations. The semantic awareness embedded within the transformer architecture enables the model to accurately interpret and represent the nuances of motion, resulting in more realistic and contextually relevant animations.

This ablation study elucidates the critical role of semantic awareness in transformer-based motion dynamics, validating the design choices implemented in RIA-Net and establishing a foundation for future innovations in the field of image-to-video synthesis. The potential for further advancements in this domain is immense, particularly as the integration of semantic frameworks continues to evolve, paving the way for more sophisticated and high-fidelity animation generation techniques.

5.6 Comparison with the state-of-the-art approaches

The performance comparison between our proposed approach and other existing approaches is presented in Table 2 for the VoxCeleb dataset, Table 3 for the TaiChiHD dataset, and Table 4 for the TED-talks dataset. We do experiments using the same number of epochs (100) for all datasets to guarantee a fair comparison with K = 10 and a consistent learning rate. As shown, our suggested method outperforms cutting-edge methods in terms of performance. This is due to the fine-grained and representative feature representation of image and video features, which improves the performance of image animation.

• 1)1) Comparison on the VoxCeleb dataset: Table 2 provided summarizes the performance metrics of various approaches for image animation, particularly focusing on the VoxCeleb dataset. This dataset is crucial for evaluating methods that synthesize realistic animations from static images by leveraging motion dynamics. Each of the metrics plays a vital role in determining the effectiveness and quality of the animations generated by different methods. The table includes several notable approaches: MRAA, LIA, TPS, CVTHead, MBVC, AMNT, and the proposed method RIA-Net (OURS). Each approach is evaluated based on its ability to produce high-quality animations that are visually coherent and temporally consistent. L1 loss measures the absolute difference between the actual video and the generated video, with a lower value demonstrating improved performance. OURS achieves the lowest L1 loss of 0.022, demonstrating its superior capability in producing animations that are nearly indistinguishable from real sequences. In contrast, MRAA and LIA report losses of 0.043 and 0.041, respectively, indicating that while they perform well, they are less effective than OURS. The Average Key Distance (AKD) evaluates the accuracy of key point locations in the generated animations. Here, OURS also excels, achieving an AKD of 1.020, significantly lower than the other methods. For instance, the next best performer, AMNT, reports an AKD of 1.120, showcasing a notable difference in performance. A lower AKD indicates that OURS better maintains the fidelity of key point positions, which is crucial for realistic motion representation.

Table 2 Comparison with the state-of-the-art approaches in the VoxCeleb dataset, demonstrating its superior capability in producing animations that are nearly indistinguishable from real sequences

Table 3 Comparison with the state-of-the-art approaches’ performance in the TaiChiHD suggests a more accurate representation of the source image in the generated output

Table 4 Comparison with the state-of-the-art approaches performance in the TED-talks

Similarly, Average Euclidean Distance (AED) measures the overall distance between the generated frames and the ground truth. OURS again leads with an AED of 0.090, outperforming its competitors. Minimizing AED is essential for ensuring that the generated animations not only look realistic but also maintain the correct spatial relationships between elements in the scene. LPIPS focuses on perceptual similarity, assessing how visually appealing the generated frames are compared to the real ones. OURS scores 0.150, which, while higher than its L1, AKD, and AED scores, still indicates strong performance relative to the other methods. Notably, AMNT has the best LPIPS score of 0.120, suggesting that it may produce more visually pleasing results, albeit at the cost of higher losses in other metrics. The results indicate that OURS effectively strengthens transformer-based motion dynamics with a generative adversarial network, as highlighted in the paper. The low L1 loss and improved AKD and AED suggest that the model successfully captures the complex motion patterns necessary for realistic animations, resolving the shortcomings of earlier approaches that have trouble maintaining temporal consistency and fidelity.

The improvements brought by OURS could be attributed to its innovative architecture, which employs a transformer encoder-decoder framework for enhanced motion understanding. This architecture is adept at capturing long-range dependencies, allowing for coherent animations that span multiple frames. Furthermore, the adversarial training component of OURS refines temporal coherence, minimizing flickering and abrupt transitions that often plague other approaches. The performance metrics on the VoxCeleb dataset reveal that OURS outperforms existing methods across key evaluation criteria. This reinforces the potential of integrating transformer dynamics with diffusion models to achieve high-quality, temporally consistent animations.

• 2)2) Comparison on the TaiChiHD dataset: Table 3, summarizing the performance metrics on the TaiChiHD dataset, provides critical insights into the efficacy of various image animation methods, evaluating their ability to generate realistic animations from static images, particularly in the context of Tai Chi movements. The metrics assessed include L1 loss, Average Key Distance (AKD) with Missing Key Point Rate (MKR), Average Euclidean Distance (AED), LPIPS (Learned Perceptual Image Patch Similarity), and Frechet Video Distance (FVD). Collectively, these metrics focus on accuracy, visual appeal, and temporal consistency, offering a comprehensive view of each method’s performance. Among the notable approaches analyzed are MRAA, LIA, TPS, LEO, MBVC, EGHVC, AMNT, and the proposed method RIA-Net (OURS). The L1 loss, which indicates the absolute difference between the generated video and the actual video, shows that OURS outperforms its competitors with a remarkable score of 0.022, significantly lower than MRAA, LIA, and TPS, which report losses of 0.063, 0.057, and 0.045, respectively. This lower L1 loss suggests a more accurate representation of the source image in the generated output.

In terms of AKD and MKR, OURS achieves an AKD of 4.420 and an MKR of 0.017, indicating effective preservation of key point positions during animation. This performance is notably superior compared to MRAA, which has an AKD of 5.626 and a higher MKR of 0.025. Such results highlight OURS’s ability to maintain fidelity in character poses, which is crucial for realistic motion representation. Furthermore, the Average Euclidean Distance (AED) measures the overall distance between generated frames and the ground truth, with OURS excelling at a value of 0.135. In contrast, MRAA and LIA report AEDs of 0.460 and 0.431, respectively, suggesting that OURS is superior in producing animations that closely resemble real sequences.

LPIPS, focusing on perceptual similarity, evaluates the visual quality of the generated frames. OURS scores 0.192, which, while slightly higher than some competitors, indicates that there is still room for improvement in visual appeal compared to MBVC, which scores 0.123. This suggests that while OURS effectively preserves motion dynamics, it may not be as visually refined as some other methods. Additionally, Frechet Video Distance (FVD), which measures the temporal consistency of the generated video, shows OURS achieving a score of 122.3, closely following AMNT at 124.21. This result indicates that OURS maintains a high level of temporal coherence in the animations produced, which is critical for applications requiring fluid motion representation.

The results from the TaichiHD dataset in Table 3 reaffirm the effectiveness of OURS in integrating transformer-based motion dynamics with generative adversarial network. The substantial improvements in L1 loss, AKD, and AED suggest that OURS captures complex motion patterns effectively while ensuring spatial fidelity in the generated animations. The model’s architecture, which combines a transformer encoder-decoder framework with an adversarial training component, plays a significant role in minimizing discrepancies between generated frames. This architectural choice enhances the model’s ability to model long-range dependencies, ensuring coherent animations over time. In conclusion, the performance metrics on the TaiChiHD dataset indicate that OURS outperforms existing methods across several key evaluation criteria. The advancements in integrating transformer dynamics with diffusion models contribute to generating high-quality, temporally consistent animations.

• 3)3) Comparison on the TED-talks dataset: The provided Table 4 outlines the performance metrics for various image animation methods evaluated on the TED-talks dataset, which is essential for assessing the effectiveness of techniques that synthesize realistic animations from static images. The key metrics analyzed include L1 loss, Average Key Distance (AKD) with Missing Key Point Rate (MKR), Average Euclidean Distance (AED), LPIPS (Learned Perceptual Image Patch Similarity), and Frechet Video Distance (FVD). These metrics are crucial for understanding the quality, accuracy, and temporal coherence of the generated animations. The table features several significant approaches: MRAA, LIA, TPS, Animate Anyone, Disco, AMNT, and the proposed method (OURS). Each method is assessed based on its effectiveness in producing realistic animations within the context of TED-talks presentations.

Starting with L1 loss, which measures the absolute difference between the generated video and the actual video, OURS achieves the lowest L1 loss of 0.021, indicating a high level of accuracy in reproducing the source content. In comparison, MRAA and LIA report L1 losses of 0.029 and 0.027, respectively. This lower L1 loss in OURS suggests a better alignment with the target animations, which enhances the overall fidelity of the output. In terms of keypoint fidelity, RIA-Net also excels with an AKD/MKR mean of (3.391, 0.0080). This indicates that it captures the movement and alignment of facial features accurately, which is essential for producing realistic animations. In contrast, other methods like MRAA and LIA show slightly higher mean values, suggesting they may struggle with precise keypoint tracking.

Further, Average Euclidean Distance (AED) evaluates the overall distance between generated frames and the ground truth. RIA-Net (OURS) scores 0.111, the best among the listed methods, indicating a strong capability to generate animations that closely resemble actual sequences. This represents a significant improvement over MRAA and LIA, which report AEDs of 0.396 and 0.399, respectively. In terms of perceptual similarity, LPIPS assesses the visual quality of generated frames. OURS scores 0.180, which is slightly higher than some competitors, suggesting that while it performs well, there is still room for enhancing visual appeal. LIA achieves the best LPIPS score of 0.110, indicating that it produces more visually pleasing results, albeit with trade-offs in other metrics.

Finally, RIA-Net achieves an LPIPS score of 0.180, reflecting closer perceptual similarity to the original images compared to its counterparts. Coupled with an FVD score of 80.82, which assesses the quality of generated videos, RIA-Net demonstrates a clear advantage in producing realistic animations. Overall, these results position RIA-Net as the most effective approach for animating faces in the TED-talks dataset, highlighting its strengths in accuracy, fidelity, and realism.

The performance metrics on the TED-talks dataset indicate that OURS outperforms existing methods across several key evaluation criteria. animations.

5.7 Qualitative visualization results

The qualitative visualization results demonstrate the effectiveness of OURS in generating realistic animations from static images. Visual examples illustrate how the model successfully captures intricate motion dynamics and expressions, showcasing smooth transitions and coherent movements. Comparisons with traditional methods reveal that RIA-Net(OURS) produces outputs with greater fidelity to the source content, highlighting the accurate representation of key features and landmarks. Additionally, the animations exhibit a strong sense of temporal coherence, reflecting the model’s ability to maintain consistency across frames. Observations of the generated sequences indicate that the integration of semantic awareness enables OURS to interpret and reproduce complex actions effectively, resulting in lifelike animations that resonate with viewers. Overall, these qualitative results underscore the advancements made in image-to-video synthesis, affirming the potential of OURS for various applications in digital content creation.

5.7.1 Visualization result comparison with state-of-the-art approach

The qualitative visualization results reveal a clear advantage of RIA-Net (OURS) over state-of-the-art methods in generating realistic animations from static images. Comparative examples illustrate that OURS produces smoother and more coherent motion sequences, effectively capturing the nuances of facial expressions and body movements. While traditional approaches, such as convolutional neural networks and other contemporary models, often exhibit artifacts and inconsistencies, OURS maintains a high level of fidelity to the source content. Notably, the animations generated by OURS demonstrate enhanced temporal coherence, with fluid transitions that closely mimic natural movement. In contrast, state-of-the-art methods tend to struggle with maintaining key point accuracy, leading to noticeable discrepancies in motion representation. These visual comparisons shown in Fig 4 0.4 highlight the superior performance of OURS, underscoring its potential for applications in various domains, including entertainment and virtual reality.

Fig. 4

Qualitative visualization results in comparison with the state-of-the-art approaches on the Tachi dataset. These visualization results demonstrate that our approach performs well in transforming motion from driving videos into images by capturing the movement in each frame, aligning the face, and tracking the shape of the face and the movement of the mouth

5.7.2 Visualization result with semantic-aware transformer

The qualitative visualization results underscore the significant advantages of OURS, which employs a semantic awareness transformer-based architecture, in generating realistic animations from static images. The visual comparisons demonstrate that OURS effectively captures intricate motion dynamics, accurately reflecting facial expressions and body language. By leveraging semantic awareness, the model interprets contextual information, resulting in animations that exhibit a deeper understanding of the source content. For instance, when generating sequences involving complex actions, OURS maintains a high degree of fidelity, with smooth transitions and coherent movements that resonate with viewers as shown in Fig. 5, but we also need some improvement as shown in Fig. 6. In contrast, traditional methods often struggle to preserve the semantic context, leading to animations that may appear disjointed or lack realism. These results highlight the efficacy of semantic awareness in enhancing the quality of generated animations, validating the architectural choices made in OURS and showcasing its potential for a wide range of applications in digital content creation. When comparisons reveal that the integration of these transformers allows the model to effectively capture and predict movements and expressions, resulting in outputs that closely align with the driving video and source images. For instance, when animating facial expressions, the model adeptly retains subtle nuances, producing lifelike results that exhibit a high degree of fidelity to the original content. Conversely, the absence of semantic awareness leads to noticeable discrepancies and inconsistencies in the generated animations, resulting in less coherent movements and reduced overall quality. These findings highlight the crucial role of semantic-aware transformers in achieving high-fidelity video predictions, reinforcing the architectural decisions made in our approach and demonstrating its potential for applications in fields such as entertainment, virtual reality, and digital storytelling, as shown in Fig. 7.

Fig. 5

Qualitative result with a VoxCeleb and TaiChiHD dataset. This demonstrates the significant contribution in achieving visually pleasing results, particularly for enhancing facial features

Fig. 6

Qualitative result with a TED-talks dataset. This demonstrates the significant contribution to achieving visually pleasing results, particularly for enhancing facial features

Fig. 7

Qualitative result with a VoxCeleb dataset. This demonstrates the significant contribution in achieving visually pleasing results, particularly for predicting unseen parts of the source image that transfer spatial and temporal facial features from both the video and the image

5.8 Subjective perceptual evaluation

To complement the quantitative metrics and provide a more comprehensive assessment of perceived animation quality, we conducted a subjective user study involving 20 human participants. For fair comparison, four animated outputs were generated from the same source image and a fixed driving video using RIA-Net (ours) and three representative baselines (MRAA, LIA, and TPS). The study covered diverse scenarios, including talking-head expressions, full-body motions, and long expressive sequences. All animations were generated at the same resolution and presented in a blind, side-by-side manner. The results, summarized in Table 5, demonstrate that RIA-Net consistently outperforms the baseline methods in terms of human perception. Specifically, RIA-Net achieves the highest Mean Opinion Scores (MOS), with 4.0 for Visual Realism, 3.85 for Temporal Coherence (smoothness and absence of flickering or jitter), 3.9 for Semantic and Identity Consistency (preservation of facial expressions, body details, clothing, and background), and an overall average score of 4.0. In contrast, baseline methods obtain MOS values ranging from 3.3 to 3.7 across these criteria. Pairwise preference evaluations further confirm the superiority of RIA-Net, with 62% of participants preferring it for temporal coherence and 70% favoring it for semantic and identity consistency. These subjective judgments strongly align with and reinforce the quantitative metrics, highlighting RIA-Net’s ability to produce perceptually more natural, flicker-free, and semantically faithful animations, particularly in challenging scenarios involving long-range and complex motions. The consistent preference for RIA-Net across all evaluation dimensions, together with substantial inter-rater agreement, confirms that the proposed semantic-aware transformer delivers noticeably superior visual quality, temporal smoothness, and semantic preservation as perceived by human observers.

Table 5 Subjective user study results (Mean Opinion Scores on 5-point Likert scale and pairwise preference percentages). Higher is better

5.9 Computational complexity and efficiency analysis

To assess the practical feasibility of RIA-Net, particularly for long video sequences, we evaluate its computational complexity, model capacity, and inference efficiency. All measurements were conducted on a single NVIDIA RTX 3090 GPU (24 GB) at a resolution of 256 × 256, using batch size 1 and FP16 precision in inference mode. We focus on the motion estimator, which is the primary transformer-based component of RIA-Net, and compare it with the keypoint-based motion estimators used in FOMM and MRAA. The evaluation metrics include the number of trainable parameters (M), FLOPs (G) for a representative sequence, peak GPU memory consumption, and inference speed measured in frames per second (FPS) across driving sequences of 256 and 512 frames.

The RIA-Net motion estimator contains 13.57 million trainable parameters and requires 8.91 GFLOPs for a representative 256-frame sequence. This parameter count is slightly lower than that of FOMM (14.21 M) and MRAA (14.20 M), while the FLOPs are higher due to transformer self-attention over motion tokens. Despite this, memory usage remains manageable, with a peak of approximately 8.5 GB for 512frame sequences, and inference speed remains practical, achieving about 8 FPS average frames processed per second during the full forward pass. This graceful scaling is enabled by our targeted design, which processes only landmark- and keypoint-derived motion tokens rather than dense pixel-level features, thereby avoiding the quadratic complexity growth common in dense transformer-based video models.

Overall, these results demonstrate that RIA-Net achieves state-of-the-art perceptual and quantitative performance without prohibitive computational costs. The modest increase in FLOPs compared to purely keypoint-based methods is justified by the substantial gains in temporal stability and semantic fidelity, particularly for long video sequences. Detailed comparisons are reported in Table 6, confirming RIANet’s favorable trade-off between performance and resource consumption for real-world applications such as entertainment, virtual avatars, and digital content creation.

Table 6 Model capacity comparison

6 Failed-case scenario of the proposed approach

Despite the advancements of OURS in generating high-quality animations, several failure cases highlight its limitations. One significant issue arises from inaccurate landmark detection, particularly in scenarios where the source image features occluded or poorly defined elements, leading to distorted facial expressions and unnatural movements. Additionally, OURS semantic misinterpretation can also occur when the input image contains ambiguous elements, causing the model to inaccurately reflect the intended motion or expression. Furthermore, while OURS performs well on specific datasets, it may have difficulties generalizing to new or diverse datasets, particularly with varying styles or cultural expressions, resulting in inconsistencies in performance. These failure cases underscore the need for improvements in landmark detection robustness, temporal coherence, semantic understanding, and generalization capabilities, paving the way for future enhancements in image-to-video synthesis.

7 Conclusion

The Realistic Image Animation (RIA-Net) successfully addresses the challenges associated with image animation through the integration of semantic-aware transformers and adversarial network. By focusing on accurate landmark detection and long-range motion dynamics, RIA-Net generates smooth, coherent, and visually appealing animations from static images and driving videos. The findings underscore the critical role of semantic awareness and long-range dependency modeling in generating high quality, realistic animations. Our experiments demonstrate that the proposed method RIA-Net, significantly outperforms traditional approaches across multiple performance metrics. The marked improvements in L1 loss, Average Key Distance, and Average Euclidean Distance emphasize the advantages of using transformers for capturing intricate motion patterns and maintaining spatial fidelity. Additionally, the ablation studies reveal that a semantic-aware transformer is essential for ensuring the accuracy of key point placements, which directly influences the overall prediction and quality of the generated animations. The substantial declines in performance when these components are omitted reinforce their necessity in achieving high-fidelity animations that accurately reflect the source content.