Benchmark Metrics
Comprehensive evaluation metrics for story visualization, assessing consistency, adherence, quality, diversity, and human perception.
1. Character Consistency (CRef / CIDS)
Evaluates whether characters maintain consistent appearance across an image sequence. Focuses on self-similarity of character identity features and cross-similarity with reference images. Also known as Character Identification Similarity (CIDS) in ViStoryBench.
Calculates character similarity across frames using CLIP image features (referencing Theatregen, Story-Adapter). Suitable for general characters.
Face recognition models specialized for real human faces, extracting facial features for similarity computation.
Evaluates consistency for anime-style characters, often requiring detection of individual characters within images.
Measures the model's ability to distinguish different characters in multi-character scenes, avoiding identity mismatches.
First uses Grounding DINO to detect and crop main characters based on text descriptions, then calculates their similarity with reference images in feature spaces like CLIP, FID, or DreamSim.
A cross-modal fine-grained matching metric for evaluating the perceptual similarity between generated and reference images.
Applies methods for evaluating character consistency in videos to image sequences, tracking character stability across the sequence.
Uses a pre-trained character classifier to identify characters in generated images and compares with ground truth labels to calculate F1 score or accuracy.
Utilizes Vision Language Models (VLMs) to compare characters in generated images with manually annotated ground truth character information for consistency verification.
2. Style Consistency (SRef / CSD)
Evaluates whether the artistic style remains consistent across an image sequence, including style cross-similarity with reference images and style self-similarity among generated images. ViStoryBench uses CSD (CLIP Style Disentanglement) for this.
Quantifies self-similarity (within generated images) and cross-similarity (between generated and reference images) through CSD-CLIP feature analysis. First, features are extracted using a CLIP vision encoder, then CSD layers separate content and style features, and finally, cosine similarity between style feature embeddings is computed.
Utilizes Vision Language Models like Qwen or GPT-4o to score the style consistency of generated image sequences.
A variant of FID (Fréchet Inception Distance) tailored for artistic styles, used to evaluate the realism and diversity of generated images in terms of artistic aesthetics.
Evaluates style similarity by analyzing frequency domain features of images or distances between deep features extracted by VGG networks; considered a more advanced exploratory method.
3. Prompt Adherence
Assesses how well the generated images align with the storyboard descriptions (prompts), including character interactions, shooting methods, accuracy of the number of on-stage characters, and individual actions.
Uses VLMs to evaluate whether key entities (objects, colors, character count & description, behaviors) appear as prompted. Specifically, Onstage Character Count Matching (OCCM) is calculated using the formula: `100 * exp(-|Detected - Expected| / (epsilon + Expected))`.
Evaluates whether the shooting method in the generated image (e.g., long/medium shot, high/low angle) aligns with the "Shot Perspective Design" in the storyboard.
Evaluates whether scene attributes, time, location, etc., in the generated image align with the "Setting Description" in the storyboard.
Evaluates whether the emotion and atmosphere conveyed by the generated image align with the descriptions in the prompt (e.g., character interactions, static descriptions).
After character segmentation using Grounding DINO, VLMs are used to verify if individual character actions and expressions conform to the "Static Shot Description" in the storyboard.
4. Generation Quality
Evaluates the visual quality and aesthetic appeal of generated images, and whether there is an issue of "copy-pasting" reference images.
Uses Aesthetic Predictor V2.5 (based on SigLIP) to evaluate image aesthetics, with scores ranging from 1-10. Scores above 5.5 are generally considered excellent quality. Focuses on whether images are blurry, noisy, etc.
Measures the similarity in feature space distribution between generated images and real images, commonly used to assess the quality of realistic images.
A CLIP-based image quality assessment model that predicts human preference rankings for images, reflecting the overall attractiveness of generated images.
Detects whether the model excessively relies on or directly copies character reference images. For single-image input models, compares the similarity difference of output characters with the input image and an alternative reference image; this item is not applicable to multi-image input models.
Analyzes whether generation quality decreases as the length of the image sequence increases.
5. Diversity
Measures the model's ability to generate multiple different outputs from a single prompt or to exhibit variation in sequence generation.
Evaluates the clarity and diversity of a batch of generated images. For multiple results generated from a single prompt, IS can reflect the diversity of their content.
Calculates the variance of CLIP features for all generated images in a story sequence. Higher variance indicates more content or style variation among images, thus higher diversity.
6. Human Evaluation
Subjective assessment of generated results by human evaluators, typically covering environmental consistency, character identification consistency, and subjective aesthetics.
Assesses the use of camera language, the rationality and aesthetics of composition, and the effectiveness of visual storytelling.
Evaluates the fluency of visual presentation, the rationality of plot development, and the overall comprehensibility of the story through methods like questionnaires.
Assesses the continuity and consistency of visual elements (such as lighting, color, object placement) in adjacent or related images within an image sequence.
For comic generation tasks, evaluates the appeal, narrative guidance, and visual impact of panel layouts.