Beyond Surface Metrics: Modeling Abstractivity in Summarization via Action-Based Spectrums

Traditional views of abstractive summarization treat it as a free-form rephrasing of the source text, yet this notion is vague and difficult to measure; existing evaluations rely on surface-level metrics —such as novel \n\ -gram counts, extractive fragment coverage, or extractive fragment density— that obscure the stylistic nuances that distinguish summaries along an abstractivity spectrum. We therefore propose a fine-grained, continuous characterization that does not claim to unify extractive and abstractive summarization into a single continuous dimension, but rather presents them as two endpoints of a spectrum defined by the frequency of text-transformation actions (sentence reduction, combination, paraphrasing, reordering, etc.) drawn from the foundational summarization literature. To operationalize this view, we introduce CLAsum, a human-annotated dataset comprising 600 unique samples; these arise from 200 document–summary pairs sourced from CNN/DailyMail and XSum datasets, each annotated by three independent annotators who assign one of five perceived abstractivity levels and record the prevalence of seven abstractivity-inducing actions per pair. Our analysis shows a strong Pearson’s correlation of 0.6 between the overall frequency of these actions and human perception of abstractivity, which validates the spectrum. Moreover, automatic prediction of the resulting Abstractivity-Inducing Features (AIFs) captures signals that existing metrics miss and yields significant performance gains of up to 9.1% in abstractivity-level measurement on the evaluated regression methods. This work delivers both conceptual clarity and practical tools for diagnosing and informing the design of controllable, style-aware summarization systems capable of producing summaries with precisely specified levels of abstractivity, thereby advancing the field toward truly user-tailored, task-adaptive summarization.