Attention-based Approaches for Text Analytics in Social Media and Automatic Summarization
Abstract
Nowadays, society has access, and the possibility to contribute, to large amounts of the content present on the internet, such as social networks, online newspapers, forums, blogs, or multimedia content platforms. These platforms have had, during the last years, an overwhelming impact on the daily life of individuals and organizations, becoming the predominant ways for sharing, discussing, and analyzing online content. Therefore, it is very interesting to work with these platforms, from different points of view, under the umbrella of Natural Language Processing. In this thesis, we focus on two broad areas inside this field, applied to analyze online content: text analytics in social media and automatic summarization. Neural networks are also a central topic in this thesis, where all the experimentation has been performed by using deep learning approaches, mainly based on attention mechanisms. Besides, we mostly work with the Spanish language, due to it is an interesting and underexplored language with a great interest in the research projects we participated in. On the one hand, for text analytics in social media, we focused on affective analysis tasks, including sentiment analysis and emotion detection, along with the analysis of the irony. In this regard, an approach based on Transformer Encoders, based on contextualizing pretrained Spanish word embeddings from Twitter, to address sentiment analysis and irony detection tasks, is presented. We also propose the use of evaluation metrics as loss functions, in order to train neural networks for reducing the impact of the class imbalance in multi-class and multi-label emotion detection tasks. Additionally, a specialization of BERT both for the Spanish language and the Twitter domain, that takes into account inter-sentence coherence in Twitter conversation flows, is presented. The performance of all these approaches has been tested with different corpora, from several reference evaluation benchmarks, showing very competitive results in all the tasks addressed. On the other hand, we focused on extractive summarization of news articles and TV talk shows. Regarding the summarization of news articles, a theoretical framework for extractive summarization, based on siamese hierarchical networks with attention mechanisms, is presented. Also, we present two instantiations of this framework: Siamese Hierarchical Attention Networks and Siamese Hierarchical Transformer Encoders. These systems were evaluated on the CNN/DailyMail and the NewsRoom corpora, obtaining competitive results in comparison to other contemporary extractive approaches. Concerning the TV talk shows, we proposed a text summarization task, for summarizing the transcribed interventions of the speakers, about a given topic, in the Spanish TV talk shows of the ``La Noche en 24 Horas" program. In addition, a corpus of news articles, collected from several Spanish online newspapers, is proposed, in order to study the domain transferability of siamese hierarchical approaches, between news articles and interventions of debate participants. This approach shows better results than other extractive techniques, along with a very promising domain transferability.
Keywords
- Resumen automático
- Analítica de textos
- Medios de comunicación social
- Aprendizaje profundo
- Enfoques basados en la atención
- Text analytics
- Automatic summarization
- Social media
- Deep learning
- Attention-based approaches
Publications during the thesis (26)
- 2021 JournalCross-domain Polarity Models to evaluate User eXperience in E-learning
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría , Rosario Sanchis-Font , María José Castro Bleda
Neural Processing Letters · DOI: 10.1007/s11063-020-10260-5
- 2021 JournalNASca and NASes: Two Monolingual Pre-Trained Models for Abstractive Summarization in Catalan and Spanish
Vicent Ahuir Esteve , Lluís Felip Hurtado Oliver , José Ángel González Barba , Encarna Segarra Soriano
Applied Sciences · DOI: 10.3390/app11219872
- 2021 JournalTWilBert: Pre-trained Deep Bidirectional Transformers for Spanish Twitter
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría
Neurocomputing, 426, 58-69 · DOI: 10.1016/j.neucom.2020.09.078
- 2020 ConferenceELiRF-UPV at TASS 2020: TWilBERT for SentimentAnalysis and Emotion Detection in Spanish Tweets
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría , José Arias Moncho
Iberian Languages Evaluation Forum
- 2020 JournalExtractive summarization using siamese hierarchical transformer encoders
José Ángel González Barba , Lluís Felip Hurtado Oliver , Encarna Segarra Soriano , Fernando García Granada , Emilio Sanchis Arnal
Journal of Intelligent & Fuzzy Systems, 2409-2419 · DOI: 10.3233/JIFS-179901
- 2020 JournalSelf-Attention for Twitter Sentiment Analysis in Spanish
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría
Journal of Intelligent & Fuzzy Systems, 2165-2175 · DOI: 10.3233/JIFS-179881
- 2020 JournalTransformer based Contextualization of Pre-trained Word Embeddings for Irony Detection in Twitter
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría
Information Processing & Management, 57(4) · DOI: 10.1016/j.ipm.2020.102262
- 2019 ConferenceAggressiveness detection through deep learning approaches
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría , Víctor Nina-Alcocer
Iberian Languages Evaluation Forum
- 2019 JournalApplying Siamese Hierarchical Attention Neural Networks for multi-document summarization
José Ángel González Barba , Delonca, Julien , Emilio Sanchis Arnal , Fernando García Granada , Encarna Segarra Soriano
Sociedad Española para el Procesamiento del Lenguaje Natural, 111-118 · DOI: 10.26342/2019-63-12
- 2019 JournalChoosing the Right Loss Function for multi-label Emotion Classification
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría
Journal of Intelligent & Fuzzy Systems, 4697-4708 · DOI: 10.3233/JIFS-179019
- 2019 ConferenceELiRF-UPV at IroSvA: Transformer Encoders for Spanish Irony Detection
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría
Iberian Languages Evaluation Forum
- 2019 ConferenceELiRF-UPV at SemEval-2019 Task 3: Snapshot Ensemble of Hierarchical Convolutional Neural Networks for Contextual Emotion Detection
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría
International Workshop on Semantic Evaluation, 195–199
- 2019 ConferenceELiRF-UPV at TASS 2019: Transformer Encoders for Twitter Sentiment Analysis in Spanish
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría
Workshop on Semantic Analysis at SEPLN
- 2019 JournalSiamese hierarchical attention networks for extractive summarization
José Ángel González Barba , Lluís Felip Hurtado Oliver , Encarna Segarra Soriano , Fernando García Granada , Emilio Sanchis Arnal
Journal of Intelligent & Fuzzy Systems, 4599-4607 · DOI: 10.3233/JIFS-179011
- 2019 JournalSummarization of Spanish Talk Shows with Siamese Hierarchical Attention Networks
José Ángel González Barba , Lluís Felip Hurtado Oliver , Encarna Segarra Soriano , Fernando García Granada , Emilio Sanchis Arnal
Applied Sciences, 1-13 · DOI: 10.3390/app9183836
- 2018 ConferenceELiRF-UPV at MultiStanceCat 2018
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría
Workshop on Evaluation of Human Language Technologies for Iberian Languages
- 2018 ConferenceELiRF-UPV at SemEval-2018 Task 10: Capturing Discriminative Attributes with Knowledge Graphs and Wikipedia
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría , Encarna Segarra Soriano
International Workshop on Semantic Evaluation, 968–971 · DOI: 10.18653/v1/S18-1159
- 2018 ConferenceELiRF-UPV at SemEval-2018 Task 11: Machine Comprehension using Commonsense Knowledge
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría , Encarna Segarra Soriano
International Workshop on Semantic Evaluation, 1034–1037 · DOI: 10.18653/v1/S18-1172
- 2018 ConferenceELiRF-UPV at SemEval-2018 Tasks 1 and 3: Affect and Irony Detection in Tweets
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría
International Workshop on Semantic Evaluation, 565–569 · DOI: 10.18653/v1/S18-1092
- 2018 ConferenceELiRF-UPV en TASS 2018: Análisis de Sentimientos en Twitter basado en Aprendizaje Profundo
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría
Workshop on Semantic Analysis at SEPLN, 37-44
- 2018 ConferenceELiRF-UPV en TASS 2018: Categorización Emocional de Noticias
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría
Workshop on Semantic Analysis at SEPLN, 103-109
- 2017 ConferenceELiRF-UPV at IberEval 2017: Classification Of Spanish Election Tweets (COSET)
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría
Workshop on Evaluation of Human Language Technologies for Iberian Languages, 55-60
- 2017 ConferenceELiRF-UPV at IberEval 2017: Stance and Gender Detection in Tweets
José Ángel González Barba , Lluís Felip Hurtado Oliver , Ferran Pla Santamaría
Workshop on Evaluation of Human Language Technologies for Iberian Languages, 193-198
- 2017 ConferenceELiRF-UPV at SemEval-2017 Task 4: Sentiment Analysis using Deep Learning
José Ángel González Barba , Ferran Pla Santamaría , Lluís Felip Hurtado Oliver
International Workshop on Semantic Evaluation, 714-718
- 2017 ConferenceELiRF-UPV at SemEval-2017 Task 7: Pun Detection and Interpretation
Lluís Felip Hurtado Oliver , Encarna Segarra Soriano , Ferran Pla Santamaría , Carrasco, Pascual , José Ángel González Barba
International Workshop on Semantic Evaluation, 431-434
- 2017 ConferenceELiRF-UPV en TASS 2017: Análisis de Sentimientos en Twitter basado en Aprendizaje Profundo
Lluís Felip Hurtado Oliver , Ferran Pla Santamaría , José Ángel González Barba
Workshop on Semantic Analysis at SEPLN, 29-34