amsqr at WASSA 2024 Empathy and Personality Shared Task: Dynamic Personality Profiling in Interactions

This post describes the system submitted to WASSA 2024 shared task on empathy detection, emotion classification and personality detection in interactions. The main focus of this research was the detection of personality traits from a group of users and their essays written in reaction to news articles. Empirical results suggest that in the absence of user-specific features such as demographic information or empathy indexes, text embeddings extracted from their interactions using Transformer models are useful to infer users’ personality. However, the best evaluation score could be achieved only with an interaction-agnostic model.

Introduction

The OCEAN model, also known as “the Big Five”personality test (Digman, 1990), are 5 personality traits that measure the openness to experience, conscientiousness, extroversion, agreeableness and neuroticism. This model can be used to better understand an individual’s overall personality and behavior by ranking each trait through a series of interconnected scales.

Personality profiling has been found to have many different use cases, ranging from enabling personalization technologies to uncovering new research insights, such as candidate screening during recruitment (Agarwal et al., 2023), improving online marketing campaigns (Desmonda et al., 2024), predicting financial outcomes (Exley et al., 2022) and understanding social media use (Wu, 2023) among others.

The link between criminality and individual personality traits has been also frequently revisited (Tharshini et al., 2021), with a growing importance over time (Ozer and Akbas, 2023) and with an emphasis in understanding predictive variables (O’Riordan and O’Connell, 2014) for both real life and cybercrime scenarios (van de Weijer and Leukfeldt, 2017; Hani et al., 2024).

From the psycholinguistic perspective, it has been shown that each of the five dimensions is characterized by different styles in language usage (Stajner and Yenikent, 2020). While traditional Big 5 assessments rely on self-reports using questionnaires, Natural Language Processing (NLP) approaches to personality detection have gained popularity with the expansion of social media, thus allowing anyone to infer people’s personality traits simply by analyzing their social media posts, without any special training for psychological assessment (Peters and Matz, 2024). However, based on recent advances, there is still room for improvement in this area when comparing the performance of NLP personality profiling systems against human judgment (Barriere et al., 2022, 2023).

Related Work

Vora et al. (2020) and Beck and Jackson (2020) survey personality prediction models. Sutton et al. (2023), Srinivas et al. (2023) and Gruschka et al. (2023) build models by finetuning or pretraining BERT (Devlin et al., 2019). Other Transformer architectures such as DeBERTa (He et al., 2020) has been also successfully used (He et al., 2021). The recent advances in the Large Language Models (LLM) field have shown that zero-shot approaches to personality detection are also viable (V Ganesanet al., 2023). For non-NLP solutions, Ghosh et al. (2022) developed a support vector machine (SVM) model using only demographic features.

Results

Best results were achieved using Random Forest regression using non-textual features in a multi-ouput scenario. The feature importance plot shows that interpersonal reactivity indexes (IRI) are strong predictors of user personality. Likewise, other factors such as gender, race or education seemed to have little to no impact.

Conclusions and Future Work

Based on this analysis and the results of the last 2 previous editions of WASSA, we can conclude that personality detection using NLP still remains a challenging task. However, low-cost, large-scale personality profiling could pose a future threat to privacy and consent, specially if this data is misused or wrongly assessed (Peters and Matz, 2024). An analysis of safeguarding measures to disrupt NLP-based automatic profiling (Mosquera, 2022) is left to a future work.

References

Raj Agarwal, Roshni Mukherjee, Nitin More, and Suvarna Pawar. 2023. Big five (o.c.e.a.n): A new way to recruit. pages 1–7.

Valentin Barriere, João Sedoc, Shabnam Tafreshi, and Salvatore Giorgi. 2023. Findings of WASSA 2023 shared task on empathy, emotion and personality detection in conversation and reactions to news articles. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 511–525, Toronto, Canada. Association for Computational Linguistics.

Valentin Barriere, Shabnam Tafreshi, João Sedoc, and Sawsan Alqahtani. 2022. WASSA 2022 shared task: Predicting empathy, emotion and personality in reaction to news stories. In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 214–227, Dublin, Ireland. Association for Computational Linguistics.

Emorie Beck and Joshua Jackson. 2020. A megaanalysis of personality prediction: Robustness and boundary conditions.

Diera Desmonda, So Jimmy, and Mohammad Annas. 2024. Understanding the influences of hedonic personality towards advertising avoidance on social media. The South East Asian Journal of Management, 18:81–104.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

John M Digman. 1990. Personality structure: Emergence of the five-factor model. Annual review of psychology, 41(1):417–440.

Jim Exley, Patrick C. Doyle, John Grable, and W. Keith Campbell. 2022. Ocean wealth profiles: A latent profile analysis of personality traits and financial outcomes. Personality and Individual Differences, 185:111300.

Soumitra Ghosh, Dhirendra Maurya, Asif Ekbal, and Pushpak Bhattacharyya. 2022. Team IITPAINLPML at WASSA 2022: Empathy detection, emotion classification and personality detection. In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 255–260, Dublin, Ireland. Association for Computational Linguistics.

Fabio Gruschka, Allison Lahnala, Charles Welch, and Lucie Flek. 2023. Domain transfer for empathy, distress, and personality prediction. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 553–557, Toronto, Canada. Association for Computational Linguistics.

Umema Hani, Osama Sohaib, Khalid Khan, Asma Aleidi, and Noman Islam. 2024. Psychological profiling of hackers via machine learning toward sustainable cybersecurity. Frontiers in Computer Science, 6.

Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding Enhanced bert with disentangled attention. Preprint, arXiv:2006.03654.

Alejandro Mosquera. 2022. Alejandro Mosquera at politices 2022: Towards robust spanish author profiling and lessons learned from adversarial attacks. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2022) co-located with the Conference of the Spanish Society for Natural Language Processing (SEPLN 2022), XXXVIII International Conference.

Murat Ozer and Halil Akbas. 2023. The relevance of personality to criminal behavior. In Sevgi Güney, editor, Criminal Behavior, chapter 6. IntechOpen, Rijeka.

Cáit O’Riordan and Michael O’Connell. 2014. Predicting adult involvement in crime: Personality measures are significant, socio-economic measures are not. Personality and Individual Differences, 68:98–101.

Heinrich Peters and Sandra Matz. 2024. Large language models can infer psychological dispositions of social media users. Preprint, arXiv:2309.08631.

Addepalli Sai Srinivas, Nabarun Barua, and Santanu Pal. 2023. Team_Hawk at WASSA 2023 empathy, emotion, and personality shared task: Multi-tasking multi-encoder based transformers for empathy and emotion prediction in conversations. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 542–547, Toronto, Canada. Association for Computational Linguistics.

Sanja Stajner and Seren Yenikent. 2020. A survey of automatic personality detection from texts. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6284–6295, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Adam Sutton, Almog Simchon, Matthew Edwards, and Stephan Lewandowsky. 2023. You are what you read: Inferring personality from consumed textual content. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 28–38, Toronto, Canada. Association for Computational Linguistics.

NK Tharshini, Fauziah Ibrahim, Mohammad Rahim Kamaluddin, Balan Rathakrishnan, and Norruzeyati Che Mohd Nasir. 2021. The link between individual personality traits and criminality: A systematic review. International journal of environmental research and public health, 18(16):8663.

Adithya V Ganesan, Yash Kumar Lal, August Nilsson, and H. Schwartz. 2023. Systematic evaluation of GPT-3 for zero-shot personality estimation. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 390–400, Toronto, Canada. Association for Computational Linguistics.

Steve van de Weijer and Eric Leukfeldt. 2017. Big five personality traits of cybercrime victims. Cyberpsychology, Behavior, and Social Networking, 20.

Hetal Vora, Mamta Bhamare, and Dr Kumar. 2020.Personality prediction from social media text: An overview. International Journal of Engineering Research and, V9.

Lin Wu. 2023. Relationship between social media use and personality. Lecture Notes in Education Psychology and Public Media, 9:325–330.