Opinion extraction from texts has received a great interest over a decade (Liu, 2015). This interest has increased with the rise of the social web, and the possibility to widely broadcast emotions, evaluations and opinions. Extraction methods rely on a variety of approaches going from bag-of-words representations, to more sophisticated models that deal with various context-dependent phenomena such as contextual effects deriving from intensifiers, and hedging and other related discourse-level phenomena, including discourse structure and coherent relations. Although current systems achieve relatively good results on objective vs. subjective classification, polarity analysis still need further improvement to address more sophisticated language devices, such as figurative language.
Figurative language makes use of figures of speech to convey non-literal meaning, i.e., meaning that is not strictly the conventional or intended meaning of the individual words in the figurative expression. Figurative language encompasses a variety of phenomena, including metaphor, oxymoron, idiomatic expressions, puns, humor, irony and sarcasm. Figurative language detection has gained relevance recently, due to its importance for efficient sentiment analysis (Maynard and Greenwood 2014; Ghosh et al. 2015, Benamara et al, 2017). This shared task focuses on irony, sarcasm and humor.
Irony is a complex linguistic phenomenon widely studied in philosophy and linguistics (Grice 1975; Sperber and Wilson 1981; Utsumi 1996). Glossing over differences across approaches, irony can be defined as an incongruity between the literal meaning of an utterance and its intended meaning. For example, to express a negative opinion towards a cell phone, one can either employ a literal form using a negative opinion word, as in This phone is a disaster , or a non-literal form by using a positive word, as in What an excellent phone!! In computational linguistics, irony is often used as an umbrella term that includes sarcasm, although some researchers make a distinction between irony and sarcasm, considering that sarcasm tends to be harsher, humiliating, degrading and more aggressive (Clift 1999).
Figurative language detection and its role in sentiment analysis has been the focus of past shared tasks SemEval 2015 Task 11 (Ghosh et al. 2015) and SENTIPOLC@Evalita in both 2014 and 2016 editions (Basile et al., 2014; Barbieri et al., 2016) on respectively English and Italian tweets. For the first time, DEFT proposes to study these complex phenomena on French tweets. The task is open to everyone from industry and academia.
The goal of DEFT this year is sentiment analysis of French tweets and identification of French tweets containing figurative language devices. The evaluation campaign is divided into three tasks with an increasing level of complexity. Participants may choose to participate in one or more tasks.
--(Basile et al, 2014). Valerio Basile, Andrea Bolioli, Malvina Nissim, Viviana
Patti, and Paolo Rosso. 2014. Overview of the Evalita 2014 SENTIment POLarity
Classification Task. In Proc. of EVALITA 2014, pages 50–57, Pisa, Italy. Pisa
University Press.
--(Barbieri et al, 2016). Francesco Barbieri, Valerio
Basile, Danilo Croce, Malvina Nissim, Nicole Novielli, and Viviana Patti. 2016.
Overview of the Evalita 2016 SENTIment POLarity Classification Task. In Proceedings
of Third Italian Conference on Computational Linguistics (CLiC-it 2016) and Fifth
Evaluation Campaign of Natural Language Processing and Speech Tools for Italian.
Final Workshop (EVALITA 2016), Napoli, Italy, December 5-7, 2016., volume 1749 of
CEUR Workshop Proceedings. CEUR-WS.org.
--(Benamara et al, 2017). Farah Benamara, Maite Taboada, Yannick Mathieu.
Evaluative language beyond bags of words: Linguistic insights and Computational
Applications. To appear in Computational Linguistics.
--(Clift, 1999).
Rebecca Clift. 1999. Irony in conversation. Language in Society, 28:523–553.
--(Grice, 1975). Herbert Paul Grice, Peter Cole, and Jerry L Morgan. 1975.
Syntax and semantics. Logic and conversation, 3:41–58.
--(Ghosh et al,
2015). Aniruddha Ghosh, Guofu Li, Tony Veale, Paolo Rosso, Ekaterina Shutova,
John Barnden, and Antonio Reyes. 2015. Semeval-2015 task 11: Sentiment
Analysis of Figurative Language in Twitter. In Proceedings of SemEval 2015,
Co-located with NAACL, page 470-478.
--(Liu 2015) Bing Liu. Sentiment Analysis: Mining Opinions, Sentiments, and Emotions.
2015. Cambridge University Press, Cambridge
--(Maynard and Greenwood, 2014). Diana
Maynard and Mark A Greenwood. Who cares about sarcastic tweets? Investigating the impact
of sarcasm on sentiment analysis. In Proceedings of the Ninth International Conference on
Language Resources and Evaluation, LREC 2014, pages 4238–4243.
--(Sperber and
Wilson, 1981). Dan Sperber and Deirdre Wilson. Irony and the use-mention distinction.
Radical pragmatics, 49:295–318.
--(Utsumi, 1996). Akira Utsumi. 1996. A unified
theory of irony and its computational formalization. In Proceedings of COLING, the 16th
conference on Computational Linguistics-Volume 2, pages 962–967.
The DEFT 2017 dataset is composed of French tweets about hot topics (politics, sport, artists, locations, Arab Spring, environment, racism, health, social media) discussed in the French media from Spring 2014 until Autumn 2016. We removed duplicates, retweets and tweets containing pictures which would need to be interpreted to understand the figurative content.
Data format for each task
Each participating team will initially have access to the training data only. Later, the unlabelled test data will be released (see the timeframe below). After the assessment, the labels for the test data will be released as well.
In each stage, participants will have a CSV file containing for each tweet the text of the tweet. Emojis in tweets are remplaced by their corresponding code in Unicode.
The CSV file has the following format: (see examples of tweets on the Annotation guidelines page)
(Identifiers are defined for the task)
id tweet objective
id tweet positive
id tweet negative
id tweet mixted
id tweet nonfigurative
id tweet figurative
id tweet objective
id tweet positive
id tweet negative
id tweet mixted
For each task, participants have to send a CSV file similar to the one given during the training stage.
For each task, participants can submit a maximum of three runs. If participants want to submit more runs, please contact the organisers.
The evaluation will be performed according to the standard metrics known in literature (precision, recall and F-measure).
Once participants receive the test dataset, they are morally engaged to submit a run by sending us their results and a paper describing their systems. The paper consists of an abstract and a technical report including a brief description of their approach, an illustration of their experiments, in particular techniques and resources used, and an analysis of their results for the publication in the proceedings. Papers can be written in French or English (for non French).
Papers must be submitted in PDF format, following the TALN/RECITAL conference style mentioned (TALN template).
During the annotation campaign, annotators had access to the text of the tweet only without having any additional information about the context of the tweet such as author profiles, retweets, conversation thread, hashtags indcating figurative language (#ironie, #humour, ...), etc. For this reason, participants are asked to use the text of the tweets given by the organizers and not tweets edited on Twitter. If needed, annotators could use the web pages pointed out by the URL present in the text of the tweet (these pages may have disappeared since the tweet collection).
We focus on the opinions expressed by the tweet's author towards a target. A tweet may contains several opinions as well as several targets. Opinion can be explicit (using explicit poistive or negative opinion words) or implicit.
The overall polarity of a tweet can be one of the following four mutually exclusive classes:
Syrie: Le Pentagone affirme avoir tué le chef de Khorasan - Monde - lematin.ch
#Poutine critique les Etats-Unis sur la question des frappes en #Syrie #SaveSyria
Cécile Duflot: "je ne crois pas que DSK soit en mesure de donner des leçons"
Je reprends le replay de #DALS
A voir ce soir sur Fr3 l'excellent doc sur l'affaire du #Carlton #dsk @France3tv: L'affaire du Carlton : doc. Inedit
Ce soir la reprise du #MeilleurPatissier @M6 @LMP_M6 je dis oui 👍
"La Russie ne va pas imposer la paix à coup de bombes" Ok Obama l'hôpital la charité tout ça.
#Valls n'est pas le super héros qu'il s'imagine être
Fillon qui souhaite bonne chance à Valls. Je sais pas si je pleure ou si je ris. J'hésite.
C'est un mal pour un bien si la hollande va pas a l'EURO, les joueurs pourront profiter des coffee shop de leur pays au moins
J'ai faillit pleurer devant la fin de fast and furious 7
Et #Sarkozy qui veut nous refaire le coup du nouveau traité comme en 2005
We focus on three figurative language devices: irony, sarcasm and humor. If a tweet contains at least one figurative language statement, it is considered as figurative otherwise non firgurative.
Humor:
Si Morandini meurt subitement dans son émission "vous êtes en direct" on pourra dire qu'il est morandirect? #MDRRRR
#USElection A force de claironner, Trump-pete
L'Angleterre le seul pays qui quitte deux fois l'euro en 4 jours #Brexit #Angleterre #ANGISL
#Remaniement Franck Ribéry : « Je n’ai jamais pensé être ministre de l’éducation » [Interview] - https://edukactus.wordpress.com/2014/03/27/
Irony and sarcasm
Ca va bien en Corée du Nord, ils ouvrent un super parc aquatique ! #northkorea
Les arbitres étaient digne de la ligue 1 de football tellement ils étaient bons
J'adore le taff , manger en 5 minutes et travailler jusqu'a 20h c'est top 😄
C'est dommage, Émilie avait l'air bien #adp #ADP2016
Le régime de Bachar al-Assad regagne du terrain en #Syrie ►
Olivier Besancenot invité politique de #ONPC ! 👍
This task is similar to task 1 except that there are figurative tweets in the dataset. Figurative tweets included in this task are tweets containing irony and sarcasm. Humorous tweets have been discarded.
Here are some examples of figurative tweets annotated according to the overall polarity of the opinion they convey. Examples of polarity annotation of non figurative tweets are given in annotation guidelines of task 1.
#DSK songe à déménager en Ontario s'il est libéré des accusations de proxénétisme... http://www.ledevoir.com/societe/justice/345944/maisons-closes-la-cour-d-appel-de-l-ontario-invalide-la-loi-federale
@lesinrocks La pollution n'est pas une si mauvaise chose : gratuité des transports et journalistes inspirés http://ow.ly/uClzW
#Hollande est vraiment un très bon diplomate… #Algérie http://www.leparisien.fr/international/blague-de-hollande-des-officiels-algeriens-denoncent-une-provocation-21-12-2013-3430495.php
Bonne nouvelle! #chômage "@lemondefr: Alerte : le taux de chômage atteint son plus haut depuis juillet 1997
Un truc avec DSK, mais quoi ? Aucun site internet n'en parle. Surement parce qu' on ne sait rien de ce qu'il s'est réellement passé ?
#Sarkozy contre le droit du sol ? Modifier le nom d'un parti vous change un homme !
Farah Benamara, Cyril Grouin, Jihen Karoui, Véronique Moriceau, Isabelle Robba
Analyse d'opinion et langage figuratif dans des tweets : présentation et résultats du Défi Fouille de Textes DEFT2017
Actes de l'atelier DEFT2017 associé à la conférence TALN, 26 juin 2017, Orléans, France.