Motivations

Opinion extraction from texts has received a great interest over a decade (Liu, 2015). This interest has increased with the rise of the social web, and the possibility to widely broadcast emotions, evaluations and opinions. Extraction methods rely on a variety of approaches going from bag-of-words representations, to more sophisticated models that deal with various context-dependent phenomena such as contextual effects deriving from intensifiers, and hedging and other related discourse-level phenomena, including discourse structure and coherent relations. Although current systems achieve relatively good results on objective vs. subjective classification, polarity analysis still need further improvement to address more sophisticated language devices, such as figurative language.

Figurative language makes use of figures of speech to convey non-literal meaning, i.e., meaning that is not strictly the conventional or intended meaning of the individual words in the figurative expression. Figurative language encompasses a variety of phenomena, including metaphor, oxymoron, idiomatic expressions, puns, humor, irony and sarcasm. Figurative language detection has gained relevance recently, due to its importance for efficient sentiment analysis (Maynard and Greenwood 2014; Ghosh et al. 2015, Benamara et al, 2017). This shared task focuses on irony, sarcasm and humor.

Irony is a complex linguistic phenomenon widely studied in philosophy and linguistics (Grice 1975; Sperber and Wilson 1981; Utsumi 1996). Glossing over differences across approaches, irony can be defined as an incongruity between the literal meaning of an utterance and its intended meaning. For example, to express a negative opinion towards a cell phone, one can either employ a literal form using a negative opinion word, as in This phone is a disaster , or a non-literal form by using a positive word, as in What an excellent phone!! In computational linguistics, irony is often used as an umbrella term that includes sarcasm, although some researchers make a distinction between irony and sarcasm, considering that sarcasm tends to be harsher, humiliating, degrading and more aggressive (Clift 1999).

Figurative language detection and its role in sentiment analysis has been the focus of past shared tasks SemEval 2015 Task 11 (Ghosh et al. 2015) and SENTIPOLC@Evalita in both 2014 and 2016 editions (Basile et al., 2014; Barbieri et al., 2016) on respectively English and Italian tweets. For the first time, DEFT proposes to study these complex phenomena on French tweets. The task is open to everyone from industry and academia.

Task description

The goal of DEFT this year is sentiment analysis of French tweets and identification of French tweets containing figurative language devices. The evaluation campaign is divided into three tasks with an increasing level of complexity. Participants may choose to participate in one or more tasks.

1. Task 1: Polarity analysis of non figurative tweets
   Given a tweet that does not contain any figurative language devices, classify it into one of the following four classes: positive, negative, mixed (both positive and negative) and objective.


2. Task 2: Figurative language detection
   Given a tweet, classify it into two classes: figurative vs. non figurative. Figurative tweets are tweets that contain ironic, sarcacstic or humorous statements.


3. Task 3: Polarity analysis of figurative and non figurative tweets
   Given a tweet that contains or not figurative language devices, classify it into one of the following four classes: positive, negative, mixed (both positive and negative) and objective.

References

--(Basile et al, 2014). Valerio Basile, Andrea Bolioli, Malvina Nissim, Viviana Patti, and Paolo Rosso. 2014. Overview of the Evalita 2014 SENTIment POLarity Classification Task. In Proc. of EVALITA 2014, pages 50–57, Pisa, Italy. Pisa University Press.
--(Barbieri et al, 2016). Francesco Barbieri, Valerio Basile, Danilo Croce, Malvina Nissim, Nicole Novielli, and Viviana Patti. 2016. Overview of the Evalita 2016 SENTIment POLarity Classification Task. In Proceedings of Third Italian Conference on Computational Linguistics (CLiC-it 2016) and Fifth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2016), Napoli, Italy, December 5-7, 2016., volume 1749 of CEUR Workshop Proceedings. CEUR-WS.org.
--(Benamara et al, 2017). Farah Benamara, Maite Taboada, Yannick Mathieu. Evaluative language beyond bags of words: Linguistic insights and Computational Applications. To appear in Computational Linguistics.
--(Clift, 1999). Rebecca Clift. 1999. Irony in conversation. Language in Society, 28:523–553.
--(Grice, 1975). Herbert Paul Grice, Peter Cole, and Jerry L Morgan. 1975. Syntax and semantics. Logic and conversation, 3:41–58.
--(Ghosh et al, 2015). Aniruddha Ghosh, Guofu Li, Tony Veale, Paolo Rosso, Ekaterina Shutova, John Barnden, and Antonio Reyes. 2015. Semeval-2015 task 11: Sentiment Analysis of Figurative Language in Twitter. In Proceedings of SemEval 2015, Co-located with NAACL, page 470-478.
--(Liu 2015) Bing Liu. Sentiment Analysis: Mining Opinions, Sentiments, and Emotions. 2015. Cambridge University Press, Cambridge
--(Maynard and Greenwood, 2014). Diana Maynard and Mark A Greenwood. Who cares about sarcastic tweets? Investigating the impact of sarcasm on sentiment analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, pages 4238–4243.
--(Sperber and Wilson, 1981). Dan Sperber and Deirdre Wilson. Irony and the use-mention distinction. Radical pragmatics, 49:295–318.
--(Utsumi, 1996). Akira Utsumi. 1996. A unified theory of irony and its computational formalization. In Proceedings of COLING, the 16th conference on Computational Linguistics-Volume 2, pages 962–967.

Data and evaluation metrics

Data

The DEFT 2017 dataset is composed of French tweets about hot topics (politics, sport, artists, locations, Arab Spring, environment, racism, health, social media) discussed in the French media from Spring 2014 until Autumn 2016. We removed duplicates, retweets and tweets containing pictures which would need to be interpreted to understand the figurative content.

Data format for each task

Each participating team will initially have access to the training data only. Later, the unlabelled test data will be released (see the timeframe below). After the assessment, the labels for the test data will be released as well.

In each stage, participants will have a CSV file containing for each tweet the text of the tweet. Emojis in tweets are remplaced by their corresponding code in Unicode.

The CSV file has the following format: (see examples of tweets on the Annotation guidelines page)

(Identifiers are defined for the task)

• Task 1: 4882 tweets
  

  id  tweet  objective

  id  tweet  positive

  id  tweet  negative

  id  tweet  mixted



• Task 2: 7317 tweets
  

  id  tweet  nonfigurative

  id  tweet  figurative



• Task 3: 6399 tweets
  

  id  tweet  objective

  id  tweet  positive

  id  tweet  negative

  id  tweet  mixted

Evaluation

For each task, participants have to send a CSV file similar to the one given during the training stage.

For each task, participants can submit a maximum of three runs. If participants want to submit more runs, please contact the organisers.

The evaluation will be performed according to the standard metrics known in literature (precision, recall and F-measure).

Once participants receive the test dataset, they are morally engaged to submit a run by sending us their results and a paper describing their systems. The paper consists of an abstract and a technical report including a brief description of their approach, an illustration of their experiments, in particular techniques and resources used, and an analysis of their results for the publication in the proceedings. Papers can be written in French or English (for non French).

Papers must be submitted in PDF format, following the TALN/RECITAL conference style mentioned (TALN template).

Annotation guidelines

During the annotation campaign, annotators had access to the text of the tweet only without having any additional information about the context of the tweet such as author profiles, retweets, conversation thread, hashtags indcating figurative language (#ironie, #humour, ...), etc. For this reason, participants are asked to use the text of the tweets given by the organizers and not tweets edited on Twitter. If needed, annotators could use the web pages pointed out by the URL present in the text of the tweet (these pages may have disappeared since the tweet collection).

Task 1: Sentiment analysis of non figurative tweets

We focus on the opinions expressed by the tweet's author towards a target. A tweet may contains several opinions as well as several targets. Opinion can be explicit (using explicit poistive or negative opinion words) or implicit.

The overall polarity of a tweet can be one of the following four mutually exclusive classes:

• Objective: The author does not give his own opinion. He mainly relates a fact or an event, quotes a statement or resums an extract (journal titles, etc.).
  
  

  Syrie: Le Pentagone affirme avoir tué le chef de Khorasan - Monde - lematin.ch

  #Poutine critique les Etats-Unis sur la question des frappes en #Syrie  #SaveSyria

  Cécile Duflot: "je ne crois pas que DSK soit en mesure de donner des leçons"

  Je reprends le replay de #DALS

• Positive: The author expresses only a positive opinion towards a fact, an event or a quotation.
  
  

  A voir ce soir sur Fr3 l'excellent doc sur l'affaire du #Carlton #dsk @France3tv: L'affaire du Carlton : doc. Inedit

  Ce soir la reprise du #MeilleurPatissier @M6 @LMP_M6 je dis oui 👍

• Negative: The author expresses only a negative opinion towards a fact, an event or a quotation.
  
  

  "La Russie ne va pas imposer la paix à coup de bombes" Ok Obama l'hôpital la charité tout ça.

  #Valls n'est pas le super héros qu'il s'imagine être

• Mixed: The author expresses either a subjective statement i.e., a point of view without explicity giving a positive or a negative opinion, or a mixed opinion (both positive and negative).
  
  

  Fillon qui souhaite bonne chance à Valls. Je sais pas si je pleure ou si je ris. J'hésite.

  C'est un mal pour un bien si la hollande va pas a l'EURO, les joueurs pourront profiter des coffee shop de leur pays au moins

  J'ai faillit pleurer devant la fin de fast and furious 7

  Et #Sarkozy qui veut nous refaire le coup du nouveau traité comme en 2005

Task 2: Figurative language detection

We focus on three figurative language devices: irony, sarcasm and humor. If a tweet contains at least one figurative language statement, it is considered as figurative otherwise non firgurative.

Figurative

Humor:

• Pun:
  

  Si Morandini meurt subitement dans son émission "vous êtes en direct" on pourra dire qu'il est morandirect? #MDRRRR

  #USElection A force de claironner, Trump-pete

• Joke:
  

  L'Angleterre le seul pays qui quitte deux fois l'euro en 4 jours #Brexit #Angleterre #ANGISL

• Parody:
  

  #Remaniement Franck Ribéry : « Je n’ai jamais pensé être ministre de l’éducation » [Interview] - https://edukactus.wordpress.com/2014/03/27/

Irony and sarcasm

Ca va bien en Corée du Nord, ils ouvrent un super parc aquatique ! #northkorea 

Les arbitres étaient digne de la ligue 1 de football tellement ils étaient bons

J'adore le taff , manger en 5 minutes et travailler jusqu'a 20h c'est top 😄

Non figurative

C'est dommage, Émilie avait l'air bien #adp #ADP2016

Le régime de Bachar al-Assad regagne du terrain en #Syrie ►

Olivier Besancenot invité politique de #ONPC ! 👍

Task 3: Sentiment analysis of figurative and non figurative tweets

This task is similar to task 1 except that there are figurative tweets in the dataset. Figurative tweets included in this task are tweets containing irony and sarcasm. Humorous tweets have been discarded.

Here are some examples of figurative tweets annotated according to the overall polarity of the opinion they convey. Examples of polarity annotation of non figurative tweets are given in annotation guidelines of task 1.

• Objective: The author expresses an objective statement which can be ironic (ironic quotation, ironic parody, ironic journal article title, etc.) or relates a fact or an event which happens in an ironic context (twist of fate, situational irony, etc.).
  

  #DSK songe à déménager en Ontario s'il est libéré des accusations de proxénétisme... http://www.ledevoir.com/societe/justice/345944/maisons-closes-la-cour-d-appel-de-l-ontario-invalide-la-loi-federale

• Positive: The author expresses only a positive opinion towards a fact, an event or a quotation
  

  @lesinrocks La pollution n'est pas une si mauvaise chose : gratuité des transports et journalistes inspirés http://ow.ly/uClzW

• Negative: The author expresses only a negative opinion towards a fact, an event or a quotation
  

  #Hollande est vraiment un très bon diplomate… #Algérie http://www.leparisien.fr/international/blague-de-hollande-des-officiels-algeriens-denoncent-une-provocation-21-12-2013-3430495.php

  Bonne nouvelle! #chômage "@lemondefr: Alerte : le taux de chômage atteint son plus haut depuis juillet 1997

• Mixed:The author expresses either a point of view without explicity giving a positive or a negative opinion, or a mixed opinion (both positive and negative).
  

  Un truc avec DSK, mais quoi ? Aucun site internet n'en parle. Surement parce qu' on ne sait rien de ce qu'il s'est réellement passé ?

  #Sarkozy contre le droit du sol ? Modifier le nom d'un parti vous change un homme ! 

Proceedings

Presentation of the task and results

Farah Benamara, Cyril Grouin, Jihen Karoui, Véronique Moriceau, Isabelle Robba
Analyse d'opinion et langage figuratif dans des tweets : présentation et résultats du Défi Fouille de Textes DEFT2017
Actes de l'atelier DEFT2017 associé à la conférence TALN, 26 juin 2017, Orléans, France.

Download proceedings (in French)

Committee

Scientific committee

• Patrice Bellot, LSIS, Aix-Marseille Université
• Caroline Brun, XRCE-XEROX, Grenoble
• Béatrice Daille, LINA, Université de Nantes
• Guy Lapalme, RALI, Université de Montréal
• Patrick Paroubek, LIMSI, CNRS, Université Paris-Saclay

Organization committee

• Farah Benamara, IRIT, Université de Toulouse
• Cyril Grouin, LIMSI, CNRS, Université Paris-Saclay
• Jihen Karoui, IRIT, Université de Toulouse
• Véronique Moriceau, LIMSI, CNRS, Univ. Paris-Sud, Université Paris-Saclay
• Isabelle Robba, LIMSI, CNRS, UVSQ, Université Paris-Saclay