PSB 2016 Social Media Mining Shared Task Workshop



Task 1 Description

Task 2 Description

Task 3 Description







Task 1 Evaluation:


For this task, the evaluation metric will be the ADR F-score. The binary annotation consists of two classes: ADR and non-ADR. The intent of this task is to devise automatic classification techniques for detecting ADR assertive user posts. As such, the evaluation is based on the harmonic mean of the recall and precision for the ADR class. The ADR F-score has been previously used for evaluation of systems performing this task: Portable automatic text classification for adverse drug reaction detection via multi-corpus training


The system with the highest ADR F-score on the test set will be considered to be the winner.


Task 2 Evaluation:


F-measure is also used as the metric for evaluation in this task. True positives, true negatives and false negatives for a system are identified via approximate matching. The F-measure is then computed from these values. The evaluation will be similar to the one described in this paper: Pharmacovigilance from social media: mining adverse drug reaction mentions using sequence labeling with word embedding cluster features


Task 3 Evaluation:


For this task, the evaluation metric used will be accuracy: # correct/total.

# correct is the number of correct system predictions. A system prediction will be considered correct if the predicted CUI is identical, is a synonym, or has is-a relationship to the gold standard concept.




© DIEGO LAB 2015 Competition Organisers.