CoNLL-2005 Shared Task: 

Semantic Role Labeling: Data & Software


Introduction    F.A.Q.    References    CoNLL Conferences


CoNLL-2005 :       Description&Goal       Examples       Data&Software      Systems&Results 


CoNLL-2004 :      Summary Page (data, systems & results)



Data

The current data distribution (released on March 4th, 2005) corresponds to the Wall Street sections 02-21 as training set, and section 24 as development set.

Each dataset is distributed split into many separate folders, each grouping files of different annotations (see details in the README file):
Standard UNIX commands can be used to merge all the necessary files into a single training/development complete file with all the information. As an example, we distribute the make-trainset.sh shell script, which constructs a complete training set (20 sections) containing words, syntax (Collins full parses), named entities, verb senses, and propositions.

Words and gold parse trees aligning with the previous annotations are to be extracted from the Penn Treebank II collection distributed through LDC or from the CoNLL-2005 shared task collection freely distributed through LDC under an  evaluation license. Use 'wsj-removetraces.pl' and 'wsj-to-se.pl' utilities (from the srlconll software package) to convert Penn Treebank annotation style into the column-based format of the shared task (details in the README file).



CoNLL-2005 Shared Task Release
released: March 04, 2005
last update: March 14, 2005
   NOTE:
The data of this distribution is to be completed with the Wall Street Journal sections 02-21 and 24 of the Penn Treebank II collection. For participants not owning a valid license on the Penn Treebank II collection, LDC is providing an evaluation license allowing to freely download and use the part corresponding to the CoNLL-2005 shared task datasets during competition time.

    Distribution Package, containing all annotation layers, README file, and scripts

conll05st-release.tar.gz last updated on March 14, 2005


Files of the Distribution (those not forming part of annotation layers)

README Description file
CHANGES Changes on the release since March 04, 2005
make-trainset.sh Script for constructing a complete training set
make-devset.sh Script for constructing a complete development set


Annotation Layer Status Released Description

props.tar.gz stable 2005/03/04 Correct props for the sample (targets + args)

synt.upc.tar.gz stable 2005/03/11 Partial parses of UPC processors [GM03] and [CM03]

synt.col2.tar.gz stable 2005/03/10 Full syntax of Collins' [Col99], model2, w/ WSJ NTs

synt.col2h.tar.gz stable 2005/03/04 Full syntax of Collins' [Col99], model2, w/ lexicalized NTs

synt.cha.tar.gz stable 2005/03/14 Full syntax of Charniak' [Cha00]

ne.tar.gz stable 2005/03/11 Named Entities of (Chieu and Ng 03)

senses.tar.gz stable 2005/03/04 Correct senses of target verbs, extracted from Propbank
These will not be provided for the test set.

 
    CoNLL-2005 SHARED TASK : TEST DATA OF WSJ AND BROWN (posted February 2006)
    conll05st-tests.tar.gz : includes correct propositions and input annotations



Software

If you find bugs in the distributed software, please get in touch asap with the organizers at srlconll <at> lsi.upc.edu.

srl-eval

The srl-eval.pl program is available for download. (last updated: April 21, 2005)

The srl-eval.pl program is the official script for evaluation of CoNLL-2005 Shared Task systems. It expects two parameters: The first is the name of the file containing correct propositions; the second is the name of the file containing predicted propositions. Both files are expected to follow the format of "props" files (first column: target verbs; remaining columns: args of each target verb). It is required that both files contain the same sentences and the same target verbs. The program outputs performance measures based on precision, recall and F1. The overall F1 measure will be the measure used to compare the performance of systems.

Use the option "-latex" to produce a table of results in LaTeX.

The program is implemented in Perl, object oriented, with each object embedded in the script. It is part of the srlconll software package.

srlconll scripts

The srlconll package contains a suite of scripts to process the data of the CoNLL-2004 and CoNLL-2005 Shared Tasks.
The package includes the srl-eval.pl script, baseline systems, and other scripts to change formats and perform other utilities with data files.

Package : srlconll-1.1.tgz
Posted on December 1, 2005. For description and usage, see the README file included in the package.
This version includes the script to convert from PropBank-1.0 to the CoNLL-2005 formats. It is also being updated to process NomBank-0.8 (very preliminary). See README-databuilt for more information.

Package : srlconll-1.0.tgz
Posted on March 4, 2005. For description and usage, see the README file included in the package.
(This is an updated version of the srlconll-beta.tgz package released on February 17, 2005)

The package is implemented in Perl, and contains a library of objects which might be useful to participants. Feel free to use the code and develop further functionalities. Feedback is welcome!

Baseline

For now, the baseline system of CoNLL-2004 is available : srl-baseline04.pl

We expect to come up with alternative baseline systems. If you have a baseline system, please let us know.

collins-fixp.pl

In Collins' trees the punctuation terminals are included within the preceding phrases. This issue makes the correspondance between syntactic nodes and propositional arguments not direct at all.

On April 19, we've made available a script for fixing some of the punctuation terminals. It solves most of the simple cases, although, obviously, it fails at the complex ones. Download the script from :

    http://www.lsi.upc.edu/~srlconll/collins-fixp.pl  (April 19, 2005)

Once downloaded, read the documentation by executing :
 $ perl collins-fixp.pl -h
Example of usage:
 $ gunzip -c devel/synt.col2/devel.24.synt.col2.gz | collins-fixp.pl -s
 
If you find bugs or inconsistencies, report them to us as soon as possible.
Ideas/code to improve the script are more than welcome.

Official Resources


Last update: February 23, 2006. Xavier Carreras, Lluís Màrquez.