Corpora@Stanford

Overview

This page has replaced an older corpus inventory page as of 04/01/2004. If you for some reason want to access the old page that is still possible.

Beside the corpora that we own on CD (which you can get from the Corpus TA, many corpora are installed and ready-to-use on either the AFS space or the corpus computer (CC). Some additional speech corpora may be available in the phonetics lab (see also the tip below). Please contact the Phonetics RA or the Corpus TA if you have questions about speech corpora. Although this page is not intended to give an overview of available online corpora (outside of Stanford), a very small selection is actually provided on this page. Nevertheless, we strongly encourage you to take advantage of the variety of freely accessible online corpora - for some links to sites that will provide you with an overview of the colorful world of online corpora, please browse & click through our subjectively construed list of the top 10 info-sources "out there".

This page contains the following information:

• Recent acquisitions
• LDC Corpora
• Non-LDC Corpora
• Corpora on the WWW

Tip-1: Since this page is not a database you may find it useful to just use your browser search function if you are interested in a specific corpus, or corpora for a specific language, or any other key word (e.g. "syntactically annotated"). Try it!

Tip-2: Interested in prosody? See Florian Jaeger's page of annotated links to prosodically annotated speech corpora.

Tip-3: If you cannot find your the corpus you were looking for - see if we can order it! Our department is a member of the LDC (Linguistic Data Consortium) which gives us free access to a lot of corpora. It may also be able to order other corpora. Simply tell the corpus TA what you need, but have a look at the information on "ordering corpora from the LDC" first, or browse the web to see whether what you need can be found online (and maybe for free). Also keep in mind that our inventory may be outdated since our resources to maintain this list are limited.

Tip-4: Thematically grouped corpora on AFS: There are a couple of thematic groups of corpora on AFS:

• IE (Information Extraction):
  /afs/ir/data/linguistic-data/IE
  • Corporate Acquisitions Annotated Reuters Texts:
    /afs/ir/data/linguistic-data/IE/CorpAcq-Reuters-Freitag
  • Kristie Seymore's Information Extraction Data:
    /afs/ir/data/linguistic-data/IE/Kristie-Seymore-IE
  • MUC3-4 (Message Understanding Conference):
    /afs/ir/data/linguistic-data/IE/MUC/MUC3-4
  • MUC-6 (Message Understanding Conference) Text collection, LDC96T10:
    /afs/ir/data/linguistic-data/IE/MUC/MUC6/LDC96T10-Muc6
  • MUC-6 (Message Understanding Conference), LDC2003T13:
    /afs/ir/data/linguistic-data/IE/MUC/MUC6/LDC2003T13-muc6
  • Mooney Job Data:
    /afs/ir/data/linguistic-data/IE/Mooney-Job-Data
  • Census 1990 Names:
    /afs/ir/data/linguistic-data/IE/census1990names
• Text Categorization:
  /afs/ir/data/linguistic-data/TextCat
  • 20Newsgroups:
    /afs/ir/data/linguistic-data/TextCat/20Newsgroups
  • DavidLewis (Reuters, TREC-AP):
    /afs/ir/data/linguistic-data/TextCat/DavidLewis
  • Spam Filtering:
    /afs/ir/data/linguistic-data/TextCat/Spam-Filtering
• TREC (Information Retrieval Text Research Collection):
  /afs/ir/data/linguistic-data/TREC/
  • TREC-1 (=Tipster 1): [each user needs to sign license]
    /afs/ir/data/linguistic-data/TREC/TREC-1
  • TREC-2 (=Tipster 2): [each user needs to sign license]
    /afs/ir/data/linguistic-data/TREC/TREC-2
  • TREC-3 (=Tipster 3): [each user needs to sign license]
    /afs/ir/data/linguistic-data/TREC/TREC-3
  • TREC-4:
    /afs/ir/data/linguistic-data/TREC/TREC-4
  • TREC-5:
    /afs/ir/data/linguistic-data/TREC/TREC-5
  • OHSUMED-TREC-9:
    /afs/ir/data/linguistic-data/TREC/OHSUMED-TREC-9
• WSD (Word Sense Disambiguation):
  /afs/ir/data/linguistic-data/WSD
  • DSO Sense-Tagged, LDC97T12:
    /afs/ir/data/linguistic-data/WSD/DSO-Sense-Tagged
  • Leacock's Data:
    /afs/ir/data/linguistic-data/WSD/leacock
  • Pedersen's Data:
    /afs/ir/data/linguistic-data/WSD/pedersen
  • Senseval1:
    /afs/ir/data/linguistic-data/WSD/senseval/senseval1

Recently acquired corpora

We've acquired a fair number of corpora and tools recently. Notably we've now got several new treebanks at Stanford and we update some older corpora to newer versions:

• Arabic Treebank: Part 1 v3.0
• Multiple Translation Arabic
• ACE Time Normalization (TERN) 2004 English Training Data v1.0
• BBN/AUB DARPA Babylon Levantine Arabic Speech and Transcripts
• Chinese Treebank 5.0
• Levantine Arabic QT Training Data Set 3 Speech
• Levantine Arabic QT Training Data Set 3 Transcripts
• Fisher English Training Speech Part 1 Transcripts
• Fisher English Training Speech Part 1 Speech
• Prague Arabic Dependency Treebank 1.0
• Switchboard Cellular Part 2 Audio
• Talkbank Ethology Data: Field Recordings of Vervet Monkey Calls
• Arabic English Parallel News Part 1
• Arabic News Translation Text Part 1
• Santa Barbara Corpus of Spoken American English 3


• Acquisitions that predate 01/01/2004 are listed on the old corpus inventory page.

LDC corpora

D:/

/afs/ir/data/linguistic-data/ldc/LDC-tarfiles/

How to read the LDC catalog numbers: Each LDC corpus has a unique catalog number. The first three digits are always 'LDC'. The next two digits represent the year in which the corpus was released. The third part of the catalog number is a single digit representing the corpus type (Lexicon, Speech or Text). The final digits uniquely distinguish that corpus from other corpora of that type. Note that availability of corpora is not necessarily restricted to members of the release year. To see what membership years a certain corpus is available for, click on the catalog number and check the detailed listing for the corpus.

Tip 1: You can also search the LDC Catalog directly by type and source or by year or the projects. You may also use the general catalog search.

Non-LDC corpora

The non-LDC corpora are listed alphabetically below. As for the LDC corpora, an AFS directory location and/or the abbreviation "CC://" indicate whether that corpus is installed on AFS or the Corpus Computer (CC) or on both. If the path starts with "CC://", the corpus is installed on the Corpus Computer. Exchange "CC://" by the following path to derive the location of the corpus on CC:

D:/

Corpora on the WWW (a very small collection)

This is only a very small collection of online corpora, please see the top 10 info-sources page for links to sites with far more information.

• British National Corpus
  By Beth Levin: You can find information on doing searches by looking in the SARA Manual (http://thetis.bl.uk/CHAP4/) this manual is intended for the full version of the BNC, but it contains information on pattern searching in section 3. With some ingenuity, you can even do searches by lexical category. Since you only get 50 randomly chosen examples of the pattern you are searching for at a time, if there are 80-100 or more examples of this pattern, search for it again since you may get some more examples. (We now have the BNC in AFS space, but we haven't installed the SARA server yet. But you can use gsearch with it.)
  
  Note - You may also find it useful to have the the BNC Basic Tagset open in a separate window while doing your first searches in the BNC.

• COBUILD Corpus
  By Beth Levin: This web site also has material from a whole range of written and spoken sources, used in the development of the Collins COBUILD dictionaries and ESL materials. It allows for some relatively sophisticated searching; click on ``query syntax'' for details. Particularly nice is ``@'': read@ searches for all the inflected forms of read! You only get 40 examples at a time, but can partially get around this in the same way as with the BNC. The other shortcoming is that the window of text is very small -- often not even a whole sentence. If you want more text, you have to search for the string of text defining the left or right edge of the example.
  
  Note - There is also the Cobuild Concordance and Collocations Sampler.
• Lexis-Nexis Academic Universe
  By Beth Levin: This web site has material from major newspapers, wire services, and television news programs. Although it is not a ``balanced'' corpus, it has so much text that you can find things you won't find anywhere else (well, maybe through a web search!); for example, outgraze, outdefend, outweed or he reads himself quasi-blind and the fans spin themselves dizzy. The other major drawback is that it is not designed for linguists, so that you cannot take full advantage of what's there and you also need to work around search procedures that were designed for research into current events.

  From the Lexis-Nexis home page choose ``General News''. From ``General News'' choose the ``More Options'' tab; using the ``Basic'' tab will only allow searches of the title and first paragraph of an article! On the ``More Options'' search page, always be sure to click on ``Headline'' and then select ``Full text'' from the popup menu that comes up. You can select from a range of sources; the default is ``Major Newpapers''; similarly, you can also select from a range of dates. When you get the list of results, click on ``Expanded List''; this will show you the part of the text with your search pattern for the results.

  There are tips at the bottom of the search page that tell you how to construct searches. A few pointers. You can use ``!'' as a wild card for truncation on the right edges of words. Also helpful is ``pre/n'', for some ``n'', which allows you to search for a word that precedes another within an window of n words; ``read! pre/2 way'' will find patterns such as reads her way or reading our slow way, etc. (But ``read!'' will also find reader.) Another disadvantage: Lexis-Nexis has a large list of stop words -- words that you can't search for -- including just about any determiner, auxiliary, and preposition.

• OED new version (slow and requires a graphical browser) or old version (fast and lynx-friendly)
• DIALOGUE DIVERSITY CORPUS: Version 2.0
  From their website: The DDC gives direct access to a set of dialogue transcripts (13 sources, more than 12 hours of dialogue, all in English.). It also gives a set of links and methods for indirect access to hundreds of additional dialogues (principally in English.) Many sources provide speech data as well as transcripts. The emphasis is on free or inexpensive access.

  Volume 2.0 presents access to hundreds of dialogues that were not represented in the original release in October 2002. It is more diverse in terms of situations and dynamic patterns. Access to oral history interviews, the Watergate tapes (by several paths), diverse regional varieties of English (both British and international), the just-emerging American National Corpus (ANC), the U. S. Supreme Court, and other originally non-linguistic sources are presented for the first time.

  The dialogues in this corpus occurred in a very diverse collection of interactive situations. Thus it is a data resource for studies of the breadth of coverage of particular dialogue models, and for studies that compare dialogue from different situations.

• TITUS corpus and search engine [signed license conditions & user agreement need to be faxed to the number stated on the user agreement].
  By Florian Jaeger: TITUS is a large collection of Indo-European text that can be searched with the TITUS search engine. Several types of searches are available. You can search specific texts, restrict the search to specific languages (e.g. Farsi) or language groups (e.g. Avestan, Lithuanian, or simply Old Prussion) or combinations of these restrictions. Wildcards (e.g. '*') or logical operators (e.g. 'AND' or 'OR') can be used, too. The output is a list of texts that contain matches to your search. The site also contains lexica, links to several other text databases, tutorials, etc. A rich source for anyone working on Indo-European languages.
• COSMAS II is a German giga corpus with almost 2 billion (!) text words. It is accessible via the COSMAS II Online Client. Unfortunately, all help and information available for this corpus is given in German.
  By Florian Jaeger: COSMAS II contains morphosyntactic annotated parts, speech, formal writings (news texts, novels, etc.), as well as special sociolinguistic corpora. You can load subcorpora (so called 'archives') and construct searches using a text or a graphical interface - both of which need some time getting used to (but it's worth it). The searches have almost regular expression power; you can search for tagged morphosyntactic information; save your searches and filter your results. A inflection and derivation operator, '&' is also availabel.
• Right in front of your door you can find the Text@Humanities project of the Human Digital Information Service, a collection of online searchable (literary) texts drawn from American English, Irish, other varieties of English, German, French and Spanish.
• The W3-Corpora Search Engine is an online search engine on a large collection of corpora (still in prototype stage but looks promising).