Guides: Text Mining : Newspapers as Corpora

Text Mining Newspapers Accessed via GU Library Databases

Most library newspaper databases prohibit text mining. Exceptions are listed below:

Gale Digital Scholar Lab
The Gale Digital Scholar Lab is a single research platform where you can apply natural language processing tools to raw text data (OCR) from Georgetown's Gale Primary Sources holdings, including The Economist, The Times, and historic newspaper collections.
New York Times Metadata API
The New York Times Developer Portal provides API-level access to NYT article metadata. Full text of articles is NOT available through this service. API calls will generally return metadata (headline, byline, date, keywords, etc.) and a URL to the full article.

For more detailed information on text mining library paid databases, read the guide here.

Specialized Text Mining Tools for Newspapers

There are some tools that allow you to conduct text analysis using contemporary and historic newspapers, while prohibiting or limiting full-text download. Some of these targeted tools are listed below, with a description of their content and capabilities:

Impreso Project
The Impreso Project contains 200 years of historical news and radio data. The sources in Impreso are European, primarily from Switzerland and Luxembourg. The Impreso app lets you use a graphical user interface to compile a dataset and run analysis. The Impreso data lab lets researchers use a Jupyter notebook to compile and analyze data.
Media Cloud
Media Cloud is an open source and open data platform for storing, retrieving, visualizing, and analyzing online news. News data is available in 20 languages and includes current day news. Full text data is not available in Media Cloud due to copyright restrictions. Data is displayed in widgets showing attention over time, total attention, top words, and more.
News on the Web Corpus
The News on the Web corpus contains 23.5 billion words of data from web-based newspapers and magazines from 2010 to the present time. The corpus can be searched and accessed for free via the website. A paid account is required to download the corpus.

Sources of Historic Newspaper Text Data

Gale Digital Scholar Lab
The Gale Digital Scholar Lab is a single research platform where you can apply natural language processing tools to raw text data (OCR) from Georgetown's Gale Primary Sources holdings, including The Economist, The Times, and historic newspaper collections.
Chronicaling America
Chronicling America provides free access to millions of historic American newspaper pages. Coverage is between 1777-1963.

Utilizing Creative Commons News as a Corpus

If you require a customized and contemporary corpus of news articles that cannot be acquired at any of the above sources, you may be able to manually compile a corpus of articles from free, publicly-available, online news sources. News sources that are published with a Creative Commons CC-0 BY attribution (for example, ProPublica or OpenNewswire) would be the most straightforward.

Researchers must always do their due diligence, read and adhere to the Terms of Service for the data source, and comply with local laws around copyright, data collection, and privacy. Check out the Web Scraping a Corpus page of this guide for more information.