-
Notifications
You must be signed in to change notification settings - Fork 1
Wikipedia
Download it there: http://burnbit.com/download/276314/enwiki_20140203_pages_articles_xml_bz2 (
On the server it is in /data/team11/wikipedia/
The filtered dataset is available here.
To test on a smaller dataset, export a few articles/categories from this page.
There are no easy way to get all weather events related content. Categories may loop, or subcategories may diverge to unrelated topics (eg. drought -> drought resistant plants, floods in spain -> geography of spain, etc.).
Steps:
1 - filter articles by selecting only those part of a relevant category.
2 - Extract information from them.
3 - Store information in a predefined format
- blizzard (cold++, wind++)
- cyclone / hurricane / typhoon (wind++, rain++)
- derecho (wind++)
- drought (temperature++, rain--)
- nor'easter (temperature--, wind++)
- storm (wind++, rain++)
- tornado (wind++)
- wave (heat / cold) (temperature++ / temperature--)
- weather event
A CSV file (tab separated) with the following columns Title, Category, Start date, End date, Location(s)
e.g.
2003 European heat wave \t Heat Wave \t 1 June 2003 \t 31 August 2003 \t Western Europe
-
Wikihadoop: profides streaming interface to wiki dump (use compressed file, low memory use). How to use it(Too outdated to be useful, use XMLInputFormat instead)
-
List relevant categories. This page is a good starting point (tip: type $("span[title="expand"]").click(); in the javascript console to quickly expand categories). Could be a more generic list, i.e. only keywords that should be in the article category (e.g. "Storm" will automatically include "European windstorms", etc.)✓ Extract information, either from title when possible (e.g 2003 European heat wave), or from article body when necessary (e.g. List of European tornadoes in 2012), or from infobox if available (e.g Hurricane Katrina)