ExtractingDataFromNCMEC.ipynb

{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Extracting Data from NCMEC"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<a href=\"http://www.missingkids.com\">The National Center for Missing & Exploited Children (NCMEC)</a> provides information about missing children from all over the world, but does not provide a public API to access the data. However monitoring the network requests on the search page exposes the underlying API that drives the web search. We can send requests directly through their API.\n",
      "\n",
      "In this notebook, I want to show how to retrieve missing persons data from NCMEC using requests. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import json\n",
      "import requests"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Create a new requests session so that the correct cookies are sent with each request."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "session = requests.Session()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now use the API to request all missing children in the state of California."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "state_abbrev = \"CA\"\n",
      "response = session.get(\"http://www.missingkids.com/missingkids/servlet/JSONDataServlet?action=publicSearch&searchLang=en_US&search=new&subjToSearch=child&missState=\" + state_abbrev + \"&missCountry=US\")\n",
      "print \"{} {}\".format(response.status_code, response.reason)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "200 OK\n"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We get a paginated response with at most 9 missing children per page. First let's look at page 1."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "response = session.get(\"http://www.missingkids.com/missingkids/servlet/JSONDataServlet?action=publicSearch&searchLang=en_US&goToPage=1\")\n",
      "dct = json.loads(response.text)\n",
      "print json.dumps(dct, sort_keys=True, indent=4, separators=(',', ': '))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{\n",
        "    \"labels\": {\n",
        "        \"ageNow\": \"Age Now\",\n",
        "        \"days\": \"Day(s)\",\n",
        "        \"missingFrom\": \"Missing From\",\n",
        "        \"missingSince\": \"Missing Since\",\n",
        "        \"months\": \"Month(s)\"\n",
        "    },\n",
        "    \"persons\": [\n",
        "        {\n",
        "            \"age\": 12,\n",
        "            \"approxAge\": \"\",\n",
        "            \"caseNumber\": \"982012\",\n",
        "            \"caseType\": \"\",\n",
        "            \"firstName\": \"AMIR\",\n",
        "            \"hasPoster\": true,\n",
        "            \"hasThumbnail\": true,\n",
        "            \"inDay\": false,\n",
        "            \"inMonth\": false,\n",
        "            \"isChild\": true,\n",
        "            \"langId\": \"en_US\",\n",
        "            \"lastName\": \"ABDOU\",\n",
        "            \"middleName\": \"KAREEM\",\n",
        "            \"missingCity\": \"SAN RAMON\",\n",
        "            \"missingCountry\": \"US\",\n",
        "            \"missingCounty\": \"Contra Costa\",\n",
        "            \"missingDate\": \"Apr 30, 2003 12:00:00 AM\",\n",
        "            \"missingState\": \"CA\",\n",
        "            \"orgName\": \"National Center for Missing & Exploited Children\",\n",
        "            \"orgPrefix\": \"NCMC\",\n",
        "            \"posterTitle\": \"\",\n",
        "            \"race\": \"\",\n",
        "            \"seqNumber\": 1,\n",
        "            \"thumbnailUrl\": \"/photographs/NCMC982012c1t.jpg\"\n",
        "        },\n",
        "        {\n",
        "            \"age\": 14,\n",
        "            \"approxAge\": \"\",\n",
        "            \"caseNumber\": \"1229923\",\n",
        "            \"caseType\": \"\",\n",
        "            \"firstName\": \"CELESTE\",\n",
        "            \"hasPoster\": true,\n",
        "            \"hasThumbnail\": true,\n",
        "            \"inDay\": false,\n",
        "            \"inMonth\": false,\n",
        "            \"isChild\": true,\n",
        "            \"langId\": \"en_US\",\n",
        "            \"lastName\": \"ACOSTA\",\n",
        "            \"middleName\": \"\",\n",
        "            \"missingCity\": \"LA VERNE\",\n",
        "            \"missingCountry\": \"US\",\n",
        "            \"missingCounty\": \"Los Angeles\",\n",
        "            \"missingDate\": \"Mar 17, 2014 12:00:00 AM\",\n",
        "            \"missingState\": \"CA\",\n",
        "            \"orgName\": \"National Center for Missing & Exploited Children\",\n",
        "            \"orgPrefix\": \"NCMC\",\n",
        "            \"posterTitle\": \"\",\n",
        "            \"race\": \"\",\n",
        "            \"seqNumber\": 1,\n",
        "            \"thumbnailUrl\": \"/photographs/NCMC1229923c1t.jpg\"\n",
        "        },\n",
        "        {\n",
        "            \"age\": 33,\n",
        "            \"approxAge\": \"\",\n",
        "            \"caseNumber\": \"780684\",\n",
        "            \"caseType\": \"\",\n",
        "            \"firstName\": \"JORGE\",\n",
        "            \"hasPoster\": true,\n",
        "            \"hasThumbnail\": true,\n",
        "            \"inDay\": false,\n",
        "            \"inMonth\": false,\n",
        "            \"isChild\": true,\n",
        "            \"langId\": \"en_US\",\n",
        "            \"lastName\": \"ACOSTA\",\n",
        "            \"middleName\": \"\",\n",
        "            \"missingCity\": \"VAN NUYS\",\n",
        "            \"missingCountry\": \"US\",\n",
        "            \"missingCounty\": \"Los Angeles\",\n",
        "            \"missingDate\": \"Nov 1, 1992 12:00:00 AM\",\n",
        "            \"missingState\": \"CA\",\n",
        "            \"orgName\": \"National Center for Missing & Exploited Children\",\n",
        "            \"orgPrefix\": \"NCMC\",\n",
        "            \"posterTitle\": \"\",\n",
        "            \"race\": \"\",\n",
        "            \"seqNumber\": 1,\n",
        "            \"thumbnailUrl\": \"/photographs/NCMC780684c1t.jpg\"\n",
        "        },\n",
        "        {\n",
        "            \"age\": 12,\n",
        "            \"approxAge\": \"\",\n",
        "            \"caseNumber\": \"1203501\",\n",
        "            \"caseType\": \"\",\n",
        "            \"firstName\": \"DUCHESS\",\n",
        "            \"hasPoster\": true,\n",
        "            \"hasThumbnail\": true,\n",
        "            \"inDay\": false,\n",
        "            \"inMonth\": false,\n",
        "            \"isChild\": true,\n",
        "            \"langId\": \"en_US\",\n",
        "            \"lastName\": \"ADAMS\",\n",
        "            \"middleName\": \"SAVANAH\",\n",
        "            \"missingCity\": \"LANCASTER\",\n",
        "            \"missingCountry\": \"US\",\n",
        "            \"missingCounty\": \"Los Angeles\",\n",
        "            \"missingDate\": \"Sep 30, 2012 12:00:00 AM\",\n",
        "            \"missingState\": \"CA\",\n",
        "            \"orgName\": \"National Center for Missing & Exploited Children\",\n",
        "            \"orgPrefix\": \"NCMC\",\n",
        "            \"posterTitle\": \"\",\n",
        "            \"race\": \"\",\n",
        "            \"seqNumber\": 1,\n",
        "            \"thumbnailUrl\": \"/photographs/NCMC1203501c1t.jpg\"\n",
        "        },\n",
        "        {\n",
        "            \"age\": 13,\n",
        "            \"approxAge\": \"\",\n",
        "            \"caseNumber\": \"1203501\",\n",
        "            \"caseType\": \"\",\n",
        "            \"firstName\": \"TINA\",\n",
        "            \"hasPoster\": true,\n",
        "            \"hasThumbnail\": true,\n",
        "            \"inDay\": false,\n",
        "            \"inMonth\": false,\n",
        "            \"isChild\": true,\n",
        "            \"langId\": \"en_US\",\n",
        "            \"lastName\": \"ADAMS\",\n",
        "            \"middleName\": \"VICTORIA\",\n",
        "            \"missingCity\": \"LANCASTER\",\n",
        "            \"missingCountry\": \"US\",\n",
        "            \"missingCounty\": \"Los Angeles\",\n",
        "            \"missingDate\": \"Sep 30, 2012 12:00:00 AM\",\n",
        "            \"missingState\": \"CA\",\n",
        "            \"orgName\": \"National Center for Missing & Exploited Children\",\n",
        "            \"orgPrefix\": \"NCMC\",\n",
        "            \"posterTitle\": \"\",\n",
        "            \"race\": \"\",\n",
        "            \"seqNumber\": 2,\n",
        "            \"thumbnailUrl\": \"/photographs/NCMC1203501c2t.jpg\"\n",
        "        },\n",
        "        {\n",
        "            \"age\": 15,\n",
        "            \"approxAge\": \"\",\n",
        "            \"caseNumber\": \"1180655\",\n",
        "            \"caseType\": \"\",\n",
        "            \"firstName\": \"BERANIA\",\n",
        "            \"hasPoster\": true,\n",
        "            \"hasThumbnail\": true,\n",
        "            \"inDay\": false,\n",
        "            \"inMonth\": false,\n",
        "            \"isChild\": true,\n",
        "            \"langId\": \"en_US\",\n",
        "            \"lastName\": \"AGAPITO\",\n",
        "            \"middleName\": \"TERESA\",\n",
        "            \"missingCity\": \"VAN NUYS\",\n",
        "            \"missingCountry\": \"US\",\n",
        "            \"missingCounty\": \"Los Angeles\",\n",
        "            \"missingDate\": \"May 7, 2010 12:00:00 AM\",\n",
        "            \"missingState\": \"CA\",\n",
        "            \"orgName\": \"National Center for Missing & Exploited Children\",\n",
        "            \"orgPrefix\": \"NCMC\",\n",
        "            \"posterTitle\": \"\",\n",
        "            \"race\": \"\",\n",
        "            \"seqNumber\": 1,\n",
        "            \"thumbnailUrl\": \"/photographs/NCMC1180655c1t.jpg\"\n",
        "        },\n",
        "        {\n",
        "            \"age\": 11,\n",
        "            \"approxAge\": \"\",\n",
        "            \"caseNumber\": \"1180655\",\n",
        "            \"caseType\": \"\",\n",
        "            \"firstName\": \"JAZIBI\",\n",
        "            \"hasPoster\": true,\n",
        "            \"hasThumbnail\": true,\n",
        "            \"inDay\": false,\n",
        "            \"inMonth\": false,\n",
        "            \"isChild\": true,\n",
        "            \"langId\": \"en_US\",\n",
        "            \"lastName\": \"AGAPITO\",\n",
        "            \"middleName\": \"\",\n",
        "            \"missingCity\": \"VAN NUYS\",\n",
        "            \"missingCountry\": \"US\",\n",
        "            \"missingCounty\": \"Los Angeles\",\n",
        "            \"missingDate\": \"May 7, 2010 12:00:00 AM\",\n",
        "            \"missingState\": \"CA\",\n",
        "            \"orgName\": \"National Center for Missing & Exploited Children\",\n",
        "            \"orgPrefix\": \"NCMC\",\n",
        "            \"posterTitle\": \"\",\n",
        "            \"race\": \"\",\n",
        "            \"seqNumber\": 2,\n",
        "            \"thumbnailUrl\": \"/photographs/NCMC1180655c2t.jpg\"\n",
        "        },\n",
        "        {\n",
        "            \"age\": 18,\n",
        "            \"approxAge\": \"\",\n",
        "            \"caseNumber\": \"1180655\",\n",
        "            \"caseType\": \"\",\n",
        "            \"firstName\": \"WENDY\",\n",
        "            \"hasPoster\": true,\n",
        "            \"hasThumbnail\": true,\n",
        "            \"inDay\": false,\n",
        "            \"inMonth\": false,\n",
        "            \"isChild\": true,\n",
        "            \"langId\": \"en_US\",\n",
        "            \"lastName\": \"AGAPITO\",\n",
        "            \"middleName\": \"\",\n",
        "            \"missingCity\": \"VAN NUYS\",\n",
        "            \"missingCountry\": \"US\",\n",
        "            \"missingCounty\": \"Los Angeles\",\n",
        "            \"missingDate\": \"May 7, 2010 12:00:00 AM\",\n",
        "            \"missingState\": \"CA\",\n",
        "            \"orgName\": \"National Center for Missing & Exploited Children\",\n",
        "            \"orgPrefix\": \"NCMC\",\n",
        "            \"posterTitle\": \"\",\n",
        "            \"race\": \"\",\n",
        "            \"seqNumber\": 3,\n",
        "            \"thumbnailUrl\": \"/photographs/NCMC1180655c3t.jpg\"\n",
        "        },\n",
        "        {\n",
        "            \"age\": 7,\n",
        "            \"approxAge\": \"\",\n",
        "            \"caseNumber\": \"1221385\",\n",
        "            \"caseType\": \"\",\n",
        "            \"firstName\": \"BRENDA\",\n",
        "            \"hasPoster\": true,\n",
        "            \"hasThumbnail\": true,\n",
        "            \"inDay\": false,\n",
        "            \"inMonth\": false,\n",
        "            \"isChild\": true,\n",
        "            \"langId\": \"en_US\",\n",
        "            \"lastName\": \"AGUILERA\",\n",
        "            \"middleName\": \"\",\n",
        "            \"missingCity\": \"LOS ANGELES\",\n",
        "            \"missingCountry\": \"US\",\n",
        "            \"missingCounty\": \"Los Angeles\",\n",
        "            \"missingDate\": \"May 2, 2012 12:00:00 AM\",\n",
        "            \"missingState\": \"CA\",\n",
        "            \"orgName\": \"National Center for Missing & Exploited Children\",\n",
        "            \"orgPrefix\": \"NCMC\",\n",
        "            \"posterTitle\": \"\",\n",
        "            \"race\": \"\",\n",
        "            \"seqNumber\": 1,\n",
        "            \"thumbnailUrl\": \"/photographs/NCMC1221385c1t.jpg\"\n",
        "        }\n",
        "    ],\n",
        "    \"status\": \"success\",\n",
        "    \"subject\": \"Missingkids Child Search Result\",\n",
        "    \"thisPage\": 1,\n",
        "    \"totalPages\": 54,\n",
        "    \"totalRecords\": 486\n",
        "}\n"
       ]
      }
     ],
     "prompt_number": 28
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "How many pages are in the search results?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "int(dct[\"totalPages\"])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 29,
       "text": [
        "54"
       ]
      }
     ],
     "prompt_number": 29
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "How many missing children are there in total for California?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "int(dct[\"totalRecords\"])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 23,
       "text": [
        "486"
       ]
      }
     ],
     "prompt_number": 23
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can also request more details per case. First isolate the first case."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "person = dct[\"persons\"][0]\n",
      "print json.dumps(person, sort_keys=True, indent=4, separators=(',', ': '))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{\n",
        "    \"age\": 12,\n",
        "    \"approxAge\": \"\",\n",
        "    \"caseNumber\": \"982012\",\n",
        "    \"caseType\": \"\",\n",
        "    \"firstName\": \"AMIR\",\n",
        "    \"hasPoster\": true,\n",
        "    \"hasThumbnail\": true,\n",
        "    \"inDay\": false,\n",
        "    \"inMonth\": false,\n",
        "    \"isChild\": true,\n",
        "    \"langId\": \"en_US\",\n",
        "    \"lastName\": \"ABDOU\",\n",
        "    \"middleName\": \"KAREEM\",\n",
        "    \"missingCity\": \"SAN RAMON\",\n",
        "    \"missingCountry\": \"US\",\n",
        "    \"missingCounty\": \"Contra Costa\",\n",
        "    \"missingDate\": \"Apr 30, 2003 12:00:00 AM\",\n",
        "    \"missingState\": \"CA\",\n",
        "    \"orgName\": \"National Center for Missing & Exploited Children\",\n",
        "    \"orgPrefix\": \"NCMC\",\n",
        "    \"posterTitle\": \"\",\n",
        "    \"race\": \"\",\n",
        "    \"seqNumber\": 1,\n",
        "    \"thumbnailUrl\": \"/photographs/NCMC982012c1t.jpg\"\n",
        "}\n"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now request more information about the child."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "detailed_response = session.get(\"http://www.missingkids.com/missingkids/servlet/JSONDataServlet?action=childDetail&caseNum={}&orgPrefix={}\".format(person[\"caseNumber\"], person[\"orgPrefix\"]))\n",
      "\n",
      "print json.dumps(json.loads(detailed_response.text), sort_keys=True, indent=4, separators=(',', ': '))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{\n",
        "    \"childBean\": {\n",
        "        \"age\": 12,\n",
        "        \"altContact\": \"Contra Costa District Attorney's Office (California) 1-925-646-2502 (daytime) or 1-925-646-2441 (nighttime) or FBI (Sacramento, California) 1-415-553-7400\",\n",
        "        \"approxAge\": \"\",\n",
        "        \"birthDate\": \"May 27, 2001 12:00:00 AM\",\n",
        "        \"caseNumber\": \"982012\",\n",
        "        \"caseType\": \"familyAbduct\",\n",
        "        \"circumstance\": \"Amir's photo is shown age-progressed to 12 years.  He may be in the company his mother.  They are believed to have left the country and traveled to Egypt.  The child is of Egyptian descent.\",\n",
        "        \"eyeColor\": \"brown\",\n",
        "        \"firstName\": \"AMIR\",\n",
        "        \"hairColor\": \"black\",\n",
        "        \"hasAgedPhoto\": true,\n",
        "        \"hasExtraPhoto\": false,\n",
        "        \"hasPhoto\": true,\n",
        "        \"hasPoster\": true,\n",
        "        \"hasThumbnail\": true,\n",
        "        \"height\": 24,\n",
        "        \"heightInInch\": true,\n",
        "        \"inDay\": false,\n",
        "        \"inMonth\": false,\n",
        "        \"isChild\": true,\n",
        "        \"langId\": \"en_US\",\n",
        "        \"lastName\": \"ABDOU\",\n",
        "        \"maxHeight\": 0,\n",
        "        \"maxWeight\": 0,\n",
        "        \"middleName\": \"KAREEM\",\n",
        "        \"missingCity\": \"SAN RAMON\",\n",
        "        \"missingCountry\": \"US\",\n",
        "        \"missingCounty\": \"Contra Costa\",\n",
        "        \"missingDate\": \"Apr 30, 2003 12:00:00 AM\",\n",
        "        \"missingProvince\": \"\",\n",
        "        \"missingState\": \"CA\",\n",
        "        \"namUs\": \"\",\n",
        "        \"ncic\": \"\",\n",
        "        \"orgContactInfo\": \"1-800-843-5678 (1-800-THE-LOST)\",\n",
        "        \"orgLogo\": \"NCMC_en_US.gif\",\n",
        "        \"orgName\": \"National Center for Missing & Exploited Children\",\n",
        "        \"orgPrefix\": \"NCMC\",\n",
        "        \"otherChildList\": [],\n",
        "        \"otherCsawList\": [],\n",
        "        \"photoMap\": \"c1e1\",\n",
        "        \"profileNarrative\": \"\",\n",
        "        \"race\": \"white\",\n",
        "        \"repSightURL\": \"\",\n",
        "        \"seqNumber\": 1,\n",
        "        \"sex\": \"male\",\n",
        "        \"userLangId\": \"en_US\",\n",
        "        \"weight\": 30,\n",
        "        \"weightInPound\": true\n",
        "    },\n",
        "    \"msg\": \"\",\n",
        "    \"status\": \"success\"\n",
        "}\n"
       ]
      }
     ],
     "prompt_number": 27
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We now have enough information to crawl through and collect information about each missing child!"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}