Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Factor out handling of filer_id. Add more IEC data. #351

Merged
merged 1 commit into from
Nov 2, 2014

Conversation

mikeubell
Copy link
Contributor

This should fix #311 issues.
Factor out the handling of filer_id (usually the recipient of a contribution):
There are committee ids which are not integers, map them to unique integers.
Make sure we update pending ids when we find them.
Add the IEC info from form 496
Special case Kaplan, since there is an IEC that matches better than her committee.
Update map.csv (early and often).

state: row['tran_state'],
zip: row['tran_zip4'])
.first_or_initialize
.tap { |p| p.update_attributes(employer: row['tran_emp'], occupation: row['tran_occ']) }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So here we're creating Party::Individual records despite the fact that row['entity_cd'] is actually 'Other'. What exactly is this getting us?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a comment. I was fixing this when the other stuff came op. Forgot about it.
The Firemans Union PAC reported all the contributions from members as OTH. That makes them look like a business rather than a person.

On Nov 1, 2014, at 6:30 PM, Tom Dooner [email protected] wrote:

In backend/fetchers/contribution.rb:

  •      # contributor is "Other"
    
  •      Party::Other.where(name: row['tran_naml'] || 'unknown')
    
  •                  .first_or_initialize
    
  •                  .tap { |p| p.update_attributes(city: row['tran_city'], state: row['tran_state'], zip: row['tran_zip4']) }
    
  •    when 'IND', 'OTH'
    
  • if row['entity_cd'] == 'IND' || !row['tran_namf'].nil? the
    
  •   # contributor is an Individual
    
  •   full_name = row.values_at('tran_namf', 'tran_naml', 'tran_nams')
    
  •          .join(' ')
    
  •          .strip
    
  •   Party::Individual.where(name: full_name,
    
  •               city: row['tran_city'],
    
  •               state: row['tran_state'],
    
  •               zip: row['tran_zip4'])
    
  •            .first_or_initialize
    
  •            .tap { |p| p.update_attributes(employer: row['tran_emp'], occupation: row['tran_occ']) }
    
    So here we're creating Party::Individual records despite the fact that row['entity_cd'] is actually 'Other'. What exactly is this getting us?


Reply to this email directly or view it on GitHub.

@tdooner
Copy link
Member

tdooner commented Nov 2, 2014

Cool – I pulled this locally and it seems to work. I didn't see any problems with your code, though using semicolons in Ruby is unnecessary, and so too is the "then" in if ... then.

I'll merge this and deploy it.

tdooner added a commit that referenced this pull request Nov 2, 2014
Factor out handling of filer_id. Add more IEC data.
@tdooner tdooner merged commit d6fe046 into openoakland:master Nov 2, 2014
@tdooner
Copy link
Member

tdooner commented Nov 2, 2014

Done. Here's the output when I ran it:

Running `bundle exec ruby backend/load_data.rb` attached to terminal... up, run.6414
-- drop_table("parties")
   -> 0.0938s
-- drop_table("iecs")
   -> 0.0142s
-- drop_table("employers")
   -> 0.0074s
-- drop_table("contributions")
   -> 0.0104s
-- drop_table("payments")
   -> 0.0091s
-- drop_table("summaries")
   -> 0.0070s
-- drop_table("maps")
   -> 0.0119s
-- drop_table("category_contributions")
   -> 0.0088s
-- drop_table("employer_contributions")
   -> 0.0096s
-- drop_table("whales")
   -> 0.0062s
-- drop_table("multiples")
   -> 0.0071s
-- drop_table("lobbyists")
   -> 0.0144s
-- drop_table("imports")
   -> 0.0061s
-- create_table(:parties)
   -> 0.1455s
-- create_table(:committee_maps)
   -> 0.0191s
-- create_table(:iecs)
   -> 0.0209s
-- create_table(:employers)
   -> 0.0223s
-- create_table(:contributions)
   -> 0.0639s
-- create_table(:payments)
   -> 0.0300s
-- create_table(:summaries)
   -> 0.0151s
-- create_table(:maps)
   -> 0.0212s
-- create_table(:category_contributions)
   -> 0.0227s
-- create_table(:employer_contributions)
   -> 0.0189s
-- create_table(:whales)
   -> 0.0106s
-- create_table(:multiples)
   -> 0.0114s
-- create_table(:lobbyists)
   -> 0.0408s
-- create_table(:imports)
   -> 0.0091s
Loading Employer Map
Loading Lobbyist data
Fetching Contribution data (Form 497) from Socrata:
    Downloading: http://data.oaklandnet.com/resource/qact-u8hq.json?%24limit=1000&%24order=thru_date+ASC&%24offset=0
 Gay and Lesbian Victory Fund Committee To Support Rebecca Kaplan for City of Oakland Mayor 2014 invalid id:COAK-1538
filer_id is 9000001
 Neighbors for a Safer Oakland 2014 invalid id:Pending
Pending Neighbors for a Safer Oakland 2014
    Downloading: http://data.oaklandnet.com/resource/qact-u8hq.json?%24limit=1000&%24order=thru_date+ASC&%24offset=1000
Fetching Contribution data (Schedule A) from Socrata:
    Downloading: http://data.oaklandnet.com/resource/3xq4-ermg.json?%24limit=1000&%24order=thru_date+ASC&%24offset=0
Pending Friends of Nyeisha Dewitt for City Council 2012
    Downloading: http://data.oaklandnet.com/resource/3xq4-ermg.json?%24limit=1000&%24order=thru_date+ASC&%24offset=1000
 Barbara Parker City Attorney Officeholder Account invalid id:Pending
Pending Barbara Parker City Attorney Officeholder Account
 Barbara Parker City Attorney Officeholder Account invalid id:Pending
 Barbara Parker City Attorney Officeholder Account invalid id:Pending
 Barbara Parker City Attorney Officeholder Account invalid id:Pending
 Barbara Parker City Attorney Officeholder Account invalid id:Pending
 Barbara Parker City Attorney Officeholder Account invalid id:Pending
    Downloading: http://data.oaklandnet.com/resource/3xq4-ermg.json?%24limit=1000&%24order=thru_date+ASC&%24offset=2000
Updating Barbara Parker City Attorney Officeholder Account 1352921
 Metropolitan Greater Oakland Democratic Club-M60 invalid id:S10088
filer_id is 9000002
 Metropolitan Greater Oakland Democratic Club-M60 invalid id:S10088
filer_id is 9000002
 Metropolitan Greater Oakland Democratic Club-M60 invalid id:S10088
filer_id is 9000002
 Committee against Measure A invalid id:Pending
Pending Committee against Measure A
Updating Committee against Measure A 1362064
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
Skipping Oakland Police Officer's Association - Political Action Committee:INC85
Skipping Oakland Police Officer's Association - Political Action Committee:INC86
Skipping Oakland Police Officer's Association - Political Action Committee:INC87
Skipping Oakland Police Officer's Association - Political Action Committee:INC88
Skipping Oakland Police Officer's Association - Political Action Committee:INC89
Skipping Oakland Police Officer's Association - Political Action Committee:INC90
 Patrick McCullough Mayor 2014 invalid id:Pending
Pending Patrick McCullough Mayor 2014
    Downloading: http://data.oaklandnet.com/resource/3xq4-ermg.json?%24limit=1000&%24order=thru_date+ASC&%24offset=3000
 Metropolitan Greater Oakland Democratic Club-M60 invalid id:S10088
filer_id is 9000002
 Sam Washington For Oakland Mayor 2014 invalid id:Pending
Pending Sam Washington For Oakland Mayor 2014
 Sam Washington For Oakland Mayor 2014 invalid id:Pending
 Sam Washington For Oakland Mayor 2014 invalid id:Pending
 Sam Washington For Oakland Mayor 2014 invalid id:Pending
 Sam Washington For Oakland Mayor 2014 invalid id:Pending
 Sam Washington For Oakland Mayor 2014 invalid id:Pending
 Sam Washington For Oakland Mayor 2014 invalid id:Pending
 Sam Washington For Oakland Mayor 2014 invalid id:Pending
    Downloading: http://data.oaklandnet.com/resource/3xq4-ermg.json?%24limit=1000&%24order=thru_date+ASC&%24offset=4000
Skipping Oakland Police Officer's Association - Political Action Committee:INC108
Skipping Oakland Police Officer's Association - Political Action Committee:INC109
Skipping Oakland Police Officer's Association - Political Action Committee:INC110
Skipping Oakland Police Officer's Association - Political Action Committee:INC111
Skipping Oakland Police Officer's Association - Political Action Committee:INC112
    Downloading: http://data.oaklandnet.com/resource/3xq4-ermg.json?%24limit=1000&%24order=thru_date+ASC&%24offset=5000
    Downloading: http://data.oaklandnet.com/resource/3xq4-ermg.json?%24limit=1000&%24order=thru_date+ASC&%24offset=6000
    Downloading: http://data.oaklandnet.com/resource/3xq4-ermg.json?%24limit=1000&%24order=thru_date+ASC&%24offset=7000
 Sokhom Mao for City Council 2014 invalid id:PENDING
Pending Sokhom Mao for City Council 2014
 Sokhom Mao for City Council 2014 invalid id:PENDING
 Sokhom Mao for City Council 2014 invalid id:PENDING
 Sokhom Mao for City Council 2014 invalid id:PENDING
 Sokhom Mao for City Council 2014 invalid id:PENDING
 Sokhom Mao for City Council 2014 invalid id:PENDING
 Sokhom Mao for City Council 2014 invalid id:PENDING
 Sokhom Mao for City Council 2014 invalid id:PENDING
 Sokhom Mao for City Council 2014 invalid id:PENDING
 Sokhom Mao for City Council 2014 invalid id:PENDING
 Sokhom Mao for City Council 2014 invalid id:PENDING
 Sokhom Mao for City Council 2014 invalid id:PENDING
 Sokhom Mao for City Council 2014 invalid id:PENDING
 Sokhom Mao for City Council 2014 invalid id:PENDING
 Sokhom Mao for City Council 2014 invalid id:PENDING
 Patrick McCullough Mayor 2014 invalid id:Pending
 Sam Washington For Oakland Mayor 2014 invalid id:Pending
 Sam Washington For Oakland Mayor 2014 invalid id:Pending
 Sam Washington For Oakland Mayor 2014 invalid id:Pending
 Sam Washington For Oakland Mayor 2014 invalid id:Pending
    Downloading: http://data.oaklandnet.com/resource/3xq4-ermg.json?%24limit=1000&%24order=thru_date+ASC&%24offset=8000
    Downloading: http://data.oaklandnet.com/resource/3xq4-ermg.json?%24limit=1000&%24order=thru_date+ASC&%24offset=9000
Skipping Oakland Police Officer's Association - Political Action Committee:INC117
Skipping Oakland Police Officer's Association - Political Action Committee:INC128
Skipping Oakland Police Officer's Association - Political Action Committee:INC129
    Downloading: http://data.oaklandnet.com/resource/3xq4-ermg.json?%24limit=1000&%24order=thru_date+ASC&%24offset=10000
    Downloading: http://data.oaklandnet.com/resource/3xq4-ermg.json?%24limit=1000&%24order=thru_date+ASC&%24offset=11000
    Downloading: http://data.oaklandnet.com/resource/3xq4-ermg.json?%24limit=1000&%24order=thru_date+ASC&%24offset=12000
Fetching Loan data (Schedule B1) from Socrata:
    Downloading: http://data.oaklandnet.com/resource/qaa7-q29f.json?%24limit=1000&%24order=thru_date+ASC&%24offset=0
 Committee to Elect Michael V. Johnson City Council 2014 invalid id:COAK-1536
filer_id is 9000003
    Downloading: http://data.oaklandnet.com/resource/qaa7-q29f.json?%24limit=1000&%24order=thru_date+ASC&%24offset=1000
Fetching Summary data from Socrata:
    Downloading: http://data.oaklandnet.com/resource/rsxe-vvuw.json?%24limit=1000&%24order=thru_date+ASC&%24offset=0
    Downloading: http://data.oaklandnet.com/resource/rsxe-vvuw.json?%24limit=1000&%24order=thru_date+ASC&%24offset=1000
    Downloading: http://data.oaklandnet.com/resource/rsxe-vvuw.json?%24limit=1000&%24order=thru_date+ASC&%24offset=2000
    Downloading: http://data.oaklandnet.com/resource/rsxe-vvuw.json?%24limit=1000&%24order=thru_date+ASC&%24offset=3000
    Downloading: http://data.oaklandnet.com/resource/rsxe-vvuw.json?%24limit=1000&%24order=thru_date+ASC&%24offset=4000
    Downloading: http://data.oaklandnet.com/resource/rsxe-vvuw.json?%24limit=1000&%24order=thru_date+ASC&%24offset=5000
    Downloading: http://data.oaklandnet.com/resource/rsxe-vvuw.json?%24limit=1000&%24order=thru_date+ASC&%24offset=6000
    Downloading: http://data.oaklandnet.com/resource/rsxe-vvuw.json?%24limit=1000&%24order=thru_date+ASC&%24offset=7000
    Downloading: http://data.oaklandnet.com/resource/rsxe-vvuw.json?%24limit=1000&%24order=thru_date+ASC&%24offset=8000
    Downloading: http://data.oaklandnet.com/resource/rsxe-vvuw.json?%24limit=1000&%24order=thru_date+ASC&%24offset=9000
    Downloading: http://data.oaklandnet.com/resource/rsxe-vvuw.json?%24limit=1000&%24order=thru_date+ASC&%24offset=10000
    Downloading: http://data.oaklandnet.com/resource/rsxe-vvuw.json?%24limit=1000&%24order=thru_date+ASC&%24offset=11000
    Downloading: http://data.oaklandnet.com/resource/rsxe-vvuw.json?%24limit=1000&%24order=thru_date+ASC&%24offset=12000
    Downloading: http://data.oaklandnet.com/resource/rsxe-vvuw.json?%24limit=1000&%24order=thru_date+ASC&%24offset=13000
Fetching IEC data from Socrata:
    Downloading: http://data.oaklandnet.com/resource/6ejr-39gh.json?%24limit=1000&%24order=thru_date+ASC&%24offset=0
    Downloading: http://data.oaklandnet.com/resource/6ejr-39gh.json?%24limit=1000&%24order=thru_date+ASC&%24offset=1000
    Downloading: http://data.oaklandnet.com/resource/6ejr-39gh.json?%24limit=1000&%24order=thru_date+ASC&%24offset=2000
    Downloading: http://data.oaklandnet.com/resource/6ejr-39gh.json?%24limit=1000&%24order=thru_date+ASC&%24offset=3000
    Downloading: http://data.oaklandnet.com/resource/6ejr-39gh.json?%24limit=1000&%24order=thru_date+ASC&%24offset=4000
    Downloading: http://data.oaklandnet.com/resource/6ejr-39gh.json?%24limit=1000&%24order=thru_date+ASC&%24offset=5000
    Downloading: http://data.oaklandnet.com/resource/6ejr-39gh.json?%24limit=1000&%24order=thru_date+ASC&%24offset=6000
    Downloading: http://data.oaklandnet.com/resource/6ejr-39gh.json?%24limit=1000&%24order=thru_date+ASC&%24offset=7000
    Downloading: http://data.oaklandnet.com/resource/6ejr-39gh.json?%24limit=1000&%24order=thru_date+ASC&%24offset=8000
    Downloading: http://data.oaklandnet.com/resource/jkj3-8yq3.json?%24limit=1000&%24order=thru_date+ASC&%24offset=0
 Gay and Lesbian Victory Fund Committee To Support Rebecca Kaplan for City of Oakland Mayor 2014 invalid id:COAK-1538
filer_id is 9000001
    Downloading: http://data.oaklandnet.com/resource/jkj3-8yq3.json?%24limit=1000&%24order=thru_date+ASC&%24offset=1000
Run analysis

Thanks for adding this stuff. It adds a whole new dimension to the site.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Display Independent Expenditure data
2 participants