Make it work for the new AP 2016 directory structure and prepare for integration with elex

[anthonyjpesce]

No description provided.

[jeremyjbowers]

Are you working on this, @anthonyjpesce? I am working on a PR for you.

[anthonyjpesce]

Not since we talked, thank you!
On Fri, Jan 15, 2016 at 10:46 AM Jeremy Bowers [email protected]
wrote:

Are you working on this, @anthonyjpesce https://github.com/anthonyjpesce?
I am working on a PR for you.

—
Reply to this email directly or view it on GitHub
#98 (comment)
.

[palewire]

I just pushed a patch to master that tries to adapt our existing framework to integrate the datestamped result files we haven't used in the past. I suspect something like this will work in a pinch.

[jeremyjbowers]

Sorry I'm so behind on this. The world's turned upside down, you know.

[palewire]

Really? Havnt heard. Off for a swim now.

On Tue, Jan 26, 2016, 4:39 AM Jeremy Bowers [email protected]
wrote:

Sorry I'm so behind on this. The world's turned upside down, you know.

—
Reply to this email directly or view it on GitHub
#98 (comment)
.

[palewire]

Next we'll see if I can substitute in the elex data models for ours, which seems to be the straightest line to merging the two libraries. Wish me luck. I'm going to a dark place. https://www.youtube.com/watch?v=hLwrPDRRnf8

[jeremyjbowers]

@palewire for president. https://www.youtube.com/watch?v=6v0Q7moYmnI

[palewire]

elex Race model just slipped in without too much trouble. This is only going to get uglier but it just might be possible.

[palewire]

@jeremyjbowers, if I don't survive this battle I'm counting on you to tell my story https://www.youtube.com/watch?v=DEW5p3vGo10

[eads]

If the models line up, the CLI should Just Work(tm).

On Tue, Jan 26, 2016 at 4:03 PM, Ben Welsh [email protected] wrote:

@jeremyjbowers https://github.com/jeremyjbowers, if I don't survive
this battle I'm counting on you to tell my story
https://www.youtube.com/watch?v=DEW5p3vGo10

—
Reply to this email directly or view it on GitHub
#98 (comment)
.

David Eads | http://recoveredfactory.net

"Medical statistics will be our standard of measurement: we will weigh life
for life and see where the dead lie thicker, among the workers or among the
privileged." -- Rudolf Virchow

[palewire]

That's my hope. My stopping point for today might be just refactoring them into the existing library here but I agree that theoretically it's possible after that.

Though there are some init functions you guys are calling I don't totally understand just yet that might cause an issue.

Also, when this is done one of the most important things will be to proof how accurately I've mapped the fields from the FTP to your schema. It's the same data of course, but the CSV headers from the FTP are slightly different and I'm mapping a bunch at once.

[palewire]

Okay. I have a basic refactor stuffing all the data in the elex classes. Many things are probably broken, and it isn't even close to integration with your Election class, but it is possible, I think, to make an alernative call to our library and have it "serialize" out data identical for an application. which we will try tomorrow during the AP test as a fallback source of data.

[palewire]

[palewire]

Alrighty, after today's AP data test I believe we have squashed the most glaring bugs from my work yesterday. We ran two separate crons, one for the API and one for the FTP, using this simple fork in our download command.

            if options['ftp']:
                logger.debug("Pulling data from FTP")
                election = Election(
                    election_date=date,
                    username='myuser',
                    password='mypass',
                )
            else:
                logger.debug("Pulling data from web API")
                election = api.Election(
                    electiondate=date,
                    testresults=True,
                    liveresults=False,
                    is_test=True
                )

Since python-elections is now returning the same classes as elex, the rest of our code remained 100% the same. Everything worked. Results came out identical (as far as we could tell) in our current front end application.

This is probably enough of a hack to get through Iowa and "work" as a fallback.

Taking the next step of fully integrating the FTP calls within elex is probably within reach, but I'm reluctant to dive into that refactoring without guidance from @jeremyjbowers and @eads. Do you have any opinions on how our classes here should be merged with your existing code?

[jeremyjbowers]

This is fantastic. I'll dig in on this tonight.

contact([{ "twitter": "@jeremybowers", "phone": "727-667-0510", "github": "
http://github.com/jeremyjbowers" }])

On Wed, Jan 27, 2016 at 1:49 PM, Ben Welsh [email protected] wrote:

Alrighty, after today's AP data test I believe we have squashed the most
glaring bugs from my work yesterday. We ran two separate crons, one for the
API and one for the FTP, using this simple fork in our download command.

        if options['ftp']:
            logger.debug("Pulling data from FTP")
            election = Election(
                election_date=date,
                username='myuser',
                password='mypass',
            )
        else:
            logger.debug("Pulling data from web API")
            election = api.Election(
                electiondate=date,
                testresults=True,
                liveresults=False,
                is_test=True
            )

Since python-elections is now returning the same classes as elex, the rest
of our code remained 100% the same. Everything worked. Results came out
identical (as far as we could tell) in our current front end application.

This is probably enough of a hack to get through Iowa and "work" as a
fallback.

Taking the next step of fully integrating the FTP calls within elex is
probably within reach, but I'm reluctant to dive into that refactoring
without guidance from @jeremyjbowers https://github.com/jeremyjbowers
and @eads https://github.com/eads. Do you have any opinions on how our
classes here should be merged with your existing code?

—
Reply to this email directly or view it on GitHub
#98 (comment)
.

[palewire]

Okay. I am going to shave away a little bit at it today, trying to bring our big class a few baby steps closer to yours, but not matter what we do I think there's probably a difficult merge ahead.

[palewire]

[eads]

[jeremyjbowers]

OMFG it is totally working.

[jeremyjbowers]

In [1]: from elections import Election
In [2]: e = Election(electiondate=2016-02-01, username="****", password="****")
In [3]: len(e.candidate_reporting_units)
Out[3]: 1800

[schwanksta]

[palewire]

So you merged it somehow?

[jeremyjbowers]

Example loader script here: https://github.com/newsdev/elex-ftp-loader

[jeremyjbowers]

@palewire Did not try to do the gigantic merge for IA/NH. I'm cool with having this be separate for now. It is a fallback, after all.

[palewire]

I agree that now is a good time to punt. Should we aim to try to land it for South Carolina?

[palewire]

[palewire]

I am going to broaden this ticket's name to aim for creating something that is closer to snapping into elex. First off, I'd like to get my tests working again so I'm not flying blind.