Heavy XLSX processing using Python and C++

Example of handling huge (over a million lines) xlsx data using Python and C++

Table of Contents

• Introduction
• Dependencies
• Acknowledgments

##Introduction

###Motivation This is part of a real life project where I needed to handle some pre-processed results of a mechanical testing. Basically, a machine would spit output in sine wave form with each line representing a point in time. Unfortunatly, these waves would have a propagating phase difference due to information being acquired in different frequencies and ocasionally some noise.

###What does it do? For the purpose of these tests, my solution attempts to sync peaks and valleys by applying some crude criteria (number of points, min/max criteria until next peak or valley can be detected...)

###How does it do? A Python script is used for handling user input trough a (gooey) gui, converting data to csv with xlsx2csv and passing it to a C++ program that does the processing.

##Dependencies Python 2.7 (because gooey only works on it!) recommended way of installing deps is by using pip

• Gooey
• wxPython
• xlsx2csv (included)
• fast-cpp-csv-parser (included)

##Acknowledgments Performance is all thanks to the awesome work of

• Ben Strasser at fast-cpp-csv-parser
• Dilshod Temirkhodjaev at [xlsx2csv] (https://github.com/dilshod/xlsx2csv)