Skip to content

Commit

Permalink
added a readme and described example use case
Browse files Browse the repository at this point in the history
  • Loading branch information
Jennings Anderson committed Oct 9, 2018
1 parent ee45fc7 commit 67cc4aa
Show file tree
Hide file tree
Showing 20 changed files with 129 additions and 1,930 deletions.
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
This utility is built on a modified version of tile-reduce by Mapbox, original license and
This utility is built on a modified version of tile-reduce by Mapbox, original license and
copyright below:

ISC License
Expand Down
48 changes: 48 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Stream-Reduce

This is a simplified framework built off Mapbox's [tile-reduce](//github.com/mapbox/tile-reduce) to perform map-reduce functions against large files of line-delimited JSON. This simply removes all of the `tile` processing and instead passes each new line in the file to the map script.

This is currently used for geometry-reconstruction of historical OSM objects as output from the [OSM-Wayback](//github.com/osmlab/osm-wayback) utility

### Example Implementation

#### File: `example/index.js`

var count = 0;

streamReduce({
map: path.join(__dirname, 'map.js'), //Map function
file: path.join(__dirname, 'file.jsonseq'), //Input file (lines of JSON)
maxWorkers:10 // The number of cpus you'd like to use
})
.on('reduce', function(res) {
count+=res
})
.on('end', function() {
console.log("Finished with count value: "+count)
});

#### File: `example/map.js`

module.exports = function(line, writeData, done) {
var object = JSON.parse(line.toString());
done(object.count)
})

#### file.jsonseq

{"count":10}
{"count":10}
{"count":10}
{"count":10}
{"count":10}

#### Use
$ node examples/index.js

>>Starting up 2 workers... Job started.
>>Processing lines from file: examples/file.jsonseq
>>5 lines processed in 0s.
>>Finished, value of count: 50


5 changes: 5 additions & 0 deletions examples/file.jsonseq
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{"count":10}
{"count":10}
{"count":10}
{"count":10}
{"count":10}
16 changes: 16 additions & 0 deletions examples/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
var streamReduce = require('../src');
var path = require('path')

var count = 0;

streamReduce({
map: path.join(__dirname, 'map.js'),
file: path.join(__dirname, 'file.jsonseq'),
maxWorkers:5 // The number of cpus you'd like to use
})
.on('reduce', function(res) {
count+=res
})
.on('end', function() {
console.log("Finished, value of count: "+count)
});
8 changes: 8 additions & 0 deletions examples/map.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
'use strict';

module.exports = function(line, writeData, done) {

var object = JSON.parse(line.toString());

done(null, object.count)
}
280 changes: 0 additions & 280 deletions geometry-reconstruction/map-geom-reconstruction.js

This file was deleted.

Loading

0 comments on commit 67cc4aa

Please sign in to comment.