ReduceMyMap

A MPI-based map-reduce program designed to take in some documents and to output frequency of words in every document.

Prerequisities

• Sometin' able to compile C++17
• Visual Studio may help but it's not necessary
• Microsoft MPI if you're on Windows or Open MPI if you're on Linux
• nlohmann/json

Usage

This should be obvious, but build the application and run the application you just built. The program should receive 2 arguments:

• -i /path/to/input to tell the program where to read files from
• -o /path/to/output to tell the program where to write files to

A temporary directory will also be created, but you should live with the idea that you can't change that easily.

The program will loop through all files in the directory, read them and apply the mapping to each one.

Description

Pseudocode for whatever the fuck happens here:

Master:

initTaskList();
while not done:
	checkForAvailableSlaves();
	if nextTask is barrier:
		waitForAllSlavesToFinish();
	sendJobsToSlaves();
killAllSlaves();

Servant:

while not done:
	tellMasterImNotBusy();
	getNextJob();
	if nextJob is map:
		map(document);
	if nextJob is reduce:
		reduce(letter);
	if nextJob is finish:
		die();

License

• This thing is distributed under Apache 2.0. See LICENSE.
• This thing uses nlohmann/json which is distributed under MIT License
• Gitignore file from github/gitignore.

Additional details

Contact

You can find me here to ask questions.