Adding a hash transformation

[Mohammad-nassar10]

Today, Arrow-flight module has three actions to transform columns. The first transformation is redact which replaces the values of a column with a given fixed "redactValue". The other transformations are delete and filter columns.
We want to add another possible way of transforming columns in a way where different column values will be replaced with different values. One possible way is to replace each value by its hash value. This transformation can hide the origin values but give the ability to detect equal values (that have the same hash).
A possible hash function is "hashlib.sha256()" which takes objects of type "bytes", return hashes with 256 bits long, and it is consistent (returns the same hash in all different machines). It is required that the values of the columns are of type "string" in order to convert them to "bytes" using "encode()" function.
The transformation changes each value in the requested columns by the hexadecimal representation of the hash value that is returned from "hashlib.sha256()".

[froesef]

The mover has a digest transformation function that pretty much does this. The supported hashes are:

• md5
• sha1
• sha2
• crc32
• murmur3
  To be clear this is currently available via special options of a BatchTransfer but cannot be configured in a Blueprint. We should find a good way to merge the two configurations and also have the same hashing functions (and results). I'd hope python and JVM land returns the same results. Here is our chance to validate it :D

https://github.com/IBM/the-mesh-for-data-mover/blob/master/src/main/scala/com/ibm/m4d/mover/transformation/DigestColumnsTransformation.scala#L42

[Mohammad-nassar10]

@froesef It appears that the hash functions return same results for the same inputs.
I have two questions. First, is there a simple way to run the mover with an example to see what is the result after "digest columns transformation"?
Second, do you have an idea how can we test this addition? To check that the results from arrow-flight-module are equal to those from the mover for same transformation.

[froesef]

I added a local data store that the mover can use for local testing. An example configuration for the mover can be found here.
The simplest way to run this would probably be to manually run the AppTest that is executing this configuration.

I'll add some tests for the different hash functions with some example values so we'll have a baseline.
I think the check if the arrow flight and mover are outputting the same results should be done once (manually). automated cross repository tests are too much for this as they'll basically compare two libraries that we'll use.

[froesef]

I added some tests for the hashes that I use here. You'll find the original string in the tests below. I hope this helps you to set up similar tests for the arrow-flight module. If anything is unclear please ping me.