decimal.DivisionImpossible raised when handling schema items with high levels of precision

[tomelliff]

I'm trying to pull data out of a Postgres database with the pipelinewise-tap-postgres and load it as JSONL with this target. Unfortunately when this target encounters data that is held as a numeric type in Postgres it raises the following error:

target-jsonl | Traceback (most recent call last):
target-jsonl |   File "/home/tom/git/meltano-projects/gitlab-metrics/.meltano/loaders/target-jsonl/venv/lib/python3.7/site-packages/jsonschema/_validators.py", line 131, in multipleOf
target-jsonl |     failed = instance % dB
target-jsonl | decimal.InvalidOperation: [<class 'decimal.DivisionImpossible'>]
target-jsonl | Traceback (most recent call last):
target-jsonl |   File "/home/tom/git/meltano-projects/gitlab-metrics/.meltano/loaders/target-jsonl/venv/lib/python3.7/site-packages/jsonschema/_validators.py", line 131, in multipleOf
target-jsonl |     failed = instance % dB
target-jsonl | decimal.InvalidOperation: [<class 'decimal.DivisionImpossible'>]
target-jsonl | 
target-jsonl | During handling of the above exception, another exception occurred:
target-jsonl | 
target-jsonl | Traceback (most recent call last):
target-jsonl |   File "/home/tom/git/meltano-projects/gitlab-metrics/.meltano/loaders/target-jsonl/venv/bin/target-jsonl", line 8, in <module>
target-jsonl |     sys.exit(main())
target-jsonl |   File "/home/tom/git/meltano-projects/gitlab-metrics/.meltano/loaders/target-jsonl/venv/lib/python3.7/site-packages/target_jsonl.py", line 94, in main
target-jsonl |     state = persist_messages(input_messages, config.get('destination_path', ''), config.get('do_timestamp_file', True))
target-jsonl |   File "/home/tom/git/meltano-projects/gitlab-metrics/.meltano/loaders/target-jsonl/venv/lib/python3.7/site-packages/target_jsonl.py", line 59, in persist_messages
target-jsonl |     validators[o['stream']].validate(float_to_decimal(o['record']))
target-jsonl |   File "/home/tom/git/meltano-projects/gitlab-metrics/.meltano/loaders/target-jsonl/venv/lib/python3.7/site-packages/jsonschema/validators.py", line 129, in validate
target-jsonl |     for error in self.iter_errors(*args, **kwargs):
target-jsonl |   File "/home/tom/git/meltano-projects/gitlab-metrics/.meltano/loaders/target-jsonl/venv/lib/python3.7/site-packages/jsonschema/validators.py", line 105, in iter_errors
target-jsonl |     for error in errors:
target-jsonl |   File "/home/tom/git/meltano-projects/gitlab-metrics/.meltano/loaders/target-jsonl/venv/lib/python3.7/site-packages/jsonschema/_validators.py", line 323, in properties_draft4
target-jsonl |     schema_path=property,
target-jsonl |   File "/home/tom/git/meltano-projects/gitlab-metrics/.meltano/loaders/target-jsonl/venv/lib/python3.7/site-packages/jsonschema/validators.py", line 121, in descend
target-jsonl |     for error in self.iter_errors(instance, schema):
target-jsonl |   File "/home/tom/git/meltano-projects/gitlab-metrics/.meltano/loaders/target-jsonl/venv/lib/python3.7/site-packages/jsonschema/validators.py", line 105, in iter_errors
target-jsonl |     for error in errors:
target-jsonl |   File "/home/tom/git/meltano-projects/gitlab-metrics/.meltano/loaders/target-jsonl/venv/lib/python3.7/site-packages/jsonschema/_validators.py", line 134, in multipleOf
target-jsonl |     raise decimal.DecimalException(f"instance: {instance}, dB: {dB}")
target-jsonl | decimal.DecimalException: instance: 299.52638, dB: 1E-38

Note that the final line was caused by me monkey patching the _validators.py file with the following change:

6a7,9
> import decimal
> import traceback
> 
127c130,146
<         failed = instance % dB
---
>         try:
>             failed = instance % dB
>         except decimal.DecimalException:
>             traceback.print_exc()
>             raise decimal.DecimalException(f"instance: {instance}, dB: {dB}")

so that I could see the precision of the records causing the issue.

This seems to be happening when handling the following schema coming from the Postgres tap:

          "duration": {
            "type": [
              "null",
              "number"
            ],
            "exclusiveMaximum": true,
            "maximum": 100000000000000000000000000000000000000000000000000000000000000,
            "multipleOf": 1e-38,
            "exclusiveMinimum": true,
            "minimum": -100000000000000000000000000000000000000000000000000000000000000
          },

I'm aware of the fix for an issue with decimal numbers in 95e9fdb but without seeing any tests or anything in the commit message I don't know what exactly was fixed there. It doesn't seem to have solved this issue anyway.

It looks like this was fixed elsewhere by a combination of https://github.com/meltano/target-postgres/pull/6/files and https://github.com/meltano/target-postgres/pull/7/files in another target by checking the precision required for the validation of the 2 values and then increasing the precision to that if it's not high enough already.

[visch]

Hit this today as well.

target-jsonl          | Traceback (most recent call last):
target-jsonl          |   File "/home/visch/git/oracle2mssql/oracle2mssql/.meltano/loaders/target-jsonl/venv/bin/target-jsonl", line 8, in <module>
target-jsonl          |     sys.exit(main())
target-jsonl          |   File "/home/visch/git/oracle2mssql/oracle2mssql/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/target_jsonl.py", line 94, in main
target-jsonl          |     state = persist_messages(input_messages, config.get('destination_path', ''), config.get('do_timestamp_file', True))
target-jsonl          |   File "/home/visch/git/oracle2mssql/oracle2mssql/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/target_jsonl.py", line 59, in persist_messages
target-jsonl          |     validators[o['stream']].validate(float_to_decimal(o['record']))
target-jsonl          |   File "/home/visch/git/oracle2mssql/oracle2mssql/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/jsonschema/validators.py", line 129, in validate
target-jsonl          |     for error in self.iter_errors(*args, **kwargs):
target-jsonl          |   File "/home/visch/git/oracle2mssql/oracle2mssql/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/jsonschema/validators.py", line 105, in iter_errors
target-jsonl          |     for error in errors:
target-jsonl          |   File "/home/visch/git/oracle2mssql/oracle2mssql/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/jsonschema/_validators.py", line 300, in properties_draft4
target-jsonl          |     for error in validator.descend(
target-jsonl          |   File "/home/visch/git/oracle2mssql/oracle2mssql/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/jsonschema/validators.py", line 121, in descend
target-jsonl          |     for error in self.iter_errors(instance, schema):
target-jsonl          |   File "/home/visch/git/oracle2mssql/oracle2mssql/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/jsonschema/validators.py", line 105, in iter_errors
target-jsonl          |     for error in errors:
target-jsonl          |   File "/home/visch/git/oracle2mssql/oracle2mssql/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/jsonschema/_validators.py", line 127, in multipleOf
target-jsonl          |     failed = instance % dB
target-jsonl          | decimal.InvalidOperation: [<class 'decimal.DivisionImpossible'>]

[visch]

Should be fixed with #5

[andyhuynh3]

Closing this as it is fixed with #5