Show command with nested json yields "Object of type ndarray is not JSON serializable"

[racheldaniel]

Describe the bug

dbt-databricks throws Object of type ndarray is not JSON serializable when previewing nested json (dbt show)

Steps To Reproduce

• Open a databricks dbt project on your machine
• create a new model and paste the code snippet below into it
• run dbt show --select your_new_model
• This will throw Object of type ndarray is not JSON serializable

with src_data as (
 
   select * from (
   values
       ('first json', '[{"obisCode":"1.8.0","interval":"00:05:00","rollOverLimit":999999.999,"values":[{"value":0,"registrationTime":"2023-01-31T21:16:35.5848643Z","measurementTime":"2023-01-31T21:15:00.0000000Z","quality":0,"isConfident":true}]}]')
   ) as (filename, series)
 
),
cast_types as (
 
   select
       * except(series)
       , from_json(series,'
           array<
               struct<
                   obisCode:string,
                   interval:string,
                   rollOverLimit:double,
                   values:array<
                       struct<
                           value:float,
                           registrationTime:timestamp,
                           measurementTime:timestamp,
                           quality:int,
                           isConfident:boolean
                       >
                   >
               >
           >
       ') as series
   from src_data
 
),
explode_series as (
 
   select
       *
       , explode(series)
   from cast_types
 
)
select * from explode_series

Expected behavior

I expect this to result in previewed code as with other adapters:

| filename   | series                         | obiscode | interval | rolloverlimit   | values               | ... |
| ---------- | -------------------- | -------- | -------- | ------------- | -------------------- | --- |
| first json | [{"obisCode":"1.8... | 1.8.0    | 00:05:00 |   999,999.999 | [{"value": 0, "re... | ... |

Screenshots and log output

Traceback in logs:

{"data": {"exc_info": "Traceback (most recent call last):\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 372, in safe_run\n    result = self.compile_and_execute(manifest, ctx)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 323, in compile_and_execute\n    result = self.run(ctx.node, manifest)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 419, in run\n    return self.execute(compiled_node, manifest)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/show.py\", line 38, in execute\n    adapter_response, execute_result = self.adapter.execute(\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/utils.py\", line 72, in wrapper\n    return func(*new_args, **new_kwargs)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/impl.py\", line 150, in execute\n    return super().execute(sql=sql, auto_begin=auto_begin, fetch=fetch, limit=limit)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/base/impl.py\", line 290, in execute\n    return self.connections.execute(sql=sql, auto_begin=auto_begin, fetch=fetch, limit=limit)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/connections.py\", line 798, in execute\n    table = self.get_result_from_cursor(cursor, limit)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/sql/connections.py\", line 132, in get_result_from_cursor\n    return dbt.clients.agate_helper.table_from_data_flat(data, column_names)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/clients/agate_helper.py\", line 127, in table_from_data_flat\n    value = json.dumps(value, cls=dbt.utils.JSONEncoder)\n  File \"/usr/lib/python3.8/json/__init__.py\", line 234, in dumps\n    return cls(\n  File \"/usr/lib/python3.8/json/encoder.py\", line 199, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n  File \"/usr/lib/python3.8/json/encoder.py\", line 257, in iterencode\n    return _iterencode(o, 0)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/utils.py\", line 351, in default\n    return super().default(obj)\n  File \"/usr/lib/python3.8/json/encoder.py\", line 179, in default\n    raise TypeError(f'Object of type {o.__class__.__name__} '\nTypeError: Object of type ndarray is not JSON serializable\n"}, "info": {"category": "", "code": "Z011", "extra": {}, "invocation_id": "4d4b3d45-15e5-4f27-8045-3ae4fb04c5b0", "level": "debug", "msg": "Traceback (most recent call last):\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 372, in safe_run\n    result = self.compile_and_execute(manifest, ctx)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 323, in compile_and_execute\n    result = self.run(ctx.node, manifest)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/base.py\", line 419, in run\n    return self.execute(compiled_node, manifest)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/task/show.py\", line 38, in execute\n    adapter_response, execute_result = self.adapter.execute(\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/utils.py\", line 72, in wrapper\n    return func(*new_args, **new_kwargs)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/impl.py\", line 150, in execute\n    return super().execute(sql=sql, auto_begin=auto_begin, fetch=fetch, limit=limit)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/base/impl.py\", line 290, in execute\n    return self.connections.execute(sql=sql, auto_begin=auto_begin, fetch=fetch, limit=limit)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/databricks/connections.py\", line 798, in execute\n    table = self.get_result_from_cursor(cursor, limit)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/adapters/sql/connections.py\", line 132, in get_result_from_cursor\n    return dbt.clients.agate_helper.table_from_data_flat(data, column_names)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/clients/agate_helper.py\", line 127, in table_from_data_flat\n    value = json.dumps(value, cls=dbt.utils.JSONEncoder)\n  File \"/usr/lib/python3.8/json/__init__.py\", line 234, in dumps\n    return cls(\n  File \"/usr/lib/python3.8/json/encoder.py\", line 199, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n  File \"/usr/lib/python3.8/json/encoder.py\", line 257, in iterencode\n    return _iterencode(o, 0)\n  File \"/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/utils.py\", line 351, in default\n    return super().default(obj)\n  File \"/usr/lib/python3.8/json/encoder.py\", line 179, in default\n    raise TypeError(f'Object of type {o.__class__.__name__} '\nTypeError: Object of type ndarray is not JSON serializable\n", "name": "LogDebugStackTrace", "pid": 50, "thread": "Thread-8", "ts": "2023-10-26T19:13:37.437631Z"}}

System information

The output of dbt --version:

  - installed: 1.6.6

This bug was originally found in dbt v.1.5.0-latest with accompanying adapter, confirmed to be present in 1.6.6 as well.

The operating system you're using:
This has been confirmed on MacOS Ventura 13.5.2 and Ubuntu 20.04

The output of python --version:
python 3.8.10

Additional context

Add any other context about the problem here.

[benc-db]

Thanks for reporting, will investigate.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue.

[benc-db]

Closing in favor of #897