Can feast support null values for an entity column?

[danbaron63]

I have a use case where we have a table containing features about banking transactions for users. This table contains keys for multiple entities, e.g.

--------+-------------------+-----------------+--------+----------
user_id | feature_timestamp | counterparty_id | amount | direction
--------+-------------------+-----------------+--------+----------
1       | 2024-01-01        | 2               | 100.0  | in 
2       | 2024-02-01        | 1               | 102.0  | in 
3       | 2024-03-01        | null            | null   | null 
4       | 2024-04-01        | 5               | 100.0  | in 
3       | 2024-03-01        | 4               | 300.0  | in 
4       | 2024-04-01        | null            | null   | null 
--------+-------------------+-----------------+--------+----------

Where user_id is an entity and so is counterparty_id. The records with null values imply that no transactions took place for that month for a given user.

However this breaks feast when null values are supplied in an entity column. To recreate this you can run the driver_stats quick start but with some null values, when running python test_workflow.py you will get the following error message:

Materializing 2 feature views to 2024-10-16 15:06:13+01:00 into the sqlite online store.

driver_hourly_stats from 2024-10-15 15:06:13+01:00 to 2024-10-16 15:06:13+01:00:
  0%|                                                                         | 0/1 [00:00<?, ?it/s]WARNING:root:Cannot use sqlite_vec for vector search
  0%|                                                                         | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/Users/dan.baron/code/feast-demo/entity_serialisation/feature_repo/test_workflow.py", line 130, in <module>
    run_demo()
  File "/Users/dan.baron/code/feast-demo/entity_serialisation/feature_repo/test_workflow.py", line 22, in run_demo
    store.materialize_incremental(end_date=datetime.now())
  File "/Users/dan.baron/code/feast-demo/.venv/lib/python3.10/site-packages/feast/feature_store.py", line 1296, in materialize_incremental
    provider.materialize_single_feature_view(
  File "/Users/dan.baron/code/feast-demo/.venv/lib/python3.10/site-packages/feast/infra/passthrough_provider.py", line 332, in materialize_single_feature_view
    raise e
  File "/Users/dan.baron/code/feast-demo/.venv/lib/python3.10/site-packages/feast/infra/materialization/local_engine.py", line 173, in _materialize_one
    self.online_store.online_write_batch(
  File "/Users/dan.baron/code/feast-demo/.venv/lib/python3.10/site-packages/feast/infra/online_stores/sqlite.py", line 115, in online_write_batch
    entity_key_bin = serialize_entity_key(
  File "/Users/dan.baron/code/feast-demo/.venv/lib/python3.10/site-packages/feast/infra/key_encoding_utils.py", line 90, in serialize_entity_key
    val_bytes, value_type = _serialize_val(
  File "/Users/dan.baron/code/feast-demo/.venv/lib/python3.10/site-packages/feast/infra/key_encoding_utils.py", line 23, in _serialize_val
    raise ValueError(f"Value type not supported for feast feature store: {v}")
ValueError: Value type not supported for feast feature store: 

Does anyone have any suggestion of how we can supply nulls to feast as entities without breaking the serialisation logic? Thanks!

EDIT: I took a look at using TTL in place of null records (on the basis that after a given period a record won't be served as it is considered stale) but AFAIK this only applies to historic data and not to online serving. Could someone please correct me if I'm wrong on this though and whether TTL could be used with the online store and then we can exclude the null values altogether whilst still achieving the same purpose of representing that no transactions took place over a given period.