Hadoop integration with offline store

[Andrew32bit]

Hello! is it possible to see in near future hadoop integration with offline FS?

[woop]

We're going to try to make it super easy to add new stores. Have a look at the interface we've started building over here:

feast/sdk/python/feast/offline_store.py

Line 81 in a541b1d

class OfflineStore(ABC):

It should be ready for contribution in a few weeks.

[Andrew32bit]

@woop is any news about this feature ?

[woop]

Hey @Andrew32bit. We'll be shipping the latest Feast in line with www.applyconf.com later this month. Right now we're cleaning up these interfaces. Essentially you'll need to extend a "provider" interface to add support for an offline store.

One thing that isn't clear to me: Are you looking for Hive or HDFS support?

[Andrew32bit]

@woop im looking both =)It will be ok for HDFS .How can I customize get_offline_features() ?

[elukey]

+1 as well, but I'd add an extra constraint, namely Kerberos support. Precisely I'd need any client of the Feature Store to be able to use get_offline_features and force only the Feature Store to authenticate properly to HDFS (mainly because from Kubernetes it is a little hard to set up Kerberos in Hadoop's context).

[elukey]

Hi! I am very interested in integrating Feast with an existing Hadoop + Hive + Spark environment (Kerberos auth, on premise). The ideal scenario would be to be able to query Hive to get historical features while training models, and IIUC this may become possible if somebody implements the OfflineStore() API.

After reading https://github.com/feast-dev/feast/blob/master/examples/minimal/minimal_ride_hailing.ipynb I am wondering what is best though, IIUC Spark can be used to pull data into dataframes when calling get_historical_features(), so possibly having HDFS support would suffice? Any help in understanding this would be really appreciated :)

[nicelgueta]

• 1 on this.
  Exactly same boat of understanding as you after reading minimal ride hailing. Using Impala instead of Hive but rest is same with Kerberos auth on prem. Help with further understanding of how this should work would be great!

[woop]

Hi folks, thanks for being so patient on the replies. These discussion threads sometimes aren't as visible as normal issues.

We simply need to implement the offline store interface in order to support Hadoop/Hive. Right now we're working on documentation in order to make it super clear to users how connectors can be built. We expect to have the documentation out in the next 2-3 weeks.