Approach to store scrapped data in database (postgres)

[acanimal]

Hi, I recently discovered Crawlee and I'm trying to figure out how can I store the scraped data in database instead in local directorio storage.

Is there any plugin for that? How must I proceed to implement one? Must I code my own class that implements StorageClient interface? If so how must I injected later to be used.

Thanks!

[B4nan]

If you'd like to use postgres for all the storages, then yes, you'd have to implement new storage client with all the interface methods. We have alternative storage implementation called https://github.com/apify/apify-storage-local-js - it uses sqlite under the hood, so you could probably use that as a base for a postgres storage implementation.

In the long run, we want to simplify this interface and provide more storage implementations.

You could as well just use the memory storage for everything except datasets, and store data directly (via your own custom helper) to the postgres.