Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for IndexedDB #1699

Open
dsseng opened this issue Feb 7, 2019 · 39 comments
Open

Add support for IndexedDB #1699

dsseng opened this issue Feb 7, 2019 · 39 comments
Assignees
Labels
feat new feature (which has been agreed to/accepted) web related to Web APIs

Comments

@dsseng
Copy link
Contributor

dsseng commented Feb 7, 2019

It'll be useful to have IndexedDB in Deno.
Related to #1657.

@manyuanrong
Copy link
Contributor

Should we use pure TS implementation or on the Rust side?

@manyuanrong
Copy link
Contributor

Or use ready-made C++ implementations
https://cs.chromium.org/chromium/src/content/browser/indexed_db/

@kitsonk
Copy link
Contributor

kitsonk commented Mar 13, 2019

This is really complicated when you dig into it.

There is no Rust implementation of IndexDB (there is a crate that appears to be squatting, but it really isn't an implementation). Servo does not support IndexedDB, they have a long standing issue to implement. So on the Rust front it is sort of a dead end.

In Node.js, there are no realistic implementations either. Again there is a package squatting on indexeddb, but it isn't really a good package at all. The one thing that seems to have promise is IndexedDBShim which sits on top of WebSQL, and there are native implementations of WebSQL on Node.js using sqlite3.

jsdom doesn't implement IndexedDB or WebSQL, though there is this issue talking about implementing the IndexedDB shim mentioned above, I assume requiring the WebSQL implementation. It looks like it is stalled.

There is also an in memory IndexedDB API written in TypeScript.

There is what looks like a realistic Rust create for sqlite3.

So outside of using a C++ implementation, which would be a break in the pattern, the only realistic path I can easily see is that to use the in memory IndexedDB written in TypeScript, which would not give persistent storage, or we could adapt the in memory version and wire it up to a Rust sqlite3 bindings to persist the data.

I don't think just persisting out to JSON really works, because IndexedDB is a bit more complex than that, it contains concepts like transactions and other database features which is why almost all the browser implementations are based on sqlite3 IIRC.

@kitsonk
Copy link
Contributor

kitsonk commented Apr 24, 2020

Just updating this, servo still doesn't have it, and Gecko is looking at refactoring, but it is sqlite3 under the hoods. Here is a the most recent post that lays things out most relevantly: servo/servo#6963 (comment).

It would be really good to get localStorage/IndexedDB/kv-store in to Deno, IMO.

@anlexN

This comment has been minimized.

@kitsonk
Copy link
Contributor

kitsonk commented May 18, 2020

@anlexN your comment is borderline abusive. The issue is well discussed here of how it is a complex issue. Please keep your conversation civil and on-topic.

@kitsonk kitsonk added feat new feature (which has been agreed to/accepted) web related to Web APIs labels Nov 5, 2020
@kitsonk kitsonk changed the title IndexedDB Add support for IndexedDB Nov 5, 2020
@jacobbogers
Copy link

jacobbogers commented Apr 6, 2021

HI, just exploring deno a bit, i thought (my mental model) is that deno implemented chromium aswell, (so it 100% compatible (as much as possible) with all webapis)

is the source code of the implementation of indexDB (firefox chrome) open sourced?

@kitsonk
Copy link
Contributor

kitsonk commented Apr 6, 2021

@jacobbogers Deno does not implement Chromium, it only implements V8. All the Web APIs need to be implemented independently. Chromium tends to implement these in C++ using the v8 binding to inject objects into the isolate.

Yes, both Firefox and Chromium are open source and their code is available, but that doesn't make it magically work for Deno.

@jacobbogers
Copy link

@jacobbogers Deno does not implement Chromium, it only implements V8. All the Web APIs need to be implemented independently. Chromium tends to implement these in C++ using the v8 binding to inject objects into the isolate.

Yes, both Firefox and Chromium are open source and their code is available, but that doesn't make it magically work for Deno.

thanks for explaining, well at least if the source is avail (Firefox/Chrome) then we can kindof get some ideas /inspiration how to implement it,

IndexDB is a tuff nut to crack, it has transactions but has no explicit commit (its weird and ingenious way of doing transactions) this would not be easy to build from scratch.

@kitsonk
Copy link
Contributor

kitsonk commented Apr 7, 2021

@jacobbogers we have ideas/inspiration. One of the biggest factors was that we didn't have easy access to the V8 serialize/deserialize routines, which you need to support "structured cloning" which is needed to be able to store objects. We have that now. The other problem is the underlying DB as well as support under Rust. LevelDB (the one Google created and is in Chromium) is a bit of a mess. Firefox and Safari use SQLite as well as Servo looked at SQLite (can't remember how far they got). The biggest problem is the analogues between how Chromium and Firefox operate, and how Deno operates mean that there is very little overlap. For implementing web standards, we have had a lot better luck looking at a JavaScript shim/polyfill and the specification itself, than we have looking at other browser implementations.

There are the wpt for IndexedDB which helps ensure the standard is implemented correctly, and we now have the wpt integrated into our test process.

@brainless
Copy link

brainless commented May 25, 2021

Hey everyone just reading the discussions. I think IndexDB for Deno might end up being used in more varied (or larger) ways than in a browser. For example, it is OK to expect data eviction in the browser, but if developers use IndexDB as a persistent backend store (like say SQLite) then the they might want further control of the underlying storage, its space allocation, path, etc.. Ability to backup/restore also comes to mind.

Maybe my thought process is incorrect, just that I felt it is perhaps a good time to share these now, before we have a ready implementation.

@kitsonk
Copy link
Contributor

kitsonk commented May 25, 2021

then the they might want further control of the underlying storage, its space allocation, path, etc..

Then that isn't IndexedDB, that is a native SQLite plugin.

@brainless
Copy link

then the they might want further control of the underlying storage, its space allocation, path, etc..

Then that isn't IndexedDB, that is a native SQLite plugin.

You are right @kitsonk this is not IndexDB. I am just saying that people might end up using IndexDB in a different way. Maybe just great documentation will help in case people start using IndexDB as a traditional backend store.

@tracker1
Copy link

tracker1 commented Jun 7, 2021

Small input on this one, investigating leveldb as a potential underlying option to implement IndexedDB on top of, and it looks like most of the rust crates are just bindings to the C library for leveldb and the single rust native implementation hasn't been updated in a long time with a warning.

I do think a rust native leveldb implementation with an IndexedDB compatibility layer over it would be a very good approach. Though, there aren't enough roots planted in that space (yet). Not sure if there is any interest in implementing such a thing in native rust, exposing a C api that matches the existing library for testing, well outside the scope of Deno.

@crowlKats
Copy link
Member

I am currently implementing IndexedDB. It will be based on SQLite as that was the decision that was made when picking the backend for WebStorage.

@Platzer
Copy link

Platzer commented Jun 12, 2021

then the they might want further control of the underlying storage, its space allocation, path, etc..

Then that isn't IndexedDB, that is a native SQLite plugin.

You are right @kitsonk this is not IndexDB. I am just saying that people might end up using IndexDB in a different way. Maybe just great documentation will help in case people start using IndexDB as a traditional backend store.

Think about application Migration in case of a multi application server, it would be nice to just copy a single file instead of extracting a table from a file in some sys directory. 🧐

@crowlKats
Copy link
Member

@Platzer well the idb file will be accessible just like the localstorage file.

@Platzer
Copy link

Platzer commented Jun 14, 2021

@Platzer well the idb file will be accessible just like the localstorage file.

Sorry I hadn't time to try it out, found it with deno info. This should work fine, thx 💪

@petamoriken
Copy link
Contributor

Unfortunately, IndexedDB is not useful at the moment. Specifically, transactions cannot be fully wrapped in Promise because they are auto-committed after the all requests and microtasks are completed. This should be solved by a new API in the future, but we don't know when that will be.

w3c/IndexedDB#34 (comment)

@johnspurlock
Copy link
Contributor

IndexedDB is not useful at the moment

Just wanted to say that while IndexedDB is not ideal, it is most certainly useful, and would be very useful in Deno for writing isomorphic browser/server-side libraries and apps.

@janus-reith
Copy link

I am currently implementing IndexedDB. It will be based on SQLite as that was the decision that was made when picking the backend for WebStorage.

Hey @crowlKats, I think support for IndexedDB in Deno would be awesome, have you been able to make progress on this, or did you face any major roadblocks?

Btw hope I won't get too far off topic with this, but with new architecture options like Cloudflare Workers emerging, and hoping Deno Deploy would eventually add such a feature too, there could be a lot of use cases for IndexedDB on the server side. (Although I'm not sure wether storage options with a a standard-compliant implementation would actually be flexible enough for use cases I have in mind.)

@crowlKats
Copy link
Member

Hey @crowlKats, I think support for IndexedDB in Deno would be awesome, have you been able to make progress on this, or did you face any major roadblocks?

I havent. A bit after i wrote that comment, i stopped, as:
A) i didnt have the time for such a complex api
B) the api sucks to either make or use
C) I am not all too familiar with database related things
Maybe at some point i will pick it up again

Regarding storage on Deploy:
Having a KV storage is something that we are considering

@janus-reith
Copy link

I havent. A bit after i wrote that comment, i stopped, as: A) i didnt have the time for such a complex api B) the api sucks to either make or use C) I am not all too familiar with database related things Maybe at some point i will pick it up again

Regarding storage on Deploy: Having a KV storage is something that we are considering

Thanks for the quick response!

@lucacasonato lucacasonato self-assigned this Feb 2, 2022
@aaronhuggins
Copy link

aaronhuggins commented Feb 7, 2022

Anyone looking for a pony or poly fill, I've got IndexedDBShim working here based on a WebSQL ponyfill here.

@jsejcksn
Copy link
Contributor

jsejcksn commented Feb 26, 2022

@crowlKats I think having KV storage is a more accessible first goal than an IDB implementation. Have you considered this idea, and, if so, what attributes would you expect such a feature to have? Here are some examples of what I mean by that:

  • each store is namespaced (many kv stores, each accessed by its ID/name)
  • each store's API is map-like
  • (even if structured clone or circular references would be an issue) at least all JSON types should be supported as values, i.e.
    type JsonValue = boolean | number | null | string | JsonValue[] | { [key: string]: JsonValue | undefined };
    type Value = JsonValue | undefined;
  • async?
  • backed by SQLite?

@crowlKats
Copy link
Member

This issue is mainly about CLI, and there seem to be demand for IDB, so we will implement IDB. Then for feature parity, we probably will implement IDB on deploy as well. For CLI we cant just create some new API, it goes against our principles of being web compatible.

@jsejcksn
Copy link
Contributor

jsejcksn commented Feb 26, 2022

For CLI we cant just create some new API, it goes against our principles of being web compatible

I meant on the Deno namespace, where web-compatibility isn't a criteria (many such APIs already exist there).

e.g. Deno.kvStorage for an object or Deno.KVStorage for a class, etc.

@crowlKats
Copy link
Member

But that has to do with web compat, or rather, in this case, incompatibility. What we put in the deno ns and what we dont is carefully considered based on if there is a web api that could take care of it.

@jsejcksn
Copy link
Contributor

jsejcksn commented Feb 26, 2022

I think I now understand the miscommunication (and perhaps I didn't make it clear in my initial message): my suggestion is entirely unrelated to web compatibility. It's simply about offering a built-in API for persistent KV storage of native data types.

(You might consider my suggestion analogous to WICG/kv-storage.)

@crowlKats
Copy link
Member

Yes, I understand that, but we wont go that route, since there is a web compatible solution: IDB. having 2 different APIs on a later date would be suboptimal; I wouldnt consider this as an option. KV-Storage is a dead spec. We have had this kind of discussion multiple times internally already, and we havent reached a final conclusion yet

@jsejcksn
Copy link
Contributor

We have had this kind of discussion multiple times internally already, and we havent reached a final conclusion yet

Are there published notes from those? (I was actually searching for kv storage and that's how I found this issue.)

@crowlKats
Copy link
Member

No, since this is an ongoing discussion

@aaronhuggins
Copy link

@crowlKats I have some interest in implementing this, but I don't want to take something away that you really want to do. If it is okay, should I start from your branch in #14035? Or just cut a fresh branch independent of yours and open a separate PR?

@crowlKats
Copy link
Member

@aaronhuggins The PR isnt too far from completion, but still a bit of work left. thing is, it currently isnt high priority, so leaving this on the backburner for a bit. I would advise against starting your own PR

@jsejcksn
Copy link
Contributor

Ref: #18232

Is this groundwork for a persistent storage API in Deno Deploy?

@bartlomieju
Copy link
Member

@jsejcksn that's correct.

@benatkin
Copy link

benatkin commented Apr 18, 2023

a bit tangential, but IIRC localStorage doesn't work in web workers. It is in IndexedDB. I would hope/imagine that Deno maintains this behavior. However it isn't documented here. https://deno.land/[email protected]/runtime/web_storage_api

@idranme
Copy link
Contributor

idranme commented Jul 13, 2023

@crowlKats I have some interest in implementing this, but I don't want to take something away that you really want to do. If it is okay, should I start from your branch in #14035? Or just cut a fresh branch independent of yours and open a separate PR?

I'm kind of looking forward to it.

@alexgleason
Copy link
Contributor

Now that Deno.KV is a thing, would it make sense to make IndexedDB a wrapper around the Deno.KV API instead of yet another SQLite? That way IndexedDB on Deno Deploy would take advantage of FoundationDB, and it would mean improvements to Deno.KV locally (see: #18657) would improve the IndexedDB implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feat new feature (which has been agreed to/accepted) web related to Web APIs
Projects
None yet
Development

Successfully merging a pull request may close this issue.