Feature/json schema article #6787

pubkey · 2025-01-28T10:12:51Z

DO NOT MERGE

docs-src/docs/article-json-schema.md

jviotti · 2025-01-29T12:09:34Z

docs-src/docs/article-json-schema.md

+
+While RxDB reuses the core JSON Schema specification, it also extends that schema to introduce RxDB-specific features. Like in other NoSQL databases, you manually define which fields to encrypt, which ones to index, and how to interpret specific fields for queries. RxDB puts these configurations directly into the JSON Schema with additional properties:
+
+- `primaryKey`: Specifies which field in the document serves as the primary key.


Interesting stuff btw. One line of work we did in the past was helping Oracle define a standard "database" extension to JSON Schema: https://github.com/json-schema-org/vocab-database/blob/main/database.md (which they implement at Oracle). I wonder if that same database vocabulary could be a good fit for RxDB and if it could be extended for the things you do that they don't!

Also, we should separately discuss how to register your extension keywords as a proper "vocabulary" as the next version of JSON Schema will disallow unrecognised keywords! I would love to help you out with this.

Also, we should separately discuss how to register your extension keywords as a proper "vocabulary" as the next version of JSON Schema will disallow unrecognised keywords! I would love to help you out with this.

Oh this is new to me. RxDB could always fall back to having a separate config-object, outside of the schema.

jviotti · 2025-01-29T12:12:08Z

docs-src/docs/article-json-schema.md

+
+- `primaryKey`: Specifies which field in the document serves as the primary key.
+- `indexes`: Defines which fields (or combination of fields) RxDB indexes. You can have single-field indexes or compound indexes.
+- `version`: Indicates the version of the schema. Whenever you change your schema, you must increment this version so RxDB can handle migrations or other adjustments. This is important because data migration on a client-side database can be tricky when you have many clients out there that update your app at different points in time.


I think it's OK for the blog post. For your later consideration, the current "standard" way of doing versioning is with $id, and making the version part of that. For cases like RxDB, URNs or Tag URIs can be interesting!

I will think about these. RxDB is so complex already and having the user to learn about the URI instead of a version number might be more confusing than what is solves.

jviotti · 2025-01-29T12:23:42Z

docs-src/docs/article-json-schema.md

+
+Performance is a critical factor in deciding whether to validate documents at runtime, especially in production environments. The following tables illustrate a basic comparison of initialization time (time-to-first-insert) and bulk insertion speed for different validators on two RxDB storages
+
+The RxDB team ran performance benchmarks using two storage options:


Can you describe the exact browser version and machine characteristics you used to show the numbers below?

I will add this.

jviotti · 2025-01-29T12:25:49Z

docs-src/docs/article-json-schema.md

+
+Over time, RxDB has evolved its usage of JSON Schema, learning from real production experiences and feedback from the community. Here are some key takeaways:
+
+- Avoid inlined `required` fields: Some validators let you write `"required": true` inside the property definition, but the official JSON Schema specification requires you to declare an array of required fields at the parent object level. Following the standard from the start avoids confusion and ensures broader validator compatibility.


This is historical. JSON Schema Draft 3 used the boolean form of required. The issue here really touches on what I mentioned before: most schema authors avoid $schema, thus set no versions, and everybody gets confused about what version of JSON Schema they are actually using. Can you revise this paragraph, maybe rephrasing it as some validators being obsolete? (as required: true was indeed valid ages ago!)

jviotti · 2025-01-29T12:26:50Z

docs-src/docs/article-json-schema.md

+
+- Avoid inlined `required` fields: Some validators let you write `"required": true` inside the property definition, but the official JSON Schema specification requires you to declare an array of required fields at the parent object level. Following the standard from the start avoids confusion and ensures broader validator compatibility.
+
+- Keep Custom Fields at the Top Level: Originally, RxDB allowed custom definitions (`index`, `encrypted`, etc.) to appear deeply nested. This caused performance hits because the library had to traverse large schema objects to find them. By placing these fields at the top level, RxDB can parse and apply them much faster, improving startup times.


I now wonder if these custom fields even need to be part of the schema. Maybe you have a JSON object with these properties and a schema property? Then you don't have to deal with the complexity of creating a custom JSON Schema vocabulary, etc

I will think about this. Maybe RxDB should move them to somewhere else.

jviotti · 2025-01-29T12:27:31Z

docs-src/docs/article-json-schema.md

+
+- Keep Custom Fields at the Top Level: Originally, RxDB allowed custom definitions (`index`, `encrypted`, etc.) to appear deeply nested. This caused performance hits because the library had to traverse large schema objects to find them. By placing these fields at the top level, RxDB can parse and apply them much faster, improving startup times.
+
+- Error messages are not standardized: Each validator produces a different structure for error messages. If your app logic inspects these errors, you risk partial or complete rewrites if you ever switch validators. Decide early on which validator meets your needs and plan on sticking with it long-term.


This is a huge one we are constantly battling against. Can you mention the Standard Output Formats I touched on before? If you can express your desire for validators to comply, that would be wonderful!

Yes, I add this.

docs-src/docs/article-json-schema.md

jviotti · 2025-01-29T12:30:28Z

Awesome stuff! I left a few comments just to polish things a bit. I also left some suggestions for improving RxDB's JSON Schema integration. Of course, those don't have to be solved for the case study, but would love to separately collaborate on them :)

Once we feel the draft is ready, we can send a PR to the official JSON Schema website to get reviews from the rest of the TSC and get it published.

Thanks a lot for putting this together! It already looks very interesting

Co-authored-by: Juan Cruz Viotti <[email protected]>

pubkey · 2025-01-29T15:45:48Z

@jviotti Thank you for the feedback. I updated all parts, please check.

pubkey added 5 commits January 28, 2025 11:09

ADD article

e66c6ce

FIX

74dfa99

FIX typo

ae07e27

REFACTOR article

36a21c5

FIX typo

8a6c0e1

jviotti suggested changes Jan 29, 2025

View reviewed changes

pubkey and others added 8 commits January 29, 2025 16:05

Update docs-src/docs/article-json-schema.md

b0d8f4f

Co-authored-by: Juan Cruz Viotti <[email protected]>

Update docs-src/docs/article-json-schema.md

e6bc968

Co-authored-by: Juan Cruz Viotti <[email protected]>

Update docs-src/docs/article-json-schema.md

68a126c

Co-authored-by: Juan Cruz Viotti <[email protected]>

Update docs-src/docs/article-json-schema.md

02eba6b

Co-authored-by: Juan Cruz Viotti <[email protected]>

Update docs-src/docs/article-json-schema.md

68ff07c

Co-authored-by: Juan Cruz Viotti <[email protected]>

Update docs-src/docs/article-json-schema.md

7524a50

Co-authored-by: Juan Cruz Viotti <[email protected]>

Update docs-src/docs/article-json-schema.md

1c2cbea

Co-authored-by: Juan Cruz Viotti <[email protected]>

ADD suggested changes

057a793

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/json schema article #6787

Feature/json schema article #6787

pubkey commented Jan 28, 2025

jviotti Jan 29, 2025

jviotti Jan 29, 2025

pubkey Jan 29, 2025

jviotti Jan 29, 2025

pubkey Jan 29, 2025

jviotti Jan 29, 2025

pubkey Jan 29, 2025

jviotti Jan 29, 2025

jviotti Jan 29, 2025

pubkey Jan 29, 2025

jviotti Jan 29, 2025

pubkey Jan 29, 2025

jviotti commented Jan 29, 2025

pubkey commented Jan 29, 2025


		While RxDB reuses the core JSON Schema specification, it also extends that schema to introduce RxDB-specific features. Like in other NoSQL databases, you manually define which fields to encrypt, which ones to index, and how to interpret specific fields for queries. RxDB puts these configurations directly into the JSON Schema with additional properties:

		- `primaryKey`: Specifies which field in the document serves as the primary key.


		Performance is a critical factor in deciding whether to validate documents at runtime, especially in production environments. The following tables illustrate a basic comparison of initialization time (time-to-first-insert) and bulk insertion speed for different validators on two RxDB storages

		The RxDB team ran performance benchmarks using two storage options:


		Over time, RxDB has evolved its usage of JSON Schema, learning from real production experiences and feedback from the community. Here are some key takeaways:

		- Avoid inlined `required` fields: Some validators let you write `"required": true` inside the property definition, but the official JSON Schema specification requires you to declare an array of required fields at the parent object level. Following the standard from the start avoids confusion and ensures broader validator compatibility.


		- Avoid inlined `required` fields: Some validators let you write `"required": true` inside the property definition, but the official JSON Schema specification requires you to declare an array of required fields at the parent object level. Following the standard from the start avoids confusion and ensures broader validator compatibility.

		- Keep Custom Fields at the Top Level: Originally, RxDB allowed custom definitions (`index`, `encrypted`, etc.) to appear deeply nested. This caused performance hits because the library had to traverse large schema objects to find them. By placing these fields at the top level, RxDB can parse and apply them much faster, improving startup times.


		- Keep Custom Fields at the Top Level: Originally, RxDB allowed custom definitions (`index`, `encrypted`, etc.) to appear deeply nested. This caused performance hits because the library had to traverse large schema objects to find them. By placing these fields at the top level, RxDB can parse and apply them much faster, improving startup times.

		- Error messages are not standardized: Each validator produces a different structure for error messages. If your app logic inspects these errors, you risk partial or complete rewrites if you ever switch validators. Decide early on which validator meets your needs and plan on sticking with it long-term.

Feature/json schema article #6787

Are you sure you want to change the base?

Feature/json schema article #6787

Conversation

pubkey commented Jan 28, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jviotti commented Jan 29, 2025

pubkey commented Jan 29, 2025