Neo metaphysics #809

alloy · 2017-11-14T17:45:29Z

This bootstraps schema stitching inside MP. In the below example you’ll see the use of MP’s and Convection’s schemas combined and used.

{
  # This root field comes from MP’s schema.
  artist(id: "marina-abramovic-1") {
    name
  }
  # This root field comes from Convection’s schema.
  submission(id: 186) {
    # This field exists in Convection’s schema
    artist_id
    # This field is generated in the `mergedSchema` module by fetching
    # an `Artist` from MP’s schema using the above `artist_id` field.
    artist {
      name
    }
  }
}

{
  "data": {
    "artist": {
      "name": "Marina Abramović"
    },
    "submission": [
      {
        "artist_id": "marina-abramovic-1",
        "artist": {
          "name": "Marina Abramović"
        }
      }
    ]
  }
}

TODO

Neo metaphysics #809 (comment)

alloy · 2017-11-14T17:45:37Z

I think that, if we feel this works well, we should start removing types from MP’s schema that are expressed in service’s own GraphQL schemas and stitch them in. The end goal would be to have pretty much nothing left in MP but stitching code.

alloy · 2017-11-14T17:47:49Z

Here’s a few issues I’ve written up previously about other (long term) things to consider in this new stitching world:

alloy · 2017-11-14T17:53:56Z

index.js

+mergedSchema().then(merged => {
+  schema = merged
+  app.listen(port, () => info(`Listening on ${port}`))
+})


Maybe all of the app setup should just move inside this promise and then schema can just be a const.

alloy · 2017-11-14T17:55:43Z

mergedSchema.js

+          fragment: `fragment SubmissionArtist on Submission { artist_id }`,
+          resolve: (parent, args, context, info) => {
+            const id = parent.artist_id
+            return mergeInfo.delegate("query", "artist", { id }, context, info)


This is basically at runtime generating a query like:

{ artist(id: "<result of artist_id>") { # ... } }

Interesting note is that we may not want to have root fields for each type that we stitch into other types. Maybe we can use the node root field for this? 🤔

The mergeSchemas author explained to me that this should be done in a few steps:

Add the required root field to the original schema.

Merge the schemas and setup the delegator to use the root field.

Transform the merged schema to remove the root field (the delegator refers to the original schema).

alloy · 2017-11-14T18:01:39Z

package.json

@@ -53,6 +54,7 @@
    "graphql": "^0.11.7",
    "graphql-depth-limit": "^1.1.0",
    "graphql-relay": "0.5.3",
+    "graphql-tools": "https://github.com/alloy/graphql-tools/releases/download/v2.7.3-alpha.1/graphql-tools-2.7.3-alpha.1.tgz",


This uses my fork ardatan/graphql-tools#484

alloy · 2017-11-14T18:06:46Z

mergedSchema.js

+    schemas: [localSchema, convectionSchema, linkTypeDefs],
+    // Prefer others over the local MP schema.
+    onTypeConflict: (_leftType, rightType) => {
+      console.warn(`[!] Type collision ${rightType}`) // eslint-disable-line no-console


@orta There’s currently a bunch of consignments related type name conflicts, presumably because you’ve manually added these to MP’s schema before:

19:04:06 web.1 | [!] Type collision ID 19:04:06 web.1 | [!] Type collision Submission 19:04:06 web.1 | [!] Type collision String 19:04:06 web.1 | [!] Type collision Asset 19:04:06 web.1 | [!] Type collision JSON 19:04:06 web.1 | [!] Type collision Boolean 19:04:06 web.1 | [!] Type collision Int

The ID, String, Boolean, and Int ones are built-in though, it seems weird that graphql-tools would trigger this callback for those 🤔

Yeah, that's tricky, to remove Submission entirely in metaphysics - I'd have to recreate a bunch of mutations on convection, feasible, but time consuming

That said, the submission type should be the exact same object - so maybe it's fine to return the convection one

WRT the JSON type - I would assume they have the same behavior in terms of representing an any - I bet we get these primitives for every Ruby GraphQL

The mutations already exist in convection-- might need some updating but the basic functionality is there.

Ah, and indeed the mutations are already automatically available in the stitched schema, I haven’t tried them yet, though.

alloy · 2017-11-14T18:20:25Z

mergedSchema.js

+    fetch,
+    uri: process.env.CONVECTION_GRAPH_URL,
+    headers: {
+      Authorization: `Bearer ${process.env.CONVECTION_TOKEN}`,


@sweir27 I guess this actually needs to be specified by the client at runtime, as it differs per user, right? I think that should be easy using the ‘link’ API.

Yeah, the convection loaders have the ability to convert a grav token into a convection token. We'll need something like this for at least convection / impulse

Ooohhh, we can probably just access those data loaders (including the token loader) during stitching resolving. Will take a look 👍

alloy · 2017-11-14T20:11:02Z

mergedSchema.js

+    return {
+      ...context.headers,
+      headers: {
+        // authorization: `XApp ${config.GRAVITY_XAPP_TOKEN}`,


@sweir27 Does this look like what you had in mind (to allow MP to fetch Convection’s schema on startup)?

We chatted about this - it's a hard one, maybe looking into giving Metaphysics 2 client apps, one for the APIs we forward onwards and another for the server to server access. Then the server2server one can have privileged access for getting the schema.

Just read the discussion in #platform-humans and that makes sense to me. My only question is how easy it is for MP to identify on each incoming request if the user has access to the more privileged app role? cc @mzikherman

But once we have that we can just load 2 stitches schemas and execute against the correct one depending on the access the authenticated user has.

Hmm, one thing that comes to mind is that in the more privileged stitched schema we would probably have stitching code that refers to the more privileged fields and thus leak schema details through MP being OSS 🤔

One solution I can think of is to have another app that imports all of MP and then extends the schema with privileged details and make that closed source? Probably only slightly more cumbersome, mostly in having to remember to add fields to the right app.

Filed a new issue to discuss further #810

alloy · 2017-11-15T05:00:16Z

mergedSchema.js

+    if (tokenLoader) {
+      return tokenLoader().then(({ token }) => {
+        return {
+          ...context.headers,


Note to self, this is wrong, it needs to add all headers to the below object instead.

ashkan18 · 2017-11-15T15:08:35Z

mergedSchema.js

+    }
+  `
+
+  return mergeSchemas({


This basically happens once during start up of MP, so one thing to consider is, lets say Convection's GraphLQ schema got updated and there was a new field, we'd ONLY get that update if we restart MP, right?

For now we might be fine with manual restarts of MP, but eventually if this became more problematic (as we add more and more services) it might make sense to use event stream for reloading schema. Each time an app updated it schema we'd refresh this.

Correct 👍 And that sounds like a good solution to keep in mind.

alloy · 2017-11-15T16:37:25Z

mergedSchema.js

          headers: {
+            ...context.headers,


Maybe we don’t even really want to pass on (all) headers that a client specifies? I can imagine that we do want e.g. the request ID that @sweir27 had been working on.

Yeah, we'd want the request ID - but that's probably it (at least in convection I've not seen any header handling outside of auth)

Note: with #818 there's more headers we want to pass through

craigspaeth · 2017-11-16T15:50:57Z

index.js

+          ...loaders,
+        },
+        formatError: graphqlErrorHandler(req.body),
+        validationRules: [depthLimit(queryLimit)],


Nice! Wasn't aware of GraphQL depth limit.

I… dislike it so much 😞 It’s such an arbitrary 🔨

Of course I understand why we’d want it at the moment, but in the long term I see only allowing persisted queries in production (or limiting depth query for not persisted queries) as a better solution.

craigspaeth · 2017-11-16T16:00:59Z

mergedSchema.js

+function createConvectionLink() {
+  const httpLink = createHttpLink({
+    fetch,
+    uri: `${process.env.CONVECTION_API_BASE}/graphql`,


Seems like this uri might be the main piece of convection-unique code here. I wonder if there's an opportunity to wrap this boilerplate in a way that'd make it easy to do just point to the urls we're stitching.

The other Convection specific code is using the convectionTokenLoader. I agree that this can probably be made more generic, but was punting on that until we add another service.

craigspaeth · 2017-11-16T16:53:35Z

Very exciting stuff 👏 I think bootstrapping stitching inside MP with the intent to whittle it down to mostly just stitching code and having logic in downstream GraphQL APIs is a great strategy. This is a very exciting step towards a beautiful microservice story where MP is a single orchestrator of downstream GraphQL APIs and we can eliminate redundant fields/logic/caching layers. It'd also be cool to consider writing a HAL > GraphQL stitching library that could introspect the Hypermedia links (and JSON schemas?) and generate GraphQL schemas + resolvers for it 🤔 💥 .

One general comment not to block this PR is that I'm noticing a lot of code that I struggle to follow in these glue code layers. e.g. There's a lot of factory/dependency-injection-like patterns being used in the loader/stitching layers and I wonder if there're simpler ways to organize these modules. It seems like the tools encourage this as well, so maybe they're partly responsible. In any case, just a general comment—not sure how to make it better myself.

👏 amazing work!

orta · 2017-11-16T21:43:27Z

artsy/convection#122 is pretty close to being ready

stubailo · 2017-11-18T06:06:33Z

Hey, just found this from the PR here: ardatan/graphql-tools#484

Can you help me figure out how that function existing is messing things up? I see the PR just deletes it, but wanted to learn more.

alloy · 2017-11-18T20:49:52Z

@stubailo The test in that PR should illustrate it well if you comment out the deletion change.

In short, the problem is that after selecting a subset of fields from the original schema the function in the merged schema may not return true if the selection doesn’t satisfy the data it’s looking for.

For example, a isTypeOf implementation for the Artist type in the merged schema that looks for the data to contain name and birthday keys will fail to pass data returned from the original schema with the following query:

{
  artist(id: “banksy”) {
    name
  }
}

alloy · 2017-11-22T15:10:22Z

@saolsen Did a bit of refactoring re tracer work ff4083f and I also pass these IDs along to Convection a3c3c13

alloy · 2017-11-22T15:15:19Z

lib/requestIDs.js

+  return {
+    "X-Request-Id": requestID,
+    "x-datadog-trace-id": traceId,
+    "x-datadog-parent-id": parentSpanId,


@saolsen I noticed that these last two are not strings. I’m sure that all entries in response.headers get coerced to strings by Express before the response is sent to the client, but it was slightly unexpected to see these objects in there. Do you think we can add toString() calls here?

This is currently using my fork that has ardatan/graphql-tools#484

alloy · 2017-11-22T23:21:38Z

@craigspaeth

It'd also be cool to consider writing a HAL > GraphQL stitching library that could introspect the Hypermedia links (and JSON schemas?) and generate GraphQL schemas + resolvers for it 🤔 💥 .

I don’t know how well that would work when we want to design the schema without necessarily following the same shape in which these endpoints return data. Maybe an example of what you have in mind could help?

One general comment not to block this PR is that I'm noticing a lot of code that I struggle to follow in these glue code layers. e.g. There's a lot of factory/dependency-injection-like patterns being used in the loader/stitching layers and I wonder if there're simpler ways to organize these modules. It seems like the tools encourage this as well, so maybe they're partly responsible. In any case, just a general comment—not sure how to make it better myself.

It’s true and I agree that it can be a bit much when reading for the first few times. It does work nicely with how graphql-js works and especially testing, though (no more need for rewiring). I’m on the fence, because thus far everybody ended up understanding how it works.

damassi · 2017-11-23T05:37:15Z

lib/mergeSchemas.js

+      return rightType
+    },
+    resolvers: mergeInfo => ({
+      Submission: {


@alloy - can you explain how adding in additional schemas will work in relation to the above line? Is this a namespace?

No this is not a namespace, it’s the way in which schemas are created using graphql-tools https://github.com/apollographql/graphql-tools/blob/master/README.md. In this case the code you’re referring to is the implementation for the schema extension just above (in IDL).

orta · 2017-11-23T14:45:46Z

OK, it's time - let's give this a shot.

alloy assigned mzikherman Nov 14, 2017

alloy requested review from orta, sweir27, craigspaeth and joeyAghion November 14, 2017 17:48

alloy commented Nov 14, 2017

View reviewed changes

alloy requested a review from saolsen November 14, 2017 18:07

alloy commented Nov 14, 2017

View reviewed changes

alloy requested review from dblock and ashkan18 November 14, 2017 18:45

alloy commented Nov 14, 2017

View reviewed changes

alloy commented Nov 15, 2017

View reviewed changes

alloy mentioned this pull request Nov 15, 2017

Schema stitching for server to server access with more privileged access. #810

Open

ashkan18 reviewed Nov 15, 2017

View reviewed changes

alloy commented Nov 15, 2017

View reviewed changes

craigspaeth reviewed Nov 16, 2017

View reviewed changes

orta mentioned this pull request Nov 20, 2017

Removes convection representations from metaphysics in anticipation of stitching #819

Merged

alloy force-pushed the neo-metaphysics branch from 7057776 to a3c3c13 Compare November 22, 2017 15:01

alloy commented Nov 22, 2017

View reviewed changes

alloy added 13 commits November 23, 2017 00:07

[package] Add graphql-tools and related deps.

55be8e2

This is currently using my fork that has ardatan/graphql-tools#484

[Stitching] Get bare bones MP + Convection running.

36c5fe6

[Stitching] Add MP’s Artist type to Convection’s Submission type.

e6b24f5

[package] Remove unneeded dev dep.

28d846a

[Stitching] Enable type name conflict logging.

a3e29bc

[Stitching] Fetch Convection schema on startup using just XApp token.

ce0bb0c

[Stitching] Cleanup required env vars.

0369323

[Stitching] Spread client request headers into right location.

ee447c7

[Stitching] Don’t use authentication when requesting convection schema.

ce9f6bb

Fix linter warnings.

4b81220

[Stitching] Move module into lib

4781a3a

[tracer] Refactor code into its own module.

11b9a83

[Stitching] Pass all request IDs on to convection.

ab06a51

alloy force-pushed the neo-metaphysics branch from a3c3c13 to ab06a51 Compare November 22, 2017 23:07

alloy added 2 commits November 23, 2017 00:11

[package] Update graphql-tools to release that contains our fix.

83d16f4

[env] Disable warning about __id

c7f7776

alloy assigned orta and unassigned mzikherman Nov 22, 2017

damassi reviewed Nov 23, 2017

View reviewed changes

orta merged commit 5cd73c2 into master Nov 23, 2017

orta deleted the neo-metaphysics branch November 23, 2017 14:46

Neo metaphysics #809

Neo metaphysics #809

Conversation

alloy commented Nov 14, 2017 • edited Loading

TODO

alloy commented Nov 14, 2017

alloy commented Nov 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

craigspaeth commented Nov 16, 2017

orta commented Nov 16, 2017

stubailo commented Nov 18, 2017

alloy commented Nov 18, 2017

alloy commented Nov 22, 2017 • edited Loading

Choose a reason for hiding this comment

alloy commented Nov 22, 2017

damassi Nov 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

orta commented Nov 23, 2017

alloy commented Nov 14, 2017 •

edited

Loading

alloy commented Nov 22, 2017 •

edited

Loading

damassi Nov 23, 2017 •

edited

Loading