2022 W04


Hello and welcome to my first edition of weeknotes - something that I think might be a nice addition to this blog. Shout out to Simon Willison’s blog for the inspiration and consistent source of excellent info.

Neo4J GraphQL Subscriptions

The neo4j/graphql library is great for getting an API up and running and performing some interesting graph queries. However, for an application that requires notifications of changes, it suffers from a lack of support for subscriptions. This was a big problem for a proof-of-concept chat application that I’m exploring at the moment.

Within neo4j there is support for database changes to trigger notifications pushed to a kafka topic, however, this approach was massively overkill for my proof-of-concept application. I wanted a way to subscribe to changes without having to support that with lots of infrastructure. Along the way I encountered a number of problems and new topics that I had to learn. The rest of this page is a summary of what I went through in case it helps someone again later. The final code can be found on GitHub.

Limitations

The only way to trigger updates at the database level is to use the neo4j streams kafka integration to publish data to a kafka topic. If you’re like me and:

  1. have never worked with kafka
  2. don’t think it’s necessary for your little use case
  3. don’t want to support the infrastructure it entails

then you’re a bit stuck for database-level tiggers.

This leaves us with dealing with the events at the application level. This isn’t something I particularly wanted to do either but after a little reading it seemed like something that could be achieved by combining a few techniques.

Landscape

Before this goes any further, I want to set the scene. Doing this will help our understanding of how each piece plays its part later.

We’re focusing on a GraphQL server, implemented with Apollo. The server is largely generated for us by using the neo4j/graphql library. We connect to a free-tier neo4j cluster provided kindly by neo4j aura and send data back and forth.

Apollo does the heavy-lifting for the server, neo4j does the heavy lifting for the data and the real value-add is our implementation of the data structure defined using the neo4j/graphql library. They have a good getting started guide so I won’t go into how that works.

Goals

Before we get into the code, it’s helpful to lay out the various bits and pieces of our approach so we know what we’re aiming for.

Subscription GraphQL schema

We must have the schema in our type definitions for our server and run them through the neo4j/graphql package. This is the same as writing a subscription for any other GraphQL server and the package will just ignore them for now when generating code for us.

Subscription transport with graphql-ws

In order to enable subscriptions in Apollo, you need to follow a bunch of steps listed in their documentation. Only after getting to the end of those steps do you find that the library that underpins all of it, subscriptions-transport-ws, is not actively maintained. Bummer.

Instead of that transport, we can use graphql-ws. We’ll have to implement support for this transport in both the client and the server later. This isn’t particularly well documented but I aim to cover parts of that here.

Listening for events with neo4j/graphql

For the subscription transport to be useful, it needs to send some data. Subscriptions can’t be generated by neo4j/graphql so we’ll have to implement some resolvers for the subscriptions.

Our resolvers need to push data over the transport as events happen in our application so we’ll also need a way to trigger events. In Apollo, we can intercept operations using plugins. We’ll write a plugin that intercepts successful mutations that we want to listen for and triggers an event in response.

Recap

  1. Write our subscription type definitions
  2. Implement transport protocol for subscriptions
  3. Trigger events in response to certain operations
  4. Push data over the subscription transport in response to events

Step 1 I shall leave to the reader as I’m already assuming you have knowledge beyond that point.

Implement transport protocol

Before we have our transport set up, we have a GraphQL server that looks something like this:

const app = express();
const neoSchema = new Neo4jGraphQL({ typeDefs, driver });
const apolloServer = new ApolloServer({
  schema: neoSchema.schema,
});
await apolloServer.start();
apolloServer.applyMiddleware({ app });
await new Promise(resolve => {
  app.listen(4000, () => resolve(undefined))
})
console.log(`🚀 Server ready at http://localhost:4000${apolloServer.graphqlPath}`);

We’re setting up an express application and applying the Apollo server as middleware on that application. Note that here I’ve already converted to use apollo-server-express and we’ve set up our server to use the generated typeDefs and driver for our neo4j instance.

In order to serve subscriptions over the WebSocket protocol, we need to set up another server and route connections to it. Our code becomes a little more complicated:

const app = express();
const neoSchema = new Neo4jGraphQL({ typeDefs, driver });
const apolloServer = new ApolloServer({
  schema: neoSchema.schema,
});
await apolloServer.start();
apolloServer.applyMiddleware({ app });
await new Promise(resolve => {
  const server = app.listen(4000, () => {
    const wsServer = new WebSocketServer({
      server,
      path: '/graphql',
    });
    useServer(
	  {
        schema: neoSchema.schema,
      },
      context: (ctx, _msg, _args) => ctx,
    }, wsServer)
  })
  resolve(undefined);
})
console.log(`🚀 Server ready at http://localhost:4000${apolloServer.graphqlPath}`);

We set up another server that serves requests over the WebSocket protocol and we initialise it using the useServer function provided by graphql-ws and the same schema that we use for the rest of the GraphQL sever.

The importance of the line context: (ctx, _msg, _args) => ctx cannot be overstated here. This is essential for our upcoming integration with the Neo4jGraphQL instance that we created further above. It’s a missing piece that isn’t documented anywhere (but here!)

Trigger events in response to operations

There’s no perfect way to do this that I found. In this approach we get inside the request/response lifecycle in our Apollo server using a custom plugin.

We’re interested in responses that completed successfully, so we want to step in right before the response is sent and emit an event. It helps to have an example to talk about for this section; I was specifically working on monitoring for when a new message was created in my chat application so we’ll continue to use this scenario as our example as it seems simple enough for everyone to understand straight away.

First we need to create a new event emitter that will act as the focal point for our events:

const eventEmitter = new EventEmitter()

Then, we amend our apollo server to point to our new custom plugin:

const apolloServer = new ApolloServer({
  schema: neoSchema.schema,
  plugins: [
    operationEventPlugin(eventEmitter, 'CreateMessage', 'CreateMessage'),
  ],
});

Let’s have a look at that operationEventPlugin. This is (not terribly great) code that I wrote to see if this was possible.

type OperationEventPlugin = (eventEmitter: EventEmitter, operationName: string, eventName: string) => PluginDefinition;

const operationEventPlugin: OperationEventPlugin = (eventEmitter, operationName, eventName) => ({
  async requestDidStart(_initialRequestContext) {
    return {
      async willSendResponse(requestContext) {
        if (requestContext.operationName != operationName) return;
	    if (requestContext.errors?.length) {
          console.error(requestContext.errors);
          return
        }
	    console.debug(`${operationName} finished with no errors. Emitting event.`);
	    eventEmitter.emit(eventName, requestContext.response.data);
      }
    }
  }
});

This little plugin listens for a particular operation name that we choose, in our case CreateMessage. During the willSendResponse phase, we step in and emit an event with the data that will be sent back in the original operation.

Push data over the subscription transport in response to events

The last piece of the puzzle is to send data back to the client when an event occurs. We can adapt our graphql-ws server and create a resolver for the subscription that we defined earlier. In this example, the subscription I defined was messages.

useServer({
  schema: neoSchema.schema,
  roots: {
    subscription: {
      messages: async function*() {
        for await (const event of on(eventEmitter, 'CreateMessage')) {
          const [e] = event;
          yield e.createMessages;
        }
      }
    }
  },
  context: (ctx, _msg, _args) => ctx,
}, wsServer)

That’s it!

Thoughts

There are a lot of holes in this approach. Many of them are clear to me and seem to have obvious solutions that I’ll try out soon, some of them seem a bit unclear and I’m 100% there are a few that I’ve not thought about yet.

Either way, this seemed like a novel approach to neo4j subscriptions that helped me bridge a gap with a much easier setup than the currently existing options.

Relationship-Centered Care

I went to a talk by one of my colleagues, Luke Tanner, this week and he had some fascinating points on why relationship-centered care is so different to the current dominant form of care that we get in the UK at the moment.

There are too many points to go into here and I’m not sure if I’ve fullt digested them yet but I found this image extremely helpful as a summary so I’ll repeat it here. All credit to Luke for this.

A summary of the differences between Institution, Personal and Relationship-centered care models

© neverstew 2022
GitHub logo