Apollo, the data stack for modern apps

Last week at Meteor Night, I gave a 20 minute talk about why we’re building Apollo, our new GraphQL-based data stack, what the architecture of an Apollo-based app will look like, and how much progress we’ve made since the original introduction of the idea. If you’ve been closely following our Building Apollo publication here on Medium, you have probably read about many of these things already, but this talk puts everything into one coherent story.

You can either watch the video or read the post, which includes the same context using text and images!

Here’s the same talk in blog post form, with all of the slides interspersed:

---

I’m here to tell you about Meteor’s next big project, after the Meteor framework itself and the recently-launched Galaxy cloud platform. It’s the data stack for modern apps, and hopefully you’ll have a good idea of what that means to us by the end of the talk!

Meteor does a lot of things. When it started 4 years ago, the JavaScript world was a completely different place. Client-side rendering was a pretty new concept, and writing JavaScript on the server with Node.js was just becoming the hot new thing. This meant that to achieve a seamlessly integrated development experience, Meteor had to come out of the box with a lot of custom tools to fill in all the things you need to build a complete application entirely in JavaScript.

But to me, the most exciting and magical part of the development experience in Meteor has always been the data system. These days, there are many great options for the view layer, and a variety of different JavaScript build tools and containers. The data system, on the other hand, is something that is only recently seeing really exciting competition in the JavaScript space, with new tools like Redux and Relay coming out of the React ecosystem.

What is it about Meteor’s data system that makes it feel so futuristic to use?

I think the main thing is that it makes the client-server divide something that can be traversed seamlessly at will. You can often write your code as if the data you need is already on the client.

So let’s go over some of the main properties of Meteor’s data system, from a developer productivity point of view:

Declarative: you can tell the server what information you are interested in via subscriptions, and the server figures out how to get it to the client in the most efficient manner.

Semantic: instead of blobs of JSON fetched from opaque endpoints, data in Meteor is organized into collections where each object has an ID. So you always have a way of referring to a specific object.

Stateful and caching: Meteor doesn’t load the same data twice, since both the client and server are aware of what has already been sent over the network. This means you can preload a lot of data and rely on the system to only send diffs, making prolonged interaction with an app more efficient.

So what if we could get all of these properties that make Meteor so nice to work with, but adoptable in every app, even existing applications you have already built, regardless of the platform?

And we’re talking any app. Perhaps your backend is written in Rails, or Express. Maybe it’s a Microsoft SQL server database, or Dynamo, or something else. On the client, we want to start with JavaScript, but quickly branch out to have clients for Swift and Java native apps.

What will let you achieve this kind of decoupling between the different components of the system?

Meteor’s data loading system reuses many concepts from MongoDB to make the client-side environment maximally similar to the server. The more we thought about decoupling the two, the more we realized that we needed a new query language that was optimized for the needs a client has when it’s loading data from an application server.

After thinking about a lot of different options, we determined that GraphQL was a perfect fit for these requirements.

It doesn’t assume anything about your backend, and lets you pass any arguments you need to tell the server how to assemble your data. It also lets you reduce the number of roundtrips you need to fetch the data for a particular part of your UI, in most cases to exactly one.

This was further validated by ideas in the Meteor community, including Arunoda’s ventures into teaching GraphQL, the alternative stack of Meatier, and widely used Meteor packages like publish-composite which reproduce some of the functionality inherent in GraphQL.

For those who aren’t familiar with GraphQL, here’s a super quick example of what one GraphQL query and response looks like:

My colleague’s post Intro to GraphQL has a great list of materials if you want to dive a bit deeper.

And there’s one more thing we really want to focus on in this new project. When you use Meteor today, there’s a huge “wow” factor when you get started, since you can often write dramatically less data management boilerplate than in other platforms. It’s been described as “magic”, because it can be really hard to understand what is actually going on under the hood.

For Apollo, we want to keep that “wow” factor and the ultra-productive experience, but at the same time giving you clarity about what exactly it’s doing. We want developers working with Apollo to be able to understand the ins and outs of their data stack.

So we’ve gone over a lot of the abstract ideas, what would an application built with Apollo actually look like?

The architecture

Here’s a 10,000 foot view of what an application built on Apollo will look like, from an architectural point of view. This is a diagram of a relatively complex application, with many clients and backends — if you’re building a small application from scratch it will be simpler. But we want to make sure we are thinking ahead to the larger apps out there.

If you’ve spent some time recently working on a large modern app built on top of REST APIs and microservices, this diagram might look pretty familiar. In a situation where you have multiple backends and microservices, the data you get almost never looks exactly like what you want for the frontend.

These days, most apps end up with an API server in the middle to translate the data between the microservice APIs and the frontend requirements. People are starting to call this the backend-for-frontend, or BFF. That’s exactly what Apollo gives you:

A principled, unified, and scalable API for all of your client apps, agnostic to what is going on with the backend.

To be clear, this isn’t a new concept — this is something the authors of GraphQL are talking about in every talk they give. Our goal is to make a simple and coherent data stack around these ideas.

Let’s dive down for a second and look at the lifecycle of a query in this system:

While you can get a lot of benefits of using GraphQL through simply sending POST requests to your server, it becomes much better once you have a smarter client that can avoid fetching data you already have, and give you other features that we’ll talk about below. Let’s compare what we’re thinking about with Apollo and GraphQL to Meteor’s current data stack:

OK, so now we’ve learned about the general architecture. So what are we actually building?

The components

First, let’s give credit where it’s due — there is already a vibrant ecosystem of GraphQL tools, and that’s one of the reasons we’re so excited to be working with this technology:

Here are links to the projects on the slide: GraphQL-JS, Graphene, Relay, and Sangria. We want to stand on the shoulders of giants, and make using GraphQL in your app a complete no-brainer.

This means building a GraphQL client and server that can be used to efficiently build a new app from the ground up, or slotted into an existing architecture to immediately start getting the benefits of GraphQL, including reduced roundtrips, self-documenting APIs, and abstraction between your backend services and your frontend data needs.

Apollo Server

We’ve recently started work on the Apollo Server. Read more about it from my colleague Jonas in his Medium post:The Apollo serverA GraphQL server toolkit focused on clarity and ease-of-usemedium.com

Apollo Client

We’ve been working on the Apollo Client for a few weeks now, and we’re getting ready to release an alpha build for testing. We expect to start using it in production very soon — perhaps within a week or two. It’s not quite ready for general use, but if building a simple and easy to adopt yet functional GraphQL client appeals to you, we’re actively looking for collaborators — just file an issue on the repository!apollostack/apollo-clientapollo-client – [Under construction] An easy to use and understand JavaScript GraphQL client with Relay’s most useful…github.com

---

If this is as exciting for you as it is for us, we want to hear from you. Perhaps this solves some of the problems you’ve seen in your architecture, in which case we’d love to talk to you in depth about what you’re seeing. Perhaps you’re hyped about the promise of GraphQL and you want to contribute to this new piece of technology, in which case we have a lot of tasks and ideas ready to pick up.

Please reach out or follow Apollo on Twitter, Medium, or GitHub!