Inlining store data in Ember.js

12 June 2018

Thanks to Dave Laird and Kris Khaira for reviewing this post.

In an app I'm working on, we always load some data from the back-end that very rarely changes. Since the data is needed on almost all pages, this is done in the application route:

 // app/routes/application.js
 export default Route.extend({
   beforeModel() {
     return RSVP.hash({
       currencies: this.store.findAll('currency'),
       destinations: this.store.findAll('destination'),
     });
   }
 })

In this case, the long-lived data are currencies and destinations but I suspect lots of other apps have data of this type: categories for an e-commerce app, priorities and labels for an issue tracker and instruments for a music catalog app are all such examples.

So every time our Ember app booted, two ajax requests were sent to the backend which also blocked rendering as that data was crucial for lots of components of the page.

The fact that the data was quasi-static gave me an idea: what if we could just inline it into the index.html and load it into the store from there?

Turns out the idea was legit and feasible. In the following post I want to describe how this can be achieved.

Fetching data during build

The first step is to write a script that fetches the data from the back-end and stores it in a file where the build process can pick it up.

The easiest thing is to write a node.js script that can be called from your Ember.js project that is used to build the Ember.js app.

To fetch data from the back-end in node, I used a simple http library, axios (but there is a great deal of alternatives to choose from):

$ yarn add --dev axios

 // support/scripts/fetch-data-for-inlining.js
 /* eslint-env node */
 'use strict';
 
 const axios = require('axios');
 const fs = require('fs');
 
 function fetchDestinations() {
   return axios.all([
    axios.get('https://your-api.com/destinations'),
    axios.get('https://your-api.com/currencies'),
  ])
}

fetchDestinations()
  .then(axios.spread((destinations, currencies) => {
    let mergedData = Object.assign(destinations.data, currencies.data);
    fs.writeFileSync('support/data/store-data.json', JSON.stringify(mergedData), 'utf-8');
  }));

After fetching the pieces of data, we merge them and store the resulting serialized format in a file.

Next, we can read from there and inline it into the "skeleton" of our web app, its index.html.

Inlining serialized data into a meta tag

Ember apps have the contents of their configuration (config/environment.js) serialized in a meta tag so I peeked under the hood of Ember CLI to find out how that's done as I wanted to do likewise for store data. Dave coined the term "index stuffing" for this technique and I think it's a great expression.

I've found that it uses an add-on called ember-cli-inline-content so I added it to the project:

$ yarn add --dev ember-cli-inline-content

Modifying the build pipeline usually happens in ember-cli-build.js so that's where I added the code that populates the meta tag with the desired store data:

 // ember-cli-build.js
 'use strict';
 
 const EmberApp = require('ember-cli/lib/broccoli/ember-app');
 const fs = require('fs');
 
 function readFile(file) {
   try {
     return escape(fs.readFileSync(file));
  } catch(e) {
    if (e.code === 'ENOENT') {
      return '{}';
    }
    throw(e);
  }
}

module.exports = function(defaults) {
  let app = new EmberApp(defaults, {
    inlineContent: {
      'store-data': {
        content: `<meta name='store-data'
          content="${readFile('support/data/store-data.json')}" />`,
      }
    },
    (...)
  });
}

We read the contents of the file previously written by the script and add it as the content of the store-data meta tag. As this is an optimization and we want to provide a fallback, it's important to handle the case when the file is not there. In that case, we just write an empty object so the code loading the serialized records into the store can be kept simple and assume the tag has valid JSON data.

The inlineContent snippet defines a slot called store-data so that still has to be written in the index.html for this to work. The magic word to define these slots is content-for:

 // app/index.html
 <!DOCTYPE html>
 <html>
   <head>
     (...)
     {{content-for 'store-data'}}
   </head>
   (...)
 </html>

Done, our index is now "stuffed", we can now move on to loading the serialized data.

Loading the data into the store

All right, so we have the JSON stringified data in the meta tag. We should now load it into the store so that when the app boots up, it's already there and we don't need to query the back-end. A good place to do this is an instance initializer:

 // app/instance-iniitalizers/load-store-data.js
 export function initialize(appInstance) {
   let metaTag = document.querySelector('meta[name=store-data]');
   if (metaTag) {
     let store = appInstance.lookup('service:store');
     let storeContent = JSON.parse(unescape(metaTag.content));
     store.pushPayload(storeContent);
   }
 }

export default {
  initialize
};

Honestly, I was a bit surprised that simply using store.pushPayload works but it does.

Falling back to querying the backend when pre-loaded data is missing

My first thought after I had that working was to just get rid of the ajax calls in beforeModel but that would've been rushed. As we'd already said, we want to fall back to querying the data from the back-end "live", if we find that we need it. The testing environment is a great example where we probably don't want this pre-loading behavior (in our app, we use a mock API, using Mirage fixtures) but there can be other reasons for the data not to be there.

So we need to check if we have data for each type and then still consult the API if we don't:

 // app/routes/application.js
 export default Route.extend({
   beforeModel() {
     let preload = {};
     let currencies = this.store.peekAll('currency');
     if (currencies.get('length') === 0) {
       preload.currencies = this.store.findAll('currency');
     }
     let destinations = this.store.peekAll('destination');
    if (destinations.get('length') === 0) {
      preload.destinations = this.store.findAll('destination');
    }
    return RSVP.hash(preload);
  },
});

store.peekAll is a great method and comes handy here. It checks the contents of the store for the given type and doesn't ever trigger a request to the backend. So we assume that if there's at least one record for each type in the store, it means that we had pre-loaded data and thus don't fire an API request.

As you can see from the following Network tab screenshot, the ajax request to fetch destinations is no longer fired:

And that's because the store already has them:

Making it part of the build process

To make use of this in production, where improving performance (and reducing the number of API requests) pays the most dividends, the process has to be integrated into the production build.

Let's first define our data fetching script in package.json:

 // package.json
 {
   (...)
   "scripts": {
     "start": "ember serve",
     "test": "ember exam",
     "build": "ember build",
     "fetch-store-data": "node support/scripts/fetch-data-for-inlining.js"
   },
  (...)
}

Having done that, calling the script during the build is a simple matter of running yarn fetch-store-data (or npm run fetch-store-data) which should be easy whichever CI/build platform you use.

The other pieces are encapsulated in the app: ember-cli-build.js will pick it up during ember build -e production and write the meta tag, the initializer reads from the meta tag and loads the data into the store. The app is then happy not having to make a costly request to the back-end for data that almost never changes. Or does it?

Caching considerations

Writing the serialized response for API requests into the index.html is equivalent to caching the response of those API requests so we have to ask ourselves the question: when do we have to invalidate the cache and how can we do it?

Assuming the index.html is not allowed to be cached, users get a fresh copy of the data when a new index.html is built – every time we make and deploy a build to production (users will then still have to reload the app to get the new index.html).

That might be fine in certain cases (as it was for us with destinations), but in others, more up-to-date data might be needed. As it turned out, we update currency exchange rates daily on the backend, so we ended up not inlining currencies and querying them from the API on the fly.

Since the implementation was flexible enough, this was just a matter of not fetching currencies in the fetch-data-for-inlining.js script, a simple, quick change.

A hybrid solution is also possible – inline store data initially but update it in the background or when it's needed. Currencies could be fetched from the backend when the user opens the shopping cart, instruments can be re-loaded when a dropdown on the artist form is activated.

I plan to write more about this hybrid approach in a sequel to this post.

Oh, and Vidi, the company I wrote this for, are hiring!