promises everywhere, at the same time

With nodejs, you almost always need to interact with the filesystem, HTTP resources or databases. Since we’re past the age of callback hell, we’ve introduced the next best problem, promise hell.

Typically, I’ve found you end up using these promises in two different ways, with the first way being the fairly manual and hardcoded way of dealing with them:

async function dealingWithData() {
  const resource = await this.db.getSomeResource()
  const file = await fs.readFileAsync(thatPath)
  const result = await fetch(thatUrl)
  const json = await result.json()

  // Do stuff
}

I like the above approach to dealing with promises, you have control and it’s fairly elegant (compared to older solutions), but as mentioned, it’s incredibly static.

The second usecase is when you need to deal with multiple promises in a more dynamic fashion.

Lets move on to an example I had to deal with recently, where we had to read a bunch of files from a given folder, parse them and save some information to a json file. The immediate problem with the above result is that we do not know how many files we’ll be dealing with, we’ll simply be dealing with an array of files.

It started out looking somewhat similar to this (a somewhat naive approach):

async function readFilesFromSystem(directoryPath) {
    const files = glob.sync(`${directoryPath}/**/*.js`);
    const parsedFiles = files.map(async filePath => {
        const file = await fs.readFileAsync(filePath);
        return {
            property1: file.property1,
            ...
        }
    });
}

So the initial problem is that we never received the files. parsedFiles was just each of the promises returned from readFileAsync. So how do we resolve all the of the promises after we’ve gathered them? Turns out the native Promise implementation has a good method for us; .all().

async function readFilesFromSystem(directoryPath) {
    const files = glob.sync(`${directoryPath}/**/*.js`);
    const promises = files.map(async filePath => {
        const file = await fs.readFileAsync(filePath);
        return {
            property1: file.property1,
            ...
        }
    });

    const parsedFiles = await Promise.all(promises);
    console.log(parsedFiles);
    // [{ file1 }, { file2 }]
}

Success? Well yes, in theory. We are now getting all of the file related data. Worth noting (and I found this a bit confusing to start with) is that even though your .map() returns the objects, that’s only the actual return value of the resolving Promise. This coupled with Promise.all() means each resolved value of each promised will get bunched out into an array.

Kind of confusing, but once you get used to it, it’s pretty straight forward. With one given drawback though; the order of promises resolving varies, a lot. In the above example it might not make a difference, because we end up with the file-related information regardless. But lets say you’re interested in resolving each promise in sequential order, you could do something like the example below.

async function readFilesFromSystem(directoryPath) {
    const files = glob.sync(`${directoryPath}/**/*.js`);
    const parsedFiles = [];
    for (const filePath of files) {
        const file = await fs.readFileAsync(filePath)
        parsedFiles.push({
            property1: file.property1,
            ...
        });
    }

    console.log(parsedFiles);
    // [{ file1 }, { file2 }]
}

Using the for of iterator, we can walk through the list of filePaths sequentially. I’d also like to point out that this is not in any way the only way of resolving them sequentially, there are also ways of using reduce, or introducing libraries like bluebird.

That’s the short story of how to work with resolving promises in nodejs.

Bonus round!

Sometimes you end up talking to APIs, where throttling is sometimes an issue, I’ve found the use of bluebird to be very useful, because normally Promise.all() does not deal with concurrency, it will just attempt to resolve promises as soon as possible.

const Promise = require('bluebird')

async function talkToAPI(urls) {
  await Promise.map(
    urls,
    async (url) => {
      const res = await fetch(url)
      const json = await res.json()
      // Do something with json
    },
    { concurrency: 1 }
  )
}

The above options object allows us to pass a concurrency value that states that bluebird will only resolve a new promise once the previous has been resolved. Do note that it instead of .all() uses .map(), and that we’re required to monkey-patch the Promise with bluebird.