JavaScript iterators and iterables

1 May 2018 generators iterables itereators javascript

In this article I'm going to look at iterators, iterables and the iterator protocol and what this means for how we iterate over objects that implement this protocol.

What are iterables? 🔗

Iterablesare objects that enforce the so-called iterator protocol. What on earth does this mean? It means they have logic behind them - built in, or user-defined - that governs how the data within them is iterated.

Iterables can be used with patterns and syntax that require iterables, such as the for-of construct and spread syntax, both of which we'll see in action shortly.

So arrays are iterables, for example, as are more exotic, array-like objects such as sets and maps. Feeding them to an iteration process means stepping through the values within them. Enabling this process is a built-in iterator. Object literals are not iterable (presumably as they have no guaranteed order.)

Stringsare iterables, too, because we can step through the characters within them, just like we step through the elements in an array.

let arr = ['foo', 'bar'];
for (let prop in arr) console.log(prop); "foo" "bar"
let str = "hello!";
console.log(...str); //"h" "e" "l" "l" "o" "!"

Note that the string example will perform a single console.log() action, not one per character in the string. This is because we're spreading the string's characters into an array, so it's the equivalent of doing console.log('h', 'e', 'l', 'l', 'o', '!');

Sothere we see the iterators in action, but we don't see their actual working - that's somewhat hidden from us. So how do we know they have iterators at all? Well, if they didn't, the above code wouldn't work - if we feed a non-iterable to an iterator process we get an error:

let obj = {foo: 'bar'}
for (let prop of obj); //TypeError: obj is not iterable

Furthermore, we can check if an object is an iterable by checking for a [Symbol.iterator] function - this is where the function that defines the iterator lives.

[1, 2, 3][Symbol.iterator]; //function
({foo: 'bar'})[Symbol.iterator]; //undefined - object literals aren't iterables

We'll look more at what this weird Symbol.iterator thing is a little later - it's not key for understanding and using iterators and iterables.

As well as regular functions, generator functions can also be used to return iterators. Don't worry if you don't know what generator functions are yet - they're not key to understanding iterators. I'll talk about iterators and generator functions a little later.

What are iterators? 🔗

An iterator is a mechanism to allow code that consumes your data to iterate over it.

At its heart an iterator is an object that conforms to a set of rules and conventions known as the iterator protocol. Specifically:

The object must define a next() method
This next() method is responsible, during iteration, for returning an object
This object must define a value, the next value to be returned, if any, or done, a boolean denoting whether iteration is complete or not, or both

There's nothing special about such objects; they're just perfectly normal objects. The magic comes when they're hooked into (and returned by) that function defined at [Symbol.iterator].

Becauseof this, it's possible to use iterators completely independently of host objects and wrapper functions, as we'll see a little later.

Nowwe know what the internal mechanics of an iterator are, we can understand what hidden magic is going on when we use an iterable in a operation like for (x of y). The value property returned is used to fill the x placeholder, and the loop will run until it detects that done has been set to true.

So far we've used iterators via iterator processes like spread syntax and for-of, meaning their mechanics were hidden behind the scenes. But we can actually interface with iterators manually.

let str = 'dog';
let it = str[Symbol.iterator](); //note, no arguments

Let's take a moment to see what's happening there. We learned above that strings, like arrays and some other data types, are iterables. This means we know a string has a method at [String.iterator]. This gets called automatically any time we use built-in iterator processes such as for-of, but here we're doing the iteration ourselves, manually.

Remember we learned above that a condition of the iterator protocol is that the method return an object that defines a next() method. That object is stored in our it variable - in other words, it is our iterator.

And if it is our iterator, and we know valid iterators define a next() method, there's nothing stopping us calling it ourselves!

What we do we get back from it.next()? Recall that, under the iterator protocol, next() must return an object with a value property containing the next iteration value. And so, putting it all together:

let str = 'dog';
let it = str[Symbol.iterator]();
console.log(it.next().value); //"d"

OK that's pretty cool, but we're not really iterating in the true sense - we just grabbed the first step of iteration, i.e. the first letter. How do we manually iterate right through to the end and then stop, like a built-in iterator process like for-of would?

Recall that the object returned by next() can (and in the case of built-in iterators, always does) return a done property also, denoting whether iteration has finished or not or whether there's still more items/characters to grab. We can loop over the values, and look out for done changing to true to know when to quit (otherwise we'd have an endless loop).

let str = 'foo',
    it = str[Symbol.iterator](),
    next = it.next();
while(next.done !== false) {
    console.log(next.value);
    next = it.next();
} //"f" "o" "o"

If we continue to call next() after the iterator has declared itself finished, i.e. by setting done to true, we'll get an object back with value: undefined and done: true.

So this is all very nice, but it's hardly that interesting. What's the point of manually calling built-in iterators when we have patterns like for-of and spread syntax to do it for us behind the scenes?

User-defined iterators 🔗

Thereal fun comes when we define our own iterators - that is, we define explicitly how an iterable we create should behave when iterated over.

Let's say we wanted an inifinite number factory, starting from 1.

function nextNum() {
    let index = 1;
    return {
        next() {
            return {value: index++};
        }
    };
}
let it = nextNum();
console.log(it.next().value); //1
console.log(it.next().value); //2, etc

Remember I said we could use iterators independently of host objects like arrays? This is an example. But when you think about it, of course we can! In fact, forget about iterators; the example above is just plain old, simple JavaScript; we just defined a function that returns an object with a next() method which, every time it's called, returns the next number.

Iterators only become interesting and sort of magical when they're hooked onto host objects via that all-important [Symbol.iterator] function. So let's get back to that with another example.

RememberI said earlier that an iterator is principally a means of allowing data-consuming code to easily iterate over data. Well, what if that data isn't in a flat array?

Sure, it's fine and easy iterating over something like:

let vehicles = ['car', 'bike', 'plane'];

We can just use the array's forEach() method, or loop some other way. But what if we wanted to provide an off-the-shelf means of getting ALL animals from a nested data structure?

let animals = {
    canine: ['dog', 'wolf', 'dingo'],
    cat: ['cat', 'tiger', 'puma'],
    bird: ['ostrich', 'eagle', 'goose']
};

Suddenly that's not so easy, because our animals don't all live at the same level in our hierarchy - they're not siblings, in other words. Iterator to the rescue!

animals[Symbol.iterator] = () => {
    let all = Object.values(animals),
        i = 0;
    all = [].concat(...all);
    return {
        next() {
            return i < all.length ? {
                value: all[i++],
                done: false
            } : {
                done: true
            };
        }
    }
}

Great! Let's take it for a spin:

console.log(...animals); //"dog" "wolf" "dingo" ...
for (let anml of animals)
    console.log(anml); //"dog", "wolf", "dingo" ...

What is Symbol.iterator? 🔗

So we know that, where an object has a iterator, it should be returned by a special function wrapper that lives on its [Symbol.iterator] property. But what exactly is Symbol.iterator?

It's a special, reserved kind of symbol, which is used to store the wrapper function that returns the iterator.

If you haven't met symbols before, you can read more about them over here. In short, they're a new kind of primitive data type (alongside strings, booleans etc). They're used primarily as a safe means to store things on objects without hitting naming clashes.

We're not going to get deep into symbols in this article. But just know that Symbol.iterator is itself a symbol - one that lives as a static property on the Symbol namespace.

It's here that any iteration process, e.g. for-of will look for an iterator. Storing it anywhere else, it won't get found, and a "not iterable" error will get thrown.

let obj = {"dog": "cat"};
obj.myIteratorWrapper = function() { ... };
console.log(...obj); //error - not iterable

Iterators and generators 🔗

If you're unfamiliar with generators, you can check out my three-part guide to them.

Iterators and generators have a lot in common. Generator functions are both iterators and are themselves iterable, meaning that they can be assigned as an object's iterator, and can also themselves be iterated over.

Generators work great as iterators because they have a built-in way of spitting out values sequentially, whereas with regular user-defined iterators we have to do this manually and keep track of which value to output next. Generators do this with the special yield keyword.

let arr = [1, 3, 5];
arr[Symbol.iterator] = function* () {
    for (let i=0; i < this.length; i++) yield this[i];
}
console.log(...arr); //1 3 5

See how we didn't need to manually track the next value to spit out? We did this previously by incrementing a variable in next(). Generators, like all built-in iterators, abstract this away from us and handle this internally.

One interesting difference between regular iterators and generator iterators is that generators won't declare themselves done until a return statement is hit. We can see this if we call the iterator manually rather than via an iteration process like spread syntax like we just did above.

let it = arr[Symbol.iterator]();
it.next(); //{value: 1, done: false}
it.next(); //{value: 3, done: false}
it.next(); //{value: 5, done: false}

For more on the funky world of generators, check out my three-part guide to them.

Async iterators 🔗

Speaking of generators, they're deeply intertwined with iterators when it comes to async iterators.

So far we've been working with synchronous flows. But iterators can also be asynchronous. These work in conjunction with generators and also with the async/await combo that arrived in ECMAScript 2017.

There's a couple of other important differences about async iterators. Firstly, they must be declared under Symbol.asyncIterator, not Symbol.iterator. Secondly, it's not possible to use them with spread syntax, only with for await (... of ...).

Let's put it all together and create one.

let animals = {
    animals: ['hippo', 'rhino', 'duck'],
    async *[Symbol.asyncIterator]() {
        for (let animal of this.animals) yield animal;
    }
};

(async() => {
    for await (animal of animals) console.log(animal); //"hippo", "rhino", "duck"
})();

Note I'm not actually doing anything asynchronous here (asynchronous flows can also handle synchronous values) - I'm merely illustrating syntax.

So our function there is a generator function, hence the yield keywords that we met above. So async iterators and async generators are essentially the same topic. For a deeper look at async generators, I discuss them in the third part of my my three-part Guide to JavaScript Generators.

Summary 🔗

Iterators define logic governing how code should consume your data.

Iterators are just objects, but if they conform to the iterator protocol. (Synchronous) iterators define a next() method that returns an object announcing the next value and whether iteration is complete.

By hooking them onto objects by having them returned by a function declared on the object's [Symbol.iterator] property, they will automatically be used during any iteration process such as spread syntax and for-of.

Most JavaScript "collections" - arrays, sets, even (soon) HTML node collections - have built-in iterators. Object literals do not - unless you supply your own, custom iterator, as we did with the animals example.

Async iterators allow iterables to generate and return data asynchronously, when fed to the special for await (x of y) construct.

I hope you found this article useful!

Did I help you? Feel free to be amazing and buy me a coffee on Ko-fi!