JSON Data File Collections in Jekyll

Requirements

We have three similar projects under development at the W3C WAI. Each provides information on a set of items (course, authoring tools and evaluations tools) and users of our Jekyll SSG generated web ‘app’ wil be able to sort and filter the items.

A form is provided to submit new items that will undergo a review process before eventually being added to the website dataset for future access in the app.

So that’s only two pages, assuming no pagination.

Architecture

The need for client side manipulation of the item list view (filter and sort) suggests the JSON data be made available to the client.

It would be possible to keep all state in the DOM and manipulate that directly. It would also be possible to use the now common pattern of always rendering a view of the data in the client(made popular by React).

However, as this is a low complexity app other architectures might be suitable, for example using matching data and HTMLin the client.

As the content will ony rarely change and Jekyll is a SSG it makes sense to pre-generate as much as possible, both data and HTML. Then custom client side behaviour (ie javascript code) can be minimised.

A final observation is that as we are using a SSG, data and HTML can be pre render and directly included in pages, rather than fetched a runtime in the client.

The need for scalability and pagination might impact these decisions, but initially data sets are small. But this post highlight an option that might be non obvious in terms of approach and implementation in Jekyll.

Outline

The approach is as follows:

• each item has it’s own json data file in a folder with the others
• the JSON files have a top level object (hash) s owe can access various fields
• generate a single JSON structure of the items as an array
• Jekyll is used to general a single JSON structure of items
• html for the items can be created in as similar way
• the data and initial HTML are directly embedded into the HTML files
• we can use Jekyll includes to modularise the code

So at build time we would like to:

• access the files as a collection
• sort the collection on a specific item field - eg name
• generate the JSON array from the files
• iterate over the collection, to generate HTML for example

Creating the JSON structure

The Jekyll documentation on using data files is a little lacking and some experimentation is needed. But it turns out to be really easy to generate the JSON from all the files.

Tools

Liquid has the concept of arrays, but provides a very restricted set of filters (functions) that create and operate on them. It also has the [n] access operator, plus first and last methods. Jekyll adds a few more useful filters like push.

Jekyll also adds hashes (objects) and filters that work on them. But hashes are created for you in standard variables and there is no way to create or manipulate them. The . and [] operators are provide for getting values.

But we can do what we need to with these limited programming constructs.

Attempt

Given our JSON files live in the _data directory, an initial attempt might be as trivial as using the variable site.data to access them as a collection. Then we can try things like

or
null
and

But alas results are unexpected. With a little use of the inspect filter it turns out that:

• site.data is a hash keyed by data filename and with values that are file contents are being parsed as json
• the jsonify filter also shows a hash
• when iterating using for each item is converted into a 2-tuple of [key, value]
• sorting appears to work use the filename key

Solution

The solution is straight forward. As we don’t care about the filenames we can convert the hash to an array of the values. Then all the operation work as required.

# create an empty array
# note push is non mutating

Note there’s no “return” form includes so the output is any variables we set. itemValues in this case.

Now you can do the things we tried before:

[]

As a optimisation you can put the code in an include, passing the data folder (you can use subfolders of _data to organise things) and a sort key.

{% include sort-data-folder.liquid data=site.data sortKey="name" %}