Merge several paginated JSON using "tags" and access all as Collection?

[debatz]

Hey,

I've been using 11ty for a while with tremendous success but have found a bit of a snag recently trying to do something more ambitious (for me).

I am making a price comparison website. I scrape each website I monitor and have an API available with the results, per website (using simplescraper.io) Inside the "_data" folder, I have a .js file for each website's API.

Example:

const Cache = require("@11ty/eleventy-cache-assets");

module.exports = async function() {
	let url = "XXXXXX";

	/* This returns a promise */
	return await Cache(url, {
		duration: "4h", // save for 4 hours
		type: "json" // we’ll parse JSON for you
	});
};

Now, I make a folder "website1", where I paginate over website1's JSON from the API call.

Example:

---
tags: testtag
pagination:
    data: website1
    size: 1
    addAllPagesToCollections: true
---

{{ website1.title }}

11ty is nicely paginating the JSON, but I need to put everything inside a global collection. Reason is, I have to scrape by website, but I'm hoping to access all products with Nunjucks and sort by their price in a "lowest to highest" list for the consumer to choose the cheapest option.

Because my JS skills are close to none, I have tried to add the

tags: testtag

To access it on index.njk like this, for example:

{{ collections.all }}

However, this results in the following output:

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Could you please help me see what I'm doing wrong here? I realize there might be a more elegant solution but like I said, I'm self taught and just barely scraping by. Thank you for your help.

[pdehaan]

When using pagination, I like to use the alias key as well to make it easier for me to remember the current pagination item. https://www.11ty.dev/docs/pagination/#aliasing-to-a-different-variable

Also, doesn't look like you're setting explicit permalinks, so I'm not sure how many files are being written out and what the structure of that is. But from your output, it looks like you have 18 items in the all collection. You could probably also output the collections.testtag variable and see if that has 18 items, or if you have multiple different websites and each one has a few different pages/collections.

Another trick I like to use is a special filter for dumping objects out into JSON strings for easier debugging.

// .eleventy.js

eleventyConfig.addFilter("toJson", (value, pretty=2) => JSON.stringify(value, null, pretty));

That doesn't always play nice w/ collections and complex Eleventy data due to circular references, so another handy one is:

// .eleventy.js

const inspect = require("util").inspect;
//...
eleventyConfig.addFilter("inspect", value => inspect(value));

Then you should be able to do something like this to debug the contents of each of the objects in the collections in question (versus the less helpful "[object Object]"):

<pre>
{{ collections.testtag | inspect }}
</pre>

[pdehaan]

Possibly something like this will work:

<pre>
{{ collections.testtag | first | inspect | safe }}
</pre>

<h1>Posts</h1>
<ol>
{% for item in collections.testtag %}
  <li>{{ item.data.beer.title | safe }}</li>
	<!-- item.data: {{ item.data | keys | safe }} -->
	<!-- item.data.beer: {{ item.data.beer | keys | safe }} -->
{% endfor %}
</ol>

So we have

• item — Our collection item for the current loop iteration.
  • data — The frontmatter data object.
    • beer — The pagination alias.
      • title — The title property from the current pagination item.

It feels kind of weird since it's a collection built using a data file and pagination and an alias that we're trying to loop over, so it had an unexpected .beer property.

Where my custom keys filter looks like:

const inspect = require("util").inspect;
eleventyConfig.addFilter("inspect", value => inspect(value));
eleventyConfig.addFilter("keys", obj => JSON.stringify(Object.keys(obj).sort()));

Output

<li>Ranger</li>
<!-- item.data: ["beer","beers","beerz","collections","page","pagination","permalink","pkg","site","tags"] -->
<!-- item.data.beer: ["brewery","code","comment_count","created_at","date","description","has_many","hashtags","id","is_private","like_count","number","rating","title","updated_at"] -->

<li>Written in the Dust</li>
<!-- item.data: ["beer","beers","beerz","collections","page","pagination","permalink","pkg","site","tags"] -->
<!-- item.data.beer: ["brewery","code","comment_count","created_at","date","description","has_many","hashtags","id","is_private","like_count","number","rating","title","updated_at"] -->

...

[debatz]

It works!

I had an inkling it would be some kind of unexpected property but didn't know what to do with it. Tried several times but there is no way in hell I'd make those filters.

To be completely honest with you, your custom filters go a bit over my head.

Thank you very, very much for your help.

I'm going to keep working on this and will update this thread. Possibly should make a write-up somewhere as I haven't seen this particular need of tagging a paginated json anywhere quite in this way?

I'll come back and give a full post-mortem in case someone else finds it interesting in the future.

Thanks a lot @pdehaan I'll update shortly.

[pdehaan]

I was thinking of bundling those debugging filters into a custom 11ty plugin last night since I tend to recreate them often for debugging... Here's the rough README.md snippet I have for documentation so far (mostly just links to the relevant documentation):

1. inspect — Wrapper for Node's native util.inspect() method. Useful if the object you're debugging might have circular references, or non-JSON-stringify-able data.
2. json — Wrapper for JavaScript's JSON.stringify() method. This filter takes one optional argument which is a string or number value to use for indentation, if you want pretty printed JSON objects.
3. keys — Wrapper for JavaScript's Object.keys() method. This filter will also sort the returned array of key names for the specified object.

[debatz]

That'd be nice!

11ty's data manipulation is one of its greatests advantages, and it is likely that after this project I'll be converting a 8K+ SKUs Woocommerce ecommerce into a Prismic + 11ty + Snipcart/Ecwid/CommerceJS (under evaluation) stack.

This is to say that I'm going to try to ELI5 11ty as much for me as for others and make my own "public documentation for dummies" in spanish (seems like I'm the only one, or near the only one, interested in 11ty for the moment here) so I'd be very interested in seeing that plugin.

Thanks a lot and I'll come back to update this when I get it working.

[pdehaan]

Just pushed to GitHub and published to npm: https://www.npmjs.com/package/eleventy-plugin-debug