Jekyll-based json-ld.org #461
Conversation
And yeah...it was as hard as it looks...
|
Nice work! This DRYs up the site quite a bit. 👍 |
|
Is the intent that we also update the hosting to run a build step for this site? Or was the target to host it on github? If on github, do they handle the .htaccess stuff we use like CORS, rewrite, mime types, etc? |
I don't believe they do, but Jekyl can be used to create a static site that can run on the existing platform, just not depending on PHP. Or, the host needs to be set up to setup/run the Jekyl templates. @BigBlueHat, what is your vision on this? Where do you have a branch with these updates available running? |
|
@davidlehn GitHub Pages was the target. It looks like the stuff the top-level curl -v https://wileylabs.github.io/json-ld.org/earl.jsonldResults include: Access-Control-Allow-Origin: *
Content-Type: application/ld+json@gkellogg currently the code uses absolute paths, so none of the styles nor JS load properly at https://wileylabs.github.io/json-ld.org/ However, I can make some changes that use a base_url set in the config to avoid that for demoing if you like. It looks exactly the same. Honest. 😉 Also, this is totally usable via jekyll locally and/or published as a static site where ever you'd like, but since the code's here and this place comes with free hosting, I made that the target. 😁 @davidlehn let me know if there are any other requirements I missed. |
|
There are some other .htaccess files:
The duplicated navbar code in lots of files should be using includes too. |
|
Front page has extra footer still in the main html file. |
|
Thanks @davidlehn. I'll look through that list. For now, that extra footer's removed and the site's demo-able on gh-pages here: |
|
@davidlehn I'd say with the exception of the test-suite ones (and maybe the schema.org one...) these are all doable with Jekyll's HTML-based redirects. GitHub Pages is unlikely (sadly) to let us hack Link headers and setup conneg properly. Perhaps the test-suite could stay hosted where it is, but at a URL like Thoughts? |
|
The schema.org stuff is, frankly, probably not too useful now, as JSON-LD has been so well adopted by that community. Indeed, the test use of redirection is pretty intense, but really just for the remote-doc-manifest where we go through some contortions. We could probably just put the files that are necessary on some other infrastructure, and move the tests and manifests themselves over. It's just a couple of tests where we need to set Link headers and do particular redirects. |
|
Other than expectations about the location of the tests which are built into some of the tests, my tests run pretty well off of your fork. |
|
The Specifications don't load off of your site, either through the UI or through URI substitution from the existing site. |
Forgot one...
|
@gkellogg all better. 😁 Forgot to use the |
|
@davidlehn I can create a new repo just containing the remote elements and .htaccess needed for the remote tests. This could presumably be set up similar to json-ld.org with a post/receive hook to live at something like, http://remote-tests.jsonld.org. With that, we can update the affected tests and manifests to point there, which would make the rest of the current jsonld.org independent of those requirements, so it could be hosted on gh-pages. We could also move the entire test-suite directory to a new repository and home at, say, http://tests.jsonld.org, and update the site to reference that. Alternatively, @BigBlueHat, it seems to me that Jekyl could support the existing infrastructure, so we just change the way files are managed on the existing jsonld.org. Personally, I think there's some advantage of getting off of DB infrastructure, as it's an ongoing burden for DB. cc/ @msporny |
It's not that much of a burden, we're happy to continue hosting (as our internal hosting clusters have gotten some upgrades as of late). That said, we're happy to push off to gh-pages as well. Whatever the community decides here is fine. If the community needs a VM, we're happy to provide one. If we can run everything off of gh-pages, that works as well. |
|
@gkellogg: Always a tradeoff when splitting up a repo. Might be nice to have regular and remote tests in their own repos but also nice to have one repo where PRs and commits change the spec and tests at the same time. Another issue with hosting on github is that people have requested an https site. *.github.io can default to https, but custom domains can't (correct me if I'm wrong). That might mean we need external hosting anyway. There are mixed content issues with the playground running on https, but we can perhaps use the a custom jsonld.js loader that would detect the issue and point people at the non-https playground as needed. For example, I use HTTPS Everywhere browser extension and @BigBlueHat's test site gets forced to https and the demos fail to fetch http://schema.org. The unreleased jsonld.js patches might help with this for some sites. |
|
@BigBlueHat do you feel this is ready to merge? If we pull the plug, what will get json-ld.org to run off of the statically built Jekyll pages? I had questions about how this would work above, has this been considered? What needs to happen to make this work on the DB infrastructure? Right now there's a post-receive hook that triggers util/git.php on every commit. Does something need to run to get the templates processed on install? Looking forward, some systems depend on a Travis-CI build passing before an update occurs (Heroku does this automatically, for example). We don't have such continuous integration tests now, but it might be nice to develop some to be sure basic things are running, such as the playground and test-suite, as part of the update process, but that's an issue for another day. I found some notes on Jekyll and Travis here. |
|
@gkellogg: A merge right now a would break everything. ;-) Task list:
|
|
Looks like the Sorry I can't do more on this one. Let me know if you need something, though! |
|
Sorry to be the party pooper but I'm not a big fan of this effort. What problem are we trying to solve here? I feel this actually complicates both development and deployment considerably. |
|
@lanthaler mainly I wanted to make it easier for everyone (including me) to work on the json-ld.org site. Removing (as much as possible) the PHP dependency for at least development and testing was the first target. Switching to a templating system means simpler future redesigns (which I also hope to contribute). Lastly, static hosting is just plain healthier for the future of the Web than "runs code someplace else every time you ask." I used Jekyll because it's the simplest to host here (if that were to be a thing). I'd be curious to hear about any specific complications wrt to development that this presents. The deployment cumbersomeness mostly seems related to the testing infrastructure also being mixed into the publishing space. The deployment "ease" wasn't really a target for this particular PR as I assume we can automate any "pain" away there. 😃 |
|
Well, there wasn't much PHP in the first place. The only thing PHP was used for was to get a list of previous spec drafts (and requirements, but there are only 2 of them from 2011). The rest was pure static HTML.
I was able to pull the repo, make changes, check how they look in the browser and push them back on any machine. No need to download and install a whole software stack. |
|
@lanthaler the use of PHP (for templating) being so minimal is precisely why it seemed prudent to take it out. 😃 It seems more efficient (and more sustainable) to generate those lists of files only when (or technically after) that list has changed--which is what static generation brings to the table. My aim was actually to minimize the amount of code/stuff needed to both host and develop this stuff. I realize Jekyll's a dependency for full generation now, but it's generally and easier installation than PHP and often a pretty commonly installed thing for folks who use GitHub and/or contribute to docs/sites. In the end, my aim is to provide more contributions to this site and the tooling around it. If I've barked up the wrong tree, apologies. Redirection is welcome. |
|
As the initial author of the PHP stuff for the site, I can assure everyone that it was a rushed copy-paste job from the Web Payments work with little thought put into maintenance, etc. It was assumed that editors of the site would git checkout and then serve the site up locally via apache+php for testing. The current change DOES make it more difficult for /that particular class of people/ and doing things like refreshing some of the auto-generated stuff (minutes, etc.). That said, if the use of Jekyll moves us into a position where more people can help out with the site, and we can still auto-publish then +1 to that and let's get on with it. The most important thing isn't the framework we're using... it's that someone has stepped forward, volunteered to work on the site, and is getting stuff done. We shouldn't get in the way of that unless it's clear that this is a step backwards. It seems like multiple steps forward to me, and I look forward to copying whatever @BigBlueHat does for the https://w3c-dvcg.github.io/ work. :P |
|
Is that Gemfile.lock supposed to be in source control? I understand what lock files are for but it's got "x64-mingw32" things in it which get rewritten on install on a linux host. Seems like that would cause diff churn between developers and doesn't really do a "lock". Another issue with hosting purely on github itself is the minutes. That repo was split off to avoid the huge multi gig repo size for regular checkouts. It's just checked out inside the main repo for hosting purposes. I wonder if anyone actually reads that page though. It's been broken for years and missing the first half year of meetings. Now fixed. |
|
Yeah. I'm seeing several issues around GitHub that say *not* to commit the Gemfile.lock. Sorry about the oversight.
The minutes repo could be made (via wget most easily) into a static site, stored on the gh-pages branch of that repo, and hosted at `http://minutes.json-ld.com/`. Actually, it looks like GH Pages supports submodules--which could also work once the site is static-a-fied: https://help.github.com/articles/using-submodules-with-pages/
Options. 😺
|
|
Gemfile.lock should be in the repo for deploying an application, this keeps it from failing randomly due to package failrues. When developing a gem, Thule should not be kept, so that a package maintainer becomes aware of issues in the wild. It is typical for a gemfile to be locked on one system, and used on another; this should not be an issue. |
|
Thanks for clarifying, @gkellogg! Also, @davidlehn it looks like it would be trivial to setup |
|
I downloaded and setup on my local machine. There were a couple of reports of missing resources: But, trying to run the playground failed miserably, mostly due to bad CDN requests (perhaps related to running locally), and code mirror not being installed. I presume all that should be necessary is Everything else looked great. |
|
@BigBlueHat I created a local branch jekyll which includes your commits, some conflict differences and a small change to remove prettify.css, which doesn't exist on the site (and seemingly, never did). There's interest in moving forward on this, so you could either update this PR, or we can simply close this one and open another, but I'd hate to miss the context from this PR. |
|
@msporny @davidlehn @dlongley and I had a quick call to discuss how we might proceed forward on this:
This should leave the site looking as it does now, but with pieces implemented in different places. Alternatively, we could eliminate needing to run a different server for /test-suite/tests if we give up on needing .htaccess support for the remote-doc tests. This would require writing a tool that developers would run to do remote-doc testing locally; it's not uncommon for groups to require something like this. All these solutions depend on using a proxy server such as Traffik to be able to route requests and handle TLS, which remains a single point of failure. |
|
I pushed the jekyll branch to gh-pages, and the site is now running at https://json-ld.github.io/json-ld.org using Jekyll. |
|
As a possible alternative to GH pages and the header / redirect issues it has, we use netlify which gives access to these sorts of things, plus manages HTTPS certificates and so forth. The price is also good: Free for non commercial. |
|
I'm OK with any static hosting--doesn't have to be gh-pages. The objective was simply to move things toward better uptime, easier contribution, and less hosting overhead (for all!). Pick whichever fast-hard-drive-with-URL-attachment service fits your needs best. 😁 |
|
@BigBlueHat sure, that would be great. We do need to deal with redirects in tests, but we may move that for the WG, anyway; in any case will need some redirect, as it's referenced from the 1.0 specs. Also, can't break json-ld.org/messages from another repo. |
|
Gotcha. Let's make list of such things and then pick some git-driven static hosting. There are lots of options these days! |
|
FWIW, I've been digging into improving more of the content on this site working toward building more interest and involvement in both the CG and the WG. Consequently, I've bumped into wanting/hoping we can get onto static-hosting which can be kept in sync with If this is still of interest, I'm happy to resolve the conflicts listed, and get this PR back on the rails. Cheers! |
|
Now that the test suite has been moved under the WG repos, the need for HTTP redirects has gone. I think this would make it easier to transition to Jekyll. |
|
Switched this to a "draft" to prevent an accidental merge. However, I would very much like to revisit this static site idea. @davidlehn does your list from a couple year's back still apply? |
Here 'tis. 😁
I've switched the PHP-based directory parsing to a couple Liquid template includes (...which are a bit insane to understand, but they do work).
The only PHP left (that I could find) is in the
utils/directory and it's a command line script, afaict.Consequently, this should be ready to go--pending approval. 😉
Hope it helps!
🎩