Versatile Internet

Blog

Introducing Pagespace: Manageable Websites with Node.js

Like many web developers I’m often asked to build various custom yet manageable websites. Normally for people I know who want online portfolios or who have small businesses or organizations. The de facto technology choice here is usually Wordpress. I’ve built Wordpress sites before and never really enjoyed the experience. It’s based on a technology stack I wouldn’t normally choose to use and customizations often seemed to warrant hacky solutions. I wanted a solution based on my preferred web technology: Node.JS and where customizations could be elegantly made.

This seemed like a great opportunity to use an existing open-source solution, but I had difficulty finding something that was being actively developed and close to the requirements and goals of what I had in mind. Mostly in terms of overall simplicity and the website management/editing user experience. Perhaps naively, I started to work on a new project.

Now, what I started to create was fairly simple. No big deal. In-line editing (a bit like instant.cm) and some way to manage the page structure of a web site. Naturally, this side-project snowballed away and two years later I think I have something quite significant that far exceeds these simple requirements. Something significant that I’m aware is also somewhat of a cliché. I see tweets like this and, with a small dose of imposter syndrome, I’m almost embarrassed of my work. This is probably why I’ve been reluctant to make any effort to share it so far.

However, I was kind of aware of this from day one. I said to myself if this solves my own problems and if I learn something along the way then the project is a success. It’s definitely has done that. Pagespace now runs several websites that I have personally created and real people are happy using it. The platform and its deployment use many technologies (NodeJS, ES6, Express, AngularJS, MongoDB, Nginx) which I’ve either improved my knowledge of or newly learnt.

So what is Pagespace exactly? I like to call Pagespace a “website management system” rather than a CMS. You could loosely describe it as a CMS, but its more geared towards brochure style/portfolio websites that don’t need sophisticated content management capabilities. The mantra of Pagespace is to provide developers with a framework where they can easily create websites, which in turn are easily manageable by users with little training or technical knowledge.

For developers:

Developers first integrate Pagespace into an Express app as Express middleware. This will intercept the predefined routes of Pagespace, like its admin dashboard or API, and its managed routes. i.e. the paths of the pages being managed. Unmatched routes will be passed along the middleware chain. This makes it easy to deploy the managed website pages alongside any other backend services within the same app.

Next, you create one or more templates to render pages. Templates use the Handlebars templating engine. They utilize Handlebars partials to declare and include the manageable regions of the page. Pagespace templates are server-side rendered and unopinionated about client-side technologies. Therefore, you may use whatever technology (React, Angular etc) you prefer to enrich managed content or just run alongside it. The custom templates and their client-side assets (CSS, JS, images etc) are termed collectively as a “theme”.

<body class="{{template.name}}">
    <header>
        <p><a href="/"><i>Versatile</i> Internet</a></p>
    </header>
    <nav>
        {{#Nav}} {{> Nav }} {{/Nav}}
    </nav>
    <div class="wrap">
        <aside>
            {{#Aside}} {{> Aside }} {{/Aside}}
        </aside>
        <main>
            <h1>{{page.name}}</h1>
            {{#if template.showDate}}
                <p class="date">{{formatDate page.publishedAt }}</p>
            {{/if}}
            {{#Main}} {{> Main }} {{/Main}}
        </main>
    </div>
    <footer>
        {{#Footer}} {{> Footer }} {{/Footer}}
    </footer>
</body>

A snippet of this page’s Handlebars template (which is running on Pagespace)

All you need now is to fire up a MongoDB database and the website is good to go.

For managers:

For website managers, Pagespace provides the ability to manage the pages and structure of a web site, to edit the content on each page and to upload and organize media. As a user of the management dashboard you just see three top-level sections: Pages, Publish and Media. The UX emphasis is on providing an interface that’s highly usable by non-developers.

Screenshot of the Pagespace management dashboard

Page structure of this website via the management dashboard

Pagespace also has a plugin system for controlling the way each piece of content on a page is managed and rendered. I’ve written the following plugins so far:

  • Web copy: WYSIWG style editing.
  • HTML: Inject HTML directly. Useful for things like dropping in the embed code of a Youtube video.
  • Markdown: Wrtie content as markdown. This is what this post is using.
  • Posts: This can create a blog roll, by aggregating content from a collection of pages.
  • Nav: Include navigation menus based on the site structure
  • Gallery: Display thumbnails and lightboxes for a selection of managed images.

Plugins are distributed and installed as NPM modules.

What next for Pagespace?

Now I must conclude if I have just completed a “right-of-passage” or began something with some potential that I should continue to work on. Or both.

Up to this point I haven’t been collaborative in my approach to building Pagespace. I have to get out of my bubble and get wider feedback and support. Personally, Pagespace solves my problems, but are these universal? Does this help you? Do you like it?

I would be delighted if Pagespace can go on to establish an open-source community and grow in its usage. Please get in touch if you have something to contribute. Issues, ideas or code. Or you are interested in building your own websites with Pagespace. Please email, leave a comment or contribute via the Github issues page.

Try it

If you would like to try Pagespace, there are several options:

  • Run the Docker image. For the purpose of demoing, this image contains Node.JS, MongoDB and the demo Pagespace/Express app.
  • Install it via NPM and add it to your existing Express application.
  • Generate a new application from scratch using the Slush generator
  • Clone the Github repo then run npm install && npm start.

There is also a dedicated documentation site for both developers and users.

Checkout the Github repo for an overview and the source code.

Wrapping Redux Stores in ES6 Classes

This is a quick post about a simple pattern for working with Redux stores.

When writing the Gallery plugin for Pagespace a while ago, I used it as an excuse to build something with React + Redux, where I used a simple pattern that wraps the application’s Redux store in an ES6 class.

Conceptually, I think of a React/Redux application as kind of implementing the Model View Presenter pattern, whereby each React component is a mini View-Presenter, and the Redux store is a shared application Model. A model which utilizes the well documented features of Redux, but albeit a model. I consider bi-directional communication a feature of modern MVC/MVVM frameworks not the MVC/MVP pattern itself which far predates Knockout and Angular.

Wrapping a Redux store in a class adds a simple layer of abstraction which turns the store into a more traditional model object. This results in a clean API surface for interacting with the store and provides a place to easily add asynchronous actions; without the need for extra Redux middleware like Redux Thunk.

I’ll show some code and then briefly describe the benefits of this approach. This code includes the GalleryModel class which extends the simple ReduxWrapperModel (the inheritance here is probably some minor over-engineering, but it illustrates the reusable parts if building an application with more than one model). The model is exported (CommonJS syntax) as a factory function which combines the relevant reducers, creates the store and returns a new instance of a GalleryModel.

class ReduxWrapperModel {
    constructor(store) {
        this.store = store;
    }

    subscribe() {
        return this.store.subscribe(...arguments);
    }
}

class GalleryModel extends ReduxWrapperModel {
    selectImages(selectedImages) {
        return this.store.dispatch({
            type: galleryActions.SELECT_IMAGES,
            selectedImages: selectedImages
        });
    }

    toggleSelected(image) {
        this.store.dispatch({
            type: galleryActions.TOGGLE_SELECTED,
            image: image
        });
    }

    load(images) {
        return this.store.dispatch({
            type: galleryActions.LOAD,
            images: images
        });
    }

    start() {
        const allImages = fetch('/_api/media?type=' + encodeURIComponent('/^image/'), {
            credentials: 'same-origin',
            headers: {
                Accept: 'application/json'
            }
        }).then((res) => {
            if (res.status === 200) {
                return res.json();
            } else {
                throw new Error(res.statusText);
            }
        });

        return allImages.then(data => {
            this.load(data);
            this.selectImages(data.images || []);
        }).catch(function (err) {
            console.error(err);
        })
    }

    get images() {
        return this.store.getState().availableImages;
    }

    get selectedImages() {
        return this.store.getState().selectedImages;
    }
}

module.exports = function createGalleryModel() {
    const reducer = Redux.combineReducers({
        availableImages: availableImagesReducer,
        selectedImages: selectedImagesReducer
    });
    const store = Redux.createStore(reducer, {});
    return new GalleryModel(store);
};

For the sake of brevity, I’ve omitted the actual reducers, the action constants and some methods on the class. The original code is here though.

Usage then looks like this:

const galleryModel = createGalleryModel();

//render react components
renderSelectTags(galleryModel);
renderAvailableImagesView(galleryModel);
renderSelectedImagesView(galleryModel);

galleryModel.start();

The code above is an example of the Adapter pattern. Redux becomes an implementation detail that consumers of the model need not be aware of. The reasons for this are:

  • Firstly, Redux as an API surface is a bit ugly. The user must be aware of the store, the available actions and the signature to dispatch an action. The wrapper class consists of simpler API; “action methods” that do things and some simple getters for querying state.
  • Wrapping the specifics of the Redux store means that it becomes an implementation detail that users of the model need not be aware of. This makes models more portable and future-proof against decisions to change technology. It also improves the readability of the code; particularly if future maintainers, of components that use the model, are not familiar with Redux.
  • It provides a neat place to add asynchronous code which can dispatch actions once an async operation has resolved. The example above of this is the start() method which fetches a list of images from the server and then calls the appropriate method which will dispatch an action.

Using this pattern might be a matter of taste. I suppose it re-introduces object-oriented concepts that are frowned upon because traditionally this is the approach that presents the problems that Redux sells itself on solving. But there is nothing here that does things like mutate state or create side effects. Personally, I think it keeps things tidy and readable. If it is something useful, I’m surprised I haven’t seen it anywhere before. But then, I’m not really involved in the React/Redux community. Maybe it is. Or maybe there are good reasons I’m not aware of not to do things this way.

Making Periodic Table Map

This is a write-up on a recent side project of mine, Periodic Table Map; an interactive periodic table of elements.

The Periodic Table?

Some time ago I remember having a pub conversation about what the element would make the most compact 10kg dumbbell, whilst not being so expensive that someone with a reasonable amount of disposable income couldn’t afford one; a premiership footballer maybe. A bit of Googling quickly led me to Periodic Videos, a wonderful collection of short videos about chemistry including videos, by Sir Martin Poliakoff and colleagues, on every element in the periodic table. From then on I was an armchair chemist (definitely an armchair chemist, not an amateur).

It’s fascinating to think about what stuff is actually made of. How something like Phosphorus can be used as a terrifying weapon of war, yet it has an essential biological role. How volatile sodium and chlorine are in isolation, yet as a compound it’s just table salt. How magnesium, the first thing that comes to mind of when I think of school chemistry, is inside every single molecule of chlorophyl (the stuff that makes plants green). Perhaps, if chemistry had been taught in a more relatable way I would have enjoyed it more.

The periodic table was first published by Dmitri Mendeleev in 1869 as a way of arranging the known chemical elements in trends of their properties. Its a grid, where rows are called periods and the columns are groups. Elements in the same group exhibit similar chemical properties. As you move down the periodic table the elements have more mass.

Periodic Table User Experience

Despite Mendeleev’s periodic table of elements being almost 150 years old and such a ubiquitous resource for chemists and those learning science I thought its representation on the web was a bit lacking.

The user interaction problem with the periodic table is that there is a huge amount of information associated with each element - ranging from the most fundamental (atomic number, mass, symbol) to the more in-depth and (more technical data, images, Wikipedia pages, videos, etc) that should be displayed or linked to from within the table on screen. Given the interactive nature of a web app, its also nice to include some tools to filter and display the elements based on their various properties.

Two of the best examles, The Royal Society of Chemistry’s Periodic Table and PTable, solve this problem in a similar way. The layout of the traditional periodic table leaves a space, void of elements, at the top middle In these examples, this space is initially used for introduction text or the filtering tools. When hovering over an element block, the space can then be used to show some more detailed data. Clicking then takes you to separate page or pop-up with more in-depth information and media.

Screenshot of the Royal Society of Chemistry's Periodic Table

The Royal Society of Chemistry’s Periodic Table

You are probably familiar with the tactile user experience of using online mapping tools. Most notably Googe Maps. I had the idea of applying the user interaction pattern of online maps to other contexts. Where a user could drag to move around a grid of items and zoom in to scale up the space for each items and thus use the new space for revealing more detailed information. This year’s 10K Apart contest provided a good opportunity to apply this “maps” interaction concept to the periodic table of elements, where each “item” is an element. So I built it. You can still view the entry here, although I’m not sure how long Microsoft will keep this server up. Alternatively the original code is also on Github.

Unfortunately, it didn’t win anything or get a notable mention. I’ll be honest, I was a bit gutted, it took some effort to get it finished in time and I thought the novelty at least would get it some notoriety. However, I’d neglected the keyboard accessibility of the application and I think this was quite an important judging criteria. Or maybe, there were other issues. Or maybe it was just not very good! I don’t know.

Nevertheless, I liked the concept and I thought those needing to use the periodic table would do too. So I persevered, made some improvements, did some refactoring and added some new features without the 10kb constraint. The result is periodictablemap.com.

How it works

This is a bit more technical. I’ll break the functionality down into 4 parts:

Moving

When a user drags with their mouse or finger, they move the table around. Rather than moving the position of the table’s DOM element, via its left/right CSS properties or translate transforms, dragging actually scrolls the page horizontally and vertically. This works particularly well on touch devices because it just utilizes the browser’s natural scrolling behavior which results in a smoother feel.

Zooming

A key part of the user experience is the zooming in and out. Whereby the element cells smoothly increase in size creating more space for more information to be injected. Or the opposite. This happens via a variety of input methods. Either by the pressing the physical zoom buttons or by double tapping/clicking, pinching or scrolling the mouse wheel at a specific point on the table.

This was something I just wanted to get done and move on with quickly, given the fast-approaching entry deadline for the 10K apart competition. It took me a bit longer to get it right though.

The obvious solution (to me at least) was to use CSS3 transforms to scale the table with a transition. However, this meant that whilst scaling up the entire table, each cell had to be simultaneously inversely scaled down, so more content could be added to the enlarged space at the original scale. This, in my initial attempt, was killing performance; it was very juddery. So I went back to the drawing board.

Instead, I tried simply increasing the pixel width of the entire table, letting the width of each cell increase naturally. I remember being happy with this at first, but then decided to ditch it. I can’t remember exactly why. I think it was something to do with scaling the height of the table cells not working out. I decided to revisit the CSS3 scaling method.

Rather stupidly, I realized that, in my initial attempt I was also fading in/out the additional data that was added/removed by using CSS3 transitions on opacity. I had done this first and forgotten about it. It was this that was causing the performance issue. Thousands of transitions were simultaneously occurring. Although the brief fade-in effect was nice, the zoom transition still looked satisfactory without it, so I dropped it. The result works well.

The only other niggle to fix here was the transform-origin of the scale transform. This must must be updated when the user zooms at a specific point. But changing the transform-origin causes the table’s scroll position to jump, so this must also be offset against the delta of the new vs old origin by updating the scroll position.

Periodic Table Zoom States

Sulfur in its possible zoom levels

Lazy Loading

Of course, retrieving and injecting into the DOM the expanded information of every chemical element after a zoom transition would be costly, particularly for the closest zoom level. Therefore, only the elements within the viewport at the start of the zoom transition are requested. Other elements are then subsequently requested and injected only after appearing within viewport after every scroll event.

Filtering

The final main piece of functionality is the filtering. Nothing out of the ordinary here. Event listeners listen to the filter control events. A model of the table is in memory which is used to find the elements which match the current properties of the filter. Elements which don’t match are faded out.

Performance

The performance of Periodic Table Map is one of its highlights. Building it, at first, as a 10k apart entry means good performance is in its DNA. Although the latest version’s size has increased a bit since the 10k apart submission, it is still very fast to load.

Because it’s server-side rendered, the initial rendering of table requires just 11.5kb to be transferred over the wire in a single HTTP request. This consists of the HTML and the CSS, which is in-lined. The Javascript is an additional 5.2kb to transfer obfuscated into one file and loaded using the defer attribute. This means the script will be downloaded asynchronously, but not block HTML parsing, it will only be executed after the HTML parsing is complete, which is, in this instance, good because the functionality the JS provides isn’t needed immediately.

Given the original size constraints of the project, the Javascript uses no frameworks or libraries; it’s “vanilla” JS. To digress a little I want to emphasize that this isn’t such a big deal and it doesn’t mean spaghetti code. With no framework you can still use design patterns to structure and write maintainable Javascript code. The role of frameworks is to typically do the plumbing and wiring required to help implement design patterns and to provide abstractions for common tasks; rendering a view, for example. In a relatively simple app, like Periodic Table Map, its quite easy to manage without this, resulting in a much lighter code footprint.

On the server, some Nginx proxy caching means the Node JS server does very little and responses are rapid. Naturally resources are all gzipped too.

Conclusions

Personally, I really enjoy using Periodic Table Map. The most detailed zoom level creates an experience that encourages you to kill time and continue browsing. It keeps the user journey flowing. More so than if the in-depth details for each element were on separate web pages. I think some of the filtering capabilities are neat too. Particularly, being able to quickly see what elements are in a given state at any temperature. You might ask, “what elements are liquid at 100°C?” Or to go back in time and see what the periodic table looked like in the past using the year filter. For example, its easy to see what elements were known to Mendeleev when he first published his periodic table in 1869.

Obviously I’m writing this with a bias, but I hope others feel the same way and that Periodic Table Map becomes a popular learning and reference tool. Please let me know if you have any feedback. I would also welcome any help on publicising Periodic Table Map.

Periodic Table Map showing the known elements in 1869

Periodic Table Map showing the known elements in 1869

Oh, and the answer to the question at the start of the post. Well, Iridium and Osmium are the densest naturally occurring metals. Osmium oxidizes in air, however, to form the highly toxic Osmium Tetroxide so Osmium can be eliminated purely on practicality. A 10kg dumbbell of Iridium would probably be prohibitively expensive even for the very wealthy. Another contender that I think is most suitable is Tungsten*.

*Being strictly an armchair chemist, this is probably wrong and/or lacking sufficient detail.

http://periodictablemap.com


Client-side Logging with Browser Bunyan

In my Node/Express app, Pagespace, and various CLIs I’m working on, I try to provide a decent amount of logging. As recommended by Joyent, I use Bunyan as my chosen logging framework. Bunyan’s premise is to store logs in a structure format, JSON. It also has some other nice features I find useful such as streams, child loggers and serializers.

Bunyan is also great for client side Javascript logging in more complex frontend applications. The ability to log to multiple streams has been useful. One such application I have been working on can run in mobile webviews as part of an Android or iOS app. In such scenarios, it has been useful to stream log records to a native bridge as well as the webview console. Another example I’ve worked on is a web app for running W3C widgets. The widget runs in an iframe, while the parent frame provides various client-side tools for developing and debugging a widget. The code running in the iframe uses Bunyan logging with two streams; one logs to the standard browser console, while the other posts messages to the parent frame which outputs the logs records as HTML.

Running Bunyan is possible in the browser by way of Browserify. But, unfortunately, this means it still pulls in a bunch of dependencies which aren’t relevant in the browser environment and a bit too bloated for my liking. Therefore, I created a fork of Bunyan, Browser Bunyan, for running a lightweight build of Bunyan in the browser.

Browser Bunyan can be used via Browserify or as a standalone build and is available via Bower and NPM. It also ships with a bonus out-of-the-box stream for pretty printing logs to the browser console:

var log = bunyan.createLogger({
    name: 'play',
    streams: [
        {
            level: 'info',
            stream: new bunyan.ConsoleFormattedStream()
            type: 'raw'
        }
    ]
});

log.info('hi on info');

Screenshot of browser console

Integrating with Angular

Pagespace’s dashboard application is written using Angular. I use Browser Bunyan as Angular’s logging implementation like this:

adminApp.config(function($provide) {
    $provide.decorator('$log', function($delegate) {
        $delegate = bunyan.createLogger({
            name: 'myLogger',
            streams: [{
                level: 'info',
                stream: new bunyan.ConsoleFormattedStream(),
                type: 'raw'
            }]
        });
        return $delegate;
    });
});

The Github repo for Browser Bunyan is here