A Quick Note on Site Speed & Comments

54 comments

The eagle-eyed among you will have spotted that we’ve temporarily removed a few of the more advanced features of the website. Most notably, the “Most active articles” list on the side of the homepage. The less eagle-eyed will have also spotted that in recent months LR has gradually been getting slower sometimes to load, reaching a point in the last couple of weeks where the site has even begun throwing errors occasionally when it tries to load a page.

It’ll probably not surprise anyone to learn that these two things are related. In essence, LR has a problem – albeit a good one to have:

We have an ever-increasing number of readers making an ever increasing number of valuable comments.

Why is this a problem?

Weirdly, this is a problem because while we all want the internet to be a space full of long articles featuring lots of intelligent debate, in reality that isn’t what most of the internet is actually like.

The vast majority of sites which support comments, therefore, operate on the assumption that they are of low value. That they are something they can outsource to Facebook or another third party and ignore, or which they will mostly delete when they start to impact speed because they are spam.

Indeed over the last five years or so, this has even subtly driven the way the architecture of many web-based Content Management Systems is designed, leading to more of a focus on ‘core’ functionality and social / third party integration than on optimization of heavy comment load.

For most people that isn’t a problem, but here in LR Towers it presents a unique one – because our continued focus on longform writing and your continued dedication to writing good comments (and lots of them) places a strain on our server resources that increasingly the very architecture of those resources isn’t set up to deal with.

Just as a rough example, in the last two years we have had as many comments as in the first seven years of the site. Indeed if the current trend continues, we will be on course to have about 30,000 comments this year alone – excluding spam comments.

Which is a problem – albeit a nice one to have (and a reason why our volunteer moderators, LBM and Malcolm, deserve an infinite amount of free pints)!

Why is it an issue now?

Dealing with the increasing issues this is causing is perhaps the main reason why you haven’t seen many posts by me personally over the last few months. I’ve actually got a big Uber piece in the works, but right now my “LR time” has largely needed to be focused on trying to find new ways to make sure things run smoothly for you. This has meant gradually bringing in multiple layers of caching, building on increased server resources which the launch of the magazine has helped subsidize.

It’s just that now we’re hitting the point where we’ll need to make some larger changes behind the scenes to ensure things run smoothly for some time to come.

What can we do about it?

The good news is that we’ve got several things in the works that should help, and one way readers can too:

A new site design

For a few months now I’ve been working in the background on a new site design. This is nothing revolutionary and we’re due one anyway. But for us every little efficiency matters now, and it’s built to avoid making certain large queries in certain places which then slow everything else down. This’ll bring with it some restructuring of content, but also an opportunity for us to highlight older content, and content series (such as those on rail studies, and our various industry interviews with senior people) better as well. So it’s a win-win really.

Creating dedicated archive and events sites

In the shorter term, you’ll soon notice that we’ll fully split some of our older posts, and our events site, onto slightly different domains – archive.londonreconnections.com and events.londonreconnections.com. These won’t change in functionality (although commenting will be closed on archive). It’s just that three smaller sites will run faster than one large one.

The reader one: encourage your firms to subscribe or work with us

Of course the simple truth is that all the above is necessary because, ultimately, LR is a volunteer effort. We don’t get paid to write, and we’re fine with that. We love what we do.

Server resources don’t come for free, however, and whilst launching the magazine has helped ensure that meeting those costs is less of a pull on our own pockets than it was before, as this shows there is always more server resource we would pay for if we had the choice. And the less we’re focused on that, the more time we have to write. This is, ultimately, why we’re always keen to work with companies to tell stories, where appropriate, run adverts in the magazine or even just sell corporate print or digital subscriptions.

That last one is certainly something with which our readers can help. If you think that a subscription to LR would be of interest to people within your company, or that your company has a good story we can help them tell, then do remember to mention it when the subject comes up. Corporate subscription contact details can be found in our shop and we’ve got a site that explains our partner model here for those who would like to collaborate with us on written pieces.

Why don’t you start a forum! It’d help.

Sorry, but that’s not what we do. And trust me – it really wouldn’t.

Our overriding ethos has always been that we’re about sharing good information and allowing discussion around that information. Forums aren’t that.

There are also several other places on the web that already fulfill that role well (you will find plenty of LR people on District Dave and Skyscraper City).

As someone who has managed forum communities before, I can also tell you that Forums are insanely resource-intensive to manage with the size of the community that, from our comment base, we know we’d attract. Put simply, if we had to run a forum then we wouldn’t have time to write. And the writing is what we’re here, and what we love, to do.

The conclusion: don’t panic, we’re not!

Like I say, there are good problems to have and bad ones. This is definitely one of the good ones!

Luckily, from an LR perspective, doing this stuff is my “day job,” so it’s just a matter of pushing through and implementing the things we need to implement to keep things running smoothly and indeed make them run smoother.

Just bear with us a bit while we do that, and my apologies if it means you see the odd bit of functionality sacrificed for speed in the short term. Wherever possible, that’ll only be a temporary thing and efficiencies we introduce elsewhere will let us put some of the more resource-intensive functions back into play again in future. As well as bring you some new ones.

All the best, and thanks as always for helping make LR what it is – and stay tuned today for the second part of Pedantic’s look at the Kent Rail Study.

John Bull
April, 2017

Written by John Bull