On Tuesday 8th January 2009, Adam Dabrowski stood watching the trams go by in his home town of Lodz, Poland. Adam had long been fascinated by the trams and how Lodz’ tram system worked. This ran deeper than just an interest in the sight, sounds and mechanics of Lodz’ ageing rolling stock, however. What Adam was really fascinated by was how the whole thing was controlled.

As another tram began to pass over the junction in front of him, Adam looked down at his hand. In it he held what, from the outside, appeared to be a standard universal TV remote. In reality, though, it was much more than that.

The idea for it had first come to Adam whilst he was in electronics class at school, but it had taken him months to actually create. Indeed building it had involved careful research online, as well as some late night trespassing in tram depots to get more data. Finally, though, it had all come together and now Adam was pretty certain that it would work.

As the wheels of the tram’s first carriage passed over the junction and it began turning to the right, Adam raised his hand and pointed his jury-rigged infra-red remote control at the points. With a solid ‘thunk’, they suddenly switch back to the left. A split second later, the tram’s second bogie hit the reversed points and, with a screech of metal, the tram derailed.

With a mixture of joy and surprise, Adam turned and began to walk away.

Piłsudskiego Avenue, Lodz. Courtesy Zorro2212.

Everything is vulnerable

By the time he was apprehended by Polish police, Adam Dabrowski had caused several more incidents on the Lodz network, luckily resulting in only four injuries. The subsequent investigation would reveal that in the course of his fascination for all things light rail and electronic, Adam had discovered that Lodz’ infra-red based signalling system had a critical flaw. In the right circumstances, it was possible to record the signal sent in one place to a set of points and play the signal back to get the same result in another location.

It was a crude, but effective, cyber attack. Adam was no ‘cyber terrorist’ in the traditional sense (despite being branded as such in the press at the time), but for Lodz’ light rail network the end result was the same. Even before Lodz, transport operators had become increasingly aware of the impact that a concerted cyber attack could have on the services they provided. Lodz served as a reminder that it wasn’t just laptops and desktops they needed to worry about. In an increasingly digital world, anything could be compromised.

The lock out

On 25th November 2016, bargain hunters across America woke early. It was ‘Black Friday’, the day after Thanksgiving, when many shops across the United States open earlier than usual to begin a traditional sales period. In San Francisco, many of those shoppers soon began to descend on the city’s public transport system, the MUNI. There these bargain hunters were greeted by an unexpected surprise – the MUNI was free.

It soon became clear to those travellers – and the world – that this wasn’t due to the beneficence of San Francisco’s transit authority. It was because the entire network’s ticketing system had been hacked, along with a significant percentage of its regular computing systems.

That hack was not something that the transit authority could try and quietly deal with and control. That they were subject to a ‘ransomware’ attack was writ large across the screens of their ticketing machines network-wide, which proclaimed, in broken English:

‘You Hacked. ALL data encrypted.’

Behind the scenes they had also received notification from a hacker group operating under the alias of ‘Andy Saolis’ (a common pseudonym used for this style of attack) as to what the price for returning control of the system would be – 100 bitcoin, or approximately $73,000. It was also accompanied by a further threat which the operator had no way of verifying – to release over 30GB of customer and employee personal and financial data.

Luckily, for the Transit Authority, their backups were unaffected. Nonetheless, it would not be until the following Sunday that they would be able to collect fares again. In fare-box terms alone this represented a $1.5m hit. The overall financial impact and the reputation damage, however, was much greater.

Information and Operational Technology

The nature of the attacks on the Lodz and San Francisco systems were very different. In this, they highlight two of the differing types of cyber attack that rail operators can face. On the one hand, as was the case in San Francisco, operators can be subjected to the type of attack with which most people would associate ‘hacking’. That is, a direct attack on the IT systems that provide the supporting communications and information management infrastructure on which a rail network relies. Lodz, meanwhile, was an attack on what might be regarded as the operational technology (OT) layer – the hardware and software that controls the physical systems which keep the network moving.

In many ways, the possibility of an attack on its OT layer is something that the rail industry has had to be aware of, and deal with, for several decades. Indeed the Lodz attack itself actually highlights just how long this attack vector has existed – it was an infra-red attack, targeted at a very old signalling integration. Nonetheless, it is a growing issue. Just as the ‘internet of things’ is beginning to have its impact on people’s homes – through everything from WIFI kettles to heating systems controlled by iPhone apps – so digital technology continues to penetrate and permeate the control layer of rail’s signalling systems and more.

On the one hand, that brings an enormous range of speed, safety and efficiency benefits. On the other, it drastically changes the way rail’s operators and infrastructure managers have to think about the control systems they run and maintain. As the industry has increasingly discovered, however, this is not as simple as an operator transferring its standard IT approach to its OT as well.

“For a while now, there’s been an appreciation, more than anything else, that OT systems and the challenges that critical national infrastructure operators such as rail face when managing them, are not necessarily the same as those for IT systems.” Says Dr. Alex Tarter, Head of Cyber Consulting at Thales.

“One can glibly say: ‘OT is not IT’. But what that really means is that IT people don’t necessarily understand how OT operates. It’s just a different environment. It’s populated by people in boots, hard-hats and high-vis, not people sitting around in offices installing CAT-5 cables or doing cyber risk assessments and vulnerability investigations.”

Getting secure

How to successfully meld IT security policies and techniques with OT systems is one of the biggest cyber-security issues facing the rail industry. One element of the path forward seems clear – ‘security by design’. This is something that all the major signalling and system manufacturers are all increasingly factoring in by default – and which operators themselves are increasingly mandating up front. It means that the systems being implemented today must include the need for strong cyber-security at the design stage. It’s a simple concept, but one that is not always proving easy to implement, partly because rail operators, and governance groups, are still in the process of defining the universal frameworks within which all systems should operate. Nonetheless, progress is being made.

Security by design, however, only addresses one element of the wider OT cyber-security problem. For whilst it ensures that a system is secure at launch, it doesn’t address what are arguably the two bigger issues – the constant risk of future threats (whether due to existing unknown software flaws, or those introduced through later patches) and the need to secure older, existing systems (which were not designed for security) from attack.

Dealing with the first problem means operators accepting that threats evolve and therefore protection must do too. That’s made ‘cyber as a service’ something increasingly important to the industry. For smaller operators, this means using dedicated outside cyber monitoring services to keep their infrastructure secure. For the larger operators (such as Network Rail or TfL) it often means using the same external services as a way of supporting and educating internal cyber-security teams when new threats arise. Often those services also provide a route to tackling the second problem as well – building the procedures and mitigation measures required to secure ageing infrastructure that is now exposed to new threats.

Crossing disciplines

Trying to secure existing systems and keeping new ones secure, however, brings with it a major challenge – one which again stems from the traditional differences between the way IT and OT have been approached. The techniques for dealing with the issues in both now overlap, but this doesn’t mean that skills in one area are instantly transferable to the other. This is something that both operators and individuals often underestimate.

“A cyber-security consultant, can’t just wander in and say: ‘Well, we understand cyber-security and here’s how they’ll attack you.’” Says Dr. Tarter.

“Because they’ll be told: ‘Yes, but you don’t understand the physical environment we’re working in.’ And they’ll be right. Those people will be unable to tie the genuine cyber risks into the real-world impact.”

“It can be as simple as understanding what is actually running on a system. You can’t just point at an issue and say: ‘Well there’s a flaw there. Someone could introduce malware.’ Because if nothing on that system is running windows or another protocol that the malware is going to care about, then it’s not going to run.”

“That’s one of the biggest challenges rail operators, actually all critical infrastructure companies, have. Indeed our policy, and recommendation, is always that you have to match up a cyber-security expert with a domain expert.”

The concept of meshing cyber-security and domain expertise is not new. Perhaps unsurprisingly it can trace its origins, as an approach, to the world of defence. Indeed whilst they may not have invented it, the British Ministry of Defence (MOD) were certainly an early pioneer. As it re-launched and upgraded Britain’s surface and subsurface naval capacity, the MOD recognised that it was now essentially managing a number of floating industrial control systems.

Those control systems were well designed to be protected against kinetic attacks, and their commanders and crews were well trained in the limits of that protection in combat. The same level of protection and understanding was not being factored into the digital aspect of those control systems, however. Without that, they had no way of ensuring that their digital systems weren’t under (or over) designed, and were robbing their commanders of a critical piece of knowledge in combat. Put bluntly, a captain needed to know which computer systems he had to protect most in battle as much as which physical areas of his ship.

Indeed the adoption of this mentality is one reason why both the MOD and the cyber-security industry didn’t react to ‘revelations’ in the mainstream press that Britain’s nuclear submarine fleet was still using Windows XP. It didn’t matter. Both the MOD and the Navy were confident that they had the combined cyber and domain knowledge to know where that system was used, if it was on a critical attack path and, if so, that on a design – and command – level they had taken the necessary steps to mitigate it.

Taking that same concept of cross-domain expertise into rail is now something that it is critical for operators to do themselves, or get help to do.

“This is actually one area where the nature of Thales as a business, and the range of what we do, means we’re quite lucky.” Admits Dr. Tarter. “But also because we’ve embedded that culture. You need people who are cyber-security and domain experts. Don’t just hire IT people. You have to hire OT people and then we make them cognisant of both disciplines.”

“That’s important because it’s exactly the same in critical infrastructure.” He explains. “Here, the operators get told: ‘We’ve done a risk assessment to ISO 27000. We’ve followed the process and here are the outstanding risks.’ But the business owners may still not understand whether a cyber attack is still going to stop them running the railway.”

“This is because, fundamentally, they don’t understand what is an unacceptable impact and what isn’t, as well as what their operational staff can work around and what they can’t. For that, you have to have domain experts and cyber experts. Because the domain people will understand the effect for each cause. They’ll know what will actually impact the system and what will not. They understand the whole system overall and have that operational knowledge.”

The human factor

The need for expertise and understanding extends beyond the design and management layer as well. This is one area in which the rail industry’s IT and OT security issues overlap. Indeed nothing highlights the risks here better than the San Francisco hack. It would be easy to imagine that MUNI’s systems were penetrated by a dedicated team of Hollywood-style attackers working away in a monitor-lit darkened room. The reality is very different. In fact, the evidence suggests that it was a ‘spray and pray’ attack, in which a transport authority worker unwittingly uploaded the ransomware virus to MUNI systems via a zip file or USB device.

“This is why the human factor has to be at the heart of an operator’s approach.” Dr. Tarter says. “Speaking with my consultant hat on, good cyber-security practice means sitting down with people and saying: ‘Okay, what do you actually need to do?’ We then work out how we can design a system that helps them operate in a more secure manner, but which also makes it really hard for them to act in an insecure manner.”

“When it comes to critical infrastructure, we have to build a ‘secure by default’ mentality into people as well as systems.” He continues. “Apple is a great example of this, where the iPhone, by default, says ‘Hey! Use your fingerprint!’. They make security straight forward and easy to do. Now, when it comes to rail, you can’t just adopt the same practices. There are real world operational aspects to running railways that we can’t ignore. Not least, we can’t expect that people who have had 50 years of operating in one manner in the rail industry will operate in a different manner after a one half-hour training course in anti-phishing. That’s ludicrous.”

“But that’s fine!” He says. “We just have to embrace that and actually make sure that standard operating procedures are designed, by default, to be secure.”

With people as much as systems, however, secure by default is only half the battle. Proactive monitoring services are again key.

“People will deviate from those operating procedures. That’s just human nature.” Dr. Tarter says. “So it’s important that you try to make it so they have to deviate in an insecure manner that you can easily detect and mitigate.”

“We had one system, for example, where the operators and the maintainers were from two different companies.” He says. “Which is not unusual in rail, but from a cyber-security perspective is always problematic. By policy, the maintaining engineers shouldn’t have been modifying the system without letting the operators – and their cyber-security team – know. But because the computer ports were easily accessible, they could just wander up and plug in to do configurations. When challenged, their response would be: ‘Well I did send someone an email to say I was going to do it.’”

“So we put that system, with its easily accessible ports, inside a locked box with an audible alarm which was also tied in, over the network, back to the security operations centre.”

“Now if an engineer wanted to go in and make modifications he had to open the box. And if he opened the box and didn’t tell anybody, then an alarm would go off and someone in the control room would by definition hear about it. Straight forward, but effective.”

As an example, it is one that actually highlights nicely many of the issues that getting cyber-security right in rail means facing – a mix of physical and digital systems, processes that need to cross corporate boundaries and, beneath all that, the overall need to keep people moving.

At root this is, of course, the core cyber-security challenge facing the railway – in London, the UK and ultimately the world. Because for rail, like all elements of critical infrastructure, not providing a service during, or because of, a cyber attack often simply isn’t an option.

Falling back to ‘wetware’

The need to operate a service during an incident is something that is particularly pertinent to cities like London. One of the risks that political proponents of ‘driverless trains’ and reduced staffing levels on the Underground (and Britain’s railways in general) often overlook is the impact that this can have if the network is under attack or potentially compromised digitally.

A ransomware attack on the ticketing or information layer is relatively easy to operate around. Something that targets control systems (the OT layer) though can have a much greater impact. Here, there naturally has to be a higher burden of proof when it comes to demonstrating that a problem has been fully assessed and remediated. And until that burden of proof is met, those digital control systems can’t be used.

As both TfL and the mainline operators push towards more efficient (and thus less human-resource-intensive) systems, this leads to a new problem. The systems that allow an operator to be more robust and efficient in normal operation not only become a barrier to operation when a cyber incident occurs, but potentially magnify its consequences.

“Right now, the rail industry has the ability to flip to manual – to proceed at the slowest speed, with visual inspection and do it that way.” Dr Tarter explains. “So at least you can continue to operate. With a lot less capacity and efficiency, yes, but it’s not a case of fundamentally stopping moving.”

“One of the things we see in disaster recovery exercises is that all critical infrastructure relies increasingly heavily on machines and computers though. And if you’ve moved to a model where you’ve reduced staffing levels thanks to technical efficiencies, then you suddenly discover that going manual requires more resources than you actually have available to you – or at least more trained resources.”

For rail operators, whether they realise it or not, disaster recovery is thus increasingly about making sure their ‘wetware backups’ are as good as their software backups. People, it seems, have become something of the modern equivalent of the strategic steam reserve. They are the old, trustworthy failsafe on which the railways may one day need to depend.

All of this highlights the complexity and importance of understanding cyber-security in the railway context. Because as technology becomes increasingly embedded in railway systems, both the risk of, and scale of impact from, a cyber incident grows.

“In a regular enterprise, the IT systems support the business.” Agrees Dr. Tarter. “These days in critical infrastructure like rail, they are the business.”

“If those systems aren’t working, then as an operator you’re either not providing a service or can only do so by not making money. That’s really not a position in which you want to be.”

If you enjoy what you read, then why not support our writing through buying our magazine.

With thanks to Thales for their assistance and sponsorship of this look at cyber-security on the railways.

Written by John Bull
John Bull is the Editor of London Reconnections. A transport journalist and historian, his writing often focuses on the political or strategic challenges facing London's transport network and beyond.