Notes from 2022

This is Part 3, where we discuss what happened to the HFC network, why its rollout was paused, and what can be done to salvage it.

Part 1 focuses on the cultural deficiencies at NBNCo.

Part 2 focuses on the failings of FTTN.

Part 4 explains why the LTE network was poorly thought out from the start.

Part 5 briefly touches on the economics of the NBN.

Part 6 hammers home the fact that all of this is a result of shameless and disgusting short sighted political manoeuvring by the Liberal Party to protect the interests of Rupert Murdoch and other corporate friends such as Telstra.

An NBN restrospective - Part 3.

In 2015, it became apparent to both the Government and the board of NBNCo that continuing to push forward with FTTN would result in a national uprising. Interestingly, the revelation that FTTN is untenable came at around the same time Malcolm Turnbull usurped the Prime Ministership from Tony Abbott.

Returning to FTTP, as per Labor’s original plan, was simply politically unacceptable. The only other technology capable of delivering high speed, fixed line broadband is Hybrid Fibre Coaxial (HFC), but HFC networks are ruinously expensive to deploy, and in the process of being replaced with FTTP in the rest of the world. Luckily for the Government, the solution to the cost problem was right there, buried under a big T. But first, a history lesson. A very long, but ultimately necessary, history lesson.

In the mid to late 1990s, Telstra and Optus were engaged in a fierce battle over pay TV supremacy. Optus was first to market with an HFC network, carrying the OptusVision service. Telstra followed suit, entering a partnership with Rupert Murdoch’s Fox. FOX would bring the subscription TV content and TELstra would bring the infrastructure required to deliver it.

Optus’s network utilised a Motorola-designed system called CableComm to also deliver telephony services over HFC, in a bid to rid themselves of Telstra’s monopoly on communications. Since Telstra had no need for such a system, it designed its network predominately for Foxtel. Optus and Telstra then locked themselves into what amounted to little more than a dick waving contest.

Rollout targets were aggressive, and both networks’ footprints expanded massively over the course of relatively little time. In order to meet their aggressive rollout targets, however, Optus and Telstra both elected to cut corners in the design of their networks. These cut corners are irrelevant for the delivery of cable television, but become very important later

In 1996, the first version of the Data Over Cable Service Interface Specification (DOCSIS) standard was released, allowing HFC networks, which were at the time used almost exclusively for cable television, to act as a broadband data network. Since that time, DOCSIS has become the industry standard for broadband delivery over HFC networks.

Telstra were able to call Optus’s bluff with their near infinite cash reserves and when the latter was acquired by SingTel, who pulled the plug on further HFC investment, Telstra too saw no further need to continue rolling out their own network. Both networks were left pretty much to rot for the next 20-odd years, with only essential maintenance being performed to keep services running.

Instead of seeing an old, tangled up mess of rotting coaxial cables, NBNCo saw a free lunch. A free lunch that cost them around $3bn when all was said and done.

After purchasing Optus’s HFC network for $800m, NBNCo quickly found out that not only had Optus neglected to perform routine maintenance, but they had also seriously oversubscribed the network in an effort to maintain revenue in the face of relentless competition from Foxtel/Telstra. Contention ratios across the network were usually well in excess of 600:1. That is, 600 homes sharing a single link back to the exchange. Due to the lack of upgrades over the years, this shared link was usually 1Gbps, meaning that each subscriber on a given coaxial segment was allotted 1.7Mbps downstream. Optus’s network in its twilight years was plagued with complaints of poor speeds and poor reliability.

After running trials in Redcliffe, NBNCo deemed Optus’s network so far gone that it was beyond cost-effective remediation, and ultimately elected to abandon it completely, to be replaced with yet another worthless stop-gap technology, Fibre to the Curb/Kerb (FTTC). $800m down the gurgler so far.

The company also purchased the Telstra HFC network - this time for an undisclosed amount via a renegotiated $11bn asset lease deal, originally put in place to allow NBNCo the use of Telstra’s cable ducts and pits, as well as space in their exchanges, fees for remediating unsafe pits and compensation for the forced decommissioning of its HFC and PSTN assets. This renegotiated deal instead put in place an asset transfer for the HFC and PSTN networks, and awarding them even more public money for the privilege. Most importantly, it also gave Telstra a spectrum license to continue using the HFC network to deliver Foxtel indefinitely, with NBNCo to foot the maintenance bill for this. It is estimated that the total value of the HFC asset transfer was somewhere between $1.1bn and $1.3bn.

Someone neglected to tell NBNCo that Telstra’s network design was based on two very major assumptions that made it unsuitable for what NBNCo required of it.

• The network would be used primarily for pay TV
• The network would have a takeup rate of ~30% of passed premises

The first assumption allowed Telstra to get away with using lower bandwidth equipment on the coaxial side of the network since Foxtel only uses a relatively small amount of bandwidth compared to the total a regular HFC network would be capable of. The second allowed Telstra to under-provision the network relative to the number of premises it passed. Simply put, these were cost cutting measures intended to allow Telstra to keep up with and leapfrog Optus’s HFC rollout. Telstra were able to get away with providing DOCSIS services since there was so much spare spectrum available on the network on frequencies that their cheap equipment could still work happily with, and services were so expensive anyway that few in the network’s footprint took up said services.

Having not been made aware of these shortcomings, NBNCo gleefully began relentlessly connecting new customers to the network. CEO Bill Morrow said that the purchase of the HFC network would “…shave years off the rollout…” This was before people started noticing their Foxtel and Telstra Cable services starting to sporadically drop out, when previously they had been stable for years.

HFC is a shared medium. That is, end users all share a single coaxial trunk cable back to the optical node. With each end user you add to this coaxial segment, you lower the signal level to each of the other users. NBNCo tried taking an HFC network barely suitable for 30% uptake to 100% uptake without making the necessary upgrades to the equipment in the field. As signal levels across the network fell, dropouts, fuzzy Foxtel pictures and slow speeds became more and more apparent until late in 2017, NBNCo announced that it would be stopping all new HFC connections until further notice to investigate the issues.

A prolonged investigation into the cause of these disruptions made multiple significant findings:

• Telstra had used low quality fittings and connectors, and had not replaced old ones when they had worn out
• The amplifiers and nodes Telstra used were ill-equipped to handle the frequencies NBNCo were forced to use (Foxtel and BigPond taking the good spectrum for themselves)
• Telstra had oversubscribed the network, but to a lesser degree than Optus
• Poor maintenance standards meant that the network was prone to RF noise ingress
• Much of the network’s declared footprint was not serviceable and no HFC infrastructure actually existed where it was said to

All of these physical shortcomings were compounded as NBNCo went around disturbing the infrastructure and adding connections. NBNCo kept new connections on ice for around a year while they conducted expensive remedial works. This work included installing new nodes, replacing faulty 30 year old amplifiers and taps, fixing noise ingress points, replacing old connectors and fittings and building out the network to fill in the gaps in the footprint.

The sales freeze was ostensibly lifted in April 2018, however many were still not able to order a service until around August of the same year. This “pause” in the HFC rollout as it has come to be known cost the company $900m, bringing the total amount of taxpayer money stolen on HFC to around $3bn, with the total rollout cost blowing out to $51bn. This is $8bn more than the projected cost of a full FTTP rollout.

It is accepted in the industry that while HFC/DOCSIS itself is still more than suitable for delivering high speed broadband, when done correctly. The assumption that simply buying Telstra’s and Optus’s HFC would be a silver bullet for NBNCo comes from data on HFC networks in the United States – HFC networks which cover most of the population, have take-up rates nearing 100%, and as such are aggressively maintained and upgraded by their operators. The Australian HFC networks could not have been further from their US counterparts if they tried.

The Telstra/NBNCo network is, to put it simply, an obsolete and failing 90s network with some late 2000s quality of life features gaffer taped on to keep it barely functioning for another 5 years at best; good taxpayer money after bad in the Liberal Party’s pursuit of facial salvation.

This network is currently the designated technology for again around 30% of the population. Unlike FTTN, however, DOCSIS is unaffected by distance from the node, is mostly immune to noise ingress and doesn’t care if Uranus is in retrograde. As such, the service quality is much, much higher and a full FTTP overbuild is largely unwarranted.

That said, in its current form, the HFC network will never be able to keep up with bandwidth demand. NBNCo are already having difficulty getting 1Gbps plans on the network, plans which have existed on the company’s own FTTP network since early 2013.

So, without throwing the proverbial chessboard and starting over, how do we fix the HFC network? Luckily, the people who design, and the US network operators who ratify the DOCSIS standards are a little more forward thinking than NBNCo could ever hope to be in its current iteration. Built in to the latest version of DOCSIS are a number of enhancements which prepare compliant HFC networks for the future.

One of the major limiting factors to HFC network performance is the number of amplifiers between the node and the end user. Current best practice is to utilise an N+0 architecture. That is, the Node plus zero amplifiers between it and the end user’s cable modem. The NBNCo network is currently at an N+3 architecture, which makes it largely incompatible with the latest DOCSIS standard, DOCSIS 3.1. This is especially true if you take into consideration that there remain many of the old Telstra/Foxtel amplifiers still in service, as they were deemed to be working to an acceptable level for DOCSIS 3.0.

While NBNCo have been progressively enabling DOCSIS 3.1 on select parts of the network, this is mostly due to its efficiency gains rather than for increasing performance for the end user. D3.1’s increased efficiency means that NBNCo can maintain current performance without performing any physical upgrades to the network.

One of the most important and effective upgrades NBNCo will need to make in the immediate future is moving the physical plant to N+0 Distributed Access Architecture (DAA). N+0 takes fibre deep into the HFC network, maximising network performance by minimising the amount of noise-prone and lossy coaxial cable between the node and the user, as well as lowering contention ratios to a level where the shared bandwidth is not even noticeable.

N+0 and DAA go hand in hand. Where N+0 deals purely with the physical side of the network, DAA is an upgrade that fundamentally shifts how the network itself is provisioned. Traditional HFC networks have the Cable Modem Termination System (CMTS) and RF combiner located centrally at the headend. The RF DOCSIS signal generated by the CMTS is then modulated into an analog light beam and sent along the fibre to the node, where it is converted back into an electrical RF signal.

DAA introduces a profound paradigm shift with the Remote PHY (R-PHY) unit. This device replaces the CMTS at the headend, and instead resides in the node itself. Instead of the DOCSIS signal being generated at the headend and sent to the node in an analog fashion, it is instead generated at the node which talks to the headend digitally for service provisioning. In other words, the communication between the headend and the R-PHY is fully compatible with PON, the fibre architecture used to roll out FTTP.

DAA has huge implications for NBNCo, as it gives the company a simple and cheap upgrade path to full FTTP for when the time comes. Upgrading an N+3 DOCSIS 3.0 network is senseless, as the fibre component is incompatible with PON, and the whole lot would either need to be duplicated or upgraded in situ, which would cause mass service disruptions. By being fully PON compatible, DAA allows NBNCo to simply attach the fibre equivalent of a double-adaptor to the input of the R-PHY device and bypass it with FTTP while leaving DOCSIS services active and uninterrupted. In most cases, this multiport is installed at the time the DAA fibre is installed in anticipation of FTTP upgrades.

DAA also allows NBNCo to reduce maintenance costs across the HFC network, which is vital as the company currently has a network-related operating expenditure of around $2.3bn. Removing expensive, old, power hungry and short-lived active HFC equipment can lower this figure significantly. Importantly, it also lowers the cost of deploying FTTP to new developments in the HFC footprint, which common sense would dictate should receive FTTP from the get-go.

In terms of end user outcomes, DAA enables the efficient and reliable delivery of ultra-speed broadband plans in excess of 10Gbps download, whereas there would be no hope of delivering such services over the current N+3 DOCSIS 3.0 architecture. For those users who engage in NBNCo’s Technology Choice Program, it also makes upgrades to FTTP significantly cheaper, encouraging its uptake before even NBNCo are ready to do the upgrades themselves. Cynics would also note that every user-pays upgrade NBNCo makes through the TCP is one less the company has to pay for when it goes around upgrading the network itself.

Continue reading in Part 4.