FAA halts router installs as experts question how one change grounded the nation

Network World ·

FAA halts router installs as experts question how one change grounded the nation

After flights were grounded for more than four hours on November 13, the Federal Aviation Administration has put a halt to further installs of the equipment at the heart of the outage. “We have paused installation of the Juniper router,” the FAA told Network World on Tuesday. The agency did not respond when asked about how long the pause will last. According to the FAA, its legacy copper network became unstable at around 3:30 p.m. Eastern time on Friday afternoon, when data started flooding it at a rate of 1,000 connection attempts every 4 to 5 seconds. “This disrupted the system and created instability with aircraft communications and radar feeds,” the agency said in a statement released Sunday. The network began experiencing instability at 17 of 22 air route traffic control centers and also spread into several terminal radar approach controls, the FAA said. The agency traced the source of the traffic to a radar site in Amarillo, Texas, and a radio communication site in El Dorado, Arkansas. L3Harris, a network service provider, had deployed a new router. When L3Harris blocked the two locations, the network stabilized, the FAA said. “The broadcasting function was also disabled on all the new routers as a precaution.” But it took until 8:15 that evening for the entire system to go back to normal operations and for air traffic to resume, the agency said. “We worked as quickly as possible to identify, isolate and address the cause,” it said. “Because the flying public’s safety is our top priority, we waited for a period of time to ensure the system was stable before we lifted the ground stop and allowed flights to resume.” According to the agency, the outage was not related to the ongoing modernization of the network but rather due to “the legacy TDM system causing instabilities.” TDM—time-division multiplexing—is an analog-era digital transmission method that dates back to the 1960s. The outage was limited to the FAA Telecommunications Infrastructure network, which is a private backbone connecting the 22 regional air route traffic control centers and airport control towers—more than 4,400 FAA facilities in total. The FAA is currently in the process of upgrading the entire system from copper to fiber. An L3Harris spokesperson tells Network World that the company is “supporting the FAA in its investigation to understand the conditions that led to the Nov. 13 ground stop of our nation’s air travel.” L3Harris, together with its telco partners and the FAA, has already “completed more than 70% of the planned airspace communications conversion from legacy copper wires to a fiber optic system,” the spokesperson says. However, according to a statement from the democratic members of the Senate Committee on Commerce, Science and Transportation, the transition plan also included the conversion of 3,648 routers to Juniper devices. However, only 400 have been installed to date—and only 69 are operational. HPE, which now owns Juniper, told reporters that its routers had been working properly on Friday. “Based on what we know today, the device functioned as it was configured to by the installer. It did not fail or act improperly,” the company said in a statement on Sunday. As of this writing, HPE had not responded to Network World requests for comment. Accelerating pace of change Kris Beevers , CEO and co-founder at NetBox Labs, a network and infrastructure management company, was in an airport on Friday when the outage took place. “I was in Fort Lauderdale flying home for the weekend,” he says. “And when it was apparent that it was going to take some time, I grabbed a hotel and booked a flight for the next day.” NetBox Labs is the company behind the NetBox open-source network and infrastructure source of truth used by more than 10,000 organizations, including airline networks, critical infrastructure, and banking, data centers, and other enterprise networks. “So, we’re pretty close to these problems,” Beevers says. And, so far, there is no indication that anything malicious happened on Friday. “This network is not one that is generally accessible to outside actors,” he says. “But it’s actually extremely common for network configuration issues or network changes to manifest in something that looks like a denial-of-service attack internally.” It’s similar to what happened with the Outlook outage last week, he added. The general public doesn’t usually hear much about how networks work, Beevers says, but people are likely to be hearing a lot about them now. “I think this rapid-fire sequence of outages really emphasizes for us how important these networks are to the way the whole world works right now,” he says. “Everything is powered by networks.” While these network environments have been operational for decades, the rate of change is accelerating—and that contributes to their reliability. “Things tend to go wrong at moments of change,” Beevers says. Configuration and change management failures are responsible for 41% of network-related outages, according to this year’s Uptime Institute’s Data Center Resiliency Survey, making them the single biggest factor , ahead of hardware and software failures. “I’ve seen many companies shut down their factory, store, hospital, or trading floor because a seemingly routine change had unforeseen consequences,” says Bernadette Nixon , CEO of NetBrain, which makes network observability software . Still, the number of high-profile outages in recent days—FAA, Outlook, Monday’s bank outages, and an NSA outage on Tuesday—is raising eyebrows. “I can’t say for certain whether this is or isn’t a cyber attack,” says Mike Wiacek , co-founder and CTO at Stairwell, a cybersecurity company. “But four major outages in under a week look suspiciously like what I’d expect one to look like.” Prior to Stairwell, Wiacek founded Google’s Threat Analysis Group and co-founded Chronicle, Alphabet’s cybersecurity company. And if the outages are connected to a cybersecurity incident, it has all the hallmarks of a nation-state attack, Wiacek adds. “Deniability is the point,” he says. “And notice that no one has been physically hurt. That makes a military response hard to justify, while the economic damage and loss of public trust are real.” The fact that public officials immediately ruled out the possibility of an attack is also concerning, he says. “You can’t compress months of forensic work into a few hours.”

Источник: Network World