Sunday, November 22, 2026
English edition

Development

How FAA can modernize US national airspace system without emergency ground stops

November 20, 2026 Development Source: Ars Technica

How FAA can modernize US national airspace system without emergency ground stops

Share this article

Specifically, the L3Harris contractor was deploying a new Juniper Networks router to replace a 25-year-old Cisco router at the El Dorado site. But the subsequent incident prompted the FAA to pause installation of the Juniper routers and turn off those that had already been installed, falling back on the older Cisco routers, according to the Associated Press. FAA employees typically conduct risk assessments and coordinate with contractors, such as L3Harris technicians, on system maintenance or upgrades, according to The Professional Aviation Safety Specialists (PASS) union that represents FAA technicians. But in a statement, the union said that none of its FAA technicians were involved in the upgrades that led to the ground stop. Similarly, CNN reported that the router replacement at the remote radio communications site in El Dorado was done by an L3Harris engineer without direct FAA oversight. The union and two sources told CNN that the FAA technician merely escorted the contractor into the facility and then turned his attention elsewhere without knowing what work the contractor was doing. To get a former insider perspective, Ars conducted a lengthy interview with Philip Mann, a principal consultant at Vector Strategic Consulting LLC, where he advises on aviation safety and AI governance. Mann previously worked at the FAA in multiple roles for 17 years, starting as a technician at Denver International Airport and eventually becoming a training program manager at the FAA Academy in Oklahoma City. (This interview has been condensed and edited for clarity.) Ars Technica: What do you see as some of the potential contributing factors that led to the nationwide ground stop? Philip Mann: I spent a lot of time as a technician, and then did a bunch of things in between. And later on, I was an instructor at the FAA Academy, and then I did training program management. So I’ve seen a lot of this from all sides about how it’s supposed to work, and the thing that stood out to me the most immediately when I saw this is the time during the day in which the work was being done. As an FAA technician, this is managed at the field person level. This is not like a top-down command-control thing. It is your job to cultivate a little credulity and think, “Okay, this thing I’m working on, what’s the worst thing that could happen? How far can it reach? What’s the blast radius?” This impacted a nationwide network, so by default the blast radius is the largest possible blast radius, so that basically means you take the maximum possible mitigations. To kind of contrast that, when I worked at Denver [International Airport], a lot of times I was working on things that were on a single runway, like one approach lighting system or one electronic system, and the blast radius was just that runway, not even the whole airport, just that specific runway, and sometimes only that one end of that one runway. So you can still use the rest of it just fine. And then maybe we work on a radar site, and a radar site might have a little more of a regional area. Most of the ones I worked on were kind of short-range radar, so it might be like a 50-mile radius or something. In which case you look at what the edges are, and you assume that if something goes wrong, that whole area just goes dark for air traffic controllers. That was the thing that immediately struck me: How clear it is that basic impact analysis and basic risk management seems to have just been completely gone. It even goes a little bit further than that. As a technician, we used to have this rule that you don’t break anything on a Friday. We would actually not schedule certain work past lunch, and it wasn’t to get people off early. It was basically because, if something broke, somebody was going to have to come in on Saturday to fix it. You break it Thursday, people are going to be in on Friday anyway. And then we had the commandment: “Thou shalt not be on the front page of the newspaper,” and it seems like all of that was missing here. L3Harris should know better, because Harris has a history. Back in 2009, they misconfigured something and caused a big network outage. At the time, they had only identified a small handful—I want to say it was like a couple dozen potential areas of impact—and it wound up being like six times that. So they should know better. (Ars note: The 2009 network incident was caused by a Harris technician incorrectly configuring a router that directs air traffic data through the FAA telecommunications network. That error ended up delaying thousands of travelers and grounded hundreds of flights nationwide.) I don’t know how it was done on site, but when I did this work, quite often when contractors were working, I was the person who was there that got them cleared to the fence, that got their badges, that got them scheduled, that did the coordination with the airport authorities, and whatever else we had to do. So there was always an FAA person that was the point of contact, and even on some facilities it might have to be an FAA person that physically pushes the button to turn the system back on. Like in a lot of cases, a contractor cannot do that, and it might be quite literally as simple as just flipping a switch. But the thing is, that action puts that system onto the national airspace system. That has to be done by the government. It can’t be done by a contractor. Now, I don’t know if that’s the case here. I’m just saying it’s typical. Ars Technica: The Professional Aviation Safety Specialists (PASS) union representing FAA technicians put out a statement saying that, “historically contractors have followed a protocol where FAA employees conduct a risk assessment and coordinate the maintenance of the system or equipment.” The union then noted that “PASS-represented technicians” were not involved in the upgrades on the “contractor owned and maintained routers” that led to the ground stop. Mann: In my experience, the union a lot of times is one of the groups that raises their hand and says, “Hey, hold on, we need a qualified person. What does that actually mean? Like, who on this list of people is qualified? Does that mean that they’ve had the class? Does that mean they’ve got a certification? Does that mean they’ve been here for a certain period of time?” But they’ll figure that out, and the answer is not always the same. So, in a different FAA system at a different time, we had to get somebody who had been to a specific set of schools and, in one case, some specific vendor schools. The FAA at that time had only sent like a half-dozen people through those schools because they’re really expensive and really long. They had a couple in the east, a couple in the west, a couple in the middle, and every time they did one of these projects they would grab one of those people and put them on there, because that’s what a qualified person is. Maybe they had somebody for the FAA that was there, but maybe that person was a new hire who was administratively qualified to be on site, meaning they had their background check and their driver’s license and had their access badges and all that. But maybe they didn’t even have a networking background or whatever. They were just the person who had the access and the free time and could be there. But this is certainly a case, especially with the national-size blast radius of a network, that highlights the importance of having that qualified person, whatever that means. That is very much a union push to make sure that we know what the right qualified person means, that we know how to identify those people and that we’re treating those people fairly. It’s important to understand that in the union leadership, they’re all former technicians and engineers, everybody who runs that union. Even the local shop people are all fully qualified. So they all know what’s going on. Their role in all this potentially is to cover that blind spot that management and contractors share. Ars Technica: Based on what you described, it does sound like both the contractor side and the current FAA side were underestimating the potential blast radius for whatever reason. Mann: Well, yeah, and Duffy has even indicated that they mistook the physically remote location for being a network remote location, and those two don’t correspond to each other. Duffy has said that it was the physical smallness and the physical remoteness of the site that somehow let them forget about what they were actually working on. Now, at a national network scale, there’s not really a way to say, “By the way, just in case we might shut the NAS off, we might shut down all the airspace because we’re not sure if this one little router out in Arkansas someplace is going to cause problems.” I have ideas about how this might have happened, but the long and short of it is, when they plugged it in, they didn’t adequately test it before they connected it back to the network some way somehow, and it could have been misconfigured. Maybe there was an approved configuration, somebody didn’t have it, so they just kind of winged it, or it could be that somebody got the wrong configuration. But again, we’ll wait for the root cause to come out, if it comes out sometime later. Ars Technica: What are the main things that should have been done to prevent this from becoming a whole nationwide incident? Mann: An important caveat is, I don’t know if this was a misconfiguration or just somebody making a mistake, or just a bad configuration that shouldn’t have worked. I don’t know how much testing was done. But what should have happened is, with whatever necessary level of technician and engineer engagement, they should have done the blast radius assessment, the risk assessment. Duffy has already conceded they did not do a sufficient risk assessment. He has not said it exactly that way, but he has said that they misunderstood the risk in not so many words. They should have pulled up the network topology and looked at what does this actually touch. One of the systems I used to work on a lot was backup power, so backup generators attached to a radar site. I was working on the generator, but I still marked the radar site as potentially out of service, because the power generator affects the radar site, and if the generator crashes, it’s going to take the radar site down. So in a network, that’s just a million times more complex, because you really need to go, “How far could this reach?” You, of course, want to understand that at the networking level. You want to understand what all that is, but what they need to do is the proper risk assessment. Do whatever engineering, whatever tests, whatever scenario things they want to do and really, really, really embrace that it’s not what you’ve seen before. What is the boundary of what is prohibited by physics? You’re talking about an interconnected network. There is nothing in physics that says a data bit from this router can’t get to the far end of the universe if they’re all connected. And thinking about that kind of stuff, assume it might fail. That’s the first big thing, and that should have taken whatever amount of time it took. Because the benefit with this kind of a system is that same basic impact assessment, you know, plus or minus maybe a couple of relatively small variables, is going to apply to every single one of these installations across the entire country. If they’re doing one of them or doing 1,000 of them, they’re all network-interconnected devices that all do the same thing and all hit the same network. So it sounds bad when you think about doing it for every single site separately, but when you do a full network analysis, you’re like, “We’ve got to do this really hard thing one time, and then we’ve got to assess to see if there’s been any changes as we migrate this network piece by piece. Does the new router significantly change anything? But it’s pretty easy from there. Then, and this is the important part, then they need to behave as if they’re trying to keep that worst-case scenario thing from happening. Or even if they can’t keep it from happening, reduce the impact as much as possible. You’re kind of chopping off the impact from both sides a little bit. Technicians are more prepared to catch it faster, and even if it goes for a little bit, it’s not going to affect nearly as many people. In this case it’s possible—don’t know but it’s possible—if this would have been at 2 or 3 o’clock in the morning, it might have started lighting up some lights somewhere, but they could have potentially shut it down, because you don’t have so many flights and data overloading the network. It’s possible, and just my speculation, it’s possible they could have seen it, flipped it off, let everything calm down, and not had a ground stop at all. Ars Technica: The router installation issue that led to the nationwide ground stop was part of an ongoing FAA effort to modernize its overall system. What might this event indicate about the current state of the US National Airspace System? Mann: This is one of the challenges with the FAA modernization. When they do that properly, nobody sees that there’s that kind of danger out there. It’s kind of the paradox of risk analysis, right? The better you are at it, the less essential people think it is, and that’s the FAA being a victim of its own success for the last 50 years. I’m not in that camp that everything that sounds old needs to be replaced. Floppy disks are totally fine as long as they still work, I don’t care. But the agency has had a hard time modernizing, in part, I believe, because it is so safe and its risk management is normally so good that people lose sight of the fact that there isn’t modernization that needs to be done. It’s not that they don’t want to modernize, and it’s not like the FAA doesn’t want to modernize. It’s just that it’s the old squeaky wheel thing, right? The FAA is so good at what it does that when it comes time to write checks, there’s always a different squeaky wheel that needs the money more. Actually, I’m kind of happy for the current administration keeping this at the front. I don’t necessarily like how they’re doing it, but I do like that they’re making this a live issue in Congress. They’re making this a live issue even out here in different airports. I’ll talk to people and they’re like, “Hey, when are we going to modernize this?” Ten years ago, not too many people out in the public were talking about it. I think the NAS is still just as safe as it’s always been. We can kind of be grateful. Folks probably ought to understand that because they see this on the news, it means it’s weird. It’s unusual. But it’s good that we can be safe and we can still have the conversation about it, about what needs to happen to keep it safe, because we don’t want to have a catastrophic failure somewhere down the line that really is because something failed or really is because something was neglected, and then try to say, “Oh yeah, now we’ve got to shut down our airspace for six months because something happened.” I’m not a blank check person. I do not want [Congress] just writing a check and handing it to Duffy or anybody. I want them to do the proper acquisition oversight, do the proper union involvement, do the proper technical evaluations, all that kind of stuff, because that is money well spent. We could have been really unlucky. We could have lost communication to all the aircraft that were in the air entirely. We could have lost the ability to be able to manage those aircraft, and then we would have been back to doing 1950s, 1960s kind of stuff, like call the airplane and have the airplane tell us where it is. And we probably would have seen some bad outcomes from that. Ars Technica: Many people have noted the coincidence of the FAA’s SMART AI system being deployed nationwide in the same week as the router issue that led to the ground stop. The FAA has said the SMART rollout had no connection to what ensued. What is your view on all this? Mann: I get that. I think it is a coincidence. This particular router was just one of a long series of these replacements being done—they don’t take 1,000 routers and just replace them all on a Thursday. They might do one or two of them a night for a couple of years. Now, just to lean into it a little bit, it could be totally possible that the router’s capabilities, or maybe something that was configured or misconfigured, is a facilitating technology for SMART. In other words, it’s not intended to be used right now, but they’re putting in a placeholder. If you’ve got a deployment, you’ve got three steps. You’ve got an A, B, C step. Well, if you need some ridiculously high-powered, super fast switch or something at step C, you might put that in at step A, knowing you’re not using its full capability, because that’s when it makes sense to do that. Maybe it’s difficult infrastructure, physically hard to do or whatever. But you put the super-fast switch in it at A, then you do step B, then you do step C, and only at that time when you get to step C are you actually going to start using the capabilities in that switch. And those might be years apart. I can imagine something like that being a subtext. But the thing is, the SMART system looks out weeks and months in the future. It doesn’t touch the operation. It reads data, it does some stuff and makes some advising comments and makes some advising predictions, and then tells the air traffic control center folks, “Hey, y’all might want to do this.” To the best of my knowledge and in my opinion, the SMART system doesn’t touch the actual operational data. In other words, it’s not something that reaches out across the whole network. It has certain data sets that are at the control center or wherever it is. There’s certain data that it has access to that it cannot push information into. It just draws from data and does analysis, and then it provides a recommendation to the traffic managers, and then those traffic managers act or don’t act on that information in some way. SMART is a system that reads. You can think of it as a Wall Street guy who reads the ticker and makes some calculations and then hands a spreadsheet to the trader. The analyst is not the trader. The analyst is literally reading the screen, running some math, and then handing a ticket to the trader, and a trader makes the final decision. That’s kind of what happens where the system looks at what’s going on in NAS, what’s forecast to go on in NAS, weather, all that kind of stuff, and then it does some calculations and it effectively hands a slip of paper to the air traffic controllers and says, “Hey, you might want to do this.” But none of that would mean that it’s Skynet getting onto the network or something like that.