MANILA, Philippines – Imagine kicking off 2023 by flying to Manila from Hong Kong, only for the plane to make an abrupt U-turn 15 minutes away from the runway. Or picture being stuck at an airport terminal all night while someone barks into a broken megaphone, trying to restore order.
These were only some of the chaotic scenes that happened on New Year’s Day, as technical issues paralyzed Philippine airspace for hours on end, leaving more than 78,000 passengers stranded.
More than four months later and hoping to prevent a repeat of the disaster, the Civil Aviation Authority of the Philippines (CAAP) has finally repaired the faulty uninterruptible power supply (UPS) that was partly blamed for the air traffic system malfunction.
“The CAAP successfully completed the corrective maintenance activity at the Air Traffic Management Center early this morning,” the agency said in a statement on Wednesday, May 17. “Air traffic controllers reported no flight disruptions during the maintenance operation.”
Edgardo Diaz, CAAP deputy director general for operations, also confirmed that the UPS units have been replaced.
“Wala po tayo naging aberya. Hindi po nagkaroon ng shutdown. Wala po tayong mga flights na hindi po natugunan (We didn’t run into anomalies. We didn’t have any shutdown. We didn’t have any flights we didn’t address). In other words, the operation went normal,” he said in a Wednesday interview on DZBB.
So does this mean that what happened on January 1 could never happen again? The short answer is no – at least not until interventions address the root causes of the problem.
“We can only exert our best effort,” CAAP Director General Manuel Tamayo told Rappler a month after the incident. And even after further repairs, the system is not foolproof.
“Ito, with the proper maintenance, bagong parts, bagong machine, puwedeng possibility of a shutdown is less,” Diaz told ABS-CBN news, adding that it does not completely eliminate the possibility of another glitch.
(With these replacements, proper maintenance, new parts, a new machine, the possibility of another shutdown is lessened.)
That’s because the New Year’s Day air traffic fiasco was a case of failure to maintain crucial equipment for months – and even years. Regular maintenance checks by qualified personnel could have spotted the melted wires, loose connections, and faulty circuit breaker long before one fault kicked off this series of events.
For some equipment, like the automatic voltage regulator (AVR), CAAP was fully aware that it was broken and neglected to replace it for months – all until disaster struck.
What caused the New Year’s Day fiasco?
In the hazy aftermath of the January 1 incident, CAAP’s narrative about what went wrong frequently changed. CAAP initially pointed to a defective UPS as the reason for the power outage, saying that one of the UPS’ blowers “conked out.” But since then, the agency has clarified that both UPS units were still functional when the incident occurred.
So why did CAAP conduct an emergency procurement of two UPS units if both still worked? The agency said it was merely a preventive measure for future breakdowns.
The faulty UPS units that CAAP replaced were not the root cause of the power outage. To understand this, we have to separate the January 1 “glitch” into two main events: (1) the simultaneous tripping of both UPS units, and (2) the overvoltage that damaged communications and navigations equipment.
Changing the old UPS units only addresses issues in the first main event – and even then, the units did not exactly fail. They deenergized the air traffic management equipment, which was a safety feature to prevent damage when an electrical fault is detected.
What made things worse were the other lingering problems throughout the system. Here are some identified by the Senate’s independent audit team, composed of members from the Institute of Integrated Electrical Engineers of the Philippines:
- Lack of a functioning AVR
- Short circuit that occurred on a power transfer switch
- Lack of functioning redundancy to back up the system
Let’s take a closer look at each of these.
Among the most glaring issues was CAAP’s failure to repair its AVR, which has been broken since August 2022. CAAP claimed that its AVR is used only when the UPS must undergo maintenance, and that systems can still function without it.
However, the audit team disputed this, highlighting that the AVR is a critical part of the system.
“The AVR provides regulated clean power to the UPS bypass, thereby allowing its normal operation. The lack of a functioning AVR could lead to another catastrophic incident. In case of fluctuations in the electric supply, the system is still vulnerable to power interruptions and damage to equipment,” the Senate committee report on the incident noted.
Under usual engineering practice, the average time to repair crucial equipment like an AVR should be just eight hours. On New Year’s Day, CAAP’s AVR had been broken for four months. CAAP committed to have it repaired by February 9, but it took them until May 3 to fulfill this. Why the delay? An interagency report stated that the AVR was still going through the procurement process – unlike the UPS units that CAAP expedited through an emergency procurement.
This meant that from August 27, 2022, until the noon of January 1, 2023, there was no functioning AVR.
The consequences of this became evident when a fault occurred at 9:49 am, January 1, on a power transfer switch used by one of the communications equipment. The fault caused both UPS units to trip.
“CAAP technicians tried to close the maintenance bypass switch of the UPS to provide power via the bypass line. However, they failed to do so as the AVR was still in the ‘0’ position or switched off and the bypass supply was still de-energized,” the Senate report said.
Short circuit of power transfer switch
The power transfer switch, like the name suggests, transfers power to other sources in the event of a problem. On January 1, the switch itself short circuited, and protection devices downstream of the UPS also failed to stop the fault from doing further damage.
A short circuit, the Senate report noted, is considered “the worst power anomaly that not even a good UPS can sustain.”
The most likely cause of the short circuit is overheating as the switch input terminal and wiring were found to be melted. The independent audit team noted that this could have been prevented through regular inspection and maintenance by qualified personnel. (READ: Senators find no ‘proper maintenance’ of air traffic facilities)
No redundancy of equipment
Even though there were two UPS units powering critical air traffic management equipment, they were not redundant or independent of each other. The audit team reported that both UPS units had a parallel load-sharing scheme, meaning that a failure in either one would make both lose functionality.
“As evident in the New Year’s Day incident, a single fault or power outage brought down the entire CNS/ATM system,” the Senate committee report said.
Diaz confirmed that the latest upgrade done on Wednesday to the UPS now addresses this problem.
“Redundant na tayo. Halimbawa magkaroon po ng aberya ‘yung isa, ‘yun pong isa naman ay hindi maapektuhan. Ito po kasing dating nakalagay po, we found out na ‘yun pong dalawang UPS [are] not working independently. So pagkadiperensiya ‘yung isa, apektado din po ‘yung isa,” the CAAP deputy director general for operations said.
(We have redundant systems now. For example, if a problem occurs in one, it won’t affect the other. In the old system, we found out that both UPS units were not working independently. So if something happens to one, the other is also affected.)
Overvoltage destroys equipment
By 10:18 am, technicians were hoping to power up the UPS units by switching to generator sets, but power wasn’t coming into the units because the AVR was still turned off – the same way it had been since August 2022.
It wasn’t until an hour and 40 minutes later that technicians finally placed the AVR on maintenance bypass mode and got one of the UPS units up and running.
But when that happened, an overvoltage occurred, literally frying crucial equipment, like the very small aperture terminal (VSAT), which receives satellite data used by planes and air traffic management systems.
“Sparks and smoke were observed at the AMHS and Geo stationary satellite. One circuit breaker was turned off after they found it had an overcurrent of 344 volts,” the Senate report noted.
The overvoltage may have occurred because of a malfunctioning circuit breaker. This could have multiple possible causes, such as the loose connection of a neutral wire, which an interagency report noted could be “due to vibrations from high voltage running through the line.”
It could also be that the circuit breaker was “overdutied,” meaning that the available current is greater than the short circuit capacity of the circuit breaker – again, another engineering design flaw.
What does this all mean? The root causes of the January 1 fiasco can be pinned down to the lack of proper and regular maintenance work, and the lack of engineering guidelines and competency in the design of the system.
“There was an inadequate maintenance program on critical CNS/ATM equipment based on the findings of both CAAP’s safety investigation team and the Senate’s independent audit team,” the report said, highlighting a lack of regular training validation and proper maintenance training for personnel.
The technicians onsite were not familiar with how to handle the circuit breaker. They were also not allowed to open the circuit breaker for regular inspections. Tamayo explained that CAAP was “only authorized to do a certain level of maintenance,” which meant that the breaker had to be kept sealed. Checking the inner workings of circuit breaker, it seemed, was a duty reserved for its manufacturer.
Besides this, CAAP had no step-by-step troubleshooting guidelines for technicians and engineers to follow when something does go wrong. Perhaps worse, there was even an inaccurate wiring diagram displayed for the UPS, which might have further delayed the restoration of power.
CAAP also had no backup system for communications and internet connection. This meant that when power died in the Air Traffic Management Center, the agency couldn’t even immediately issue a Notice to Airmen because there was no internet.
Neither did it have a master contingency plan in the event of a disruption of air traffic services – be it due to electrical failures, military activity, or severe weather. The plans that CAAP did have on hand did not meet International Civil Aviation Organization standards.
What should be done?
There wasn’t one party responsible for everything that happened on New Year’s Day. After all, the current air traffic management system of the Philippines – described by Transportation Secretary Jaime Bautista as 10 years behind Singapore – was dreamed up way back in 1997.
Since then, the system and its issues have been passed along six presidents, 11 transportation secretaries, eight CAAP directors, and “many batches of poorly equipped engineers,” according to the Senate report.
So how do we fix this? The most immediate way forward is to follow several technical remedies suggested in the Senate report:
- Replace the broken AVR
- Conduct a power quality analysis and energy audit, as well as a vulnerability assessment and penetration test of the air traffic management system
- Modernize the country’s Air Navigation Service Equipment
- Procure backup supplies of critical equipment and spare parts for the UPS
- Procure an automatic fire suppression system for the UPS and battery rooms
- Procure explosion protection-rated equipment for the battery room
CAAP has acted on some of these recommendations – or at least it has done the first. That hasn’t stopped the agency from confidently proclaiming that its repaired UPS units and AVR “guarantee a continuous safe air traffic management within the Philippine Flight Information Region” and that “flight disruptions caused by equipment malfunction are now a thing of the past.”
“With the goal to address the lack of power system redundancy and robustness that contributed to the technical glitch last January 1, the automatic voltage regulator was made operational again and 2 New UPS were installed,” CAAP Air Navigation Service Chief Engineer Alex Balde said in a statement on Thursday, May 18.
The Senate also suggested that CAAP improve its personnel by hiring qualified electrical engineers and offering more competitive compensation packages. This comes after the agency revealed that the Philippines was losing air traffic controllers to countries that pay up to nine times more than rates here.
On the legislative side, the Senate report also urged lawmakers to exempt CAAP from the GOCC Governance Act and Salary Standardization Law. This would theoretically allow the agency to match these salaries and prevent the “brain drain phenomenon in CAAP.”
Congress is also looking to separate the regulatory and operational functions of CAAP, leaving it as the regulator of civil aviation matters and transferring the responsibility of air traffic management to another entity.
“As of now, CAAP is the regulator of civil aviation, it is the operator of the airports, and in any accidents or incidents or violations of other regulations, it is also the investigator – all built into one. So as you can clearly see, the conflict of interest in CAAP as to their duties and other matters that concerns them is really in big question,” Benjamin Solis, aviation advisor of the Safe Travel Alliance, said in a House committee hearing on Thursday.
CAAP itself supports this move. Among the suggestions floated was to transfer these functions from CAAP to a newly-created government authority – the Philippine Airports Authority – that would oversee the management of all airports in the country.
The Department of Transportation was also urged to fast-track the conduct of feasibility studies on the privatization of the Ninoy Aquino International Airport (NAIA). Bautista has repeatedly emphasized the openness of the Marcos administration to the idea of privatizing all four terminals of NAIA.
In the meantime, a consortium of conglomerates has already presented the department with a P100-billion plan to rehabilitate NAIA, something that the transportation secretary is already assessing.
“The initial study is to have a solicited [proposal]. That’s the reason why we engaged ADB (Asian Development Bank) for help, for us to be able to entertain a solicited proposal. But since there is this unsolicited proposal, we have to work on it…. We are given 35 days to do the completion check, and that’s what we’re doing,” Bautista told reporters on May 10.
And just days after these rehabilitation plans became public, flights at NAIA Terminal 3 were again paralyzed by a power outage on Labor Day, another incident among the many that continue to plague the country’s main gateway. – Rappler.com
There are no comments yet. Add your comment to start the conversation.