A Cascade of Errors: The Nationwide Banking System Meltdown in Japan”

A major mishap in Japan’s banking sector has caused a seismic shift in public trust, which took half a century to build. The National Bank Data Communication System (Zengin System), crucial for inter-bank transfers and foreign exchange transactions, experienced a catastrophic failure on October 10th, affecting 10 major financial institutions including Mitsubishi UFJ Bank and Resona Bank. For two full days, these banks were unable to process transfers, disrupting millions of transactions.

The Zengin System is operated by the National Bank Funds Settlement Network, a subsidiary of the Japanese Bankers Association (JBA), which is overseen by executives from major banks. More than a month since the incident, the issue remains unresolved, reflecting a deep crisis in the banking industry’s core infrastructure.

The problem originated from the Relay Computers (RC), which connect the Zengin System to the banks. On the morning of October 10th, as banks resumed operations after a long weekend, a corrupted program managing inter-bank fees caused the RCs to malfunction, paralyzing transfer operations. Each of the 24 RCs is linked to multiple banks, and unfortunately, the affected banks were using the very RCs that experienced the failure.

The failure’s timing couldn’t have been worse, coinciding with a post-holiday rush and pay-day, leading to a huge backlog. While the Zengin System had backup centers in Tokyo and Osaka for emergencies, this time, the system update involved a peculiar procedure that centralized the RCs in these centers, rendering the backup ineffective during the transition. This decision, prioritizing efficiency over caution, proved disastrous.

The immediate response was patchy. Banks resorted to manual processing, recording transactions on external media before updating the Zengin System. Despite yearly Business Continuity Planning (BCP) drills, this backup plan fell short in practice, leaving hundreds of thousands of transactions unprocessed.

Financial institutions criticized the National Bank Funds Settlement Network for its lackluster response, compounded by a lack of clear communication about the cause or timeline for resolution. The decision to set inter-bank fees at zero yen as a temporary fix meant that the early opportunity for a swift resolution was missed.

The fallout has left financial institutions grappling with how to compensate customers for their losses. While the JBA agreed to sincerely address customer losses caused by the system failure, the extent of compensation and whether the Network will reimburse banks remains unclear.

Presently, the Zengin System operates under a makeshift arrangement, with inter-bank fees set at zero. The failure has raised serious questions about the system’s risk management and crisis response mechanisms. NTT Data, the system’s application provider, has acknowledged its role, but the exact cause remains unknown. Japan’s Financial Services Agency has issued orders to both the Network and NTT Data for a thorough investigation and preventive measures.

The incident has inevitably delayed the planned overhaul of the Zengin System, scheduled intermittently until 2029, with the next update initially set for January 2024. In light of the recent crisis, these plans are now indefinitely postponed.

This incident not only highlights the fragility of Japan’s banking infrastructure but also underscores the critical need for robust risk management and emergency response systems in the financial sector. As the industry grapples with modernizing its systems, this recent failure serves as a stark reminder of the importance of balancing efficiency with safety and reliability.


Leave a Reply

Your email address will not be published. Required fields are marked *