If you are of a certain age, you remember Y2K. While I was not rewriting COBOL programs, I played my part. I spent the last half of 1999 scheduling after-hours downtime to update Compaq 1900 and 2500 servers with BIOS updates on a floppy disk. Hoping and praying the servers would come back up after the floppy disk stopped grinding. As I recall only two Compaq Proliant 2500‘s failed the BIOS upgrade and only one was DOA.
All the fun of Y2K was because memory space was too small to accommodate the fancy new year 2000 without thinking it was 1900. Now a similar memory size problem could cause internet disruptions very soon. The problem is called 768k Day.
768k Day is when the size of the global BGP routing table is expected to exceed 768,000 entries. at TechRadar explains that on August 12, 2014, a similar problem, occurred after Verizon (VZ) advertised 15,000 new BGP routes to the internet. Verizon’s actions caused the global BGP routing table, a file that holds the IPv4 addresses of all known internet-connected networks, to exceed 512,000 causing the 512K Day crisis.
The TechRadar article explains that in 2014, ISPs and others had configured the size of the memory for their router TCAMs (ternary content-addressable memory) for a limit of 512K route entries and some older routers suffered memory overflows which led their CPUs to crash. These crashes created significant packet loss and traffic outages across the internet with even large provider networks being affected. ZDNet says companies like Microsoft, eBay, BT, Comcast, AT&T, Sprint, and Verizon, were all impacted by 512K day
Engineers and network administrators rushed to apply emergency firmware patches to set a new upper limit which in many cases was 768k entries. The seeds of the 2019 768k crisis were sown.
Mr. speculates that in 2019 most of the large providers who felt 513K day’s impact have likely updated and maintained their infrastructures reasonably well which could lead to fewer outages. He says that there are still ‘soft spots’ smaller ISPs, data centers, and other providers who are part of the Internet’s fabric where maintenance on legacy routers and network equipment can be neglected or missed more easily.
These are the places that most likely see some issues or outages due to 768k Day. These outages will create significant packet loss and traffic outages that could have a ripple effect and sweep upstream and affect larger provider networks. network intelligence firm ThousandEyes writes, “Given the sheer size and unregulated nature of the Internet, it’s fair to say that things will be missed.”
rb-
To prepare for any potential disruptions, it is a good idea to perform some preventative maintenance on any routers that receive full internet routes. Jim Troutman, Director at the Northern New England Neutral Internet Exchange (NNENIX) told ZDNet,
The 768k IPv4 route limit is only a problem if you are taking ALL routes. If you discard or don’t accept /24 routes, that eliminates half the total BGP table size.
There is still a little time left before 768K day, at 2019-06-21 16:00 UTC 06/21/2019 the Regional Internet Registry for Europe, the Middle East, and parts of Central Asia (RIPE) reports that 86.9% of the IPv4 BGP tables they monitor are below 768K. Click here for current results
What is the big deal? Network intelligence firm ThousandEyes points out that there are many outage events that happen every day, especially on the fringes of the Internet. The number of garden variety outages could get amplified because of 768k day-related issues over the next few weeks.
Aaron A. Glenn, a networking engineer with AAGICo Berlin told ZDNet,
The Cisco 6500/7600 product line was extremely popular for an exceptionally long time in many, many places,” so don’t be surprised if some networks go offline because they forgot about 768k Day and didn’t prepare.
Related articles
- Target Suffers Pair of Outages and It’s a Reminder We’re All Just 1 Technology Fail Away From Chaos (Inc.)
Ralph Bach has been in IT long enough to know better and has blogged from his Bach Seat about IT, careers, and anything else that catches his attention since 2005. You can follow him on LinkedIn, Facebook, and Twitter. Email the Bach Seat here.