Given the recent news from Microsoft and their on-prem email service MS Exchange issue on January 1st: So, here is a blast from the past: Some people have asked me if the recently reported issues with dates on Microsoft Exchange on-prem are a repeat of Y2K. While the similarities are embarrassingly serious, the scope is not as wide as Y2K and the solution is much simpler than its predecessor event from 22 years ago. Please read on for background and details.
Gather around the fire, kids… grandpa will tell you a story
Back in the late 1990’s most legacy systems had been storing the year in two digits: mm/dd/yy locally or dd/mm/yy in the rest of the world. This turned out to be a complete lack of long-term planning given that system’s lifespan turned out to be longer than planned. At the turn of the millennium, the year would create all kinds of havoc in date operations. All new entries for the year 2000 would be entered as “00” and thus be presumed “1900” by most systems.
Did anyone think about it for a second? Nah- it was plain-and-simple short-term-sightedness.
The solution was simple, just use a larger variable that would allow four-digit years to be stored where before two were being used. The challenge was that the scope of this issue was so wide, that every system had to be checked and those that were not designed right had to be corrected. Many of my colleagues and I spent many, many, many months searching and fixing the lack of preparedness of those who coded and designed systems before me. And they did so in such a negligent way, that Y2K-Armaggeddon was upon us.
Billions and billions were spent on fixing this issue. There were predictions of major utility service interruptions. I remember that I was seriously worried and thus had prepared at the time in the same way one prepares for a hurricane: have spare food supplies, and other items.
In the end, thanks to the efforts made by the technology community, it turned out to be only a blip on the radar. It was a great cautionary tale for those in decision-making positions.
Was it really? By the looks of it, seems some people have not learned anything from it.
So, what is the correlation to this Microsoft Exchange issue?
The issue is essentially the same, only dates are stored in other default formats these days. Not that datatypes were not available back then, but people had been using text to store dates and negligently were ignoring not-so-new date and date-time data types available for years. Repeating the same mistake twenty-two years later, makes professional software designers look like amateurs.
MS Exchange on-prem stores the date in a signed 32-bit integer. This new year’s corresponding integer value is beyond the maximum number it can hold, and thus anything past January 1, 2022 errors out with an overflow. Emails hang in transport queues. Emails are not being sent or delivered to your inbox. They are essentially getting stuck in their respective queues.
Now, the scope of Y2K22 is only for those running MS Exchange 2016 and 2019 on-prem. MS Exchange on Microsoft Azure is not affected, so the scope is much more limited than Y2K ever was.
This is still an embarrassment
This issue could have been prevented with a simple patch. Instead, it came in as a surprise to everyone who was then stuck with no emails during the New Year holiday and until Microsoft issues the corresponding update. Don’t despair, there is a quick fix that will get you through the week.
Where is the fix for it?
Microsoft has provided a script that will turn off the date check process to avoid the error while they are working on a permanent patch. You can pick up the script here (https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-exchange-on-premises-transport-queues/ba-p/3049447)
Does this issue have any security implications?
The issue only happens in a date-check process and does not open a window for security breaches nor is it a failure of the AV, malware scanning engine itself. Please make sure you read through the link provided for all the technical details.
The solution can be deployed to your MS Exchange 2016 or 2019 on-prem to correct the issue.
Once the permanent solution is available you will be able to update the platform through the corresponding automatic update process.