State of Michigan IT officials are probably happy for a new week. The State of Michigan IT infrastructure took two big hits last week. The folks in Lansing had a failure on Monday 05-16-11 were nearly 25,000 employees were unable to use the state’s IT network for about three and a half hours, Kurt Weiss, public information officer for the Michigan Department of Technology, Management, and Budget (DTMB), said in a phone interview with InformationWeek. Apparently, an upgrade over the weekend to patch security holes had gone wrong somewhere, Mr. Weiss said. Access to the network was restored by 10:30 a.m.
On Wednesday 05-18-11 a disaster recovery test at the Michigan DTMB turned into a disaster when a link to a mainframe computer was broken reports MiTechNews. Around noon Wednesday, a link between the test environment and production environment was severed by human error, taking out a mainframe computer. Mr. Weiss told MiTechNews
A fiber link was broken by a state employee … We were working on a disaster recovery test, performing a test on the mainframe. During the test we went from test to real life disaster. The cord between testing and real life was severed. Corrupted files got loaded on the mainframe, and we crashed the mainframe.
The “big iron” failure affected many state offices, including 131 Secretary of State branch offices, which run 80,000 daily transactions. Other state operations also were affected, including the departments of corrections, treasury, and human services. Data stored on the mainframe that was affected included the bulk of information about driver’s license and motor vehicle registration in the state, the ability for police officers to look up driver’s license information (LEIN), or for automobile dealerships to transfer license plates for vehicles that they sold, Mr. Weiss said.
The mainframe was up and running by Wednesday night, but computer applications were still inoperable due to file corruption. The system was finally restored after 5:00 PM on Thursday according to Government Technology. The delay was caused by the data-recovery operations that were necessary as the result of file corruption during the outage. “We have had outages before, but not to this length or scale or duration,” Mr. Weiss said, “and actually not to this level of complexity. This one has been a much more difficult one to fix compared to the other outages.”
The mainframe that went down last week also is part of an old system that is in need of modernization, Weiss said, but Michigan’s budget woes have so far prevented the state from doing the upgrades it needs. “We do need to modernize all of those applications for the secretary of state,” he told InformationWeek.
Former Gateway Computers CEO and current republican governor Snyder, when asked about the outage, told MiTechNews it is another reason the state has to get the budget approved so the state can focus on upgrading the old computer equipment used by the Michigan government. Some of this equipment is more than 30 years old.
The DTMB IT department is doing a root cause analysis of both incidents and plans to publish a “lessons learned” review of them once that is complete, Weiss said. No data was lost in either incident, although some data files were corrupted during the second and had to be restored through tape backup, he said.
IT officials are re-evaluating how to do such tests in the future in light of the incident, and another test will not be performed until this study is complete, he said.
rb-
Just put it back in the cow box
So now the boys and girls in Lansing know what it is like to work with ancient equipment because the Governor is cutting funding to everything to give a tax cut to businesses. I doubt that Snyder or his cronies have ever been in line for hours just to get new tabs. I have. Michigan needs to invest in its people and infrastructure not tax breaks for businesses.
What do you think?
Invest in people and infrastructure so people want to stay in Michigan?
or
Cut spending and raise taxes to give businesses more profits?
Related article
- How To Implement a Disaster Recovery Strategy (readwriteweb.com)