Tag Archive for Fail

Sweating the Oldies

Sweating the OldiesIf you’re “sweating assets” and holding off on making major network upgrades, you’re not alone. No Jitter brings our attention to Dimension Data’s annual Network Barometer Report. The report surveyed the system integrator’s worldwide clients and found that the percentage of aging and obsolete devices in today’s corporate networks around the globe is at its highest in six years signaling that the recent global financial crisis may still have a lingering effect today.

 

More than half of devices are aging

RecessionAccording to the article, more than 51% of all devices assessed are now aging (3-5 years old) or obsolete (5 years or older). In addition, 27% of all devices are now ‘later’ in their product life-cycle and at the point where the vendor begins to cut support.

The aging was highest in Asia-Pac and the lowest in the Americas. The survey found equipment in the Americas, was considerably lower at 44%. Dimension Data ascribed this variation to regional macroeconomic conditions.

The oldest equipment

The “sweatiest” companies were in the travel/transportation vertical which had more than 50% aging/obsolete devices according to the study. Other verticals were “sweaty as well:

  • BaystacksConsumer/retail and utilities/energy, all of which had more than 50% aging/obsolete devices.
  • Automotive/manufacturing had an aging/obsolete base of 41%
  • Technology industries had a 37% aging/obsolete gear 37%
  • Construction/real estate was most up to date with 28% aging or obsolete.

The level of aging/obsolete networks hit 45% in 2012 without triggering a refresh and climbed to 48% in 2013 and reached 51% in 2014. That author suggests that either we’re long overdue for another refresh, or else we’re moving to an environment where aging network gear is the rule.

He goes on to speculate that as the BYOD/BYOEverything trend grew over the last 3 years and enterprises diverted technology spending to ad hoc device/cloud purchases, we’re looking at a fundamentally new buying environment.

Obsolete devices fail less

The survey results suggest that “sweating” network assets may be a smart strategy. They analyzed 91,000 trouble tickets from its own practices and found that “Obsolete devices fail less often than current devices. And, when they do fail, problems are quicker to resolve.” Specifically, the survey found that:

  • Obsolete devices had the lowest failure rates (compared with new and aging),
  • Aging devices had the lowest mean-time-to-repair rates among the three classes.

Old equipmentOld hands might be tempted to greet these findings with some variation of the old lament, “They don’t build ’em like they used to,” but the truth might actually be even more flattering to the organization.

Dimension Data suggests that gear that’s been in place while is supported by more mature processes, hence the decreased likelihood of breaking, and faster ability to fix when they do break. Of course, an asset-sweating strategy should have some rationale behind it–it’s’ not about just clinging to old stuff so you don’t have to deal with replacing it.

How to keep the old stuff going

Dimension Data “Tips for Sweating Assets” that included:

  • Have an accurate inventory of your entire network estate.
  • Understand the function of each device and how critical it is to the network’s uptime.
  • Know at which stage in their life cycles these devices are.
  • Have the right operational support strategy in place to resolve any performance issues or outages that may occur, as vendor support will be either limited or unavailable during later life cycle stages.
  • Ensure that the device’s capabilities are not constraining architectural changes, which have driven upgrades in other areas of the network.

rb-

The aging of network gear is not unique. Many firms are still reeling from efforts to survive the depression, recession, economic downturn. In some places, they don’t pick up the trash regularly or replace stained ceiling tiles. The Business Insider says the average age of private fixed assets is at a 50-year high. and here a chart to prove it.

 

Ralph Bach has been in IT long enough to know better and has blogged from his Bach Seat about IT, careers, and anything else that catches his attention since 2005. You can follow him on LinkedInFacebook, and Twitter. Email the Bach Seat here.

Another Cloud Implosion

Another Cloud ImplosionCode Spaces, formerly a popular cloud-based source code hosting service run by AbleBots from New Jersey was forced to close. Infosecurity reports that after an attacker managed to get access to its Amazon (AMZN) Web Services EC2 control panel and delete most of its customers’ data.  According to an explanation on the Code Spaces website, the firm was a victim of DDoS with the apparent attempt to extort “a large fee to resolve the DDOS.”

As the firm attempted to restore control of its machines, the attacker escalated the attack, the site says;

hanging out a closed sign

… the intruder had prepared for this and had already created a number of backup logins to the panel and upon seeing us make the attempted recovery of the account he proceeded to randomly delete artifacts from the panel … We finally managed to get our panel access back but not before he had removed all EBS snapshots, S3 buckets, all AMI’s, some EBS instances and several machine instances. In summary, most of our data, backups, machine configurations, and offsite backups were either partially or completely deleted.

Code Spaces marketed itself as a trusted provider offering “Rock Solid, Secure and Affordable Svn Hosting, Git Hosting and Project Management” and a “full recovery plan” with full redundancy, duplication, and distribution of the data across three different geographical data centers if things went wrong. According to the Infosecurity blog despite the marketing hype the Code Spaces sites is folding up its tent and hanging out a closed sign by saying;

cost of refunding customers who have been left will put Code Spaces in an irreversible financial position Code Spaces will not be able to operate beyond this point, the cost of resolving this issue to date and the expected cost of refunding customers who have been left without the service they paid for will put Code Spaces in an irreversible position both financially and in terms of on-going credibility.

rb-

Another high-profile Cloud Computing service goes bust. Last year when Nirvanix went belly up I wrote about the need for a cloud exit plan. Calum MacLeod, vice president of EMEA at Lieberman Software told CIO.com that security incidents like this are avoidable if companies take effective steps. He suggested firms should implement:

  • Certificate-based authentication along with normal user IDs and passwords,
  • Whitelist applications,
  • A schedule for changing Credentials every few hours for critical applications,
  • Continuous discovery of the systems and applications to check if there were any changes to account settings, like happened to Code Spaces where new privileged accounts were created to allow the attack to continue.

He concludes that the Code Spaces incident reads like a cyberattack 101 scenario, where the failure to properly manage privileged credentials ultimately was the cause of the breach.

Other suggested measure for organizations using AWS would be to enable multi-factor authentication for admin logins. Alternatively, to prevent the wholesale loss of files Amazon Glacier could be used for longer-term data archival, to augment regular offline backups.

Related articles

 

Ralph Bach has been in IT long enough to know better and has blogged from his Bach Seat about IT, careers, and anything else that catches his attention since 2005. You can follow him on LinkedInFacebook, and Twitter. Email the Bach Seat here.

Miami Pauses Tablet Project

Miami Pauses Tablet ProjectFinally some sanity in the rush to push iPads into schools. Miami-Dade school district Superintendent Alberto Carvalho decided to put a hold on the district’s $63 million 250,000+ tablet project. Mr. Carvalho cited problems in LA, Texas, and North Carolina as good reasons to slow down. “Those events put us in a position to say ‘we best pause and learn from their mistakes?” Superintendent Carvalho said according to the Miami Herald.

lack of planningThe article cites Leslie Wilson, CEO of the Michigan-based One-to-One Institute, which advocates successful digital convergence policies. She says that larger iPad efforts have seen a “high rate of failure,” which she blamed on a lack of planning.“We see precious little of that,” she said. (rb- I wrote how about the failure of leadership on these projects here and here.) Ms. Wilson applauded Mr. Carvalho’s decision to step back and look at others’ pitfalls. “When Miami-Dade’s superintendent says ‘I’m pushing the pause button,’ I say bravo. There’s no reason not to get this right,” Ms. Wilson said.

Troubled school tablet projects

Despite the troubled school tablet projects across the country, The Herald says board members push Superintendent Carvalho to move forward. “I hope we will not be delayed in terms of getting devices into kids’ hands as soon as possible,” said School Board Member Carlos Curbelo.

sticker shockLAUSD’s Apple iPad experiment still seems in shambles, costs have just jumped 14%. The LA Times reports that the Apple (AAPL) iPads will cost nearly $100 more apiece — or $770 per tablet. The earlier lower cost estimate for each iPad “preceded the actual procurement process,” the district said in response to questions from The LA Times. “The negotiated discount [i.e. $678] does not go into effect until the district has reached the $400-million spending threshold. “This sticker shock can be avoided, but only after the L.A. Unified School District has spent at least $400 million for the devices. In other words, the district would have to buy nearly 520,000 iPads before getting lower prices.

Other cost increases announced included classroom carts, that charge the iPads and keep them secure, cost rose for this first phase rose from $2.6 million to $3.2 million, according to the paper. The iPad curriculum from Pearson Education Inc. (PSO) is still being developed and not available to those schools that have received their iPads. It is unclear if LAUSD will receive a credit from Pearson for late delivery or if it will ever be available.

Project management failAnother topic at the same meeting was the progress in preparing the schools for iPads. The district reported less than half a percent (40 of the over 1,000 school network) of the schools’ Wi-Fi infrastructure was upgraded for the iPadsso far. (rb- I wonder if they have addressed the proprietary Apple Bonjour issue, and how they are going to limit the number of devices (Apple TVs, Printers, other iPads, Mac’s etc.) that can be selected.

LA parents also have expressed confusion about their responsibility for the devices. And officials have yet to purchase mechanical keyboards that will be necessary to use the iPads on new standardized tests.

Budget shifts

the board shifted iPad project costs to the general fundTo keep the overall budget in check the board shifted iPad project costs to the general fund, which is used for basic operations. That shift is relatively small, about $550,000 at this point, although it’s unclear whether more expenses would be transferred in the future. The price per device, higher than retail, includes a protective case, a limited three-year warranty, technical assistance and training, and one Apple TV setup per 20 students.

The LA Times also reports that these problems have prompted four LA schools to opt-out of the troubled program. Another LA Times article seems to show that the iPad debacle may cost controversial LAUSD Superintendent John Deasy his job.

rb-

no plans or funds on how to sustain their projectsLike I said in my last post on this issue, the LA iPad debacle is driven by the politicians with no real sense of the work required, and I am sure not enough budget to do the work. When I spoke with educational “leaders” about what they were going to do with the iPads they wanted to implement them without solid articulated plans. As is always true in education, there were no plans or funds on how to sustain their projects, they spent all of their money to get the iPads in never mind a plan.

Despite doing the right thing in Miami, the politicians are pushing the Superintendent to just throw the expensive iDevices at the students without any planning.

Related articles

 

Ralph Bach has been in IT long enough to know better and has blogged from his Bach Seat about IT, careers, and anything else that catches his attention since 2005. You can follow him on LinkedInFacebook, and Twitter. Email the Bach Seat here.

All EMU Students Dismissed by Email Mistake

All EMU Students Dismissed by Email Mistake Eastern Michigan University sent a mass email dismissing to the entire student body and an unknown number of recent EMU graduates. Julie Baker at AnnArbor.com reports that the message dismissed the students from the university and canceled all further enrollment. The article says the mass email was sent from the email of Associate Director of Academic Advising Molly D. Weir. EMU says it is investigating how it happened. The dismissal message said in part;

Eastern Michigan University“As a result of your Winter 2012 academic performance, you have been dismissed from Eastern Michigan University … you will be ineligible to register for classes … you will not be eligible to resume coursework at EMU until Summer 2013 at the earliest.

Eastern Michigan University President Susan Martin emailed a statement addressed to students, faculty and staff read: “I deeply apologize for the incorrect email many of our students received this evening indicating they were dismissed from the University. This message was a terrible mistake and I regret the undue alarm and concern it caused.”

take whatever steps are necessary to make sure it never happens againVice President of Communications Walter Kraft denied any claims that this event was the result of a breach of security or a hack. EMU is pointing the finger at a contractor. Mr. Kraft said; “An outside company that we contract with for this notification process, GradesFirst, sent the dismissal message to the entire student body instead of the file of 100 or so students who were supposed to receive it,” he said. “GradesFirst has offered an apology for its role in this matter.

VP Kraft added EMU will continue to investigate to find exactly what went wrong and take whatever steps are necessary to make sure it never happens again according to AnnArbor.com.

rb-

Margaret Thatcher

We are not amused

Why are they outsourcing their communications with their customers? Goes to show that if you let others control your message, they will screw it up.

Didn’t Longfellow say “If you want something done right; do it for yourself?”

 

Ralph Bach has been in IT long enough to know better and has blogged from his Bach Seat about IT, careers, and anything else that catches his attention since 2005. You can follow him on LinkedInFacebook, and Twitter. Email the Bach Seat here.

Michigan Disaster Recovery Test Turns Into Disaster

MichiganState of Michigan IT officials are probably happy for a new week. The State of Michigan IT infrastructure took two big hits last week. The folks in Lansing had a failure on Monday 05-16-11 were nearly 25,000 employees were unable to use the state’s IT network for about three and a half hours, Kurt Weiss, public information officer for the Michigan Department of Technology, Management, and Budget (DTMB), said in a phone interview with InformationWeek. Apparently, an upgrade over the weekend to patch security holes had gone wrong somewhere, Mr. Weiss said. Access to the network was restored by 10:30 a.m.

ESCON cableOn Wednesday 05-18-11 a disaster recovery test at the Michigan DTMB turned into a disaster when a link to a mainframe computer was broken reports MiTechNews. Around noon Wednesday, a link between the test environment and production environment was severed by human error, taking out a mainframe computer. Mr. Weiss told MiTechNews

A fiber link was broken by a state employee … We were working on a disaster recovery test, performing a test on the mainframe. During the test we went from test to real life disaster. The cord between testing and real life was severed. Corrupted files got loaded on the mainframe, and we crashed the mainframe.

Mainframe computerThe “big iron” failure affected many state offices, including 131 Secretary of State branch offices, which run 80,000 daily transactions. Other state operations also were affected, including the departments of corrections, treasury, and human services. Data stored on the mainframe that was affected included the bulk of information about driver’s license and motor vehicle registration in the state,  the ability for police officers to look up driver’s license information (LEIN), or for automobile dealerships to transfer license plates for vehicles that they sold, Mr. Weiss said.

The mainframe was up and running by Wednesday night, but computer applications were still inoperable due to file corruption. The system was finally restored after 5:00 PM on Thursday according to Government Technology. The delay was caused by the data-recovery operations that were necessary as the result of file corruption during the outage.  “We have had outages before, but not to this length or scale or duration,” Mr. Weiss said, “and actually not to this level of complexity. This one has been a much more difficult one to fix compared to the other outages.”

The mainframe that went down last week also is part of an old system that is in need of modernization, Weiss said, but Michigan’s budget woes have so far prevented the state from doing the upgrades it needs. “We do need to modernize all of those applications for the secretary of state,” he told InformationWeek.

Former Gateway Computers CEO and current republican governor Snyder, when asked about the outage, told MiTechNews it is another reason the state has to get the budget approved so the state can focus on upgrading the old computer equipment used by the Michigan government. Some of this equipment is more than 30 years old.

The DTMB IT department is doing a root cause analysis of both incidents and plans to publish a “lessons learned” review of them once that is complete, Weiss said. No data was lost in either incident, although some data files were corrupted during the second and had to be restored through tape backup, he said.

IT officials are re-evaluating how to do such tests in the future in light of the incident, and another test will not be performed until this study is complete, he said.

rb-

snyder

Just put it back in the cow box

So now the boys and girls in Lansing know what it is like to work with ancient equipment because the Governor is cutting funding to everything to give a tax cut to businesses. I doubt that Snyder or his cronies have ever been in line for hours just to get new tabs. I have. Michigan needs to invest in its people and infrastructure not tax breaks for businesses.

What do you think?

Invest in people and infrastructure so people want to stay in Michigan?

or

Cut spending and raise taxes to give businesses more profits?

 

Ralph Bach has been in IT long enough to know better and has blogged from his Bach Seat about IT, careers, and anything else that catches his attention since 2005. You can follow him on LinkedInFacebook, and Twitter. Email the Bach Seat here.