Data Center Knowledge had an article by Steven Rodin, CEO of Storagepipe Solutions, that lays out the challenges that those of us charged with managing backups face every day. Storagepipe Solutions, which has been a provider of online backup services for business since 2001, has identified several emerging storage trends that organizations will need to overcome in the future.
In the early days, the author says, organizations were primarily concerned with data protection, encryption and automation. The era of “Big Data” has changed those demands. The new demands are overwhelming most backup and storage systems. The article cites data from IBM (IBM), which claims that worldwide annual data production has actually exceeded worldwide storage capacity. Big Blue believes that demand for storage capacity is growing nearly 60 percent a year. The gap between the data that organizations produce and their ability to store it will continue to grow for years.
The Storagepipe Solutions CEO identified a number of important storage trends which are accelerating the growth rate of corporate data. He provided a few of the most important factors.
Cheaper Storage Hardware
Hard drive capacity has fallen exponentially in price ever since Moore’s Law was introduced. This has changed attitudes to backups. The article says that today, hardware is so cheap and abundant that attitudes have shifted to a “Better keep this. We may need it someday” mentality.
New technologies, such as advancements in compression, deduplication and hardware virtualization, have improved overall storage utilization and further accelerated the rate at which the cost-per-gigabyte of storing data is falling.
Cheap and Abundant Bandwidth
Internet bandwidth is no longer a bottleneck. Bandwidth availability has accelerated the growth of file sharing and online storage. Now large files are copied and distributed at an exponential rate which has caused duplicate data to become a major source of storage waste and data growth. The CEO of the firm based out of Toronto, calculates that if one person shares a 1GB file with 500 people, that’s half a terabyte of storage consumption.
Business is Going Paperless – Email has replaced letters, eBooks and tablets have nearly replaced paper books, and digital imaging has replaced photographs and x-rays. Not only are paperless offices better for the environment, but Mr. Rodin writes, they are also more productive, flexible and better able to extract value from their business data. Many industries are using more and more video (which is highly storage intensive) for marketing online, security and communication.
Enhanced Automated Data Collection Capabilities
Automated data collection is one of the fastest-growing areas in the “big data” space. With every move we make, the article says we’re generating GPS data, web traffic statistics, power usage data, surveillance video, and a broad range of data which companies and governments are collecting.
The author calls automated data collection the “Pandora’s Box” of the big data revolution. The information being collected about us through the electronic devices we use every day could present a threat to our privacy, but they also have the potential to offer tremendous value to society.
Advances in Data Analysis Technology
The blog says that until recently, data analysis was almost exclusively performed on structured relational databases, maintained and organized by humans. But now, a new approach to data storage which focuses on rapid analysis and processing of vast data volumes. Technologies like Hadoop, Cassandra, MapReduce and NoSQL have given birth to a whole new class of services, and have revolutionized the way organizations think about the data they collect. Organizations can now get more insight into their internally generated business data by integrating external feeds and databases into their reporting and analysis.
The Growing Strategic Importance of Data
In the past, data was simply a tool which assisted in decision-making and helped companies execute on their strategic objectives. But recently Google (GOOG), Facebook (FB), Apple’s (AAPL) iTunes and other brands have built their entire corporate strategy around the data they own. The DCK article states, information is power, and it’s now more powerful than ever.
Regulatory Compliance
Even if companies wanted to cut the amount of data they store, they wouldn’t always be able to. Laws like PIPEDA, HIPAA, Sox404 and many others are forcing companies to keep historical archives of their exponentially growing business data going back several years.
As this data grows, storage increasingly becomes a major business problem. Also, companies must plan for cost-efficient search and retrieval of these large historical data volumes to stay ready for an unexpected electronic discovery request.
As the scale and complexity of big data storage grows, it’ll quickly reach a point where manual handling is no longer practical, desirable, economical, or even possible. Automation will become absolutely essential when it comes to backing up big data.
Many big data applications have serious privacy implications for the customers that benefit from their use. So security will become a top priority for backup administrators. Gone are the days of unencrypted backup tapes.
The big data applications has created a whole new class of applications built on real-time data. These applications require much more frequent backups to optimize Recovery Point Objectives (RPOs). Strategic big data apps will need minimized downtime. This means smaller backup windows, built-in redundancy, and server fail-over to disaster recovery sites.
That’s why many organizations are opting to outsource their data backups by partnering with experts who run ahead of the trends and who can help with the complexity of some situations.
Related articles
Ralph Bach has been in IT long enough to know better and has blogged from his Bach Seat about IT, careers and anything else that catches his attention since 2005. You can follow him at LinkedIn, Facebook and Twitter. Email the Bach Seat here.
The folks at Whocalledmyphone.net have given us this excellent infographic. It traces the battle between the right to privacy and the prying eyes of the government. The infographic follows the erosion of privacy from the Bill of Rights to Prism. For more details, you can also check out the EFF page Timeline of NSA Domestic Spying.
Anyone who has ever shopped for a replacement ink cartridge knows they’re not cheap. In fact, printer ink is more expensive per gallon than gasoline or the blood running through your veins. This 