Enterprise infrastructures are changing rapidly as the management and visibility requirements of modern, data-driven applications are outpacing legacy data storage functionality. Gartner confirms that, with artificial intelligence and machine learning driving an explosion in data volume and variety, IT operations are outgrowing existing frameworks. Although insights from today’s vast amounts of structured, semi-structured, and unstructured data can deliver superior value, organizations are currently unable to adequately monitor or analyze this information (and between 60 percent and 73 percent of all data within an enterprise goes unused).
Cloud has been the buzz for more than a decade, and it is now seeing mass adoption among enterprises. Similarly, over the past several years, the size and scope of data pipelines have grown significantly. Just a few years ago, Fortune 500 companies were still experimenting with and testing the efficacy of ‘big data’ as they move toward a digital transformation. Yet today, the majority of those organizations have moved from big data pilots to large-scale, full production workloads with enterprise-level SLAs. Now, these organizations are most interested in maximizing the return on their big data investments and developing new use cases that create new revenue streams.
Data is staying put: Why Big Data needs the cloud
According to recent research from Sapio Research, who surveyed more than 300 IT decision makers, ranging from directors to C-suite, enterprises are overwhelmingly embracing the cloud to host their big data programs. As of January of this year, 79% of the respondents have data workloads currently running in the cloud, and 83% have a strategy to move existing data applications into the cloud. Why?
Modern data applications create processing workloads that require elastic scaling, meaning compute and storage needs change frequently and independently of each other. The cloud provides the flexibility to accommodate this type of elasticity, ensuring the computing and storage resources are available to ensure optimal performance of data pipelines under any circumstances. Many new generation data applications require data workflows to process increased traffic loads at certain times, yet little need to process data at other times – think of social media, video streaming or dating sites. For the many different organizations that encounter this type of resilience monthly, weekly, or even daily, the cloud provides an agile, scalable environment that helps future-proof against these unpredictable increases in data volume, velocity, and variety.
As an example, e-commerce retailers use data processing and analytics tools to provide targeted, real-time shopping suggestions for customers as well as to analyze their actions and experiences. Every year, these organizations experience spiking website traffic on major shopping days like Cyber Monday – and in a traditional big data infrastructure, a company would need to deploy physical servers to support this activity. These servers would likely not be required the other 364 days of the year, resulting in wasted expenditures. With the cloud, however, online retailers have instant access to additional compute and storage resources to accommodate traffic surges and to scale back down during quieter times. In short, cloud computing lacks the headaches of manual configuration and troubleshooting, as with on-premise, and saves money by eliminating the need to physically grow infrastructure.
Lastly, for organizations that handle hyper-secure, personal information (think social security numbers, health records, financial details, etc.) and worry about cloud-based data protection, adopting a hybrid cloud model allow enterprises to keep sensitive workloads on-premises while moving additional workloads to the cloud. Organizations realize they don’t have to be all in or out of the cloud. Sapio’s survey revealed that most respondents are embracing a hybrid cloud strategy (56 percent) for this reason.
The rapid increase in data volume and variety drives organizations to rethink enterprise infrastructures, particularly cloud strategies, and focus on longer-term data growth, flexibility, and cost savings. Over the next year, we will see an increase in modernized data processing systems, ran partially or entirely on the cloud, to support advanced data-driven applications and its emerging use cases.
