{"id":3568,"date":"2011-11-07T12:00:57","date_gmt":"2011-11-07T12:00:57","guid":{"rendered":"http:\/\/www.markwilson.co.uk\/blog\/?p=3568"},"modified":"2017-01-14T13:55:14","modified_gmt":"2017-01-14T13:55:14","slug":"sql-server-and-hadoop-unlikely-bedfellows-but-a-powerful-combination","status":"publish","type":"post","link":"https:\/\/www.markwilson.co.uk\/blog\/2011\/11\/sql-server-and-hadoop-unlikely-bedfellows-but-a-powerful-combination.htm","title":{"rendered":"SQL Server and Hadoop – unlikely bedfellows but a powerful combination"},"content":{"rendered":"

Big Data is hard to avoid \u2013 what does Microsoft\u2019s embrace of Hadoop mean for IT Managers?<\/em><\/p>\n

There are two words that seem particularly difficult to avoid at the moment: big data.\u00a0Infrastructure guys instinctivly shy away from data but such is its prevalence that big data is much more than just the latest IT buzzword and is becoming a major theme in our industry right now<\/p>\n

But what does \u201cbig data\u201d actually mean? It’s one of those phrases that, like cloud computing earlier, it is being “adopted” by vendors to mean whatever they want it to.<\/p>\n

The\u00a0McKinsey Global Institute describes big data<\/a>\u00a0as \u201cthe next frontier for innovation, competition and productivity\u201d but, put simply, it\u2019s about analysing masses of unstructured (or semi-structured) data which, until recently, was considered too expensive to do anything with.<\/p>\n

That data comes from a variety of sources including sensors, social networks and digital media and it includes text, audio, video, click-streams, log files and more. Cynics who scoff at the description of \u201cbig\u201d data (what\u2019s next, \u201chuge\u201d data?) miss the point that it\u2019s not just about the volume of the data (typically many petabytes) but also the variety and frequency of that data. Some even refer to it as \u201cnano data\u201d because what we\u2019re actually looking at is massive sets of very small data.<\/p>\n

Processing big data typically involves distributed computer systems and one project that has come to the fore is\u00a0Apache Hadoop<\/a>\u00a0\u2013 a framework for development of open-source software for reliable, scalable distributed computing.<\/p>\n

Over the last few weeks though, there have been some significant announcements from established IT players, not all of whom are known for embracing open source technology. This indicates a growing acceptance for big data solutions in general and specifically for solutions that include both open- and closed- source elements.<\/p>\n

When Microsoft released a\u00a0SQL Server-Hadoop (SQOOP) Connector<\/a>,there were questions about what this would mean for CIOs and IT Managers who may previously have viewed technologies like Hadoop as a little esoteric.<\/p>\n

The key to understanding what this would mean would be understanding the two main types of data: structured and unstructured. Structured data tends to be stored in a relational database management system (RDBMS), for example Microsoft SQL Server, IBM DB2, Oracle 11G or MySQL.<\/p>\n

By structuring the data with a schema, tables, keys and all manner of relationships it\u2019s possible to run queries (with a language like SQL) to analyse the data and techniques have developed over the years to optimise those queries. By contrast, unstructured data has no schema (at least not a formal one) and may be as simple as a set of files.\u00a0 Structured data offers maturity, stability and efficiency but unstructured data offers flexibility.<\/p>\n

Secondly, there needs to be an understanding of the term \u201cNoSQL\u201d.\u00a0 Commonly misinterpreted as an instruction (no to SQL), it really means\u00a0not only<\/em>\u00a0SQL \u2013 i.e. there are some types of data that are not worth storing in an RDBMS.\u00a0 Rather than following the database model of extract, transform and load (ETL), with a NoSQL system the data arrives and the application knows how to interpret the data, providing a faster time to insight from data acquisition.<\/p>\n

Just as there are two main types of data, there are two main types of NoSQL system: key\/value stores (like MongoDB or Windows Azure Table Storage) can be thought of as NoSQL OLTP; Hadoop is more like NoSQL data warehousing and is particularly suited to storing and analysing massive data sets.<\/p>\n

One of the key elements towards understanding Hadoop is understanding how the various Hadoop components work together. There’s a degree of complexity so perhaps it’s best to summarise \u00a0by saying that the Hadoop stack consists of a highly distributed, fault tolerant, file system (HDFS) and the MapReduce framework for writing and executing distributed, fault tolerant, algorithms. Built on top of that are query languages (live Hive and Pig) and then we have the layer where Microsoft\u2019s SQOOP connector sits, connecting the two worlds of structured and unstructured data.<\/p>\n

The trouble is that SQOOP is just a bridge \u2013 and not a particularly efficient one either \u2013 working on SQL data in the unstructured world involves subdivision of the SQL database so that MapReduce can work correctly.<\/p>\n

Because most enterprises have both the structured and unstructured data, we really need tools that allow us to analyse and manage data in multiple environments \u2013 ideally without having to go back and forth. That\u2019s why there are \u00a0so many vendors jumping on the big data bandwagon but it seems that a SQOOP connector is not the only work Microsoft is doing in the big data space:<\/p>\n