Why Would We Use Big Data?

Marketing Campaign Analysis, a target audience that identifies the “right” person for the “right” products.

Big Data allows marketing teams to evaluate large volumes from new data sources, like click-stream data and call detail records, to increase the accuracy of the analysis.

Web 3.0:  With over a billion users, today's Internet is arguably the most successful human artifact ever created

This 6-minute video outlines the basic themes of the European Union's Future Internet initiative. These include: an Internet of Services, where services are ubiquitous; an Internet of Things where in principle every physical object becomes an online addressable resource; a Mobile Internet where 24/7 seamless connectivity over multiple devices is the norm; and the need for semantics in order to meet the challenges presented by the dramatic increase in the scale of content and users."

Isn’t Big Data, Just More Data?

  • Which attributes would alert us to the use of Big Data? Aren’t we just using OLTP (On-Line Transactional Processing) and OLAP (On-Line Analytical Processing)? 
  • Big Data describes large volumes of a wide variety of data collected from various sources across the enterprise including transactional data from
  • Enterprise applications
  • Databases
  • Social media data
  • Mobile device data
  • Unstructured data/documents,
  • Machine-generated data

Structured V. Unstructured – Keeping Up

  • We, as auditors and business advisors, need to gain comfort with these new technologies, understand their benefits and risks, add capabilities to our workforce, and establish ground rules for both application and project governance.
  • Traditionally, “Structured” Data is getting faster and bigger.  Machines can’t keep up
  • Traditionally “Unstructured” Data leads the technologies that would allow their management and distribution, and Research

What are the New Technologies?

Reference Slide: Hadoop

  • Hadoop was built to address the challenge of indexing the entire World Wide Web every
  • 2004 - Google developed a paradigm called MapReduce
  • 2005 - Yahoo! started Hadoop as an implementation of MapReduce, 2007 - open-source project
  • Hadoop has the basic constructs needed to perform computing:
  • It has a file system, a language to write programs, a way of managing the distribution of those programs over a distributed cluster, and a way of accepting the results of those programs. Ultimately the goal is to create a single result set.
    With Hadoop, big data is distributed into pieces that are spread over a series of nodes running on commodity hardware.
  • Reference Slide: Pig

The Apache Pig project is a high-level data-flow programming language and execution framework for creating MapReduce programs used with Hadoop.

  • The abstract language for this platform is called Pig Latin and it abstracts the programming into a notation, which makes MapReduce programming similar to that of SQL for RDBMS systems.
  • Pig Latin is extended using UDF (User Defined Functions), which the user can write in Java and then call directly from the language.

Reference Slide: Job Tracker

  • A Job Tracker is the entry point for a “map job” or process to be applied to the data. A map job is typically a query written in java and is the first step in the MapReduce process. The Job
  • Tracker asks the name node to identify and locate the necessary data to complete the job. Once it has this information it submits the query to the relevant named nodes.
  • Any required processing of the data occurs within each named node, which provides the massively parallel characteristic of Map Reduce.  When each node has finished processing, it stores the results. The client then initiates a "Reduce" job.
  • The results are then aggregated to determine the “answer” to the original query. The client then accesses these results on the filesystem and can use them for whatever purpose.

Reference Slide: Hive and HiveQL

  • Apache Hive is a data warehouse infrastructure built on top of Hadoop (originally by Facebook) for providing data summarization, ad-hoc query, and analysis of large datasets.
  • Hive provides a mechanism to project structure onto this data and query the data using an SQL-like language called HiveQL.
  • HiveQL is used for business intelligence and visualization tools.

Reference Slide: HBase, HCatalog

  • HBase is a non-relational database that runs on top of the Hadoop file system (HDFS). It is columnar and provides fault-tolerant storage and quick access to large quantities of sparse data. It also adds transactional capabilities to Hadoop, allowing users to conduct updates, inserts, and deletes. It was originally developed by Facebook to serve their messaging systems and is used heavily by eBay as well.
  • HCatalog is a table and storage management service for data created using Apache Hadoop. It allows interoperability across data processing tools such as Pig, Map Reduce, Streaming, and Hive and a shared schema and data type mechanism.

Reference Slide: Flume, Oozie

  • Flume - is a system of agents that populate a Hadoop cluster. These agents are deployed across an IT infrastructure and collect data and integrate it back into Hadoop.
  • Oozie - coordinates jobs written in multiple languages such as Map Reduce, Pig and Hive. It is a workflow system that links these jobs and allows specification of order and dependencies between them.

Reference Slide: Mahout Sqoop

  • Mahout - is a data mining library that implements popular algorithms for clustering and statistical modeling in MapReduce.
  • Sqoop - is a set of data integration tools that allow non-Hadoop data stores to interact with traditional relational databases and data warehouses.

Reference Slide: NoSQL

  • NoSQL(Not only SQL) - refers to a large class of data storage mechanisms that differ significantly from the well-known, traditional relational data stores (RDBMS). These technologies implement their own query language and are typically built on advanced programming structures for key/value relationships, defined objects, tabular methods or tuples.
  • NoSQL as a term is used to describe the wide range of data stores classified as big data. Some of the major flavors adopted within the big data world today include
  • Cassandra,
  • MongoDB,
  • NuoDB,
  • Couchbase and
  • VoltDB.
Main Menu