{"id":41508,"date":"2010-06-04T11:13:39","date_gmt":"2010-06-04T18:13:39","guid":{"rendered":"https:\/\/redfindevelop.wpengine.com\/blog\/devblog\/?p=368"},"modified":"2020-10-05T13:12:35","modified_gmt":"2020-10-05T20:12:35","slug":"evolving_a_new_analytical_platform_with_hadoop","status":"publish","type":"post","link":"https:\/\/www.redfin.com\/news\/evolving_a_new_analytical_platform_with_hadoop\/","title":{"rendered":"Engineer-to-Engineer: Evolving a New Analytical Platform with Hadoop"},"content":{"rendered":"

<\/strong><\/p>\n

In the second installment of our San Francisco series of engineer-to-engineer lectures<\/a>, Jeff Hammerbacher<\/a> described the challenges of building data-intensive, distributed applications and how using Hadoop<\/a><\/ins> saved the day at Facebook.\u00a0 Speaking to an audience of approximately thirty Hadoop experts and enthusiasts hailing from all around the Bay Area, the Valley, and even Seattle, he also discussed what\u2019s wrong with today\u2019s analytical platforms and what will shape the platform of the future.<\/p>\n

And Jeff should know.\u00a0 After studying Mathematics at Harvard and wearing a suit as a quantitative analyst on Wall Street, he conceived, built, and led the data team at Facebook.\u00a0 He then went on to start Cloudera<\/a>, the leader in commercializing Apache Hadoop, where he currently works as Chief Scientist and VP of Products.\u00a0 Jeff also served as Contributing Editor for a book: Beautiful Data: The Stories Behind Elegant Data Solutions<\/a>, the proceeds of which are split between Creative Commons and Sunlight Labs.<\/p>\n

The Scoop on Hadoop<\/strong><\/p>\n

Hadoop<\/a> is an open source framework that enables data-intensive distributed applications to efficiently process gigantic amounts of data.\u00a0 It\u2019s an open source implementation of the MapReduce<\/a> approach to processing data.\u00a0 MapReduce was invented at Google to deal with the massive quantities of data necessary to index the web. \u00a0There are two main components to the system: the Hadoop Distributed File System (HDFS) which stores and maintains data across many machines, and the MapReduce engine which processes the data.<\/p>\n

But the talk didn\u2019t really go into Hadoop internals — as Jeff pointed out, the documentation is readily available online.\u00a0 Rather, the talk was about how and why Hadoop will provide the foundation on which the next generation platform for analytics will be built.\u00a0 Making bold predictions about technology is hard.\u00a0 Jeff quoted Larry Ellison\u2019s quip that \u201cthe computer industry is the only industry that is more fashion-driven than women’s fashion<\/a>.\u201d\u00a0 And yet, using real-world examples from his experience at Facebook, Jeff makes a compelling sell.<\/p>\n

Bottlenecks, Costs, the Black Box, and the Kitchen Sink<\/strong><\/p>\n

A typical architecture for large-scale data analysis includes a data source, a data warehouse, ETL (aka: \u201cextract-transform-load\u201d; the step that gets data out of and into RDBMSs and converts source data to the data warehouse\u2019s format), and business intelligence and analytics systems \u2013 all of which are usually centered around relational databases. \u00a0However, Jeff stressed that a relational database is a specialty<\/em> and not a foundation<\/em>, arguing that the abstractions provided by them are no longer useful on their own for analytical data management.<\/p>\n

One reason is that over the past few years, there has been an explosion in data volume primarily originating from machine-generated logs. \u00a0By simply tweaking an Apache log, you can grow your data volume and complexity by several orders of magnitude.\u00a0 As we\u2019ll see in Facebook\u2019s case, their relational database approach simply didn\u2019t scale and they soon needed new tools to handle the load.<\/p>\n

Another point Jeff made was that the percentage of data that actually gets stored in a relational database is shrinking.\u00a0 What do you do with all the unstructured data (accounting for 95% of the digital universe<\/a>) that doesn\u2019t necessarily make sense to persist relationally?\u00a0 Do you still need expensive relational data warehouses and proprietary boutique servers?\u00a0 Jeff\u2019s team at Facebook made a bet on commodity hardware which turned out to be a good move, ultimately pushing the complexity out of hardware and into the software layer.<\/p>\n

They also bet on open source data stores built by consumer web firms, arguing that web properties have the most representative problems: scalability and unstructured data management.\u00a0 Jeff stated that most production-quality data stores came from enterprise software firms in the mid-1990s, but now a growing percentage of the world\u2019s data is persisted in open source data stores.\u00a0 He also mentioned that a nice side-effect of adopting open source solutions is that it\u2019s much better to have a modular collection of open tools rather than an opaque abstraction.\u00a0 Why? \u00a0Because there\u2019s great benefit in being able to pick and choose solutions and understand what\u2019s going on under the hood.<\/p>\n

Jeff noted that another problem is that, in many cases, enterprise software does not service developers well. Many relational data warehouses simply just expose SQL; but to get real traction\/adoption from developers, you need more than that…\u00a0 You need open applications for analysis, not just a SQL interface.\u00a0 He feels that \u201cin addition, these data stores often expose a proprietary interface for application programming (e.g. PL\/SQL or TSQL), but not the full power of procedural programming.\u00a0 More programmer-friendly parallel dataflow languages await discovery, I think. \u00a0MapReduce is one (small) step in that direction.\u201d<\/p>\n

Where is this new platform going to come from?\u00a0 Any new platform must be centered around addressing these new user needs, which is hard to achieve by re-implementing an old spec in a new, clever way.\u00a0 He cautioned that implementing a new, successful cut of the ANSI SQL spec would be a real undertaking.\u00a0 Not only would it take ages before you had anything to show, but it would likely suffer the same scalability problems of previous implementations.<\/p>\n

Facebook and Hadoop are Now Friends<\/strong><\/p>\n

Using Facebook as a real-world example, Jeff described the challenge of measuring how changes to the site improved or impaired user experience.\u00a0 Their original data analysis system featured source data living on a horizontally partitioned MySQL tier and a cron job running Python scripts that pinged stats back to a central MySQL database.\u00a0 The main problem with this setup was that it made intensive historical analysis difficult since the source data was spread over many machines and aggregating the data to the analytics database was a slow, inefficient process.\u00a0 Plus, when it barfed, it took three days to replay the edit logs in order to diagnose the problem.<\/p>\n

So Facebook hired a data warehouse engineer to build a 10TB Oracle warehouse.\u00a0 This worked for a bit and would\u2019ve been fine for small and medium-sized businesses, but ultimately didn\u2019t scale — particularly when they turned on impression logging which generated over 400GB of data on the first day!\u00a0 This quickly grew to 1TB of data per day in 2007.<\/p>\n

You might suggest that since disks are cheap, why not throw more storage at the warehouse?\u00a0 It turns out that, in addition to the problem of data volume, there was also a bottlenecking CPU utilization problem. The ETL process ended up taking more than a day to aggregate, import, and load the necessary data for analysis.\u00a0 Jeff went on to explain that proprietary ETL vendors have lots of downsides and generally don\u2019t scale well for large sets of databases (on the order of thousands, in Facebook\u2019s case).\u00a0 In addition, when \u201cwarts\u201d start to show up in proprietary vendors, the closed nature of the software prohibits developers from tinkering with the source to diagnose and resolve problems.<\/p>\n

Meanwhile, his team started to play with Hadoop on the side as an open source alternative.\u00a0 They got a Hadoop cluster to replace the data collection and processing tiers.\u00a0 So the new architecture still has multiple data sources (log files, MySQL) but is now fed into HDFS instead.\u00a0 Work is done via MapReduce and the artifacts are then published to Oracle RAC servers for consumption by business intelligence and analysis.\u00a0 It also simultaneously publishes results back to the MySQL tier.<\/p>\n

\"Data<\/p>\n

From “Facebook\u2019s Petabyte Scale Data Warehouse using Hive and Hadoop<\/a>“, slide 21<\/p>\n

Initially, this shift was met with a lot of resistance mostly because Hadoop is Java-based and, since the majority of Facebook\u2019s services were written in C++, the developers there weren\u2019t comfortable in Java. But it wasn\u2019t long before the new platform showed its strengths:<\/p>\n