{"id":2034,"date":"2014-02-06T09:34:03","date_gmt":"2014-02-06T17:34:03","guid":{"rendered":"https:\/\/redfindevelop.wpengine.com\/blog\/devblog\/?p=2034"},"modified":"2020-10-05T13:12:18","modified_gmt":"2020-10-05T20:12:18","slug":"a-b-testing-build-or-buy","status":"publish","type":"post","link":"https:\/\/www.redfin.com\/news\/a-b-testing-build-or-buy\/","title":{"rendered":"A\/B Testing Framework: Build or Buy?"},"content":{"rendered":"

Six months ago Redfin needed an A\/B testing framework. The big question: “Do we use a service or do we build our own?\u201d After a lot of investigating we chose to build our own. This summary of our investigation should provide some useful context to anyone trying to make the same decision.<\/p>\n

What flavor do you need?\u00a0\"png;base64647be74a62654577\"<\/a><\/h3>\n

One thing I should point out is that A\/B testing is a very broad term. It can range from testing marketing copy, to tracking session behaviors, to testing interactive features that require abstract custom defined behavior. Third-party solutions usually focus on one area really well,but can’t do something else that you really need. Be clear on what you need your framework to do vs. what would be nice to have.<\/strong><\/p>\n

Buying or selling a home is a multi-month (sometime multi-year) journey which makes a customer\u2019s relationship with Redfin pretty unique. We need to measure experiment results over months, not just within a single web session. \u00a0Of course we don\u2019t want to run A\/B test for months, so we use proxy measurements that we\u2019ve correlated to long term customer behavior. We knew we needed a framework that could test interactive features and track custom defined behaviors.<\/p>\n

Bandits (so hot right now)<\/h3>\n

\"png;base64c3830d2c32744f4c\"<\/a><\/h3>\n

One other consideration when picking a framework is your need for multi-armed bandit testing<\/a>. At the highest level multi arm bandit algorithms automatically direct more traffic to the winning variant(s). This means you start getting more of the benefits of your change sooner and never end a test too soon, because you never really have to end a test (the algorithm just directs all the traffic to the winning variant). \u00a0Bandit testing is \u201cOh so hot right now\u201d and like most things that are hot, there are strong opinions on why you\u2019d be stupid not to use it<\/a> and why you might be smart not to<\/a>. Initially we were interested in bandit testing, but we decided we didn\u2019t need it.<\/p>\n

 <\/p>\n

Peer Research<\/h3>\n

We knew that all the big guys (Amazon, Google, Facebook, etc.) built their own framework, but they are waaay bigger than us and they started before commercial solutions were available. What we really wanted to know was “What are other startups our size doing? Were they buying or building?”<\/strong><\/p>\n

Luckily we’re backed by a couple of great venture capital firms who helped us connect with other companies in their networks. In total we talked with engineers and product managers at 11 companies<\/strong> including Etsy, Quora, Decide, Payscale, Zynga, Icanhascheezburger, Vizify, and more.<\/p>\n

We learned that nine of the eleven startups we talked to had built their own A\/B testing framework.<\/h2>\n

Surprised? We were.<\/span><\/h2>\n
\"Dan<\/a>
Dan McKinley: Principal Engineer at Etsy and occasional sea captain.<\/figcaption><\/figure>\n

The most useful insights from talking with other companies came from\u00a0Dan McKinley at Etsy<\/a>. If you haven’t watched\u00a0his talk<\/a>, you should. Dan was kind of this “purveyor of truth” around A\/B testing for us. In our conversation with him (and in the video) he frequently drops these nuggets of knowledge (simply stated, yet very profound). We continually found out how true they were.<\/p>\n

My favorites:<\/p>\n