{"id":282,"date":"2020-04-12T17:21:00","date_gmt":"2020-04-12T15:21:00","guid":{"rendered":"http:\/\/www.neovita.com\/thoughts\/?p=282"},"modified":"2020-04-13T05:27:37","modified_gmt":"2020-04-13T03:27:37","slug":"design-research-ab-test-change-my-mind","status":"publish","type":"post","link":"https:\/\/www.oninteractions.com\/thoughts\/design-research-ab-test-change-my-mind\/","title":{"rendered":"Design optimization: An\u00a0A\/B\u00a0test that made me change my mind"},"content":{"rendered":"\n<p class=\"wp-block-neo-blocks-preamble\">In May 2014 development of <a href=\"http:\/\/www.barnplay.se\">Barnplay.se<\/a> was winding down but two of my colleagues, Simon and Emil, was haunted by a feature they did not feel was implemented optimally.<\/p>\n\n\n\n<p class=\"list-heading\">All shows in Barnplay have episodes that kids can choose from. However, the shows also have a text about the show and some shows have other &#8220;functions&#8221;:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>sort the episodes differently<\/li><li>a link to the shows web site on barnkanalen.se<\/li><li>mark the show as a favorite or remove from favorites (only in our apps)<\/li><\/ul>\n\n\n\n<p>These &#8220;functions&#8221; where presented with a round icon and text and Simon and Emil thought this could become better. A smaller representation for the functions would also make the functions fit better in phones held in portrait position. Their suggestion was a &#8220;button&#8221; look without a text label.<\/p>\n\n\n\n<p>I however had already via usability tests proved that icon+text works. So I did not want to change to a &#8220;button&#8221; look without first proving that it was better. I decided to do an A\/B test in our tool <a href=\"https:\/\/www.optimizely.com\">Optimizely<\/a>.<\/p>\n\n\n\n<p class=\"list-heading\">While Emil created graphics for the &#8220;buttons&#8221; and Simon made them possible to display in Barnplay I set up the experiment in Optimizely. The variations in the experiment where &#8220;icon+text&#8221; versus &#8220;button&#8221;. We want the functions to be used in general but ultimately want them to generate video starts. For goals I therefore choose to track:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>taps on any of the functions<\/li><li>taps on each of the functions<\/li><li>a two step activity where the visitor has first clicked &#8220;Sort&#8221; and then started an episode<\/li><\/ul>\n\n\n\n<p>We ran the experiment for 2 days which resulted in 142 000 visits and lots of taps.<\/p>\n\n\n\n<table class=\"data-table data-table-xs-hide-col-4\" border=\"0\" cellspacing=\"0\" cellpadding=\"0\"><colgroup> <col \/> <col \/> <col \/> <col \/> <col \/> <\/colgroup>\n<tbody>\n<tr>\n<th colspan=\"5\">Tap on any function<\/th>\n<\/tr>\n<tr>\n<td>Variation<\/td>\n<td>Visits<\/td>\n<td>Taps<\/td>\n<td>Conversion Rate<\/td>\n<td>Improvement<\/td>\n<\/tr>\n<tr>\n<td>Icon+Text<\/td>\n<td>70829<\/td>\n<td>2536<\/td>\n<td>3.58% (+\/- 0.14)<\/td>\n<td>&#8212;<\/td>\n<\/tr>\n<tr>\n<td>Button<\/td>\n<td>71117<\/td>\n<td>9155<\/td>\n<td>12.87% (+\/- 0.25)<\/td>\n<td>+259.5%<\/td>\n<\/tr>\n<tr>\n<th colspan=\"5\">Tap to read about show<\/th>\n<\/tr>\n<tr>\n<td>Variation<\/td>\n<td>Visits<\/td>\n<td>Taps<\/td>\n<td>Conversion Rate<\/td>\n<td>Improvement<\/td>\n<\/tr>\n<tr>\n<td>Icon+Text<\/td>\n<td>70829<\/td>\n<td>1992<\/td>\n<td>2.81% (+\/- 0.12)<\/td>\n<td>&#8212;<\/td>\n<\/tr>\n<tr>\n<td>Button<\/td>\n<td>71117<\/td>\n<td>5584<\/td>\n<td>7.85% (+\/- 0.20)<\/td>\n<td>+179.2%<\/td>\n<\/tr>\n<tr>\n<th colspan=\"5\">Tap on link to show site<\/th>\n<\/tr>\n<tr>\n<td>Variation<\/td>\n<td>Visits<\/td>\n<td>Taps<\/td>\n<td>Conversion Rate<\/td>\n<td>Improvement<\/td>\n<\/tr>\n<tr>\n<td>Icon+Text<\/td>\n<td>70829<\/td>\n<td>393<\/td>\n<td>0.55% (+\/- 0.05)<\/td>\n<td>&#8212;<\/td>\n<\/tr>\n<tr>\n<td>Button<\/td>\n<td>71117<\/td>\n<td>474<\/td>\n<td>0.67% (+\/- 0.06)<\/td>\n<td>+20.1%<\/td>\n<\/tr>\n<tr>\n<th colspan=\"5\">Tap to make\/remove favorite<\/th>\n<\/tr>\n<tr>\n<td>Variation<\/td>\n<td>Visits<\/td>\n<td>Taps<\/td>\n<td>Conversion Rate<\/td>\n<td>Improvement<\/td>\n<\/tr>\n<tr>\n<td>Icon+Text<\/td>\n<td>70829<\/td>\n<td>5914<\/td>\n<td>8.35% (+\/- 0.20)<\/td>\n<td>&#8212;<\/td>\n<\/tr>\n<tr>\n<td>Button<\/td>\n<td>71117<\/td>\n<td>4119<\/td>\n<td>5.79% (+\/- 0.17)<\/td>\n<td>-30.6%<\/td>\n<\/tr>\n<tr>\n<th colspan=\"5\">Tap to sort episodes<\/th>\n<\/tr>\n<tr>\n<td>Variation<\/td>\n<td>Visits<\/td>\n<td>Taps<\/td>\n<td>Conversion Rate<\/td>\n<td>Improvement<\/td>\n<\/tr>\n<tr>\n<td>Icon+Text<\/td>\n<td>70829<\/td>\n<td>1815<\/td>\n<td>2.56% (+\/- 0.12)<\/td>\n<td>&#8212;<\/td>\n<\/tr>\n<tr>\n<td>Button<\/td>\n<td>71117<\/td>\n<td>2769<\/td>\n<td>3.89% (+\/- 0.14)<\/td>\n<td>+51.9%<\/td>\n<\/tr>\n<tr>\n<th colspan=\"5\">Tap to sort, then start episode<\/th>\n<\/tr>\n<tr>\n<td>Variation<\/td>\n<td>Visits<\/td>\n<td>Taps<\/td>\n<td>Conversion Rate<\/td>\n<td>Improvement<\/td>\n<\/tr>\n<tr>\n<td>Icon+Text<\/td>\n<td>70829<\/td>\n<td>800<\/td>\n<td>1.13% (+\/- 0.08)<\/td>\n<td>&#8212;<\/td>\n<\/tr>\n<tr>\n<td>Button<\/td>\n<td>71117<\/td>\n<td>1422<\/td>\n<td>2.00% (+\/- 0.10)<\/td>\n<td>+77.0%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n<p>All shows have the &#8220;Read about the show&#8221; function which is why that function has so many more taps then other functions. The stunning difference between the text+icon and button variants is a strong indicator that button works better. Link taps where quite few so the result is not so interesting. That the visitors used the favorite function less is troubling. The sorting and sort+start goals however look very promising.<\/p>\n\n\n\n<p>Despite the decrease in activity on the favorite function, buttons was chosen as the winning variant in this test. Simon implemented them fully and they are now in production.<\/p>\n\n\n\n<p class=\"wp-block-neo-blocks-conclusion\">So, I had to change my mind and I love when that happens! :)<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter\"><img loading=\"lazy\" decoding=\"async\" width=\"900\" height=\"653\" src=\"https:\/\/www.oninteractions.com\/thoughts\/wp\/wp-content\/uploads\/2014\/08\/ab-test-vald-variant.jpg\" alt=\"Barnplay Functions: A\/B test result\" class=\"wp-image-295\" srcset=\"https:\/\/www.oninteractions.com\/thoughts\/wp\/wp-content\/uploads\/2014\/08\/ab-test-vald-variant.jpg 900w, https:\/\/www.oninteractions.com\/thoughts\/wp\/wp-content\/uploads\/2014\/08\/ab-test-vald-variant-600x435.jpg 600w, https:\/\/www.oninteractions.com\/thoughts\/wp\/wp-content\/uploads\/2014\/08\/ab-test-vald-variant-250x181.jpg 250w\" sizes=\"auto, (max-width: 900px) 100vw, 900px\" \/><\/figure><\/div>\n","protected":false},"excerpt":{"rendered":"<p>My colleagues wanted to change the look and content of an important part of Barnplay.se. I thought the current one was better and set out to check whether their or my variant resulted in more taps and views.<\/p>\n","protected":false},"author":1,"featured_media":294,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[21,8,19,9],"tags":[],"series":[40],"class_list":["post-282","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ab-test","category-behavior","category-interaction-design","category-process","series-skills"],"acf":[],"jetpack_featured_media_url":"https:\/\/www.oninteractions.com\/thoughts\/wp\/wp-content\/uploads\/2014\/08\/ab-test-varianter.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.oninteractions.com\/thoughts\/wp-json\/wp\/v2\/posts\/282","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.oninteractions.com\/thoughts\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.oninteractions.com\/thoughts\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.oninteractions.com\/thoughts\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.oninteractions.com\/thoughts\/wp-json\/wp\/v2\/comments?post=282"}],"version-history":[{"count":15,"href":"https:\/\/www.oninteractions.com\/thoughts\/wp-json\/wp\/v2\/posts\/282\/revisions"}],"predecessor-version":[{"id":1647,"href":"https:\/\/www.oninteractions.com\/thoughts\/wp-json\/wp\/v2\/posts\/282\/revisions\/1647"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.oninteractions.com\/thoughts\/wp-json\/wp\/v2\/media\/294"}],"wp:attachment":[{"href":"https:\/\/www.oninteractions.com\/thoughts\/wp-json\/wp\/v2\/media?parent=282"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.oninteractions.com\/thoughts\/wp-json\/wp\/v2\/categories?post=282"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.oninteractions.com\/thoughts\/wp-json\/wp\/v2\/tags?post=282"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/www.oninteractions.com\/thoughts\/wp-json\/wp\/v2\/series?post=282"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}