How to calculate ROI of a video with A/B testing

A blogger recently showed in a post, how to calculate the ROI of a product video by using visitor segmentation. The author defines a segment with visitors that watch the entire product video and another one that doesn’t watch the video at all. For both segments, he calculates average sales per visitor, the break-even point and the ROI.

In my opinion, the ROI should better be calculated by using A/B testing.


Visitor segments to help us to put all our visitors in certain classes or groups.

With the segmentation we generate knowledge about visitors. In the example of the mentioned blog post we make a statement about visitors who show an interest in product videos, and visitors are not interested in product videos – and their respective willingness to buy.

O.K. we know that visitors who have enough patience, to watch a whole product video, are probably more interested in buying and will cause more revenue than visitors that are not interested in our videos.

Probably we would like to buy traffic with as much interested visitors as possible. Without doubt, this traffic would be more valuable to us than traffic with visitors that show less interest in promotional videos.

By segmenting visitors we learn a lot about visitors and the quality of our traffic.

But we do not know if the video has led to the purchase or whether the visitor would have bought anyway.

So how do we find out whether the product videos actually have a positive impact on the behavior of our visitors?

With testing (A / B tests). This means that show a variant of the shop with videos (experimental group) to one half o the visitors and the other a variant without videos (control group) to the other half of the visitors. Which variant is show to a visitor is purely coincidental.

In both groups of visitors, we collect all relevant information, such as the number of visitors, visits, conversion rate and the order value.

With the data collected on both groups of visitors, we compare the alternatives and we examine whether an investment in product videos makes sense for our business.

Because both groups will have sales, we need to calculate the ROI from the difference in sales.

(Image: berlin-pics /

Book Review: Jim Sterne – Social Media Metrics

For the impatient reader

A book about social media marketing with interesting case studies, striking arguments for social media efforts and good hints and tips for the implementation.

About the Author

Since the early nineties Jim Sterne is engaged in internet marketing a few years later he focused on web analytics. In 2003 he co-founded the Web Analytics Association. He is considered one of the most influential and experienced consultants and authors in the field of online-marketing and web analytics.


Jim Sterne himself provides the best summary: This book is about measuring the business value of social media, measuring the importance of social media to organizations, making the best of social for business in a community acceptable, brand enhancing way and how to gauge the value of your social media efforts.

This book is not about the measurement itself and the calculation of the relevant metrics, thus the experienced web analyst won’t find anything new in this book regarding numbers and figures. This book shows you what to do with the numbers and how to drive successful campaigns.

The book follows the customer life cycle. Marketing managers will immediately recognize the references to the AIDA model.

Jim Sterne puts much emphasis on the rarely mentioned but very important ‘soft’ factors which cause social media projects to success. E.g.:

– State clear objectives
– Capture Results
– Convince colleagues and supervisors
– Anticipate future developments

Recommendation: Read!

You will look over the shoulder of an experienced practitioner and receive valuable hints for the successful implementation of social media campaigns.


Titel:Social Media Metrics: How to Measure and Optimize Your Marketing Investment (New Rules of Social Media)
Verlag: John Wiley & Sons; Auflage: 1. Auflage (16. April 2010)
Sprache: Englisch
ISBN-10: 0470583789
ISBN-13: 978-0470583784



(Image: lupo /

Tagging or Logfile-Analysis?

There are two common measurement methods that are used for Web Analytics.

  • Measurement on the client side (tagging)
  • Measurement on the server side (log file analysis)

Let’s have a look at how a page is processed on the web:
ein Browser sendet einen Request zum Server und erhält die Antowrt

  1. A user selects a URL by typing it in, clicking a link or a choosing bookmark.
  2. The client (browser) sends a request to this URL on the internet and waits for response
  3. The client receives a response from a server
  4. The browser loads for more required data over the Internet
  5. The browser renders the page.

We should assume that the client receives exactly the data that the server sends. That is why an analysis of server log files should be as precise as a measurement at the client. But this is not the case as we will show.

The internet is not a cable connecting directly one server to a client. It is a conglomerate of many different network components and servers. Each of those components could interfere or manipulate a request by a client or the response of a server. Each request passes many hops before it reaches a server. The same is true for the responses.

the internet is not a cable but a network

Thus there is no such thing as a truth that you everybody agrees on. You might see the same thing from multiple angles and maybe come to different conclusions.
A server does not only serve human ‘visitors’, but also many programs requesting files. (crawlers) These hits also show up in the server’s log file but users behind a proxy are (mostly) invisible to the server.

The measurement close to the client is usually done with tagging method. This means that the user’s Browser sends a request to a tracking server via JavaScript. Because the request is issued via JavaScript robots and spiders are (usually) irrelevant for this measurement method. Users or browsers can be distinguished by using cookies as opposed to their IP address, thus even users behind proxies can be distinguished sufficiently well.

Pro log file analysis:

  • Everything that relates to the webserver can be determined really well.
  • No changes to the website necessary
  • You don’t need to have a tracking server

Pro tagging method:

  • Everything that affects the user can be measured really well.
  • You can collect more data e.g. revenues, orders, shopping carts, …
  • Tagging allows you to gather data from multiple websites or domains running on different servers
  • The measurement runs on the client and thus doesn’t put additional load on your webserver
  • gathers data from behind proxies

Which of the methods should I choose? -> Both methods!

For administrators who want to know which files are requested (and how often), when the Google Bot hits your systems and sets them under load and which error pages are displayed the log file analysis is very helpful.

For online marketing, e-commerce and other business relevant questions, where the users or customers and their behavior is relevant, tagging is the method of choice.

If you are interested in users, then you measure close to the user (tagging method)
If you are interested in the server, then you measure near the server (logfile analysis)