Newton’s third law states that for every action, there is an equal and opposite reaction. A version of this applies to any UX/UI tweak you make to your website – for every action you take, there may be an opposite (and undesirable) reaction.

One of the most common arguments against using the various email optin tools that Sumo offers is that while it may increase your number of emails, the resulting anger that it may cause means you get higher bounce rates, lower pages per user, less sharing, and worst of all, decreased revenue. People dismiss it as making your site “more annoying.”

Thankfully, I had the site traffic to run my very own Sumo performance review on my site.

Where We’re Testing

One of the sites I own is Examine.com. We analyze and summarize scientific research in nutrition and supplementation. We’re a pretty popular site (~50,000 people visit us every day), and we tend to be very nerdy (according to BuzzSumo, our most shared article in the past year was a study analysis – Really-low-fat vs somewhat-lower-carb – a nuanced analysis).

We had Sumo off for exactly 7 days, allowing us to compare 7 days on with 7 days off (immediately afterwards). The entire cohort was almost 750,000 visitors.

What We Tested

What we tracked for this performance review:

• Pageviews per person
• Average session duration
• Bounce Rate
• Emails collected
• Revenue generation
• Facebook Shares
• Twitter Shares

The first three (pageviews per session, average session duration, and bounce rate) we got via Google Analytics.

Emails collected and revenue generation was through our backend, Facebook shares was done via our own backend script (since we only have ~500 pages, it’s easy to ping Facebook’s OpenGraph every day), and Twitter shares we just added up manually.

I can also state that there was no random event (for example, going viral on reddit or having a major influencer share our content) that would impact the findings.

What we disabled was the Welcome Mat, List Builder, and the Scroll Box. Our regular in-content opt-ins remained.

The basic investigation was – “if we make our site ‘less annoying,’ will that improve on-site metrics, revenue, and shares?”

Here’s what we found:

Pages / Session:

GA reported that disabling Sumo actually decreased pages/session 0.37% – from 1.78 pages/session to… 1.78 pages/session.

Result: Not much difference

Average Session Duration:

GA again reported that disabling Sumo actually decreased the average session duration 0.47% – from 98 seconds to… 98 seconds.

Result: Not much difference

Bounce Rate:

Finally – a negative result. Disabling Sumo decreased the bounce rate by 1.29% – from 67.79% to 66.66%

Result: Slightly improved bounce rate without Sumo

Emails collected:

Ouch. A devastating loss of 78.7% by removing Sumo. We went from collecting 258 emails a day to 55.

Considering we sell just information and how email marketing is the backbone (see how we segmented 6000 emails to generate over $100k), this was an expensive test.

Result: Devastating loss without Sumo

Revenue Generation:

To be more clear, we wanted to see if the revenue we generated from people clicking links on the site itself (ignoring email) was impacted at all by making the site “less annoying.”

The end result was negligible – click thru rate to the sales pages went up by 2% and sales went up 1%. Since we did not hit 500 transactions (but we did hit 500+ clicks), an educated guess would be that the impact on sales was negligible.

On the other hand, the massive decrease in email optins had a very large impact on revenue.

Result: Negligible improvement in direct revenue from website (coupled with massive loss in revenue via email sequence)

Facebook Shares:

Because of how much flux there is in Facebook Shares (due to influencers), this one can spike easily from one week to another.

Disabling Sumo had a slight increase in Facebook shares of 2.3% – we went from 89 likes a day to 91 a day.

Result: negligible improvement without Sumo

Twitter Shares:

We get anywhere from 3-6 mentions a day on twitter for Examine.com.

Our twitter shares actually went down 9.1% – from 33 in a week to 30. Considering the low numbers in the first place, this is a negligible result

Result: negligible decrease without Sumo

Sumo Performance Review: Conclusion

Our investigation was to look at three key areas (on-site metrics, revenue, and shareability). For on-site and shareability, disabling Sumo had a negligible impact. On the other hand, the negative impact on revenue was significant.

To answer our original investigation – even if Sumo is perceived as “annoying,” it had no impact in on-site metrics, revenue, and shareability.

It’s a no-brainer to utilize Sumo to collect more emails.