Ron Berman (@marketsensei) / X

Ron Berman

3,582 posts

Ron Berman

@marketsensei

Game theorist and data scientist. I study online advertising, marketing analytics and how curation algorithms impact platforms @Penn @Wharton.

Philadelphia, PA

Joined November 2010

Pinned
Ron Berman
@marketsensei
Mar 26, 2022
Companies invest a lot in analytics - but are these investments valuable? @IsraeliAyelet and I studied ~1,500 online retailers and found that using a descriptive dashboard increased their weekly revenues by 4%-10%. >> #MarTech #BigData #Analytics #ecommerce #DataScience
Ron Berman
@marketsensei
Jan 1, 2022
How are effects of online A/B tests distributed? How often are they not significant? Does achieving significance guarantee meaningful business impact? We answer these questions in our new paper, “False Discovery in A/B Testing”, recently out in Management Science >>
Ron Berman
@marketsensei
Jan 1, 2022
Replying to @marketsensei
The paper is co-authored with Christophe Van den Bulte and analyzes over 2,700 online A/B tests that were run on the @Optimizely platform by more than 1,300 experimenters. Link to paper: pubsonline.informs.org/doi/10.1287/mn… Non paywalled: ron-berman.com/papers/fdr.pdf >>
Ron Berman
@marketsensei
Jan 1, 2022
Replying to @marketsensei
Generally, firms should try to test more radical variations in order to have larger impact on consumer behavior (we call this “swing for the fences”). Most variations will probably not be very impactful, but once in a while an innovation will prove to be very lucrative. (Fin)
Ron Berman
@marketsensei
Apr 18, 2023
T
19K
Ron Berman
@marketsensei
Jun 26, 2021
Replying to @Jeffely
Indeed the number of adults fully vaccinated in Israel has been quoted at 85% by officials. This is what the age distribution of those recently testing positive in last month. Majority is from last two weeks. (Green: women, blue: men, y axis is age group).
Ron Berman
@marketsensei
Mar 10, 2022
Ordered coffee with regular milk in SF. Everyone gave me the "he's such a rebel" look.
Ron Berman
@marketsensei
Jan 1, 2022
Replying to @marketsensei
Our analysis shows that in our data, hypothesis tests conducted with alpha=0.05 yield an FDR of 18%-25%. Much higher than 5%. That is, about 20% of significant effects chosen for implementation will not generate the business impact that was observed in the experiment. >>
Ron Berman
@marketsensei
Aug 25, 2021
Replying to @mike_luca @garjoh_canuck and @ronnyk
Yes. Just pick the outcome with the highest mean (Test & Roll) and you can use Latent Stratification if you really care about significance. Sometimes it helps. On T&R: ron-berman.com/2020/01/26/tes… On LS: arxiv.org/abs/1911.08438
Test & Roll: An experimental design to maximize profit from A/B Tests
From ron-berman.com
Ron Berman
@marketsensei
Mar 26, 2022
Replying to @marketsensei
The paper is forthcoming in Marketing Science and is available at pubsonline.informs.org/doi/10.1287/mk…. (Ungated version at dx.doi.org/10.2139/ssrn.3…) >> @MarketngScience
Ron Berman
@marketsensei
Jan 1, 2022
Replying to @marketsensei
We then classify effects into “null” (zero) and “non-null” (pos or neg), to understand how many experiments, on average, have an underlying zero effect. The answer is about 70%. That is, 70% of effects will not show any impact on Engagement compared to a baseline. >>
Ron Berman
@marketsensei
Jan 1, 2024
Traveling with a group of Penn Faculty on a Mission to Israel to build bridges with the Israeli academic community and bear witness to the impact of the Hamas attacks. First visit since Oct 7. Deep thoughts going through my mind rn. #penn #upenn #pennfaculty #mission #israel
8.1K
Ron Berman
@marketsensei
Jan 2, 2022
Been teaching digital marketing @Wharton for over 7 years. Have had multiple students reach hundreds of thousands of social media followers in projects. Still can't describe the (happy childish) feeling of this junior professor who reached 2,000 followers today. Hi All! >>
GIF
Ron Berman
@marketsensei
Jan 1, 2022
Replying to @marketsensei
First, we analyze the effects of all the A/B tests in our data. They are quite small. The median (and average) webpage variations have roughly zero effect on webpage Engagement. But the distribution is quite long-tailed with some variations showing big effects. >>