Conductrics Blog

The Value in Being Intentional and Customer Focused

Privacy in analytics is often considered a constraint or a barrier that limits our actions or our ability to develop powerful technological solutions. However, embedding privacy by design principles—such as data minimization—actually enhances our systems and software. Why? Because it aligns with our field’s primary value proposition: The main value of experimentation/AB Testing programs is that they provide a principled framework for organizations to act and learn intentionally so that, ultimately, we can be customer advocates. 

It is our view that privacy by design thinking is inherently intentional thinking. It’s good engineering that puts the customer first and requires one to explicitly ask ‘why’ before collecting additional information. Privacy by design helps us be outcomes-based (focus on the customer) vs. compliance/procedure-based (focus on the organization).

We have found that by following privacy by design engineering principles, we have not only been able to engineer data minimization and privacy by default designs directly into Conductrics’s software, but this has had the happy effect of also leading to multiple benefits, including data storage, computational, and reporting efficiencies.

What is Privacy by Design?

Privacy by design (PBD) is a set of engineering principles that seek to embed privacy-preserving capabilities as the default behavior directly into systems and technologies.
The seven principles, as originally specified by Dr. Ann Cavoukian, are:

1. Proactive, not reactive – That is, systems should be intentional and anticipate privacy requirements rather than responding to privacy issues after the fact 
2. Privacy as the default setting
3. Privacy embedded into design
4. Full functionality – That is, privacy should not unreasonably impair access or use. 
5. End-to-end security
6. Visibility and transparency
7. Respect for user privacy

At Conductrics INC we came across Privacy by Design back in 2014/2015 after stumbling on Article 22 of GDRP (Automated individual decision-making, including profiling).

At the time, we were looking to refactor our original predictive targeting/contextual bandit features to make it easier for our customers to quickly be able to understand which of their customers would get which experiences. For those not familiar, contextual bandits are a machine learning problem where one wants to jointly solve the problem of: 

1. Discovering and delivering the best treatment for each user in an ongoing, adaptive way (often called heterogeneous treatment effects); and 
2. The sub-problem of discovering which covariates (contextual information) are useful to find those best treatments.

When we learned about the then upcoming GDPR and Article 22 (Automated individual decision-making, including profiling), we decided that we needed to make interpretability a first class feature of our predictive targeting/bandit feature. That meant scraping the normalized radial basis function net we had built for something, simpler and more transparent. To our surprise, by using a few clever tricks, making the Conductrics predictive engine less complex, not only enabled interpretability, but made it much more computationally efficient and improved the over all efficacy the system.

However, Art 22 led us to Art 25, and its mandate to embed data minimization into the design of software. That ultimately led us to refactor all of our data collection and statistical procedures so that we could both provide all of the analysis needed for AB Testing (t-tests, partial f-tests, factorial ANOVA, and ANCOVA) and incorporate K-anonymization into our data storage and report/auditing. 

Why Bother with Privacy by Design? 

It’s a requirement. 

Art 25 of the GDPR is titled “Data Protection by Design and by Default”

“The controller shall …implement appropriate technical and organizational measures … such as data minimisation…in an effective manner and to integrate the necessary safeguards into the processing in order to meet the requirements of this Regulation and protect the rights of data subjects.”

It puts the customer first. 

Following privacy by design, we have found, aligns with ensuring that products are more useful for customers—Instead of maximizing a customer’s value to the company, one builds and offer products and services that maximize the company’s value to their customers. 

Privacy engineering and data engineering are both about privacy and about why we’re performing data analytics and experimentation in the first place. Again, our view is that the value of experimentation is that it provides a principled procedure for organizations to learn, make decisions, and act intentionally. The ultimate reason for doing this is to help teams —who are near the front line of the business, where engagement happens—to act as customer advocates.

According to Fred Reichheld, creator of NPS, “There is no way to sustainably deliver value to shareholders or be a great place to work unless you put customers first.” As the front line in customer interactions, analysts have the opportunity to accomplish what should be the main purpose of the business—to enrich customers’ lives by being mindful of what they want and anticipating their needs. 

Privacy engineering is good engineering. 

When designing a product, often solving one problem or feature can negatively affect other aspects or dimensions of the product. Much like a Rubik’s cube, there are various ways to solve just a single side of the puzzle, but these solutions are local—once you solve them, they leave the other sides of a problem in disarray. 

It turns out, happily, that for many of the statistics needed for tasks like AB Testing, most of the  statistical approaches used for inference in AB Testing (ANOVA, ANCOVA, t-test, partial f-tests, etc.) can be performed extremely efficiently on aggregate data stored in equivalence classes (similar to a pivot table—see a more technical view of this here). The use of equivalence classes data, rather than individual microdata, facilitates the ability incorporate K-anonymization as a simple to read measure of data minimization. Given an individual in a dataset, K-anonymity, roughly, is the minimum number of other users who are indistinguishable from that individual. Larger K’s in some sense can be said to provide more privacy protection.

Intentional Data Collection and Use

There are of course many use cases where it is appropriate and necessary to collect more specific, and identifiable information. The main point is not that one should not ever collect data, rather one should not collect, link, and store extra information by default just in case it might be useful, we should be intentional for each use case about what is actually needed and proceed from there.

Lastly, we should note that data minimization and other privacy engineering approaches are not substitutes for proper privacy policies. Rather, privacy engineering is a tool that sits alongside required privacy and legal compliance policies and contracts.
If you would like to learn more about Conductrics please reach out here. 

Conductrics takes privacy seriously. Conductrics AB Testing has been developed in conformance with Privacy by Design principles in order to give our clients the privacy options they need. This is important for: 1) maintaining customer trust by safeguarding their privacy; and 2) in order to comply with privacy regulations that require systems to be designed with default behaviors that provide privacy protections. 

For example, Article 25 of the GDPR requires “Data protection by design and by default” and explicitly proposes the use of pseudonymization as a means to comply with the overarching data minimization principle. Technology should be designed in such a way that by default the scope of data collected is limited to “only personal data which are necessary for each specific purpose of the processing are processed.”  In other words, if a task can be performed without the use of an additional bit of data, then, by default, that extra bit of data should not be collected or stored. 

With Conductrics, clients are able to run AB Tests without the need to collect or store AB Test data that is linked directly to individuals, or associated with any other information about them that is not explicitly required to perform the test.  Out of the box, Conductrics collects AND stores data using data minimization and aggregation methods.

This is unlike other experimentation systems that follow what can be called an identity-centric strategy. This approach links all events, experimental treatments, sales etc., back to an individual, usually via some sort of visitor Id. 

For clarity, lets look at an example. In the table below there are three experiments running; Exp1, Exp2, and Exp3. Each experiment is joined back to a customer such that there is a table (or logical relationship) that links each experiment and the assigned treatment back to a visitor. 

While this level of detail can be useful for various ad hoc statistical analysis, it also means that the experimentation system has less privacy control since all the information about a person is linked back to each experiment. 

One can use pseudonymization, where the explicitly identifiable information (the Name, Email, and often Cust ID) can be removed from the table. However, what many fail to realize is that even if we remove those fields but keep all of the other associations at the same individual level, the remaining data often still contains more information than required and might be at a higher risk of being able to leak personal information about individuals. 

Happily, it turns out that for most standard experiments there is no need to collect or store data at the individual level to run the required statistical procedures for AB Tests – or even for certain versions of regression analysis useful for things like AB Test interaction checks. Instead of collecting, linking, and storing experiment data per individual, Conductrics collects and stores experiment data that we both de-identify AND aggregate AS IT IS COLLECTED. This makes it possible to both collect and store the experimental data in an aggregate form such that the experimentation platform NEVER needs to know identifying individual level data. 

Privacy by Design: Data Minimization and Aggregation

In the context of privacy engineering, network centricity (“the extent to which personal information remains local to the client”) is considered a key quality attribute in the Privacy by Design process. This can be illustrated by contrasting so-called “global privacy” with “local privacy”. 

In the global privacy approach identifiable, non minimized data is collected and stored, but only by a trusted curator. For example, the US Gov’t collects census data about US citizens. The US Census Bureau acts as the trusted curator, with access to all of the personal information collected by the census. However, before the Census Bureau releases any data publicly, it applies de-identification and data minimization procedures to help ensure privacy. 

Local privacy, by contrast, is a stronger form of privacy protection because data is de-identified and minimized as it is collected and stored. This removes the need for a trusted curator. Conductrics is designed to allow for a local privacy approach by default, such that Conductrics only collects and stores data in its minimized form.

In the process of doing so, Conductrics is also respecting the key privacy engineering objective of dissociability – a key pillar of the NIST Privacy Framework. 

How does this data collection work?

Rather than link all of the experiments back to a common user in a customer table / relational model above, Conductrics stores each experiment in its own separate data table. For simple AB Tests each of these tables in turn only need to record three primary values for each treatment in the experiment: 1) The count of enrolments (the number of users per treatment); 2) The total value of the conversions or goals for each treatment; and 3) the total value of the square of each conversion value. Let’s take a look at a few examples to make this clear. At the start of the experiment the AB Test’s data table is empty.

AB Test 1

As users are assigned to treatments, Conductrics receives telemetry from each user’s client (browser) with only the information needed to update the data table above. For example, say a customer on the client website gets assigned to the ‘B’ treatment in the AB Test. A message is sent to Conductrics (optionally, the messages can be first passed to the organization’s own servers via Conductrics Privacy Server for even more privacy control).

This message looks something like ‘ABTest1:B’ and the AB Test’s data table in Conductrics is updated as follows:

If the user then purchases the product attached to the test, the following message is sent ‘ABTest1:B:10’. Conductrics interprets this as a message to update the Sales field for Treatment B by $10 and the Sales^2 field by 100 (since 10^2=100). Notice that no visitor identifier is needed to make this update (Note: The actual system is a bit more complex, ensuring that the Sales^2 records properly account for multiple conversion events per user, but this is the general idea.)

As new users enter the test and purchase, the AB Test’s data table accumulates and stores the data in its minimized form. So by the end of our experiment we might have a data table that looks like this:

A nice property of this is that not only has the data been collected and stored in a minimized form for increased privacy, but it is also in a form that is much more efficient to calculate statistical tests. This is because most of the required summation calculations that are required for things like t-tests and covariance tables have already been done. 

Interestingly we are not limited to just simple AB Tests. We can extend this approach to include additional segment data. For example if we wanted to analyze the AB Test data based on segments – say built on customer value levels; Low (L) ,Medium (M),and High (H), the data would be stored as follows.

There is of course no need to limit the number of segments to just one. We can have any number of combinations of segments. Each combination is known as an equivalence class. As we increase the segment detail of each equivalence classes we also increase the number (based on the cardinality of the segment combinations).

So as the number of classes increase, there is an associated increase the possibility of identifying or leaking information about persons entered into a test. To balance this, Conductrics collection of user traits/segments follows from ideas of K-Anonymization, a de-identification technique based on the idea that the collected user data should have groups of at least K other users who look exactly the same. Depending on each experiment’s context and sensitivity, Conductrics can either store each user segment/trait group in its own separate data table, effectively keeping the segment data de-linked, or store the data at the segment combination level.

This gives our clients the ability to decide how strict they need to be with how much the data is aggregated. In the near future Conductrics will be adding reporting tools that will allow clients to query each experiment to help manage and ensure that data across all of their experiments is stored at the appropriate aggregation level.  

What about the Stats? 

While statistics can be very complicated, calculations behind most of the common approaches for AB Testing ( e.g  Welch t-tests. z-tests, Chi-Square, F-Tests, etc.) under the hood are really mostly just a bunch of summing and squaring of numbers. 

For example, the mean (average) value for each treatment is just the sum of Sales divided by the number of enrollments for each treatment. By dividing the sales column by the count column for each treatment we can calculate the average conversion value per treatment. 

The calculations for the variance and standard errors are a bit more complicated but they use a version of the following general form for the variance which uses the aggregated sum of squares data field (where each value is squared first then those squared values are all added up):

It turns out that this approach can be generalized to the multivariate case, and using similar logic, we can also run regressions required for MVTs and interaction checks. Technical readers might be interested to notice that the inner product elements of the Gram Matrix for ANOVA/OLS with categorical covariates are just conditional sums that we can pluck from the aggregated data table with minimal additional calculations.

Intentional Data Collection and Use

Of course there are use cases where it is appropriate and necessary to collect more specific, and identifiable information.  However, it is often the case that we can answer the very same questions without ever needing to.  The main point is rather than collecting, linking, and storing all of the extra information by default just in case it might be useful, we should be intentional for each use case about what is actually needed, and proceed from there.

Lastly, we should note that disassociation methods and data minimization by default are not substitutes for proper privacy policies – perhaps one could argue that alternative approaches, such as differential privacy might be, but generally the types of methods discussed here are not). Rather, privacy engineering is a tool that sits alongside required privacy and legal compliance policies and contracts.

If you would like to learn more about Conductrics please reach out here. 

A First Look

In this post we will take our first look at some of the new features the Conductrics Market Research release. We will follow up over the coming weeks and months with more peeks and details of the new capabilities. 

As we roll out the new integrated platform, customers will gain access to Conductrics Market Research and see it as a new top-level capability along side A/B Testing, Predictive Targeting, and Rule-Based Targeting.

When creating a new survey, along with the standard survey creation options, there are three primary additional features that are unique to Conductrics. 

1 Survey Responses as A/B Test Goals

When creating a numeric survey question you will be given the option to ‘Track as a Goal/Conversion’ event. If this option is selected then the value of this question can be used as a goal in any of your A/B Tests – just like any other behavioral measure.   

For example, say we have a three question survey and we would like to capture ‘How Likely To Recommend’ as a survey response, but we would also like to see how an upcoming A/B Test might affect the likelihood to recommend. By clicking the option to track the Recommend question as a goal, all new A/B Tests will be eligible to use this survey response as a conversion goal.

2 Auto Create NPS Goal

While any metric survey question can be used as an A/B Testing goal, often what the C-Suite really cares about is changes to NPS. By also selecting the optional ‘Enable for Net Promoter Score (NPS)’ as shown in the image above, Conductrics will auto create 4 NPS related goals:

1. NPS score (% Promoters – % Detractors)*100
2. Number Promoters
3. Number Detractors
4. Number of Passives. 

What is so powerful about this is that you can now see NPS scores, and their associated confidence intervals and testing statistics, in your A/B Testing reports along with your sales and conversion goals.

We believe this is the only survey solution, even including dedicated survey tools, that provides confidence intervals around NPS and the ability to use NPS in A/B Tests.

3 Real-Time Customer Experience Customization 

As part of the Survey creation workflow there is also the option of assigning survey responses as user traits for filtering and enriching reporting for A/B Tests, or to creating real-time customized in-session customer experiences.

For example, if we wanted to customize the customer experience based on how a user responded to the “What is the Purpose of Your Visit?” question, we just select the ‘Retain Response as Visitor Trait’ in the question setup and the value of ‘Purpose of Visit’ will automatically be sent into an in-session Conductrics visitor trait ‘Purpose’

This makes this information immediately available to all reporting, A/B Testing, and Predictive Targeting modules within Conductrics.

To customize the in-session experience we can use Conductrics Rules-Based Targeting module. Once the above survey is saved it will auto-populate Conductrics User Targeting Conditions builder.  Below we see that the rules builder auto generated a user trait called ‘Purpose’ that has the four associated survey response values as options.  These can be used either directly or in any logical combination with any other set of in-session visitor traits.

To keep it simple, we set up a collection of rules that will trigger the relevant in-session user experience based on just the user’s responses to the ‘Purpose of Visit’ question. In the follow screen shot, we show our variations, or user experiences, on the left, and on the right we have our targeting rules that will trigger and deliver the experiences based on the user’s response to the ‘Purpose of Visit’ question. These user experiences can be content from Conductrics web creation tools, backend content like feature flags, or even different server side algorithms.

For example if a customer submits a survey and answers the ‘What is the Purpose of Your Visit?’ question with ‘To Purchase Now’’, then they will immediately be eligible for the targeting rule that will deliver ‘Product Offers’ content. 

Survey Response Data

Of course, Conductrics Research also provides survey reporting to aide human decision making. Conductrics provides simple to use filtering and summary features to help you more easily understand what your customers are hoping to communicate with you.

Along with the tabular view, there is also a card view, so that you can ‘flip through’ individual survey responses and see all an individual customer’s responses enriched with all of their associated in-session data that has been made available to the Conductrics platform.

Optionally, you can download the in-session enriched survey data for use in your favorite statistical software package.

Are Surveys Affecting Conversions?

An important question that arises with in-session market research is if offering a survey in a sensitive area of the site might adversely affect customer purchase or conversion. Conductrics provides the answer. Simply assign a behavioral conversion goal, like sales, or sign-up, to the survey. Conductrics will automatically run a special A/B Test alongside the survey to test the impact that offering a survey has on conversion by comparing all eligible users who were offered a survey vs those who were not offered.

Now your market research teams can learn where and when it is okay to place customer surveys by determining if conversions actually are affected and if so, if it is by a large enough amount to offset the value of direct customer feedback.

For over a decade, Conductrics has been innovating and delivering industry-first capabilities. We are very excited to be the first to combine direct voice of consumer data alongside traditional experimentation data to provide you with the integrated capabilities needed to create better user experiences, drive long-term loyalty and increase direct revenue. This is just a first quick look at some of the unique capabilities that are available to your experimentation and research teams that are rolling out at Conductrics. 

To learn more contact us here.