Skip to main content

Target users

Experiment-target

To begin, you'll need to determine which users will be eligible for assignment to the experiment. In the Target Users section of the experiment panel, you can select from various user groups or apply a specific rollout percentage to control the exposure of your experiment.

User Groups​

  1. All Users

    • Definition: This group encompasses every user interacting with your platform, without applying any filtering criteria.
    • Use Case: Choose this option if you want the experiment to be accessible to the entire user base.

  2. New Users

    • Definition: This group consists of users who have recently joined or interacted with your platform for the first time.
    • Use Case: Select this group when you want to test features or experiences specifically designed for first-time users.

  3. Repeat Users

    • Definition: This group includes users who have returned to your platform multiple times after their initial visit or interaction.
    • Use Case: Use this group to test features or changes aimed at users who have a history of engagement with your platform.

  4. Cohort

    • Definition: A cohort is a specific segment of users defined by particular criteria, such as demographic information, behavioral data, or custom attributes.
    • Use Case: Target a cohort when you want to focus the experiment on a specialized group of users, such as those from a particular location, users with specific purchase behavior, or any other defined criteria.
    • For more details, see How to Create a Cohort.

Rollout Percentage​

In addition to selecting user groups, you can apply a specific rollout percentage to your experiment. This approach allows you to initially expose only a small portion of users to the experiment, helping to mitigate potential risks. If the initial results are positive, you can gradually expand the experiment to a larger audience to achieve statistical significance.

Rollout Strategy​

Gradually rolling out your A/B tests is crucial to managing risks and ensuring that any potential issues are contained before they affect your entire user segment/base. The rate at which you roll out your A/B tests should be determined by the size of your total audience and how quickly you need results.

  1. Starting the Test with a Smaller Audience

    • The Rollout section allows you to gradually release your A/B tests by specifying the percentage of targeted users included in the experiment. Users are then randomly assigned to the control or variants based on their distribution weights.

  2. Increasing the Audience Size Step-by-Step

    • Initially, 100% of users who meet your targeting criteria are included in the A/B test by default. If you want to start with a smaller audience, reduce the percentage before starting the test.

    • For example, reducing the audience size to 1% will initially include only a small portion of the targeted users in the experiment, leaving the rest excluded until you decide to increase the audience size.

    Note: Once the test is launched, you cannot decrease the audience size—only increase it. This restriction is because targeting changes only apply to new users; those already assigned to a test group will remain in their assigned variant. Make sure to start with a smaller audience before publishing if you plan to increase it gradually. This approach helps Percept Insight ensure that your experiments are conducted safely and that user participation is consistently managed throughout the experiment.

  3. Expanding the Audience Gradually

    • When ready, you can roll out the experiment to a wider audience. For example, increase the audience from 1% to 50%, and later to 100%.

  4. Avoid Modifying the Distribution Rate Among Variants During the Test

    • Adjusting variant distribution at low rollout percentages can be acceptable to build intuition or explore early trends. However, doing so at higher rollout percentages can lead to severe issues.
    • Altering the distribution can introduce bias and complicate the interpretation of results.
    • Statistical significance becomes meaningless if the distribution is changed, leading to mismanaged user allocation.

  5. Choosing a Winner and Concluding the Test

    • Once your experiment has been rolled out to a sufficient portion of the audience and has achieved statistical significance, select the winning variant. You can then deploy it to your desired group of users, whether that be all users or your specific target segment.

This staged rollout plan can minimize risks associated with testing new features, ensuring that any issues are detected and mitigated early in the process.

Exclusivity​

Users assigned to this test will be exclusive. From the qualifying segment, only unique users who are not part of any other test will be selected. Once assigned, these users are excluded from all other tests until this test is completed, rolled out, or they are disqualified. This approach ensures clean and uncontaminated results, as users remain engaged in a single experiment at a time.

Note: If a user ID is not available, a unique ID will be used to distinguish between users.