Product Page Optimization: A Guide to App Store A/B Testing

Creative A/B testing is not new to the app stores. Google Play Experiments has existed for about 5 years now, allowing Android app developers to test different aspects of an app’s metadata and uncover the most effective one for conversion rate optimization.

Now, the latest release of iOS 15 product page optimization allows developers on the App Store to test up to 3 different treatments of their app product page to find the best-performing variant. So, does this long-awaited iOS feature live up to app marketers’ hopes? Let’s explore this topic in this blog, where we discuss the following:

• What is iOS 15 product page optimization (PPO)?
• Why is PPO important?
• Which assets can you test with PPO?
• Testing different assets of your product page
• Important considerations to create A/B tests
• Challenges of A/B testing
• How to set up your first PPO test
• Best practices for A/B testing on the App Store

What is iOS 15 product page optimization?

Product page optimization for iOS 15 is a new feature that allows you to optimize your store assets across the App Store, providing a new way to boost your conversion funnel on iOS. It offers the opportunity to experiment with different variants of your product page assets and find out which ones are the most compelling when getting App Store visitors to download your app/game.

Why is product page optimization important to ASO?

App marketers and developers hail iOS 15 product page optimization as a game-changer for ASO. Native to the App Store, this A/B testing feature serves to help optimize your app’s conversion rate, not only enabling you to stay ahead of the competition but also (ideally) leading to a considerable boost in your app downloads. It is a powerful tool to understand how creative asset optimization affects the decision-making process of potential users and, in turn, can help you create your winning creative set.

Which assets can you test with product page optimization?

With the iOS 15 update, you can test 3 different app creatives on the App Store – screenshots, app icons, and app preview videos – and compare up to 3 different product page versions against the original.

Developers can A/B test app icon, screenshots, and preview videos on the App Store

You can select the percentage of users to whom the treatments will be randomly shown, and also choose the traffic allocation for the 3 different versions. Tests can run for up to 90 days. You may also localize the test in different languages, but you can only do so for those languages for which you have created a local app page. Finally, you will need to submit the product page test assets for review, independent from the app binary or app build. When A/B testing app icons, you have to submit the different versions in the app binary and go through the standard review process for your app build.

The biggest difference with Google Play Experiments is that Apple only allows creative assets to be tested (on Google Play, you can also A/B test the short and long descriptions). PPO also requires you to submit your creative tests for review by Apple, while Google Experiments allow you to set up your tests anytime. Furthermore, you do not have to release a new app version each time you run a test on your Android icon, whereas icon experiments on iOS require you to include your icon in the binary of your new app build. On the App Store, the maximum duration of A/B tests is 90 days, while you can run A/B tests on Google Play for an unlimited period.

Testing different assets of your product page

Before starting to create product page tests with PPO, think through what you want to test and why. For some, revamping screenshots will be the main priority; for others, whether to add an app preview video is the principal concern. Assess which elements of your brand or product matter most to your current users and how you can make your app stand out.

Consider these best practices when testing different assets of your product page with PPO:

Test by features

Test different elements to identify the most appealing features, value propositions, and engaging visuals of your app. Does highlighting the value proposition on screenshots lead to an increased conversion rate? Or does showcasing a screenshot of an existing feature bring more downloads than a screenshot of a new feature?

For example, the travel app Expedia could experiment with which value proposition to show on the first screenshot – either the option to book with a free cancellation or the benefits of becoming a member. The test results would then help Expedia choose the creative that resonates with users the most and persuades more people to download the app.

Expedia could A/B test screenshots showcasing different value propositions

Test by theme

Testing seasonal content to find out whether updating your app icon or other assets to reflect seasonality helps increase your app conversion rate.

For instance, it is a good idea for shopping apps to experiment with creatives promoting deals and offers during holiday seasons and festivities to see which images convince users to download their app. Games can reflect the holiday season by testing different holiday-themed challenges or the seasons, such as snow or Halloween.

Test by new content or characters

Testing to understand whether displaying new content leads to more engagement than regular content, or highlighting culturally relevant content leads to increased downloads in a specific location.

For instance, the game Harry Potter: Puzzles & Spells could run tests on its icon and include different Harry Potter characters to identify which one resonates best with its audience.

Harry Potter: Puzzles & Spells could experiment with different icons to find the better-performing one

Test by creatives

This can include testing color variations of a screenshot background to find out which variation performs the best, or if adding an app preview video results in improved conversion rates.

For instance, the food & drink app Yelp recently revamped the design of its screenshots. Now, with product page optimization, the app could have actually run an A/B test to better understand the conversion uplift before implementing.

Yelp could run an A/B test to better understand the screenshot variant that leads to more conversion

Important considerations to create A/B tests on the App Store

To get the most out of your product page optimization efforts, you should keep these factors in mind:

Traffic proportion

This is the percentage of visitors that will randomly view a treatment version of your product page assets. You can choose the amount of traffic you want to send to your original product page vs alternate versions. For example, if you allocate 50% of your traffic to the test, the remaining 50% will be directed to the original product page. The 50% allocated to PPO will be distributed between each treatment (if you have two treatments, each one will get 25% of your entire traffic).

Conversion rates

It is best to think about conversion rates before allocating traffic. If you are not sure about how the new variant will perform, you can choose to direct more traffic to your original product page. However, at AppTweak, we recommend splitting equally traffic between the original page and the different treatments to get the most accurate results.

Assets and treatments

Think through the product page asset you want to include in the test and the number of versions you will create. We recommend testing one asset at a time to avoid a cluttered result.

Localization

Ask yourself which localizations are worth including in your test. For instance, if you want to perform bold A/B tests for your English app page but have to adhere to strict brand guidelines and don’t want to expose your main audience to the test, you can carry out the A/B test instead in Australia or New Zealand, which have much smaller audiences than the USA. You should also keep in mind that when adding more than one locale to your test, results will be presented for all locations at once and not show whether a variant performed better in one locale and not another.

Test duration

Consider the duration of your test. When launching your A/B test on the App Store, the tool provides a time estimate on the basis of the performance of your app product page, setting a target of 90% confidence. You need to choose your desired improvement value to get the estimate. The test should run until a considerable size of your potential users has visited all of the versions tested. However, tests cannot run longer than 90 days.

Challenges of A/B testing with product page optimization

A/B testing on the App Store, though long anticipated, has not yet lived up to many developers’ expectations. Instead, App Store developers have seemed to face a number of challenges:

1. Product page optimization variants are only shown to App Store users with iOS 15 or later. Users coming from external channels or older versions will see the default product page.
2. Before testing, all of the test treatments (icons, screenshots, and app previews) must be reviewed by the App Store and this can take up to 24 hours. App icons need to be included in the app binary of your next version update.
3. You can only run a product page test if your app build is “ready for sale.”
4. PPO will extend through the entire customer journey. Since the icon variants are in the app binary, the variant users see in the store will be the same one they see on their phone when the app has been downloaded.
5. You can run only one test at a given time.
6. When you submit a new app version, your test will end immediately. This is problematic for teams who are on a one- or even two-week release schedule. Every new version release will automatically end any running tests, thus considerably restricting the window of time in which your test will need to reach statistical reliability.
7. If you run a test for multiple locales, you won’t be able to distinguish between them in reporting. So, if you want to better understand test results per locale, you will have to create a different test per locale. Since you can only run one test at one time, you will have to do this sequentially, which can take a very long time.
8. Tests show really low confidence intervals that make it difficult to reach any conclusion about the test and decide whether to implement one of the versions tested. To help with low confidence issues, AppTweak advises to run A/B/B tests. This means you create a test with 3 variants, with 2 of the 3 variants being exactly the same. If both “B” variants show the same results, you can be more confident about the potential impact.

Learn more about A/B/B testing on Google Play

How to set up your first PPO test

We have compiled a comprehensive checklist to help you smoothly run your first iOS 15 product page optimization test:

Step 1: Create test

To create a test, select “Product Page Optimization” on the left menu. Click “Create Test” if you haven’t previously created a test, or hit the add (+) button beside “Product Page Optimization.”

Source: Apple Developer

Step 2: Give your test a name

Create a test name (up to 64 characters). Make it descriptive so you can easily identify your test when viewing results in App Analytics. For instance, “Easter 2022 Screenshot Characters Test with 3 characters.”

Step 3: Choose the number of treatments

You can include up to 3 treatments.

Source: Apple Developer

Step 4: Determine your traffic allocation

Choose the proportion of users who will be randomly selected to be shown a treatment instead of your original App Store product page. We recommend you split traffic between your default product page and the different treatments equally.

Step 5: Select the localization

If you are localizing the test, select the locale. Only locales within the current app version can be included in the experiment.

Source: Apple Developer

Step 6: Calculate your test duration

To estimate the time for you to achieve your goal, click on “Estimate Your Test Duration” and choose the improvement that you desire in the conversion rate. The test will run for a maximum of 90 days to help you determine if you have reached your desired results within that time. You can also manually end the test in that time span. Note that the more treatments you add to a test, the longer it may take to reach definite results.

Source: Apple Developer

Step 7: Launch your test treatments

Click “Create Test.” Try to limit the number of elements you want to change in a treatment at a given time so that you can easily identify the one that actually impacted the results. All your creative assets have to be accepted by App Review before you can use them. However, unless it is an app icon, you do not need to release a new version to launch the test.

Step 8: Monitor your test

On App Analytics in App Store Connect, you will be able to evaluate performance metrics – such as conversions, unique impressions, improvement, confidence level, etc. – a day after running the test. You can measure these metrics against your baseline, which, by default, is your original product page.

Step 9: Implement the treatment

On the basis of your test results, you may want to deploy a treatment to your original product page so that it shows to all the App Store visitors. Applying a treatment while your test is still running will end the current test. Therefore, before hurrying to implement a treatment or ending a test, we recommend waiting and observing whether at least one treatment performs better or worse than the current baseline.

Source: Apple Developer

Best practices for A/B testing on the App Store

With the PPO feature only launched recently on the App Store, it can take some time before you know your way around it. That’s why we have put together a few best practices to help you best take the advantage of product page optimization with iOS 15.

1. Set up your tests based on the strength of your hypothesis

Carry out internal and external research to know your product strengths, best practices, and industry trends. Next, come up with a defined hypothesis and a predicted outcome. Ask yourself how the hypothesis could affect the user and/or which stakeholders may profit from the test results. For instance, testing a new background color can be either minor or significant depending on why the new color is being tested (for example, to assess whether users prefer a particular color or to make your screenshot elements more clear).

2. Do not replicate Google Play’s A/B strategy for iOS

Another important consideration for A/B testing on iOS is to test different creatives than you would on Google Play. Apple and Google are two diverse platforms with marked differences in user behaviors, app popularities, and – most importantly – user interfaces, meaning that identical creatives would not be seen the same way.

3. Do not be in a hurry to end the test too early

It is ideal to wait until the console completes the test. For instance, if a game’s top users are more active on the weekend and you stop a test only 3 days after starting it on Monday, the results will not be accurate and might, in fact, affect your future test cycles. We, thus, recommend running tests for at least one week and in increments of 7 days.

4. Look out for external factors that can impact installs

For example, holiday seasons or festivities like Black Friday or Cyber Monday can affect the installs of shopping apps. Also, watch out for new app releases that will end a test immediately.

5. Optimize for the right audience

Choose the right audience for your app or game. Knowing your users and where they come from can help a great deal in creating the right optimization techniques. Also, if your app/game is available in other markets, localizing or culturalizing your app store assets to specific locales or cultures is an important consideration for test plans.

6. Test one asset at a time

It’s best to test only one store listing asset at a time. While testing, also ensure that your test variations are notable and significant enough to make a difference. Minor changes might not lead to desired results.

7. Iterative testing

Your optimization work is not a one-off. Remember that not all experiments produce a positive result. In that case, reassess your testing plan and test again. Analyze your test results and identify the treatment that performs the best. Once you get conclusive results, update your product page with the better-performing creative.

Conclusion

Product page optimization on iOS 15 is only in the early days of its release, and there are several things yet to be explored. But, in its current form, A/B testing on the App Store is not yet a competent solution for the mobile market and ASO. Having said that, we believe that Apple’s new custom product pages are a hopeful prospect for App Store developers with many helpful use cases.

Have more questions about A/B tests on the App Store? Don’t hesitate to shoot us a message to know more about how AppTweak can help enhance your ASO strategy with iOS 15.