‍Fifty-three percent of mobile users will uninstall apps that crash, freeze, or have errors, and 80% will only use these apps if the problems occur three times or less. This is why beta testing is the lifeblood of product development — and even more important in app development because apps are produced faster and require more on-the-fly iterations.
Beta testing is not just asking friends for their opinions on your app or game. You need to have testers who understand your app's goals and mission and can give you actionable feedback to help you improve your product and point out any flaws.
Good beta testing is a process that requires some foresight, careful planning, good recruiting, and data analysis. An effective mobile app beta testing framework is possible if you follow the 7 steps below.
Step 1: Set beta testing goals
Every good beta test starts with a goal because goals give you direction on following up and measuring your success. Without goals, the process becomes a long, drawn-out affair leading to delays. Before running a beta test program, you have to ask high-level questions first.
What are you looking to accomplish with your beta test?
If your goal is to ensure the stability of your app, you should use Ongoing beta testing. In this scenario, beta testing happens continuously. For example, if the latest version of the app is 4.1, then 4.2 is the current beta version. As soon as 4.2 is released, 4.3 is then in testing. In ongoing beta testing, you'd want to use superusers and internal beta testers who are already familiar with using the stable version of your app. Because of their familiarity with your app, they'll be able to spot bugs that affect your app's function in your newer version quickly.
On the other hand, if you’re looking for valuable feedback on new product features, then Finite beta testing might be more effective. TikTok recently added a retweet-type feature called Repost. Beta testers needed to find out if this feature worked, if the added functionality was easy to grasp and use, and whether it took away from or enhanced the app. In this case, beta testing was used more for product feedback than actual bug testing. Because this testing phase is only concerned with one feature, it has a clear and “finite” scope.
How many testers do you need?
According to Ubertesters, a good rule of thumb is “...4 or 5 beta testers for small independent projects. For casual app startups ... 10 or 25 beta testers. For B2B applications... 70 to 300 testers. ... for your gaming app ... 200 or more testers.” This number will vary with your individual needs, budget and the scope of your app. If you use too few testers, the results won’t be enough for you to qualify changes. Using too many testers runs the risk of too much feedback, which can be a problem if your team is unable to manage it all. As a result, some of the feedback might fall through the cracks.
How long should the beta test last?
Make sure to establish a time frame for when you want beta test feedback returned to you. Be specific, and hold your testers to that deadline. It’s important that you don't hang up on development while waiting for beta test results. A general rule of thumb is that tests should be no shorter than one-two weeks and no longer than 12. A majority of tests usually run about four to six weeks.
Step 2: Select your beta testing platform
A beta testing platform can help you find testers, receive and manage user feedback, and identify product improvements and application bugs.
Most testers will use TestFlight (iOS operating system) or Google Play Console (Android operating system) for mobile app beta testing. These are the two most popular and widely used native beta testing platforms available.
TestFlight is a free-to-use Apple-owned beta testing solution that manages the beta testing of your app. By sharing a link or emailing users directly, you can ask up to 10,000 testers to help test your product. It’s a simple setup, feedback is done through the application, and it fires off notifications as soon as new test builds are ready. In its WWDC 2021 event, Apple announced that TestFlight has come to Mac to manage builds and testers, collect feedback, and deploy macOS apps.
Google Play Console (GPC) is a beta test program that is Android app-specific. It offers Open Beta, which allows anyone with a link to sign up as a tester, and Closed Beta, where you can restrict the number of testers by limiting signup to email-only invitations. Another great feature is GPC’s Alpha Testing, which lets you test out features with a select group of testers before you send them to beta.
If you find that either of those are missing features you need, consider these beta testing tools, which may have more specialized components for your conditions.
- Centercode: All-around solution, good for managing large teams, setup is complicated
- Ubertesters: Great, easy-to-use design, segments problems so teams can work on specific areas, no web integration
- TestRail: Excellent all-around answer but doesn't play well with Jira
- UserTesting: Top-of-the-line customer support, great design and customization, but no web app integration
- DeployGate: Great app-sharing feature, creating tester accounts is automated but limited third-party integration, does not have reporting or feedback in-app. Bitrise offers a DeployGate Upload Step. Learn more about it here.
- TestFairy
Step 3: Recruit the right testers
What you need in beta testers is diversity — a good combination of motivation and skills that closely matches your target customers' user demographic.
Recruiting testers first depends on whether it's a new or existing product. If you're beta testing an update or new feature, you may want to dive into your existing customer base or post a question in a forum. Asking users who genuinely want to see improvements made with new revisions is another great place to start.
If the product is brand new and you don't have an established base of power users, consider using professional beta testing hires. Research spaces like Reddit, BetaList, Betabound, or Beta Family, or look to high engagement sites like Quora and The Hacker News.
Look to your product community and experts first and for free. Then consider a paid service. There’s nothing wrong with using friends and family to beta test your product, but are you going to hold friends and family to an eight-hour, five days a week schedule? And without the motivation of paying them, can you expect their commitment to be practical? If your beta testers are your power users, they'll know how to use the app and where the bugs are likely to occur.
Step 4: Gather feedback
One of the most important stages of mobile app beta testing is getting usable feedback and turning it into actionable items.
Feedback should be a continuous process. It should be given during testing and right after testing. Most test programs have feedback processes built-in. TestFlight, for instance, has the awesome shake feature, which allows testers to shake their device when they want to send feedback. There’s more than one way of getting bug reports — leaflets, online feedback forms, face-to-face meetings, and both individual and group comments. Use all of your feedback data from every source. Ask your testers which feedback delivery system is most effective for them.
You can keep feedback anonymous so that testers will be more honest, but distribute and share all feedback with every member of the testing and engineering team. It encourages cooperative suggestions for improvement. And make sure you acknowledge the feedback so that testers feel their input was worthy. You have to create a relationship with the tester by building trust and loyalty, or else they may not come back — or, worse, comment negatively.
Step 5. Organize feedback
In practice, feedback is chaotic — different users will express the same problem in different ways, and cutting through to the common issues for faster resolution is part of the beta process. You might get one piece of feedback that’s relevant to multiple features. This is why you should triage and organize feedback by adding it to your issue tracking or product feedback system (like Jira, for instance). Once organized, you then have to prioritize which issues are most important to address. Beta testers should be reporting on these basic elements of each problem they find:
- What went wrong?
- What did they expect to happen?
- What actually happened?
- Can you replicate the conditions to make it happen again?
Once you’ve received this categorized feedback, it will make it easier to implement changes because you now have a history of the particular bug. Without this checklist, you will find yourself with disorganized and confusing feedback.
Step 6: Act on feedback
Be selective with changes — not every change from beta testing should be acted on. Determine which changes are bug fixes (functional problems) and which are product improvements (enhancements to your existing product). You have to then decide what is most important (and cost-effective) to address from a developer standpoint and what can be fixed pre-release vs. post-release so that your app doesn't experience unnecessary delays.
Optionally, you may want to address minor bugs that are easy to resolve immediately (although there are no "minor" bugs to a coder!). This might help move the testing along a little faster. Make sure to update with a new beta release as soon as realistically possible.
Step 7: Reward your testers
A really good tester with passion and understanding of the product may be hard to find. Your mobile app beta testing platform should keep a list of your best testers. Since internal testers and super testers basically work for the betterment of the app or product, reward your testers after they’ve given usable feedback. Consider coupons, promo codes, product discounts, or other meaningful bonuses. This will build trust and loyalty among committed testers and allow you to keep them available for your next beta testing process. Using a tester who already knows your system is much easier than training a tester from scratch.
Build trust and loyalty among committed testers and allow you to keep them available for your next beta testing process.
Be careful, however, to keep the rewards discreet until after testing is done. If you reveal a significant reward at the beginning of the testing phase, the feedback may be too positive and complimentary, and if the premium is less than appealing to the tester, they may not join at all.
Beta testing shouldn't be an afterthought — it's invaluable in creating excellent mobile apps
Run a bad beta testing program, and you may be infinitely delayed — or worse, abandoned — when your product meets the real world. A bad beta test is almost worse than no beta testing at all and results in a poor user experience. Beta testers are there to protect your product from failure, help you find what changes should be made, and offer suggestions on what the real user may be looking for. When you effectively whip mobile beta testing into your development process, it makes your product that much better.
Bitrise is committed to ensuring mobile teams improve how they build, test, and deploy their applications. That's why we've partnered with tools like DeployGate to make mobile app beta testing a seamless process using a single Step. Use DeployGate Verified Step to distribute your new apps to testers by creating a landing page for quick sign-up and also receive testers' feedback.
Useful Resources
- Deploying an iOS app for external testing - Bitrise Docs
- iOS code signing - Bitrise Docs
- Android code signing - Bitrise Docs
- Indie Developers: Automatic deployment to the Play Store with Bitrise
- How indie developers benefit from Bitrise: Deploy automatically to your testers
- Build a Flutter iOS app and deploy it to TestFlight using Bitrise CI/CD platform
- Simplifying automatic code signing on Bitrise
‍
