A Classification of Post-Release Stability Monitoring Practices for Mobile Applications with a Large User Base
Keywords:
analytics integrity; crash reporting; flaky tests; mobile applications; phased rollout; post-release monitoring; silent failuresAbstract
A mobile release is not finished when a version reaches the store: users decide when to update, several versions stay in use, and failure signals arise on devices the release team does not control. Post-release observation is therefore a distinct engineering practice, yet research treats its objects separately, as problems of crash analysis, lifecycle testing, review mining or flaky tests. This review classifies post-release stability monitoring practices by the visibility mode of a failure, that is, by how the failure becomes observable once a version is in users' hands. The classification draws on a bounded literature corpus, platform documentation and the author's own release practice on a retail application with a large audience. Five classes are distinguished: self-announcing failures (crashes and hangs), silent failures (unexpected states; lifecycle and restoration failures), baseline-relative failures (regressions), instrument failures (analytics integrity) and gate-signal failures (flaky tests before release). A synthesis matrix crosses the classes with four release horizons. Audience scale makes only self-announcing failures visible early in a phased rollout; silent and instrument failures need instrumentation designed before release. A flaky test is treated by removing its cause, not by reruns.
References
[1] C. Rossi, E. Shibley, S. Su, K. Beck, T. Savor, and M. Stumm, "Continuous deployment of mobile software at facebook (showcase)," in Proc. 24th ACM SIGSOFT Int. Symp. Foundations of Software Engineering (FSE), 2016, pp. 12–23, doi: 10.1145/2950290.2994157.
[2] M. Nayebi, B. Adams, and G. Ruhe, "Release Practices for Mobile Apps – What do Users and Developers Think?," in Proc. IEEE 23rd Int. Conf. Software Analysis, Evolution, and Reengineering (SANER), 2016, pp. 552–562, doi: 10.1109/saner.2016.116.
[3] K. Glerum et al., "Debugging in the (very) large," in Proc. ACM SIGOPS 22nd Symp. Operating Systems Principles (SOSP), 2009, pp. 103–116, doi: 10.1145/1629575.1629586.
[4] L. Ravindranath, S. Nath, J. Padhye, and H. Balakrishnan, "Automatic and scalable fault detection for mobile applications," in Proc. 12th Annu. Int. Conf. Mobile Systems, Applications, and Services (MobiSys), 2014, pp. 190–203, doi: 10.1145/2594368.2594377.
[5] L. Fan et al., "Large-scale analysis of framework-specific exceptions in Android apps," in Proc. 40th Int. Conf. Software Engineering (ICSE), 2018, pp. 408–419, doi: 10.1145/3180155.3180222.
[6] M. Gómez, R. Rouvoy, B. Adams, and L. Seinturier, "Reproducing context-sensitive crashes of mobile apps using crowdsourced monitoring," in Proc. Int. Conf. Mobile Software Engineering and Systems (MOBILESoft), 2016, pp. 88–99, doi: 10.1145/2897073.2897088.
[7] D. Amalfitano, V. Riccio, A. C. R. Paiva, and A. R. Fasolino, "Why does the orientation change mess up my Android application? From GUI failures to code faults," Software Testing, Verification and Reliability, vol. 28, no. 1, Art. no. e1654, Jan. 2018, doi: 10.1002/stvr.1654.
[8] Z. Shan, T. Azim, and I. Neamtiu, "Finding resume and restart errors in Android applications," in Proc. ACM SIGPLAN Int. Conf. Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), 2016, pp. 864–880, doi: 10.1145/2983990.2984011.
[9] O. Riganelli, S. P. Mottadelli, C. Rota, D. Micucci, and L. Mariani, "Data loss detector: automatically revealing data loss bugs in Android apps," in Proc. 29th ACM SIGSOFT Int. Symp. Software Testing and Analysis (ISSTA), 2020, pp. 141–152, doi: 10.1145/3395363.3397379.
[10] H. Khalid, E. Shihab, M. Nagappan, and A. E. Hassan, "What Do Mobile App Users Complain About?," IEEE Software, vol. 32, no. 3, pp. 70–77, May 2015, doi: 10.1109/ms.2014.50.
[11] S. Hassan, C.-P. Bezemer, and A. E. Hassan, "Studying Bad Updates of Top Free-to-Download Apps in the Google Play Store," IEEE Transactions on Software Engineering, vol. 46, no. 7, pp. 773–793, Jul. 2020, doi: 10.1109/tse.2018.2869395.
[12] Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, "An empirical analysis of flaky tests," in Proc. 22nd ACM SIGSOFT Int. Symp. Foundations of Software Engineering (FSE), 2014, pp. 643–653, doi: 10.1145/2635868.2635920.
[13] S. Thorve, C. Sreshtha, and N. Meng, "An Empirical Study of Flaky Tests in Android Apps," in Proc. IEEE Int. Conf. Software Maintenance and Evolution (ICSME), 2018, pp. 534–538, doi: 10.1109/icsme.2018.00062.
[14] V. Pontillo, F. Palomba, and F. Ferrucci, "Test Code Flakiness in Mobile Apps: The Developer’s Perspective," Information and Software Technology, vol. 168, Art. no. 107394, Apr. 2024, doi: 10.1016/j.infsof.2023.107394.
[15] O. Parry, G. M. Kapfhammer, M. Hilton, and P. McMinn, "A Survey of Flaky Tests," ACM Transactions on Software Engineering and Methodology, vol. 31, no. 1, pp. 1–74, Jan. 2022, doi: 10.1145/3476105.
[16] A. Bertolino et al., "A Survey of Field-based Testing Techniques," ACM Computing Surveys, vol. 54, no. 5, pp. 1–39, Jun. 2022, doi: 10.1145/3447240.
[17] "Android vitals," Android Developers. [Online]. Available: https://developer.android.com/topic/performance/vitals (accessed Sep. 25, 2026).
[18] "MetricKit," Apple Developer Documentation. [Online]. Available: https://developer.apple.com/documentation/metrickit (accessed Sep. 25, 2026).
[19] S. Agarwal, R. Mahajan, A. Zheng, and V. Bahl, "Diagnosing mobile applications in the wild," in Proc. 9th ACM SIGCOMM Workshop on Hot Topics in Networks (HotNets), 2010, pp. 1–6, doi: 10.1145/1868447.1868469.
[20] A. Fabijan et al., "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments," in Proc. 25th ACM SIGKDD Int. Conf. Knowledge Discovery & Data Mining (KDD), 2019, pp. 2156–2164, doi: 10.1145/3292500.3330722.
[21] J. Harty, "Improving app quality despite flawed mobile analytics," in Proc. IEEE/ACM 7th Int. Conf. Mobile Software Engineering and Systems (MOBILESoft), 2020, pp. 21–22, doi: 10.1145/3387905.3388603.
[22] M. T. Rahman and P. C. Rigby, "The impact of failing, flaky, and high failure tests on the number of crash reports associated with Firefox builds," in Proc. 26th ACM Joint Meeting European Software Engineering Conf. and Symp. Foundations of Software Engineering (ESEC/FSE), 2018, pp. 857–862, doi: 10.1145/3236024.3275529.
[23] "Release a version update in phases," App Store Connect Help. [Online]. Available: https://developer.apple.com/help/app-store-connect/update-your-app/release-a-version-update-in-phases (accessed Sep. 25, 2026).
[24] "Release app updates with staged rollouts," Play Console Help. [Online]. Available: https://support.google.com/googleplay/android-developer/answer/6346149 (accessed Sep. 25, 2026).
[25] Y. Liu, C. Xu, and S.-C. Cheung, "Characterizing and detecting performance bugs for smartphone applications," in Proc. 36th Int. Conf. Software Engineering (ICSE), 2014, pp. 1013–1024, doi: 10.1145/2568225.2568229.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Mikita Burkouski

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who submit papers with this journal agree to the following terms.