Abstract
Many software users give feedback online about the applications they use. This feedback often contains valuable requirements information that can be used to guide the effective maintenance and evolution of a software product. Yet, not all software users give online feedback. If the demographics of a user-base aren’t fairly represented, there is a danger that the needs of less vocal users won’t be considered in development. This work investigates feedback on three prominent online channels: app stores, product forums, and social media. We directly survey software users about their feedback habits, as well as what motivates and dissuades them from providing feedback online. In an initial survey of 1040 software users, we identify statistically significant differences in the demographics of users who give feedback (gender, age, etc.), and key differences in what motivates them to engage with each of the three studied channels. In a second survey of 936 software users, we identify the top reasons users don’t give feedback, including significant differences between demographic groups. We also present a detailed list of user-rated methods to encourage their feedback. This work provides meaningful context for requirements sourced from online feedback, identifying demographic groups who are underrepresented. Findings on what motivates and discourages user feedback give insight into how feedback channels and developers can increase engagement with their user-base.
Similar content being viewed by others
Notes
App stores comprise typical sources of apps, such as the Apple app store, or the Google Play Store, where users can provide written feedback and star ratings for apps. Product forums are websites separate from store pages and devoted to specific products or companies. Social media include outlets such as Facebook, Reddit, Instagram, and allow users to comment and share feedback without special moderation, oftentimes on dedicated company pages.
References
Pagano D, and Maalej W (July 2013)“User feedback in the appstore: An empirical study,” in 2013 21st IEEE international requirements engineering conference (RE), pp. 125–134
Guzman E, Alkadhi R, Seyff N (Sep. 2016) “A needle in a haystack: What do twitter users say about software?” in 2016 IEEE 24th international requirements engineering conference (RE), pp. 96–105
Tizard J, Wang H, Yohannes L, Blincoe K, “Can a conversation paint a picture? mining requirements in software forums,” in, (2019) IEEE 27th international requirements engineering conference (RE). IEEE 2019:17–27
Guzman E, Alkadhi R, Seyff N (2017) An exploratory study of twitter messages about software applications. Requirements Engineering 22:387–412
Guzman E, Ibrahim M, Glinz M, “A little bird told me: mining tweets for requirements and software evolution,” in, (2017) IEEE 25th international requirements engineering conference (RE). IEEE 2017:11–20
Maalej W, Nabil H (Aug. 2015) “Bug report, feature request, or simply praise? on automatically classifying app reviews,” in 2015 IEEE 23rd international requirements engineering conference (RE), vol. 00, pp. 116–125
Sorbo AD, Panichella S., Alexandru CV, Visaggio CA, Canfora G (May 2017)“Surf: Summarizer of user reviews feedback,” in 2017 IEEE/ACM 39th international conference on software engineering companion (ICSE-C), pp. 55–58
Guzman E, Rojas AP, “Gender and user feedback: an exploratory study, in, (2019) IEEE 27th international requirements engineering conference (RE). IEEE 2019: 381–385
Guzman E, Oliveira L, Steiner Y, Wagner LC, Glinz M (2018) “User feedback in the app store: a cross-cultural study,” in 2018 IEEE/ACM 40th international conference on software engineering: software engineering in society (ICSE-SEIS). IEEE, pp. 13–22
Tizard J, Rietz T, Blincoe K (2020) “Voice of the users: A demographic study of software feedback behaviour,” in (2020) IEEE 28th international requirements engineering conference (RE). IEEE:55–65
Groen EC, Seyff N, Ali R, Dalpiaz F, Doerr J, Guzman E, Hosseini M, Marco J, Oriol M, Perini A et al (2017) The crowd in requirements engineering: the landscape and challenges. IEEE Softw 34(2):44–52
Johnson D, Tizard J, Damian D, Blincoe K, Clear T (2020) “Open crowdre challenges in software ecosystems,” in 2020 4th international workshop on crowd-based requirements engineering (CrowdRE), pp. 1–4
Panichella S, Di Sorbo A, Guzman E, Visaggio CA, Canfora G, Gall HC (2016) “Ardoc: App reviews development oriented classifier,” in Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, ser. FSE 2016. New York, NY, USA: ACM, pp. 1023–1027. [Online]. Available: http://doi.acm.org/10.1145/2950290.2983938
Chen N, Lin J, Hoi SCH, Xiao X, Zhang B (2014) “Ar-miner: Mining informative reviews for developers from mobile app marketplace,” in Proceedings of the 36th international conference on software engineering, ser. ICSE 2014. New York, NY, USA: ACM, pp. 767–778. [Online]. Available: http://doi.acm.org/10.1145/2568225.2568263
Khan JA, Xie Y, Liu L, Wen L, “Analysis of requirements-related arguments in user forums,” in, (2019) IEEE 27th international requirements engineering conference (RE). IEEE 2019:63–74
Tizard J “Requirement mining in software product forums,” in 2019 IEEE 27th international requirements engineering conference (RE), 2019, pp. 428–433
Buhrmester MD, Kwang T, Gosling S (2011) Amazon’s mechanical turk. Perspec Psychol Sci 6:3–5
Marder B, Gattig D, Collins E, Pitt L, Kietzmann J, Erz A (2019) The avatar’s new clothes: understanding why players purchase non-functional items in free-to-play games. Comput Human Behav 91:72–83
Bleize DN, Antheunis ML (2019) Factors influencing purchase intent in virtual worlds: a review of the literature. J Market Commun 25(4):403–420
Stade M, Seyff N, Baikenova A, Scherr SA, “Towards a user feedback approach for smart homes: An explorative interview study,” in, (2020) 4th international workshop on crowd-based requirements engineering (CrowdRE). IEEE 2020:5–10
Papadopoulos N, Martín OM, Cleveland M, Laroche M (2011) Identity, demographics, and consumer behaviors. International Marketing Review (2011)
“New Zealand census, 2018,” https://www.stats.govt.nz/information-releases/2018-census-population-and-dwelling-counts, accessed: December 2019
Likert R (1932) “A technique for the measurement of attitudes.” Archives of psychology (1932)
ISCED U (2012) “International standard classification of education 2011,”
Galitz WO (2007) The essential guide to user interface design: an introduction to GUI design principles and techniques. Wiley, Hoboken
Turk AM (2012) “Amazon mechanical turk,” Retrieved August, vol. 17, p. 2012
Guo Y, Barnes S (2007) Why people buy virtual items in virtual worlds with real money. ACM SIGMIS Database: the DATABASE for Advances in Information Systems 38(4):69–76
Etikan I (2016) Comparison of convenience sampling and purposive sampling. Am J Theor Appl Stat 5(1):1
“Qualtrics survey platform,” https://www.qualtrics.com, accessed: December (2019)
Bock O, Nicklisch A, Baetge I (2012) “hroot: Hamburg registration and organization online tool,” in WiSo-HH Working Paper Series
McHugh ML (2013) The chi-square test of independence. Biochemia medica Biochemia medica 23(2):143–149
Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3(2):77–101
Burnett M, Stumpf S, Macbeth J, Makri S, Beckwith L, Kwan I, Peters A, Jernigan W (2016) Gendermag: a method for evaluating software’s gender inclusiveness. Interact Comput 28(6):760–787
Rietz T, Maedche A, “Ladderbot: A requirements self-elicitation system,’ in, (2019) IEEE 27th international requirements engineering conference (RE). IEEE 2019: 357–362
De Oliveira GF, Ferreira B, Marques AB (2020) “USARP method: Eliciting and describing USAbility Requirements with Personas and user stories,” in ACM international conference proceeding series. association for computing machinery 10: 437–446
Ferreira B, Silva W, Barbosa SD, Conte T (2018) “Technique for representing requirements using personas: A controlled experiment,” IET Software, 12(3)
Martens D, Maalej W (2019) Towards understanding and detecting fake reviews in app stores. Emp Softw Eng 24(6):3316–3355
Acknowledgements
The data collection in Zhejiang University was supported by the Provincial Key Research and Development Plan of Zhejiang Province, China (No. 2019C03137).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Tizard, J., Rietz, T., Liu, X. et al. Voice of the users: an extended study of software feedback engagement. Requirements Eng 27, 293–315 (2022). https://doi.org/10.1007/s00766-021-00357-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00766-021-00357-1