User Experiment: Risk Assessment in Code Review
-
Hello there QT developers,
My name is Eve and I'm a master’s student working with Prof. Shane McIntosh and Prof. Mike Godfrey at the University of Waterloo. We're conducting a research study on the effectiveness of risk assessment during code review. Specifically, we're studying whether risk assessment can improve a reviewer’s awareness of risky changes, and if this assessment can affect their performance in identifying defects during code review.
We'd like to perform a study on the Qt codebase since Qt is a large, well-known open-source system with a high rate of code review coverage. Also, Qt has been widely used as a target system in prior related studies, so the results from this work could help to improve the overall understanding of Qt and its development practices.
To help you better understand who we are and what we are doing, here is the link to my page on our lab website: https://swag.uwaterloo.ca/members/xueyao-eve-yu.html. This includes the investigators’ information as well as the context of our project.
For our next step, we'd like to recruit developers from Qt-Base to participate in a short user experiment. We expect it will take each participant about 30-60 minutes in total, and the experiment be performed online and asynchronously using a simple web application. Each participant will be asked to examine three recent code changes (i.e., commits) taken from Qt-Base; they will then be asked to rank the commits by perceived riskiness and also to perform a code review on each commit.
Once the experiment is complete, we'll share our results with the Qt community (and anyone else who is interested). Additionally, if the experiment goes well, we'll release our risk assessment tool as a (free) plug-in to Gerrit. We're hoping that this "carrot" might make you feel more positive about taking part in the study and furthering research into code review. Have we managed to convince you to take part? If so, please use the link below to sign up.
Sign-up link: https://forms.gle/zG16V9jESHZt6Vvx8This study has been reviewed by and received ethics clearance from the University of Waterloo Research Ethics Board. Please note that all your experiment data will be anonymized. You will be assigned a randomly generated participation ID to access the experiment and your identifiable information will not be collected and used in any dataset or publication.
Best,
Eve