Lock n’ LoL is an app that lets people who are together limit their smartphone use. The study involved a 25-day field deployment in 2015, received news coverage at the time, and was presented at CHI 2016.Ko, M., Choi, S., Yatani, K., & Lee, U. (2016). Lock n’ LoL: Group-based Limiting Assistance App to Mitigate Smartphone Distractions in Group Activities. Proceedings of CHI 2016. doi:10.1145/2858036.2858568 My experience of deploying and operating an app also influenced how I examined this research. While reading about the features and results, I found myself considering how people were recruited and what kept them using it. However worthwhile the purpose, people first have to try an app before any changes can be observed. I find myself reading the recorded use alongside the conditions that brought it about.
In Lock n’ LoL, users create a room or join an existing one to limit their smartphone use together. People in the same room can see who is limiting their use and who is temporarily using their phone. Alongside exercising restraint individually, people sharing an activity can see one another’s status. The app does not block all use. Each hour, users can temporarily use their phones for a cumulative five minutes; beyond that, they need to ask a peer for permission. The design allows for the possibility that even while studying, people may need to answer an urgent message or look something up.
In total, 976 people registered and 873 tried limiting their use at least once. They accumulated 10,365 hours in limit mode. That amount of use is noticeable for an app that needs people to use it together. But to read these figures as evidence of a need for the app, or of the size of its effect, I think we also need to consider the circumstances in which people used it and why.
Under What Conditions Was It Used?
The campaign took place on the KAIST campus. The paper describes the deployment as running from after the midterm exams to the end of the semester. The fact that it was a period leading towards the end of the semester needs to be read alongside the results.
The usage records are connected to this context. Studying accounted for 80.27% of the activities selected when starting limit mode. This is the proportion of activity selections, rather than of participants. Most participants were undergraduate or graduate students, and there were also participants from nearby universities and secondary schools. They were not all KAIST students, but it is clear that the app was used mainly for studying by people recruited around a university.
This makes me wonder why they wanted to reduce smartphone use at that time. If the need to concentrate on studying grew as they prepared for final exams, that motivation may also have influenced their decision to use the app and how long they used it. For someone who already wanted to use their phone less, it may have been a tool that arrived when they needed it. These possibilities make it difficult to read the results apart from both the app’s features and the needs of that point in the semester. The paper does not measure how this motivation changed over the deployment period.
There were rewards as well. The first 300 people to accumulate 90 minutes of limiting received a gift voucher worth about $5, and anyone who reached the same threshold was eligible for a prize draw that included a digital camera and Bluetooth speakers. This seems likely to have helped attract the first users. Yet the researchers were also examining how long and how often people limited their use. The behavior required to receive a reward overlapped with the behavior of interest to the researchers. That is why we need to consider whether people used the app because they needed it, to meet the reward threshold, or for both reasons.
In the evaluations after use, self-reported distraction scores fell from 3.30 to 2.36. There were also responses saying the app had helped. But there was no comparison group that did not use the app, or a group with different reward conditions. The change was observed in a setting where the need to concentrate on studying, campaign rewards, and app features were present together. People who began because of a reward may still have found the app helpful, so it would also be wrong to attribute all use to the prizes. What I want to understand is how much of the observed change can be interpreted as an effect of the app itself.
How Were People Recruited to Use It Together?
The term snowball sampling also caught my attention in the recruitment method. The paper explicitly describes its use in the first pilot, in November 2014. Seven undergraduates were recruited at one university and asked to use the app with their friends for a week, resulting in seven groups with 20 participants in total. The researchers listened to their experiences and revised the initial app. After a second pilot, the field deployment in 2015 recruited users through a campus campaign and public app distribution. We cannot read this as saying that all 976 people in that deployment were recruited through snowball sampling.
This seems a fairly natural way to recruit people while revising an app intended for use with friends. Some difficulties may only emerge when people who already know one another try it. But recruitment through friendships may also bring together people with similar circumstances or dispositions. The experiences of people outside that network are less likely to be included.Statistics Canada. Non-probability sampling. We cannot readily expect responses among friends to be the same among colleagues or in groups with strong hierarchies, and that background may also shape the feedback used to revise the app.
Whose experiences are represented and how far the findings can be generalized need to be considered separately from what caused an observed change. Recruitment through friends does not, by itself, establish a confounding influence on an effect. For example, if we compare evaluations from people who use the app more and those who use it less, we need to consider whether people with a stronger prior motivation to reduce their use might both use the app more and report greater changes. Closeness to the people using it with them might also influence both use and evaluation. We would need to know whether these relationships were present before speaking more clearly about causes.MacKenzie, I. S. (2013). Human-Computer Interaction: An Empirical Research Perspective. Chapter 5, Experimental Design.
Distinguishing the pilot from the field deployment does not remove the questions about the conditions of participation in the latter. How much did the need for the app and the rewards each matter to people who saw advertisements on campus and chose to participate? Might a wish to help with the research also have played a part? The paper alone does not allow us to distinguish these motives. With the timing in the semester in mind, it remains possible that people who already had a strong motivation to reduce smartphone use were drawn to the study, or that their motivation changed during the deployment. How much these possibilities explain the results remains unknown.
The researchers did examine usage contexts. They used regression analysis to examine how activities, relationships with other users, time, and location were related to daily limiting duration and frequency. But examining the circumstances in which people use an app more is different from separating the reasons their distraction scores fell after using it. Without a comparison that distinguishes the rewards provided with the app and changes over that period, these conditions remain relevant to interpreting the causes of an effect.
The Notation p < .000
While reading the statistical results, I also paused at the notation p < .000. It appears four times in the paper. Read literally, it means that a p-value, which is a probability, is less than zero. The sixth edition of the APA manual advises reporting p < .001 when the actual p-value is below .001.The p-value reporting guidance in the sixth edition of the APA Publication Manual, as quoted in JASP’s official article “Exact P-Values Upon Request: Breaking with an APA Guideline” I suspect this notation was used in an attempt to emphasize the significance of the results through very small p-values.
Even if the notation were corrected and statistical significance confirmed, the questions about participation conditions would remain. A small p-value does not separate the influence of rewards provided alongside the app. In reading this paper, then, I think the significance of the reported difference needs to be considered alongside the grounds for interpreting that difference as an effect of the app.
What More Did the Follow-up Study Examine?
Following these questions also leads me to the subsequent research. Let’s FOCUS, published in 2017, directly cites Lock n’ LoL and carries its design into classroom settings.Kim, I., Jung, G., Jung, H., Ko, M., & Lee, U. (2017). Let’s FOCUS: Mitigating Mobile Phone Use in College Classrooms. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(3), Article 63. doi:10.1145/3130928 · Author-hosted paper. In particular, it cites Lock n’ LoL as the basis for allowing five minutes of temporary use, and reports checking the suitability of that duration in its own pilot. It builds on the earlier attempt while making the setting of use more specific.
Where Lock n’ LoL reminded users to limit their use with friends nearby, Let’s FOCUS created virtual spaces for individual classes and allowed reminders based on classroom location and class times. This connected the place and time of use for people attending the same class. What draws my attention is the way it retains the idea of limiting use together while also addressing when to turn on the app within recurring classes.
The observation went further as well. During the campaign from September 19 to October 28, 2016, rewards included gift vouchers for accumulating 20 hours of limiting. The researchers then tracked use until the end of the fall semester without offering further rewards. They reported that 117 people continued to use the app, recording an additional 1,224 hours of limiting. Alongside use during the campaign, they examined actual records of whether use continued after the rewards ended.
Having wondered earlier about the difficulty of distinguishing rewards from motives for use, I see this observation as a meaningful addition. But it follows use that began in a campaign with rewards, so it does not establish whether the same participation would have occurred without rewards from the outset. The app’s support also remained available, so this does not mean that an ability to regulate behavior without the app was maintained. Continuing to use an app after rewards end and regulating behavior without its help remain different questions.
Questions about operation remain alongside the continuation of research and design. These records do not establish that Let’s FOCUS was an update to the same app or that existing users moved to it. I am still interested in how improvements and support continued for people who had been using Lock n’ LoL. Even so, there seems to be room to go beyond thinking only about how difficult alternatives would be. There was a subsequent attempt to observe use beyond the initial recruitment with rewards and to adapt the design to its context of use.
Making an App That People Keep Using
We could consider extending the follow-up observations or recruiting a more varied group of people at other times and places for comparison. But when I imagine conducting this research myself, securing the use needed for those comparisons still seems difficult. This is an app that people actually have to use, and the research requires enough users and usage records within a limited period. For an app used together, one person installing it is not enough. There also have to be people to use it with.
Under those conditions, rewards seem likely to have been an accessible option. Beyond getting people to try the app once, researchers can attach conditions that encourage enough use to observe. That is why, from an operational perspective, I can understand the 90-minute threshold I questioned earlier. The influence of rewards still matters when interpreting effects, but proposing their removal does not resolve how to secure the use required. A proposal to recruit more varied participants beyond a campus or friendship network also leaves the question of where to find those people and how to get them to use the app together.
Why Do I Want to Go to Graduate School? | 맵시’s AI 잡지식 저장소:티스토리https://mapsycoy.tistory.com/36
At this point, I was reminded of a post I had written on my old Tistory blog. In it, I considered how technically similar services could differ according to the context that makes people want to return and tell others about them, and then discussed Duolingo as an example of what I aspired to. I was interested in the possibility of addressing neglected values while building a service that people keep using. This was also a question of the design and operation needed to sustain a worthwhile purpose.
What I considered in Duolingo then was gamification through rewards and competition. So describing what I hope for simply as “an app where people stay without rewards” is not quite right. Research participation rewards, such as the vouchers given for meeting a usage threshold in the Lock n’ LoL campaign, need to be distinguished from rewards within an app that encourage people to return through experiences of achievement and competition. What interests me is how to design reasons for continued use into the product, beyond getting people to participate once, rather than requiring the absence of any reward.
Yet this would require designing the app very well, alongside the marketing that encourages people to try it, then addressing problems in use, providing updates, and growing its user base. Marketing expenditure alone is unlikely to sustain that process. It calls for product design and operational skills as well as business judgment, and it is difficult to expect all of this from researchers working with limited time and resources. Nor does an app have to grow into a commercial service like Duolingo for its research to be valid. But asking researchers to secure enough use without campaign participation rewards may, in practice, require work on this scale.
I therefore think this paper contains both a contribution in revealing usage experiences and design issues through actual deployment, and limits to how far the observed changes can be interpreted as effects of the app itself. Its presentation at CHI and news coverage coexist with my wish to examine the participation conditions further. In Let’s FOCUS, I could also see an attempt to carry the design forward and observe use after rewards. Even with these additions, the practical need to secure enough use remains. I would like to hear what alternatives the researchers considered, why they chose this recruitment setting, timing, and reward structure, and what more they wanted to examine in the follow-up study. Understanding how they made those judgments might help me make more concrete choices when considering app operation and research together.
*Thumbnail: Lock n’ LoL logo. Still from the KAIST PR promotional video, via NDTV.
- ¹ Ko, M., Choi, S., Yatani, K., & Lee, U. (2016). Lock n’ LoL: Group-based Limiting Assistance App to Mitigate Smartphone Distractions in Group Activities. Proceedings of CHI 2016. doi:10.1145/2858036.2858568
- ² Statistics Canada. Non-probability sampling.
- ³ MacKenzie, I. S. (2013). Human-Computer Interaction: An Empirical Research Perspective. Chapter 5, Experimental Design.
- ⁴ The p-value reporting guidance in the sixth edition of the APA Publication Manual, as quoted in JASP’s official article “Exact P-Values Upon Request: Breaking with an APA Guideline”
- ⁵ Kim, I., Jung, G., Jung, H., Ko, M., & Lee, U. (2017). Let’s FOCUS: Mitigating Mobile Phone Use in College Classrooms. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(3), Article 63. doi:10.1145/3130928 · Author-hosted paper.