The third and last of a series of user studies for our app, was conducted at Zo Kinderopvang daycare center, Den Haag. Thirteen children participated in the first testing sessions, but unfortunately two of them dropped for the second session. The procedure followed was similar to the one here.
About to run a Simulated Work Tasks (SWTs) experiment using the online platform microworkers.com. The goal of this experiment is to assess the usability and domain contribution to the social commitments (SC) model that we created. Participants will perform 4 tasks each, and at the end of each task, they will try to solve the family-life domain problem through creating an agreement using a menu representation of the SC model. After that, they will have to rate how well did the options in the SC menu contribute towards the solving the problem in the scenario, using a slider.
You can try it out yourself as well, using this link. Make sure you see the explanation video first!
The second of a series of user studies for our app, was conducted at Zo Kinderopvang daycare center, Rijswijk. Twelve children participated in the two testing sessions. The procedure followed was similar to the one here.
Description of the first of a series of user studies for our app, this one conducted at De Lange Keizer daycare center, Delft. Eight children participated in the two testing sessions.
This user study was the first field test of our implemented app prototype. The app is a location sharing device for families with children in primary school (between 6-12 years of age), which allows users to create locations, check-in in these locations, share and receive check-ins from others as they choose, and create normative-based social commitments with other users regarding sharing/receiving check-ins. We believe that this additional capability (social commitments) will provide a better support for user values for children, namely social recognition, friendship, independence, and freedom (amongst other hypotheses).
We have built two editions of the app. Edition 1 comes without social commitments (a.k.a. afspraken) and edition 2 comes with social commitments. These hypotheses/research questions we are attempting to answer in this user study are:
- Within the domain of family life, e2 will provide better overall value support for children than e1.
- Within the domain of family life, e2 will be perceived more as social actor than e1.
- Within the domain of family life, e2 will be perceived more as useful/usable than e1.
- (Research question) what subset of the possible social commitments will our users use, and for what purposes/values?
- Children will expect that the presence of an app in their life will increase support to their values.
- Sub hypothesis: the children will expect that the presence of an app in their life will increase their independence (or personal freedom).
- Sub hypothesis: the children will expect that the presence of an app in their life will increase have a positive effect on their friendships (or their social recognition).
- Children will expect that the presence of an app in their life will increase support to their activities.
- Children will expect that the presence of an app in their life will remove some of the limitations they have.
The user study took place at a local day care center (Dutch: buitenschoolse opvang) in Delft. This day care center is a place where children are brought after school is over, until their parents pick them up at the end of their workday. It contains 9 different activity rooms where children can freely participate in activities, such as a painting room, music room, playground, lunchroom, etc. The day care center officials have assisted us finding children who wanted to participate in the user study.
Due to the number of participants (8 children, aged between 7 and 10 years) we opted for a within subject, counterbalancing design, meaning that children were randomly split in two groups of four, each group testing a different edition of the app in each of the two sessions.
We held an introductory session one week before the user study sessions, where the researchers met the participants, introduced the app using a tutorial video, and answered their questions about the app and the procedure of the user study.
User study sessions
The two sessions were identical, and they were one week apart. Below is a description of what happens in each session.
Children had to perform “missions” using the app. A mission is either:
- Instructional, directly pointing to a functionality in the app to help the user get acquainted with the app more. Examples: checking-in in a certain location, adding other users to their friend or family list, and creating a social commitment.
- A story-lined, simulation of a real life situation, that requires the user to figure out what needs to be done with the app without direct instructions. Examples: go to the lunchroom. Did you find any of your friends there? How can you ensure that one of your friends will inform you when they enter this location? Check who’s in the painting room. Anyone from your team? If yes, try to tell all your friends that you are both at that location. Can you see where others on your team are? Try to go out and find one of them. Hint: check your event list.
There were 37 mission cards in total (17 instructional missions, 20 real-life situation simulations, see example in photo below). The missions had an even distribution of “envisioned” solutions in terms of app functionalities and the possible types of commitments that need to be created.
Children would pick random missions from the pile, read the text, (attempt to) perform them, and then bring the mission back when they would like (whether finished successful or not) to the pile to pick another one, and so on. The user study stopped when one hour had passed in each session.
During which the first 10 minutes only instructional missions were available to be picked from the pile (in order to build some app knowledge). Afterwards, we added the rest of the missions to the pile.
- Value measuring questionnaire: answered by children at the end of sessions 2 and 3. Contains questions to measure fulfillment of the values Friendship, Social recognition, Independence, Freedom. (Hypotheses 1,5,6,7). Questions in this questionnaire were framed into a hypothetical future tense: e.g. “if I had this app in my life, it would <less easy………..nothing will change……..much easier> to find where my friends are. Before they answered the questionnaire, children watched a short video explaining how the questionnaire should be answered, with a period for questions and answers.
- Questionnaire for usability and social actorship, also answered by children at the end of sessions 2 and 3. Contained questions for usability, liking, dominance, trust, and intimacy. (Hypotheses 2,3).
- Every time a child picked a mission from the pile, the id numbers of the child, mission, and timestamp were recorded. (Research question 4)
- Behavioral sampling: every 10 minutes, one of the researchers recorded every child’s emotion/body language according to a predetermined coding schema (derived from Markopolous, Evaluating Children’s interactive Products, pp. 182, with positive codes created by reversing the negative ones already existing in the text), their location, and their engagement with others, i.e. whether in groups or alone. (Hypotheses 1,5,6,7).
- All app interactions with any user were recorded and time-stamped on the server, e.g. every check-in, every time a user added another user to a list, every time a commitment was created, accepted, rejected, etc. (Research question 4).
Was a guest lecturer this week in the HART course (Human-Agent-Robot Teamwork) at TU Delft. It was about using Normative Frameworks to make social applications more intelligent, and more adaptive to social situations.
I had so much fun participating in the 2013 edition of the Design Jam Amsterdam, that I have been in contact with the organizers and was added to the organizing team for 2014! This time the event will be hosted by the Hogeschool van Amsterdam (HvA), and will have many more sponsors. Looking forward for finding out what the design challenge will be this time, and how I could participate in making this series of events much more interesting.
Today I went back to Hendrik-Ido-Ambacht. I met with the same pilot test group at a community center. The children have used the app quite extensively, but the parents/adults, not that much. One of the explanations was given by the parents was that they already use other means of communication, like WhatsApp, SMS’s, Facebook, etc. One other important aspect, well, this group was quite familiar with each other, but in real future tests, Parents shouldn’t be allowed to make agreements with other people’s children, and vice-versa– it would be considered somewhat strange. Plenty of other feedback came out of this short, 40-minute focus group: one of the parents found that the mandatory buzzer everytime they received a check-in or an agreement request was not needed, it was a little too much. She also suggested using the Android notification center, which I have gone to great lengths to bypass for this app, because I believed it was too hidden/small fonted (especially on this screen)/difficult for children between 7-11 to understand– based on standard design-for-children advice. Apparently for the two participating children in this pilot, this was not the case. We do really underestimate what children are capable of, tech-wise I’d say.
Plenty of lessons to learn here when designing for children. First of all, plenty of literature and books on the subject are North American– where the ages at which children learn reading, writing, and the activities they do, they all differ from European school systems. Also, the use of the word children itself could mean completely different groups, as most of the advice you get from such books is aimed towards children 3-5 years of age, which still can’t read for example. If designing for an older group, one shouldn’t worry about incorporating more of the “advanced” interface elements that adults usually use. Cognitive skills improve really rapidly at this point too– even half a year or one year of growth would make design decisions somewhat obsolete.
Today I went with the stress tested, app version 2 (which includes the afspraken/agreement menu) to Hendrik-Ido-Ambacht. I met with the pilot test group at a community center, the same test group from the usability test earlier, so 2 children, 2 of their parents, and 2 other adults. I showed the tutorial video, answered some questions, and the following conversation (around 45 mins, in Dutch! that was quite the first time I had to speak Dutch for this long). After that, the test group was playing around with the app and asking questions as well. We agreed that I will return, 2 weeks later, and we would have an interview session to determine how well this pilot version of the app worked. I will have access to the database and therefore can see all of their app interactions, and I should also prepare a structure for the questions needed to be asked for the upcoming session 2 weeks from now.
Finally the app is done. Here’s a formal description, as it went in one of the papers we wrote:
The application permits its users to share check-ins in certain locations with other users of the system (such as family members and school friends), similar to the popular application Foursquare. The check-in feature was selected for our case study based on the analysis previous qualitative data with members of our target group.
Basic features and preferences
Users can place other users of the system in one of two lists (“family” or “friends”), otherwise they are placed in the list “others”. Users can select with which lists they share their check-ins, and from which lists they view shared check-ins.
Users can create locations either through placing a marker on an integrated Google Map, or simply by detecting the current GPS position. The application will create a 50m*50m location surrounding the resulting GPS coordinates, to which the users can assign a name.
When a user wants to check-in, a list of nearby, already created locations is displayed (with the option of adding a new location). The user can select their location, and confirm their check-in, to be shared with the users in the lists our user is sharing with, according to their preferences. Users who receive this check-in will get a pop-up with the sharer’s name and location information, assuming that they opted (in their own preferences) to view check-ins from the list to which the sharer belongs.
The application allows for additional, norm-based behavior customization. Drawing from the social commitments framework by Singh, and our analysis of user data, we have created the following commitment model:
A commitment (S, T, t, n, e, d) consists of S the source (creator of the commitment), T the target that has to comply with it, t the triggering condition that activates the commitment, n the normative effect (an obligation or a prohibition of sharing or viewing a check-in, from someone or a group of people), e the expiry condition that deactivates the commitment, and d the deadline by which an obligation commitment should be fulfilled.
For example, Mark (source) can create the following commitment: (1) I want Paula (target) to share her check-ins with me (normative effect), if she enters the park (triggering condition). Another example is Mark creating the commitment: (2) I want Paula to “not view” check-ins from the group “friends” (normative effect) after 9 pm (triggering condition). In the current app version, the deadline ASAP is used for obligation commitments. Expiry condition in (1) is not required, while in (2) it is implied as a certain hour (9 am).
Upon creation, commitments are sent from source to target, whereby the target can either accept or delete the commitment, or “decide later”. The commitment enters its “active” state when it’s been accepted by the target, and the trigger condition has been met. The commitment leaves its “active” state when the expiry condition has been met, but can re-enter that state if the triggering condition was met again, provided that it was not deleted.
Conflicts between preferences and an active commitment are decided in favor of the commitment. For example, if Mark is in Paula’s family list, and Paula opted in her preferences to “not share check-ins with family”, accepting commitment (1) above means Paula’s check-in will be shared with Mark if she enters the park. Conflicts between two active commitments would be solved in favor of the commitment most recently accepted.
A 3-minute tutorial video of the application (with subtitles) can be seen here.
Finally we have the app’s XML code for the interface implemented. Along with our programmer we both took part in implementing this interface. With android it can be quite tricky to create menus that update their contents interdependently, such as the Afspraken (agreements) menu, see photos. Must use runOnUiThread, which can be quite complex. Luckily StackOverflow is abundant with examples on that, like this one for example.
However, it is a still a very satisfying experience to have such a complex menu finally work flawlessly, while knowing that I’m being able to implement what is required in case we needed to change anything.