How Creators Publicly Request Feedback in an Open Online Community ?
Accepted to ACM CSCW 2020 full paper
Principle Investigator
Team Member
Duration
Skills & Tools
Prof. Steven Dow (UC San Diego, UCSD The Design Lab)
Regina Cheng (University of Washington), MaySnow Liu
4 Months
INTRO
Recently, creative workers frequently use online critique communities for feedback on their work, especially those with limited access to formal feedback resources. For example, design students like to ask for feedback for their portfolio and project work, product designers want to hear ideas about the logo they created, etc. Being aware of this trend, we want to understand:
" what strategies seekers use to request feedback and whether those strategies are effective ? "
PROGRESS
We present two studies to explore the dynamics between feedback request and feedback in subreddit r/design_critiques, which is part of Reddit.com. r/design_critiques is a large and active community dedicated to feedback exchange across a range of design domains, including website, graphic design, animation, etc. It allows everyone to publicly share their work through text-based forum post (including embedded URLs) and receive feedback in the form of public comments.
STUDY 1
Community member's opinions about how to request feedback
-
Conducted 12 Semi-structured Interviews
-
Analyzed Interview Results
-
Built Affinity Diagrams
STUDY 2
How request strategies affect community feedback
-
Data collection (post request and comment from 2013.1 - 2018.10)
-
Developed coding scheme
-
Qualitatively coded 900 sampled posts out of total 24587 posts
-
Built regression models
-
Conducted big data analysis
Our Data
All posts w/URLs; associated comments from 2013 to 2018 from r/design_critique subreddit community
24,587 feedback requests (Including title, body, author, timestamp)
98,475 pieces of feedback
Our Participants
12 active users in r/design_critique community aged over 18
On average 12.64 posts history (min = 8, max = 23)
4.04 years of experience in the community (min = 0.5, max = 7)
Building the Affinity Diagram
Sample of Interview Transcript
Affinity Diagram Process
Affinity Diagram Process
Qualitative Coding
We selected 900 post requests from the corpus of 24,867 posts by randomly sampling 150 posts from each year of the six years.
We iteratively developed a coding scheme describing the feedback request strategies in the community. The first coding scheme has 21 strategies, and we refined it into 7 at the end. Then, we binary coded for the presence of 7 strategies.
To investigate how the features of feedback requests influence the resulting feedback, we built regression models with binary-coded strategies as independent variables (IVs) and added length of request as a covariate in the model.
We analyzed several features of the resulting feedback as dependent variables (length of feedback, the quantity of feedback, waiting time for the first feedback) and created a different regression model for each. In the end, we report the effect sizes of our regression models.
Sample Coding Scheme
RESULTS
Although I cannot provide detailed information about method, discussion or result of our research until our paper is published, here's a high-level overview of our findings.
Feedback Seeker
-
Seekers often provide design (but not personal) details.
-
Seeker's disclosure of personal information may distract providers and frame them to generate biased feedback.
-
Seekers prompt specific feedback, yet still want comprehensive critique.
-
Seekers want expert's feedback, but avoid explicitly requesting it.
Feedback Provider
-
Providers want more contextual information.
-
Providers try to empathize with seekers.
-
Providers prefer specific, not vague feedback prompts.
-
Providers avoid lengthy and complicated requests.
Signaling novice led to faster and better-justified feedback
Critiquing one's design in request resulted in faster and more actionable feedback
Providing specific prompts, especially showing variants, yielded faster and more justified feedback.
Self-critiques establish context and surface meta-cognitive thinking
Last, based on our research, we presented some design implications for future feedback systems.
For more detailed information, feel free to read our full paper
MY TEAM 😊
