How Creators Publicly Request Feedback in an Open Online Community ?
Submitted to ACM CSCW 2020
Skills & Tools
Recently, creative workers frequently use online critique communities for feedback on their work, especially those with limited access to formal feedback resources. For example, design students like to ask for feedback for their portfolio and project work, product designers want to hear ideas about the logo they created, etc. Being aware of this trend, we want to understand:
" what strategies seekers use to request feedback and whether those strategies are effective ? "
We present two studies to explore the dynamics between feedback request and feedback in subreddit r/design_critiques, which is part of Reddit.com. r/design_critiques is a large and active community dedicated to feedback exchange across a range of design domains, including website, graphic design, animation, etc. It allows everyone to publicly share their work through text-based forum post (including embedded URLs) and receive feedback in the form of public comments.
Community member's opinions about how to request feedback
Conducted 12 Semi-structured Interviews
Analyzed Interview Results
Built Affinity Diagrams
How request strategies affect community feedback
Data collection (post request and comment from 2013.1 - 2018.10)
Developed coding scheme
Qualitatively coded 900 sampled posts out of total 24587 posts
Built regression models
Conducted big data analysis
All posts w/URLs; associated comments from 2013 to 2018 from r/design_critique subreddit community
24,587 feedback requests (Including title, body, author, timestamp)
98,475 pieces of feedback
12 active users in r/design_critique community aged over 18
On average 12.64 posts history (min = 8, max = 23)
4.04 years of experience in the community (min = 0.5, max = 7)
Building the Affinity Diagram
Sample of Interview Transcript
Affinity Diagram Process
Affinity Diagram Process
We selected 900 post requests from the corpus of 24,867 posts by randomly sampling 150 posts from each year of the six years.
We iteratively developed a coding scheme describing the feedback request strategies in the community. The first coding scheme has 21 strategies, and we refined it into 7 at the end. Then, we binary coded for the presence of 7 strategies.
To investigate how the features of feedback requests influence the resulting feedback, we built regression models with binary-coded strategies as independent variables (IVs) and added length of request as a covariate in the model.
We analyzed several features of the resulting feedback as dependent variables (length of feedback, the quantity of feedback, waiting time for the first feedback) and created a different regression model for each. In the end, we report the effect sizes of our regression models.
Sample Coding Scheme
Although I cannot provide detailed information about method, discussion or result of our research until our paper is published, here's a high-level overview of our findings.
Seekers often provide design (but not personal) details.
Seeker's disclosure of personal information may distract providers and frame them to generate biased feedback.
Seekers prompt specific feedback, yet still want comprehensive critique.
Seekers want expert's feedback, but avoid explicitly requesting it.
Providers want more contextual information.
Providers try to empathize with seekers.
Providers prefer specific, not vague feedback prompts.
Providers avoid lengthy and complicated requests.
Signaling novice led to faster and better-justified feedback
Critiquing one's design in request resulted in faster and more actionable feedback
Providing specific prompts, especially showing variants, yielded faster and more justified feedback.
Self-critiques establish context and surface meta-cognitive thinking
Last, based on our research, we presented some design implications for future feedback systems.
Detailed information, complete research results and full paper will be posted here after our paper publish.