How Creators Publicly Request Feedback in an Open Online Community ?

Submitted to ACM CSCW 2020

Principle Investigator  

Team Member             

Duration           
Skills & Tools              

Prof. Steven Dow (UC San Diego, UCSD The Design Lab)
Regina Cheng (University of Washington), MaySnow Liu
4 Months

INTRO

Recently, creative workers frequently use online critique communities for feedback on their work, especially those with limited access to formal feedback resources. For example, design students like to ask for feedback for their portfolio and project work, product designers want to hear ideas about the logo they created, etc. Being aware of this trend, we want to understand:

" what strategies seekers use to request feedback and whether those strategies are effective ? "

PROGRESS

We present two studies to explore the dynamics between feedback request and feedback in subreddit r/design_critiques, which is part of Reddit.com. r/design_critiques is a large and active community dedicated to feedback exchange across a range of design domains, including website, graphic design, animation, etc. It allows everyone to publicly share their work through text-based forum post (including embedded URLs) and receive feedback in the form of public comments.

​STUDY 1

Community member's opinions about how to request feedback

  • ​Conducted 12 Semi-structured Interviews

  • Analyzed Interview Results

  • Built Affinity Diagrams

​STUDY 2

How request strategies affect community feedback

  • ​Data collection (post request and comment from 2013.1 - 2018.10)

  • Developed coding scheme

  • Qualitatively coded 900 sampled posts out of total 24587 posts

  • Built regression models

  • Conducted big data analysis

Our Data

All posts w/URLs; associated comments from 2013 to 2018 from r/design_critique subreddit community

24,587 feedback requests (Including title, body, author, timestamp)

98,475 pieces of feedback

Our Participants

12 active users in r/design_critique community aged over 18
On average
12.64 posts history (min = 8, max = 23)

4.04 years of experience in the community (min = 0.5, max = 7)

Building the Affinity Diagram

Affinity%20Diagram_edited.jpg
Sample Interview transceript.png

Sample of Interview Transcript

Affinity Diagram 2.HEIC

Affinity Diagram Process

Affinity%20Diagram%203_edited.jpg

Affinity Diagram Process

Qualitative Coding

Sample Code Scheme.png

We selected 900 post requests from the corpus of 24,867 posts by randomly sampling 150 posts from each year of the six years.
 

We iteratively developed a coding scheme describing the feedback request strategies in the community. The first coding scheme has 21 strategies, and we refined it into 7 at the end. Then, we binary coded for the presence of 7 strategies.

To investigate how the features of feedback requests influence the resulting feedback, we built regression models with binary-coded strategies as independent variables (IVs) and added length of request as a covariate in the model.

 

We analyzed several features of the resulting feedback as dependent variables (length of feedback, the quantity of feedback, waiting time for the first feedback) and created a different regression model for each. In the end, we report the effect sizes of our regression models.

Sample Coding Scheme

RESULTS

Although I cannot provide detailed information about method, discussion or result of our research until our paper is published, here's a high-level overview of our findings. 

Feedback Seeker

  1. Seekers often provide design (but not personal) details. 

  2. Seeker's disclosure of personal information may distract providers and frame them to generate biased feedback.

  3. Seekers prompt specific feedback, yet still want comprehensive critique.

  4. Seekers want expert's feedback, but avoid explicitly requesting it.

Feedback Provider

  1. Providers want more contextual information.

  2. Providers try to empathize with seekers.

  3. Providers prefer specific, not vague feedback prompts.

  4. Providers avoid lengthy and complicated requests.

Signaling novice led to faster and better-justified feedback


Critiquing one's design in request resulted in faster and more actionable feedback


Providing specific prompts, especially showing variants, yielded faster and more justified feedback.


Self-critiques establish context and surface meta-cognitive thinking

Last, based on our research, we presented some design implications for future feedback systems.

Detailed information, complete research results and full paper will be posted here after our paper publish.

MY TEAM 😊

Contact me

©2020 BY ZIWEN ZENG

click to expand

Go to link