Unlike a single gold answer, preference data captures relative quality: helpfulness, honesty, style, or safety. Collection is expensive and noisy, so labs mix expert raters, crowd workers, and model-generated comparisons. Garbage preferences produce sycophantic or overly refused models even if the base checkpoint was strong.