Constrained Discretion Outperforms Binary Autonomy Decisions

Giving workers full override authority over AI inventory systems invites bias and noise. Blocking overrides wastes local knowledge. How do you structure discretion to filter signal from noise?

Method: A constrained override policy—limiting workers to two downward inventory adjustments per machine—reduced inventory by 1.28% without harming sales. Free overrides cut inventory by 1.95% but also reduced sales by 1.19%. The constraint forced workers to prioritize high-value SKUs, confirmed via local average treatment effects. Gains concentrated among experienced workers and growth-stage SKUs.

Caveats: Tested in retail inventory with clear performance metrics. Transfer to domains with ambiguous success criteria unverified.

Reflections: How does optimal override budget scale with decision complexity or worker expertise? · Would dynamic budgets (more overrides for uncertain predictions) outperform fixed caps? · Do workers game the constraint by clustering overrides on easier decisions?

AI evaluation treats expert disagreement as noise to eliminate. In creative work, professionals legitimately disagree on taste. Collapsing this into a single quality score discards the most actionable signal.

Method: The Human Creativity Benchmark collected 15,000 professional judgments across five creative domains and three workflow phases. Convergence concentrated on verifiable dimensions like prompt adherence and usability. Divergence concentrated on taste-driven dimensions like aesthetic direction. No model excelled uniformly across all phases. The separation reveals where models must be correct versus where they should remain steerable.

Caveats: Framework tested on visual creative domains. Transfer to other creative modalities (music, writing) requires validation.

Reflections: How should model training objectives change when optimizing for divergent versus convergent dimensions? · Do users prefer high-variance outputs on divergent dimensions, or does unpredictability frustrate iteration? · Can models learn to self-identify which dimensions are convergent for a given task?

Organizations deploying CLI coding agents face million-dollar token bills without knowing who will adopt, who will stick, or whether output justifies cost. Misreading any of these makes rollout expensive without changing velocity.

Method: Studying tens of thousands of Microsoft engineers over four months, first use spread primarily through social networks, not demographics. Retention correlated with coding activity level, not engineer characteristics. Adopters merged roughly 24% more pull requests than they would have otherwise, with the lift persisting across the full observation window.

Caveats: Four-month window at one organization. Longer-term retention and productivity effects unknown.

Reflections: Does the 24% PR lift translate to proportional business value, or does it reflect lower-impact contributions? · How does social diffusion interact with team structure—do agents spread within teams or across them? · What causes non-adopters to resist despite peer visibility?