The Limits of AI Engagement Metrics: What High Usage Numbers Don't Reveal
A new commentary in Communications of the ACM raises a critical point about how we measure AI success: high engagement numbers tell only part of the story.
The piece argues that focusing on metrics like user adoption, time spent, or interaction frequency with AI systems obscures what these technologies actually displace. Whether referring to displaced workers, abandoned workflows, atrophied skills, or replaced tools, engagement data provides a narrow view of AI's systemic effects.
This critique comes at a time when many AI products and research papers tout impressive user numbers or engagement statistics as evidence of value. The author suggests this approach is problematic because it treats AI adoption as inherently positive without accounting for what gets left behind or disrupted in the process.
The commentary aligns with broader concerns in technology studies about unintended consequences and the importance of looking beyond surface-level adoption metrics. Understanding what AI displaces—whether jobs, decision-making authority, or established practices—may be just as important as tracking who is using it.
For policymakers, researchers, and industry leaders, the takeaway may be that evaluating AI's societal impact requires more comprehensive frameworks than simple engagement counting.