Can a click through rate ever be more than 100%?

Recently while leveraging some UBI data for search quality evaluation we ran into the issue of having a single search lead to multiple click throughs. Imagine you click on a result, then go back, and then click again on the same result!

What to do? I wanted to share a great write up that my colleague Daniel did after discussions.

Pros of Capping CTRs at 100%

• Prevents Unrealistic Values: Capping ensures that CTRs remain bounded within the range of 0-100%, which aligns with the intuitive understanding of probabilities.

• Handles Extreme User Behavior: By limiting the influence of highly active users, capping reduces the potential for skewed results caused by outliers or excessive engagement from a small subset of users.

• Aligns with Semantic Definition: Treating CTR as a probability (clicks per impression) and capping it at 100% reinforces a clear, interpretable semantic definition of the metric.

• Focuses on Initial Engagement: Counting only the first click prioritizes the most impactful user action, which could be more meaningful in evaluating relevance or engagement.

• Simpler Metric Comparisons: Capped CTRs make it easier to compare performance across queries and documents without accounting for outliers or inflated values.

Cons of Capping CTRs at 100%:

• Loss of Granular Information: Ignoring subsequent clicks in the same session sacrifices potentially valuable data about user interaction patterns and document engagement.

• Dependent on Definition: Capping may not align with all definitions of CTR, particularly when the goal is to measure aggregate behavior ("clicks over views") rather than probabilities.

• Risk of Oversimplification: Limiting CTR at the document or session level might oversimplify complex user behavior and limit insights from highly engaged users.

• Implementation Complexity: Adjusting the CTR calculation to cap values appropriately across different levels (session, user, query) may introduce development and computational challenges.

• Uncertain Impact: Without testing on real datasets, the actual benefit of capping may not justify the effort required to implement it.

We stick with the current approach because:

• Captures Complete Interaction Data: By not capping, the metric reflects all user interactions, providing a richer picture of engagement and behavior.

• Context-Dependent Insights: Active users might provide insights into specific use cases or preferences that would be lost when capping is applied.

• Simplifies Metric Definition: A straightforward "clicks over views" calculation avoids the need to account for additional constraints like session boundaries or user activity levels.

• Flexible Interpretations: Allows different stakeholders to define and interpret CTRs based on their specific requirements, making the metric adaptable to multiple scenarios.

• Effort vs. Reward: Without evidence of significant benefits from capping, maintaining the existing calculation may be more practical.

In summary, while capping CTRs at 100% provides a clean and interpretable metric, the loss of granular interaction data, added complexity, and uncertain benefits make not capping the preferred option, at least until further testing demonstrates its impact. Once we see the effort vs. reward conversation move in the reward direction we can revisit this discussion.