"How Our Customer Feedback API Went From Hero to Zero (And Back Again)"

The Story

It started as a simple request from our Product team: "Can we automatically track what customers are saying about our products online?"

We built an API that would:

• Scan customer blog posts and forum discussions

• Identify product issues and sentiment

• Display insights in a Power Platform dashboard

The first demo blew everyone away. Real customer feedback, categorized and prioritized, updating in real-time.

Six months later, that same API was timing out, the dashboard was unusable, and the Product team was back to manually reading forums.

This is the story of how success almost killed our API, and how changing one data type saved it.

Why We Started with List (And Why It Made Perfect Sense)

When we kicked off the project, the requirements were modest:

Product Manager: "We want to track maybe 50-100 keywords that indicate issues - words like 'crash', 'slow', 'broken'. Oh, and flag posts from our top 100 enterprise customers."

Me: "That's it?"

PM: "Well, maybe add our product names and main competitors. But keep it simple - we're just testing if this provides value."

using Microsoft.Extensions.Logging;
using System.Diagnostics;

public interface ICustomerFeedbackService
{
    Task<FeedbackAnalysis> AnalyzeBlogPostAsync(BlogPost post, CancellationToken cancellationToken = default);
    Task ReloadKeywordsAsync(CancellationToken cancellationToken = default);
}

public class CustomerFeedbackService : ICustomerFeedbackService
{
    private readonly ILogger<CustomerFeedbackService> _logger;
    private readonly IKeywordRepository _keywordRepository;
    private readonly IMetricsCollector _metrics;
    private readonly FeedbackServiceOptions _options;

    // Started with Lists (seemed reasonable for small datasets)
    private List<string> _issueKeywords = new();
    private List<string> _productNames = new();
    private List<string> _vipCustomerDomains = new();
    private List<string> _competitorNames = new();

    private readonly SemaphoreSlim _reloadLock = new(1, 1);
    private DateTime _lastReload = DateTime.UtcNow;

    public CustomerFeedbackService(
        ILogger<CustomerFeedbackService> logger,
        IKeywordRepository keywordRepository,
        IMetricsCollector metrics,
        IOptions<FeedbackServiceOptions> options)
    {
        _logger = logger ?? throw new ArgumentNullException(nameof(logger));
        _keywordRepository = keywordRepository ?? throw new ArgumentNullException(nameof(keywordRepository));
        _metrics = metrics ?? throw new ArgumentNullException(nameof(metrics));
        _options = options?.Value ?? throw new ArgumentNullException(nameof(options));

        // Load keywords on startup
        Task.Run(async () => await LoadKeywordsAsync()).Wait(TimeSpan.FromSeconds(30));
    }

    public async Task<FeedbackAnalysis> AnalyzeBlogPostAsync(BlogPost post, CancellationToken cancellationToken = default)
    {
        if (post == null) throw new ArgumentNullException(nameof(post));

        using var activity = Activity.StartActivity("AnalyzeBlogPost");
        var stopwatch = Stopwatch.StartNew();

        try
        {
            // Reload keywords if stale
            if (DateTime.UtcNow - _lastReload > _options.KeywordRefreshInterval)
            {
                _ = Task.Run(async () => await ReloadKeywordsAsync(cancellationToken));
            }

            var analysis = new FeedbackAnalysis
            {
                BlogPostId = post.Id,
                AnalyzedAt = DateTime.UtcNow,
                Source = post.Source
            };

            // Extract and normalize words
            var words = ExtractAndNormalizeWords(post.Content);
            _logger.LogDebug("Analyzing {WordCount} words from blog post {PostId}", words.Count, post.Id);

            // THE PERFORMANCE KILLER - O(n) lookups for each word
            foreach (var word in words)
            {
                cancellationToken.ThrowIfCancellationRequested();

                using (var wordTimer = _metrics.StartTimer("word_analysis"))
                {
                    if (_issueKeywords.Contains(word))
                    {
                        analysis.IssuesFound.Add(new Issue { Keyword = word, Severity = CategorizeSeverity(word) });
                    }

                    if (_productNames.Contains(word))
                    {
                        analysis.ProductsMentioned.Add(word);
                    }

                    if (_competitorNames.Contains(word))
                    {
                        analysis.CompetitorsMentioned.Add(word);
                    }
                }
            }

            // Check VIP customer
            var domain = ExtractDomain(post.Url);
            if (!string.IsNullOrEmpty(domain) && _vipCustomerDomains.Contains(domain))
            {
                analysis.IsVipCustomer = true;
                analysis.CustomerTier = CustomerTier.Enterprise;
            }

            // Calculate sentiment and priority
            analysis.Sentiment = CalculateSentiment(analysis);
            analysis.Priority = CalculatePriority(analysis);

            var elapsed = stopwatch.ElapsedMilliseconds;
            _metrics.RecordHistogram("feedback_analysis_duration_ms", elapsed);

            if (elapsed > _options.SlowRequestThresholdMs)
            {
                _logger.LogWarning("Slow feedback analysis: {ElapsedMs}ms for post {PostId}", elapsed, post.Id);
            }

            _logger.LogInformation(
                "Analyzed blog post {PostId} in {ElapsedMs}ms. Found {IssueCount} issues, {ProductCount} products",
                post.Id, elapsed, analysis.IssuesFound.Count, analysis.ProductsMentioned.Count);

            return analysis;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to analyze blog post {PostId}", post.Id);
            _metrics.IncrementCounter("feedback_analysis_errors");
            throw new FeedbackAnalysisException($"Failed to analyze blog post {post.Id}", ex);
        }
    }

    private async Task LoadKeywordsAsync()
    {
        try
        {
            _logger.LogInformation("Loading keywords from repository");

            var loadTasks = new[]
            {
                LoadKeywordsCategoryAsync(KeywordCategory.Issue),
                LoadKeywordsCategoryAsync(KeywordCategory.Product),
                LoadKeywordsCategoryAsync(KeywordCategory.Competitor),
                LoadVipDomainsAsync()
            };

            await Task.WhenAll(loadTasks);

            _lastReload = DateTime.UtcNow;
            _logger.LogInformation(
                "Loaded {IssueCount} issues, {ProductCount} products, {CompetitorCount} competitors, {VipCount} VIP domains",
                _issueKeywords.Count, _productNames.Count, _competitorNames.Count, _vipCustomerDomains.Count);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to load keywords");
            throw new StartupException("Failed to initialize keyword data", ex);
        }
    }

    private async Task LoadKeywordsCategoryAsync(KeywordCategory category)
    {
        var keywords = await _keywordRepository.GetKeywordsByCategoryAsync(category);

        switch (category)
        {
            case KeywordCategory.Issue:
                _issueKeywords = keywords.Select(k => k.ToLowerInvariant()).ToList();
                break;
            case KeywordCategory.Product:
                _productNames = keywords.Select(k => k.ToLowerInvariant()).ToList();
                break;
            case KeywordCategory.Competitor:
                _competitorNames = keywords.Select(k => k.ToLowerInvariant()).ToList();
                break;
        }
    }

    private List<string> ExtractAndNormalizeWords(string content)
    {
        if (string.IsNullOrWhiteSpace(content))
            return new List<string>();

        // Simple word extraction - in production might use NLP library
        return content
            .Split(new[] { ' ', '.', ',', '!', '?', ';', ':', '\n', '\r', '\t' }, StringSplitOptions.RemoveEmptyEntries)
            .Where(w => w.Length >= _options.MinWordLength && w.Length <= _options.MaxWordLength)
            .Select(w => w.ToLowerInvariant().Trim())
            .ToList();
    }

    // ... other helper methods
}

Why List<T> was the right choice:

• Simple and readable - any developer could understand it

• With 50 items, Contains() took microseconds

• Easy to add/remove keywords

• YAGNI principle - why over-engineer for 50 items?

What Changed Over Time

Month 2: "This is amazing! Can we add more?"

• Marketing: "Add 500 industry buzzwords!"

• Sales: "Track 1,000 more customer domains!"

• Support: "Include 2,000 error message patterns!"

Down the Line: "We need competitive intelligence!"

• Add 2,000 competitor product names

• Track 5,000 industry keywords

• Monitor 3,000 technology terms

Further Down: "Let's go enterprise!"

• Import 15,000 customer domains from Salesforce

• Add 8,000 issue patterns from support tickets

• Include 5,000 feature requests terms

Our simple lists grew exponentially:

Nobody questioned it because it "still worked fine" - until it didn't.

What We Started Seeing (The Slow Death)

The Gradual Decline

Month 1: API response: 45ms
Month 2: API response: 150ms
Month 3: API response: 400ms
Month 4: API response: 800ms
Month 5: API response: 1,500ms
Month 6: API response: 3,000ms 💀

The Monday Morning Crisis

Our beautiful dashboard showed "Loading..." permanently. The Product team went back to manually reading forums.

The Investigation:

public FeedbackAnalysis AnalyzeBlogPost(BlogPost post)
{
    var sw = Stopwatch.StartNew();
    var words = ExtractWords(post.Content); // ~1,000 words per post

    foreach (var word in words)
    {
        var checkStart = sw.ElapsedMilliseconds;

        if (_issueKeywords.Contains(word.ToLower())) // Checking 8,000 items!
            analysis.IssuesFound.Add(word);

        _logger.LogDebug($"Keyword check: {sw.ElapsedMilliseconds - checkStart}ms");
    }

    // Logs showed:
    // "Keyword check: 12ms"
    // "Keyword check: 15ms"  
    // "Keyword check: 11ms"
    // ... x 1,000 words = 12,000ms total!
}

The math was brutal:

• 1,000 words per blog post

• × 30,000 total keywords to check

• \= 30 MILLION string comparisons per blog post

• × 1,000 blog posts per hour

• \= 30 BILLION comparisons per hour
  No wonder our servers were dying.

How We Fixed It (The One-Line Change That Saved Everything)

The solution was embarrassingly simple:

// Before
private List<string> _issueKeywords = new List<string>();

// After
private HashSet<string> _issueKeywords = new HashSet<string>(StringComparer.OrdinalIgnoreCase);

But we went further and redesigned for scale:

using Microsoft.Extensions.Logging;
using System.Collections.Concurrent;
using System.Diagnostics;

public class OptimizedCustomerFeedbackService : ICustomerFeedbackService
{
    private readonly ILogger<OptimizedCustomerFeedbackService> _logger;
    private readonly IKeywordRepository _keywordRepository;
    private readonly IMetricsCollector _metrics;
    private readonly IMemoryCache _cache;
    private readonly FeedbackServiceOptions _options;

    // OPTIMIZED: Using HashSets for O(1) lookups
    private HashSet<string> _issueKeywords = new(StringComparer.OrdinalIgnoreCase);
    private HashSet<string> _productNames = new(StringComparer.OrdinalIgnoreCase);
    private HashSet<string> _vipCustomerDomains = new(StringComparer.OrdinalIgnoreCase);

    // OPTIMIZED: Single dictionary for all keyword lookups
    private readonly ConcurrentDictionary<string, KeywordInfo> _allKeywords = new(StringComparer.OrdinalIgnoreCase);

    // OPTIMIZED: Pre-filter common words
    private readonly HashSet<string> _stopWords;

    private readonly SemaphoreSlim _reloadLock = new(1, 1);
    private volatile bool _isInitialized = false;

    public OptimizedCustomerFeedbackService(
        ILogger<OptimizedCustomerFeedbackService> logger,
        IKeywordRepository keywordRepository,
        IMetricsCollector metrics,
        IMemoryCache cache,
        IOptions<FeedbackServiceOptions> options)
    {
        _logger = logger ?? throw new ArgumentNullException(nameof(logger));
        _keywordRepository = keywordRepository ?? throw new ArgumentNullException(nameof(keywordRepository));
        _metrics = metrics ?? throw new ArgumentNullException(nameof(metrics));
        _cache = cache ?? throw new ArgumentNullException(nameof(cache));
        _options = options?.Value ?? throw new ArgumentNullException(nameof(options));

        // Load stop words
        _stopWords = new HashSet<string>(
            LoadStopWords(), 
            StringComparer.OrdinalIgnoreCase
        );

        // Initialize keywords asynchronously
        _ = InitializeAsync();
    }

    public async Task<FeedbackAnalysis> AnalyzeBlogPostAsync(BlogPost post, CancellationToken cancellationToken = default)
    {
        if (post == null) throw new ArgumentNullException(nameof(post));

        // Ensure service is initialized
        if (!_isInitialized)
        {
            await WaitForInitializationAsync(cancellationToken);
        }

        using var activity = Activity.StartActivity("AnalyzeBlogPost");
        activity?.SetTag("blog.id", post.Id);
        activity?.SetTag("blog.source", post.Source);

        var stopwatch = Stopwatch.StartNew();

        try
        {
            // Check cache first
            var cacheKey = $"analysis:{post.Id}:{post.ContentHash}";
            if (_cache.TryGetValue<FeedbackAnalysis>(cacheKey, out var cachedAnalysis))
            {
                _metrics.IncrementCounter("feedback_analysis_cache_hits");
                return cachedAnalysis;
            }

            var analysis = new FeedbackAnalysis
            {
                BlogPostId = post.Id,
                AnalyzedAt = DateTime.UtcNow,
                Source = post.Source,
                ContentHash = post.ContentHash
            };

            // OPTIMIZED: Extract, normalize, and deduplicate words
            var words = ExtractAndNormalizeWords(post.Content);
            var uniqueWords = new HashSet<string>(words, StringComparer.OrdinalIgnoreCase);

            // OPTIMIZED: Remove stop words
            uniqueWords.ExceptWith(_stopWords);

            _logger.LogDebug(
                "Analyzing {UniqueWordCount} unique words (from {TotalWords} total) from blog post {PostId}", 
                uniqueWords.Count, words.Count, post.Id);

            // OPTIMIZED: Single pass with O(1) lookups
            var wordAnalysisTasks = new List<Task>();
            var issuesBag = new ConcurrentBag<Issue>();
            var productsBag = new ConcurrentBag<string>();
            var competitorsBag = new ConcurrentBag<string>();

            // Process words in parallel for large posts
            if (uniqueWords.Count > _options.ParallelThreshold)
            {
                await Parallel.ForEachAsync(
                    uniqueWords,
                    new ParallelOptions 
                    { 
                        CancellationToken = cancellationToken,
                        MaxDegreeOfParallelism = Environment.ProcessorCount 
                    },
                    async (word, ct) =>
                    {
                        if (_allKeywords.TryGetValue(word, out var keywordInfo))
                        {
                            switch (keywordInfo.Category)
                            {
                                case KeywordCategory.Issue:
                                    issuesBag.Add(new Issue 
                                    { 
                                        Keyword = word, 
                                        Severity = keywordInfo.Severity ?? DetermineSeverity(word) 
                                    });
                                    break;
                                case KeywordCategory.Product:
                                    productsBag.Add(word);
                                    break;
                                case KeywordCategory.Competitor:
                                    competitorsBag.Add(word);
                                    break;
                            }

                            _metrics.IncrementCounter($"keyword_matches_{keywordInfo.Category}");
                        }
                    });
            }
            else
            {
                // Process sequentially for small posts
                foreach (var word in uniqueWords)
                {
                    if (_allKeywords.TryGetValue(word, out var keywordInfo))
                    {
                        ProcessKeywordMatch(word, keywordInfo, analysis);
                    }
                }
            }

            // Populate analysis results
            analysis.IssuesFound.AddRange(issuesBag);
            analysis.ProductsMentioned.AddRange(productsBag.Distinct());
            analysis.CompetitorsMentioned.AddRange(competitorsBag.Distinct());

            // OPTIMIZED: VIP check with O(1) lookup
            var domain = ExtractDomain(post.Url);
            if (!string.IsNullOrEmpty(domain) && _vipCustomerDomains.Contains(domain))
            {
                analysis.IsVipCustomer = true;
                analysis.CustomerTier = CustomerTier.Enterprise;
                _metrics.IncrementCounter("vip_customer_posts");
            }

            // Calculate derived fields
            analysis.Sentiment = await CalculateSentimentAsync(analysis, cancellationToken);
            analysis.Priority = CalculatePriority(analysis);
            analysis.ProcessingTimeMs = stopwatch.ElapsedMilliseconds;

            // Cache the results
            var cacheOptions = new MemoryCacheEntryOptions
            {
                SlidingExpiration = _options.CacheExpiration,
                Size = 1
            };
            _cache.Set(cacheKey, analysis, cacheOptions);

            // Metrics and logging
            RecordMetrics(analysis, stopwatch.ElapsedMilliseconds);

            return analysis;
        }
        catch (OperationCanceledException)
        {
            _logger.LogInformation("Blog post analysis cancelled for {PostId}", post.Id);
            _metrics.IncrementCounter("feedback_analysis_cancelled");
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to analyze blog post {PostId}", post.Id);
            _metrics.IncrementCounter("feedback_analysis_errors");
            throw new FeedbackAnalysisException($"Failed to analyze blog post {post.Id}", ex);
        }
    }

    public async Task ReloadKeywordsAsync(CancellationToken cancellationToken = default)
    {
        if (!await _reloadLock.WaitAsync(TimeSpan.FromSeconds(5), cancellationToken))
        {
            _logger.LogDebug("Keyword reload already in progress");
            return;
        }

        try
        {
            _logger.LogInformation("Reloading keywords");
            var stopwatch = Stopwatch.StartNew();

            // Load all keywords in parallel
            var tasks = new[]
            {
                LoadKeywordsCategoryAsync(KeywordCategory.Issue, cancellationToken),
                LoadKeywordsCategoryAsync(KeywordCategory.Product, cancellationToken),
                LoadKeywordsCategoryAsync(KeywordCategory.Competitor, cancellationToken),
                LoadVipDomainsAsync(cancellationToken)
            };

            var results = await Task.WhenAll(tasks);

            // Rebuild the unified keyword dictionary
            var newAllKeywords = new ConcurrentDictionary<string, KeywordInfo>(StringComparer.OrdinalIgnoreCase);

            foreach (var (category, keywords) in results.Where(r => r.HasValue).Select(r => r.Value))
            {
                foreach (var keyword in keywords)
                {
                    newAllKeywords.TryAdd(keyword.Text, keyword);
                }
            }

            // Atomic swap
            _allKeywords.Clear();
            foreach (var kvp in newAllKeywords)
            {
                _allKeywords.TryAdd(kvp.Key, kvp.Value);
            }

            _isInitialized = true;

            _logger.LogInformation(
                "Reloaded {TotalKeywords} keywords in {ElapsedMs}ms",
                _allKeywords.Count, stopwatch.ElapsedMilliseconds);

            _metrics.RecordHistogram("keyword_reload_duration_ms", stopwatch.ElapsedMilliseconds);
        }
        finally
        {
            _reloadLock.Release();
        }
    }

    private async Task<(KeywordCategory category, List<KeywordInfo> keywords)?> LoadKeywordsCategoryAsync(
        KeywordCategory category, 
        CancellationToken cancellationToken)
    {
        try
        {
            var keywords = await _keywordRepository.GetKeywordsByCategoryAsync(category, cancellationToken);
            var keywordInfos = keywords.Select(k => new KeywordInfo
            {
                Text = k.Text.ToLowerInvariant(),
                Category = category,
                Severity = k.Severity,
                Weight = k.Weight
            }).ToList();

            // Update specific HashSets for backwards compatibility
            switch (category)
            {
                case KeywordCategory.Issue:
                    _issueKeywords = new HashSet<string>(
                        keywordInfos.Select(k => k.Text), 
                        StringComparer.OrdinalIgnoreCase
                    );
                    break;
                case KeywordCategory.Product:
                    _productNames = new HashSet<string>(
                        keywordInfos.Select(k => k.Text), 
                        StringComparer.OrdinalIgnoreCase
                    );
                    break;
            }

            return (category, keywordInfos);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to load {Category} keywords", category);
            return null;
        }
    }

    private void RecordMetrics(FeedbackAnalysis analysis, long elapsedMs)
    {
        _metrics.RecordHistogram("feedback_analysis_duration_ms", elapsedMs);
        _metrics.RecordHistogram("feedback_analysis_issues_found", analysis.IssuesFound.Count);
        _metrics.RecordHistogram("feedback_analysis_products_mentioned", analysis.ProductsMentioned.Count);

        if (elapsedMs > _options.SlowRequestThresholdMs)
        {
            _logger.LogWarning(
                "Slow feedback analysis: {ElapsedMs}ms for post {PostId} with {WordCount} words",
                elapsedMs, analysis.BlogPostId, analysis.WordCount);
        }

        _logger.LogInformation(
            "Analyzed blog post {PostId} in {ElapsedMs}ms. Found {IssueCount} issues, {ProductCount} products. VIP: {IsVip}",
            analysis.BlogPostId, elapsedMs, analysis.IssuesFound.Count, 
            analysis.ProductsMentioned.Count, analysis.IsVipCustomer);
    }

    // ... additional helper methods
}

// Supporting classes
public class FeedbackServiceOptions
{
    public TimeSpan KeywordRefreshInterval { get; set; } = TimeSpan.FromHours(1);
    public TimeSpan CacheExpiration { get; set; } = TimeSpan.FromMinutes(30);
    public int SlowRequestThresholdMs { get; set; } = 500;
    public int MinWordLength { get; set; } = 2;
    public int MaxWordLength { get; set; } = 50;
    public int ParallelThreshold { get; set; } = 1000;
}

public class KeywordInfo
{
    public string Text { get; set; }
    public KeywordCategory Category { get; set; }
    public IssueSeverity? Severity { get; set; }
    public double Weight { get; set; } = 1.0;
}

public class FeedbackAnalysisException : Exception
{
    public FeedbackAnalysisException(string message, Exception innerException) 
        : base(message, innerException) { }
}

New Architecture In Place

The Results

Lessons We Learned

1. Success Changes Everything

What works for 100 items breaks at 10,000. Always ask: "What happens when this grows 100x?"

2. Big-O Notation Is Real Money

• List.Contains() is O(n) - fine for small n

• HashSet.Contains() is O(1) - essential for large n

• At scale, this difference is measured in dollars

Choose Data Structures for Tomorrow, Not Today

Final Thoughts

This experience taught us that performance problems don't announce themselves - they creep up as your success grows. That innocent List<string> that worked perfectly in demos became the bottleneck that almost killed our product.

The fix was simple - changing List to HashSet. But finding it required understanding what was happening at scale. Now, every time we use a collection, we ask ourselves:

• How big will this get?

• What's the primary operation?

• What's the Big-O complexity?

• What happens at 100x scale?

We aren’t alone on this Lessons learnt

High Scalability - Architecture stories from companies

Happy coding!