How we built a robust file watching system and extended Hoppscotch's collections store to support code-first API development. You can find the repository here.

In this post, I'll walk you through how we built two critical foundation pieces: a file watching system and a collections store that understands the relationship between your code and your API tests. You can read up on the motivation from here.

Task: Building a working File Watcher

The Problem: Files Are Tricky Beasts

File watching sounds simple in theory – just watch a file and tell me when it changes, right? Wrong! Files are like cats: unpredictable, sometimes temperamental, and they don't always behave the way you expect.

Here's what we had to deal with:

Build tools write files in chunks (imagine trying to read a book while someone's still writing it)
Security concerns (we don't want users accidentally watching /etc/passwd)
Cross-platform compatibility (Windows, Mac, and Linux all handle files differently)
Performance (watching thousands of files shouldn't slow down your IDE)

The Solution: A Smart, Secure File Watcher

We built our file watcher service on top of chokidar, which is like the Swiss Army knife of file watching libraries. But we added several layers of intelligence on top:

1. Security First: Path Validation

private validateFilePath(filePath: string): { isValid: boolean; errors: string[]; warnings: string[] } {
  const errors: string[] = []
  const warnings: string[] = []

  // Security check: Ensure the file is within the current working directory
  const normalizedPath = path.resolve(filePath)
  const currentWorkingDir = process.cwd()

  if (!normalizedPath.startsWith(currentWorkingDir)) {
    errors.push('File path must be within the current project directory.')
    return { isValid: false, errors, warnings }
  }

  // Check for valid OpenAPI file extensions
  const validExtensions = ['.json', '.yaml', '.yml']
  const fileExtension = path.extname(normalizedPath).toLowerCase()

  if (!validExtensions.includes(fileExtension)) {
    warnings.push(`File extension '${fileExtension}' is not a typical OpenAPI format`)
  }

  return { isValid: true, errors, warnings }
}

Think of this as a bouncer at a club – it checks IDs (file paths) and makes sure only legitimate files get through. We prevent path traversal attacks (like ../../../etc/passwd) and warn users if they're trying to watch files that probably aren't OpenAPI specs.

2. Debouncing: The Patient Waiter

Build tools often write files in chunks, which means we might get 5-10 file change events for a single logical change. That's like getting a notification every time someone types a letter in a document – insane!

Our solution? Debouncing:

private applyDebouncing(type: FileChangeEvent['type'], filePath: string): void {
  // Clear any existing timer for this file
  const existingTimer = this.debounceTimers.get(filePath)
  if (existingTimer) {
    clearTimeout(existingTimer)
  }

  // Set a new timer - wait 300ms for the file to "settle"
  const timer = setTimeout(() => {
    this.debounceTimers.delete(filePath)

    // Now emit the actual event
    this.eventEmitter.emit({
      type,
      filePath,
      timestamp: new Date()
    })
  }, 300)

  this.debounceTimers.set(filePath, timer)
}

This is like having a patient waiter at a restaurant who waits until the chef is completely done before bringing your food, rather than bringing it out ingredient by ingredient.

3. Chokidar Configuration

this.watcher = chokidar.watch([], {
  ignored: /(^|[/\\])\../, // ignore dotfiles
  persistent: true,
  ignoreInitial: true, // Don't emit events for existing files
  followSymlinks: false, // Security: don't follow symlinks
  usePolling: false, // Use native file system events (faster)
  atomic: true, // Wait for write operations to complete
  awaitWriteFinish: {
    stabilityThreshold: 100, // Wait 100ms for file to stabilize
    pollInterval: 10 // Check every 10ms
  }
})

Each of these settings was chosen for a specific reason:

atomic: true ensures we don't get events for partially written files
awaitWriteFinish is crucial for build processes that write files in chunks
usePolling: false uses native file system events for better performance

Testing: Because File Watchers Are Sneaky

File watchers are notoriously difficult to test because they involve real file system operations and timing. We used Vitest with mocking to create comprehensive tests:

describe('watchFile', () => {
  it('should successfully watch a valid file', async () => {
    const testFile = path.join(process.cwd(), 'test-spec.json')

    const result = await fileWatcher.watchFile(testFile)

    expect(result.success).toBe(true)
    expect(result.errors).toHaveLength(0)
    expect(mockWatcher.add).toHaveBeenCalledWith(testFile)
    expect(fileWatcher.isWatching(testFile)).toBe(true)
  })

  it('should reject paths outside project directory', async () => {
    const result = await fileWatcher.watchFile('/etc/passwd')

    expect(result.success).toBe(false)
    expect(result.errors).toContain('File path must be within the current project directory for security reasons')
  })
})

Smart Collections That Understand Your Code

The Problem: Collections Don't Know About Code

Traditional API collections are static – they're just a list of requests with no understanding of where they came from or how they relate to your actual code. It's like having a photo album with no dates or context about when or where the photos were taken.

For live sync to work, our collections needed to become smarter. They needed to:

Remember their origin (which development server or file they came from)
Track user customizations (so we don't overwrite carefully crafted auth headers)
Understand frameworks (FastAPI works differently than Express)
Handle conflicts intelligently (what happens when both code and user change the same thing?)

The Solution: Collections with Memory and Intelligence

We extended Hoppscotch's existing collection system with rich metadata that tracks the relationship between code and collections.

1. Framework Awareness: Teaching Collections About Code

export interface FrameworkInfo {
  name: string // e.g., 'FastAPI', 'Express', 'Spring Boot'
  version?: string // detected framework version
  icon?: string // icon identifier for UI display
  setupGuide?: string // URL or identifier for setup documentation
  commonEndpoints?: string[] // typical OpenAPI endpoints for this framework
  detectionPatterns?: {
    packageJson?: string[] // npm package names that indicate this framework
    requirements?: string[] // Python requirements.txt patterns
    gradle?: string[] // Gradle dependency patterns
    filePatterns?: string[] // File patterns that indicate this framework
  }
}

This is like making our collections smarter – they now understand the difference between a FastAPI app (which serves OpenAPI at /openapi.json) and a Spring Boot app (which uses /v3/api-docs).

2. Smart Framework Detection

We built a framework detection system that can identify development stack from multiple sources:

export function detectFrameworkFromPackageJson(packageJsonContent: string): FrameworkDetectionResult {
  const packageJson = JSON.parse(packageJsonContent)
  const dependencies = { ...packageJson.dependencies, ...packageJson.devDependencies }
  const dependencyNames = Object.keys(dependencies)

  const detectedFrameworks: FrameworkInfo[] = []

  for (const framework of FRAMEWORK_DEFINITIONS) {
    const matches = framework.detectionPatterns?.packageJson?.filter(pattern => 
      dependencyNames.some(dep => dep.includes(pattern))
    )

    if (matches && matches.length > 0) {
      detectedFrameworks.push({
        ...framework,
        version: dependencies[matches[0]]?.replace(/[\^~]/, '') // Remove version prefixes
      })
    }
  }

  return {
    detected: detectedFrameworks.length > 0,
    frameworks: detectedFrameworks,
    confidence: detectedFrameworks.length > 0 ? 0.9 : 0,
    detectionMethod: 'package-analysis'
  }
}

It's like having a detective that can look at your project and say "Ah, I see you're using FastAPI 0.104.1 with uvicorn – your OpenAPI spec is probably at /openapi.json."

3. Customization Tracking: Remembering What Users Changed

This was the trickiest part. We needed to track what users customized so we could preserve those changes during sync operations:

export interface CollectionCustomizations {
  // Request-level customizations
  requests?: Record<string, {
    hasCustomHeaders?: boolean
    hasCustomAuth?: boolean
    hasCustomScripts?: boolean
    hasCustomParams?: boolean
    hasCustomBody?: boolean
    customizedAt?: Date
  }>

  // Collection-level customizations
  collection?: {
    hasCustomAuth?: boolean
    hasCustomHeaders?: boolean
    hasCustomName?: boolean
    hasCustomDescription?: boolean
    customizedAt?: Date
  }
}

Think of this as a detailed change log that remembers every customization a user makes. When new code changes come in, we can intelligently merge them while preserving user customizations.

4. Store Actions: Making It All Work Together

We extended Hoppscotch's existing collections store with new actions for handling live sync:

/**
 * Track user customizations for intelligent merging
 */
trackCustomization(
  { state }: RESTCollectionStoreType,
  {
    collectionIndex,
    customizationType,
    itemPath,
    customizationData,
  }: {
    collectionIndex: number
    customizationType: 'request' | 'collection' | 'folder'
    itemPath?: string
    customizationData: any
  }
) {
  return {
    state: state.map((col, index) => {
      if (index !== collectionIndex) return col

      const liveSyncCol = col as LiveSyncCollection
      const existingCustomizations = liveSyncCol.liveMetadata?.customizations || {}

      // Update customizations based on type
      let updatedCustomizations: CollectionCustomizations
      // ... customization logic

      return {
        ...col,
        liveMetadata: {
          ...liveSyncCol.liveMetadata,
          customizations: updatedCustomizations,
          updatedAt: new Date()
        }
      } as LiveSyncCollection
    }),
  }
}

This action is called whenever a user modifies something in a live sync collection, ensuring we remember their changes for future sync operations.

Testing: Ensuring Intelligence Works

Testing intelligent systems is like testing a smart assistant – you need to verify it makes the right decisions in various scenarios:

it('should track user customizations', () => {
  // Track a request customization
  trackCollectionCustomization(0, 'request', '/users/0', {
    hasCustomHeaders: true,
    hasCustomAuth: true
  })

  const collections = restCollectionStore.value.state
  const liveSyncCol = collections[0] as LiveSyncCollection

  expect(liveSyncCol.liveMetadata?.customizations?.requests?.['/users/0']?.hasCustomHeaders).toBe(true)
  expect(liveSyncCol.liveMetadata?.customizations?.requests?.['/users/0']?.customizedAt).toBeInstanceOf(Date)
})

The flow

File Watcher detects change → "Hey, the OpenAPI spec file changed!"
Framework Detection kicks in → "This looks like a FastAPI app based on the file structure"
Collections Store gets updated → "Let me merge these changes while preserving user customizations"
User sees intelligent update → "Cool, new endpoints appeared but my custom auth headers are still there!"

It's like having a smart assistant that can update your music playlist with new songs from your favorite artists while keeping your custom playlists intact.

What's Next?

These two tasks laid the foundation for the entire live sync system. With robust file watching and intelligent collections, we can now build:

Real-time sync engines that detect and apply changes instantly
Conflict resolution UI that helps users handle merge conflicts
Framework-specific optimizations that work better with each development stack
Team collaboration features that sync changes across team members

Still here? Cool. I hope you are enjoying the ride so far! In the next post, I’ll focus on comparing the differences between changes.

API Live Sync #5: File Watching

Table of contents