A RAG-powered article publishing pipeline

Vishnu SVishnu S
3 min read

Motivation

For some time now, I have been maintaining a Zettelkasten to capture notes on topics I explore, whether through my work or personal learning. Over the years, this system has grown into a substantial network of interconnected notes. The strength of the Zettelkasten lies in its ability to reveal relationships between ideas that I might never have discovered otherwise. This led me to wonder: what if I could distill these connections into cohesive articles and share the insights they reveal?

Because a Zettelkasten naturally organizes information as a graph, there is an opportunity to use a large language model (LLM) to enhance the writing process. The idea is to have the LLM analyze the graph structure, surface related content already in the system, and use that context to generate a structured article outline that highlights the key points to cover. From there, I can develop the full piece in my own writing style. This could significantly accelerate my publishing workflow, especially when the foundational research and projects are already in place.

Microsoft’s GraphRAG offers a similar concept. It applies graph-based retrieval to deliver more complete and contextually grounded answers from a text corpus by leveraging relationships between nodes and the entities they refer to. While my implementation will be far simpler than GraphRAG, my goal is to achieve a similar benefit: articles that are well-structured, comprehensive, and deeply connected across diverse topics.

Solution Design

Requirements

  • Core content generation capabilities

    • Generate structured article skeletons from the Zettelkasten graph, identifying key sections and points based on the strongest links between notes

    • Continuously reassess and update suggested outlines as new content is ingested, ensuring the article ideas evolve with the knowledge base

    • Identify gaps in content completeness and flag areas that require additional research before articles can be developed

  • Content management and tracking

    • Maintain a comprehensive catalogue of all suggested articles, with clear indicators of their progress stages (idea, research in progress, writing, complete)

    • Track published articles and prevent duplicate or redundant suggestions

    • Preserve historical versions of article outlines to monitor the evolution of ideas over time

  • Graph-based analysis and prioritization

    • Model the Zettelkasten as a graph with nodes (notes) and edges (links) to uncover clusters of related information suitable for article development

    • Rank article suggestions by relevance, completeness, novelty, and expected value to readers

    • Detect emerging thematic trends and highlight cross-domain connections that present unique insights

    • Recommend priorities for article creation based on timeliness, novelty, and alignment with ongoing projects

  • System integration

    • Support ingestion from existing Zettelkasten formats such as Markdown files or Obsidian vaults, with bidirectional synchronization to keep the graph representation current

    • Integrate with a large language model to generate article outlines and summaries, allowing customization of prompts to maintain consistent writing style and tone

  • User interface and workflow management

    • Provide an intuitive UI dashboard where users can:

      • View, search, and filter suggested article ideas by topic, status, or required research effort

      • Track progress of articles through various stages from ideation to completion

      • Review, approve, modify, or reject suggested article outlines before moving to writing

      • Access historical versions of outlines and compare their evolution

    • Enable configurable outline detail levels within the UI, allowing users to control the granularity of article skeletons

    • Offer export options for article outlines in Markdown or other preferred formats, including links back to the original notes in the Zettelkasten

Implementation

Ingestion Pipeline

Content Completeness Evaluation Pipeline

Article Draft Generation Pipeline

Deployment

Evaluation

0
Subscribe to my newsletter

Read articles from Vishnu S directly inside your inbox. Subscribe to the newsletter, and don't miss out.

Written by

Vishnu S
Vishnu S