A RAG-powered article publishing pipeline


Motivation
For some time now, I have been maintaining a Zettelkasten to capture notes on topics I explore, whether through my work or personal learning. Over the years, this system has grown into a substantial network of interconnected notes. The strength of the Zettelkasten lies in its ability to reveal relationships between ideas that I might never have discovered otherwise. This led me to wonder: what if I could distill these connections into cohesive articles and share the insights they reveal?
Because a Zettelkasten naturally organizes information as a graph, there is an opportunity to use a large language model (LLM) to enhance the writing process. The idea is to have the LLM analyze the graph structure, surface related content already in the system, and use that context to generate a structured article outline that highlights the key points to cover. From there, I can develop the full piece in my own writing style. This could significantly accelerate my publishing workflow, especially when the foundational research and projects are already in place.
Microsoft’s GraphRAG offers a similar concept. It applies graph-based retrieval to deliver more complete and contextually grounded answers from a text corpus by leveraging relationships between nodes and the entities they refer to. While my implementation will be far simpler than GraphRAG, my goal is to achieve a similar benefit: articles that are well-structured, comprehensive, and deeply connected across diverse topics.
Solution Design
Requirements
Core content generation capabilities
Generate structured article skeletons from the Zettelkasten graph, identifying key sections and points based on the strongest links between notes
Continuously reassess and update suggested outlines as new content is ingested, ensuring the article ideas evolve with the knowledge base
Identify gaps in content completeness and flag areas that require additional research before articles can be developed
Content management and tracking
Maintain a comprehensive catalogue of all suggested articles, with clear indicators of their progress stages (idea, research in progress, writing, complete)
Track published articles and prevent duplicate or redundant suggestions
Preserve historical versions of article outlines to monitor the evolution of ideas over time
Graph-based analysis and prioritization
Model the Zettelkasten as a graph with nodes (notes) and edges (links) to uncover clusters of related information suitable for article development
Rank article suggestions by relevance, completeness, novelty, and expected value to readers
Detect emerging thematic trends and highlight cross-domain connections that present unique insights
Recommend priorities for article creation based on timeliness, novelty, and alignment with ongoing projects
System integration
Support ingestion from existing Zettelkasten formats such as Markdown files or Obsidian vaults, with bidirectional synchronization to keep the graph representation current
Integrate with a large language model to generate article outlines and summaries, allowing customization of prompts to maintain consistent writing style and tone
User interface and workflow management
Provide an intuitive UI dashboard where users can:
View, search, and filter suggested article ideas by topic, status, or required research effort
Track progress of articles through various stages from ideation to completion
Review, approve, modify, or reject suggested article outlines before moving to writing
Access historical versions of outlines and compare their evolution
Enable configurable outline detail levels within the UI, allowing users to control the granularity of article skeletons
Offer export options for article outlines in Markdown or other preferred formats, including links back to the original notes in the Zettelkasten
Implementation
Ingestion Pipeline
Content Completeness Evaluation Pipeline
Article Draft Generation Pipeline
Deployment
Evaluation
Subscribe to my newsletter
Read articles from Vishnu S directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by
