Enhanced Actions (RAG)

Use Retrieval-Augmented Generation to enhance AI actions with context from links, backlinks, and PDFs

Enhanced Actions leverage Retrieval-Augmented Generation (RAG) to provide AI with rich context from your vault’s linked notes and PDF files. This makes responses more accurate and contextually aware.

What is RAG?

RAG (Retrieval-Augmented Generation) enhances AI responses by:

1. Analyzing your selected text and linked documents
2. Retrieving relevant content from connected notes and PDFs
3. Augmenting the AI prompt with this contextual information
4. Generating more informed and accurate responses

RAG automatically processes links ([[like this]]), backlinks, and even PDF file references in your notes.

How It Works

Automatic Context Detection

When you run an action, Local GPT automatically:

1. Scans for Links

   Detects wiki-style links [[note]] and markdown links [text](note.md) in your selected text

2. Follows Backlinks

   Finds notes that link back to the current document

3. Processes PDFs

   Extracts text from linked PDF files

4. Retrieves Relevant Chunks

   Uses embedding models to find the most relevant content

5. Enhances Prompt

   Includes this context in the AI request

src/rag.ts

export async function startProcessing(
  linkedFiles: TFile[],
  vault: Vault,
  metadataCache: MetadataCache,
  activeFile: TFile,
  updateCompletedSteps?: (steps: number) => void,
): Promise<Map<string, IAIDocument>> {
  const processedDocs = new Map<string, IAIDocument>();
  const context: ProcessingContext = { vault, metadataCache, activeFile };

  await Promise.all(
    linkedFiles.map(async (file) => {
      await processDocumentForRAG(file, context, processedDocs, 0, false);
      updateCompletedSteps?.(1);
    }),
  );

  return processedDocs;
}

Setup

1. Install an Embedding Model

You need an embedding model to enable RAG. For Ollama users:

English Only

ollama pull nomic-embed-text

Fastest option, optimized for English text.

Multilingual

ollama pull bge-m3

Slower but more accurate for multiple languages.

2. Configure Embedding Provider

1. Open Local GPT settings
2. Find Embedding Provider
3. Select your embedding model provider
4. Choose a model with a large context window for best results

Use the largest model with the largest context window your system can handle for better results.

Supported File Types

Markdown Files

.md files are processed for text content and links

PDF Files

.pdf files are processed to extract text content

src/rag.ts

const SUPPORTED_RAG_EXTENSIONS = new Set(["md", "pdf"]);

export async function getFileContent(
  file: TFile,
  vault: Vault,
): Promise<string> {
  if (file.extension === "pdf") {
    const cachedContent = await fileCache.getContent(file.path);
    if (cachedContent?.mtime === file.stat.mtime) {
      return cachedContent.content;
    }

    const arrayBuffer = await vault.readBinary(file);
    const pdfContent = await extractTextFromPDF(arrayBuffer);
    await fileCache.setContent(file.path, {
      mtime: file.stat.mtime,
      content: pdfContent,
    });
    return pdfContent;
  }

  return vault.cachedRead(file);
}

Context Limits

You can configure how much context to include based on your model’s capabilities:

Context Limit · select · default: "local"

Show Context Limit Options

• Local (10,000 chars): For local models with smaller context windows
• Cloud (32,000 chars): For cloud-based models
• Advanced (100,000 chars): For advanced models with large context windows
• Max (3,000,000 chars): For cutting-edge models with massive context support

src/main.ts

private resolveContextLimit(): number {
  const preset = this.settings?.defaults?.contextLimit as
    | "local"
    | "cloud"
    | "advanced"
    | "max";
  const map: Record<string, number> = {
    local: 10_000,
    cloud: 32_000,
    advanced: 100_000,
    max: 3_000_000,
  };
  return map[preset];
}

Link Processing

Wiki-Style Links

This note references [[Another Note]] and [[Research Paper]].

Local GPT will retrieve content from both “Another Note” and “Research Paper”.

Markdown Links

See [this reference](notes/reference.md) for more details.

PDF References

According to [[research-paper.pdf]], the findings show...

src/rag.ts

export function getLinkedFiles(
  content: string,
  vault: Vault,
  metadataCache: MetadataCache,
  currentFilePath: string,
  includeAllMarkdownLinks = false,
): TFile[] {
  const sanitizedContent = sanitizeMarkdownForLinks(content);
  const wikiLinks = Array.from(
    sanitizedContent.matchAll(WIKI_LINK_REGEX),
    (match) => match[1],
  );
  const markdownLinks = Array.from(
    sanitizedContent.matchAll(MARKDOWN_LINK_REGEX),
    (match) => normalizeMarkdownLink(match[1]),
  ).filter((link): link is string => Boolean(link));

  return [...wikiLinks, ...markdownLinks]
    .map((linkText) => {
      const linkPath = metadataCache.getFirstLinkpathDest(
        linkText,
        currentFilePath,
      );
      return linkPath ? vault.getAbstractFileByPath(linkPath.path) : null;
    })
    .filter(isSupportedRagFile);
}

Backlinks

Local GPT also processes backlinks — notes that reference the current document:

src/rag.ts

export function getBacklinkFiles(
  file: TFile,
  context: ProcessingContext,
  processedDocs: Map<string, IAIDocument>,
): TFile[] {
  const resolvedLinks = context.metadataCache.resolvedLinks || {};
  const backlinks: TFile[] = [];

  for (const [sourcePath, links] of Object.entries(resolvedLinks)) {
    if (processedDocs.has(sourcePath) || !links?.[file.path]) {
      continue;
    }
    const backlinkFile = context.vault.getAbstractFileByPath(
      sourcePath,
    ) as TFile | null;
    if (backlinkFile?.extension === "md") {
      backlinks.push(backlinkFile);
    }
  }

  return backlinks;
}

Performance

Local GPT includes several optimizations:

PDF Caching

PDF content is cached to avoid re-processing. The cache is invalidated when the PDF file is modified.

Depth Limiting

Link traversal is limited to a maximum depth of 10 levels to prevent infinite loops and excessive processing.

Progress Tracking

A status bar shows processing progress when RAG is active, so you know the system is working.

src/rag.ts

const MAX_DEPTH = 10;

export async function processDocumentForRAG(
  file: TFile,
  context: ProcessingContext,
  processedDocs: Map<string, IAIDocument>,
  depth: number,
  isBacklink: boolean,
): Promise<Map<string, IAIDocument>> {
  if (depth > MAX_DEPTH || processedDocs.has(file.path)) {
    return processedDocs;
  }
  // ... process document
}

Status Bar Indicator

When RAG is processing context, you’ll see a status bar indicator:

Enhancing with context: 45%

This shows the progress of embedding generation and document retrieval.

Press Escape to cancel RAG processing at any time.

Example

Given this note:

Based on [[Project Goals]] and the findings in [[research.pdf]], 
we should focus on user experience.

When you select this text and run an action like “Summarize”, Local GPT will:

1. Read the content of “Project Goals”
2. Extract text from “research.pdf”
3. Find relevant sections using embeddings
4. Include this context when generating the summary

The result is a summary that’s informed by all linked documents, not just the selected text.

Next Steps

Vision Support

Learn how to analyze images with vision models

Community Actions

Explore and install community-contributed actions