Process documents & build semantic search with OpenAI, Gemini & Qdrant

🎯 Overview This n8n workflow automates the process of ingesting documents from multiple sources (Google Drive and web forms) into a Qdrant vector database for semantic search capabilities. It handles batch processing, document analysis, embedding generation, and vector storage - all while maintaining proper error handling and execution tracking. 🚀 Key Features - Dual Input Sources: Accepts files from both Google Drive folders and web form uploads - Batch Processing: Processes files one at a time to prevent memory issues and ensure reliability - AI-Powered Analysis: Uses Google Gemini to extract metadata and understand document context - Vector Embeddings: Generates OpenAI embeddings for semantic search capabilities - Automated Cleanup: Optionally deletes processed files from Google Drive (configurable) - Loop Processing: Handles multiple files efficiently with Split In Batches nodes - Interactive Chat Interface: Built-in chatbot for testing semantic search queries against indexed documents 📋 Use Cases - Knowledge Base Creation: Build searchable document repositories for organizations - Document Compliance: Process and index legal/regulatory documents (like Fair Work documents) - Content Management: Automatically categorize and store uploaded documents - Research Libraries: Create semantic search capabilities for research papers or reports - Customer Support: Enable instant answers to policy and documentation questions via chat interface 🔧 Workflow Components Input Methods 1. Google Drive Integration - Monitors a specific folder for new files - Processes existing files in batch mode - Supports automatic file conversion to PDF 2. Web Form Upload - Public-facing form for document submission - Accepts PDF, DOCX, DOC, and CSV files - Processes multiple file uploads in a single submission Processing Pipeline 1. File Splitting: Separates multiple uploads into individual items 2. Document Analysis: Google Gemini extracts document understanding 3. Text Extraction: Converts documents to plain text 4. Embedding Generation: Creates vector embeddings via OpenAI 5. Vector Storage: Inserts documents with embeddings into Qdrant 6. Loop Control: Manages batch processing with proper state handling Key Nodes - Split In Batches: Processes files one at a time with reset: false to maintain state - Google Gemini: Analyzes documents for context and metadata - Langchain Vector Store: Handles Qdrant insertion with embeddings - HTTP Request: Direct API calls for custom operations - Chat Interface: Interactive chatbot for testing vector search queries 🛠️ Technical Implementation Batch Processing Logic The workflow uses a clever looping mechanism: - Split In Batches with batchSize: 1 ensures single-file processing - reset: false maintains loop state across iterations - Loop continues until all files are processed Error Handling - All nodes include continueOnFail options where appropriate - Execution logs are preserved for debugging - File deletion only occurs after successful insertion Data Flow Form Upload → Split Files → Batch Loop → Analyze → Insert → Loop Back Google Drive → List Files → Batch Loop → Download → Analyze → Insert → Delete → Loop Back 📊 Performance Considerations - Processing Time: ~20-30 seconds per file - Batch Size: Set to 1 for reliability (configurable) - Memory Usage: Optimized for files under 10MB - API Costs: Uses OpenAI embeddings (text-embedding-3-large model) 🔐 Required Credentials 1. Google Drive OAuth2: For file access and management 2. OpenAI API: For embedding generation 3. Qdrant API: For vector database operations 4. Google Gemini API: For document analysis 💡 Implementation Tips 1. Start Small: Test with a few files before processing large batches 2. Monitor Costs: Track OpenAI API usage for embedding generation 3. Backup First: Consider archiving instead of deleting processed files 4. Check Collections: Ensure Qdrant collection exists before running 🎨 Customization Options - Change Embedding Model: Switch to te