🔄 Async Video Polling Engine - Background Job Handler for AI Video Generation

🔄 AI Video Polling Engine - Long-Running Job Handler for Veo, Sora & Seedance The async backbone that makes AI video generation production-ready ⚡🎬 !LinkedIn!Telegram --- 🎥 See It In Action !Watch the Demo 🔗 Full Demo: youtu.be/OIoJ2F1O0 --- ⚠️ Must Read First This is a companion workflow for the main AI Shorts Generator: 🔗 Main Workflow: AI Shorts Reactor This workflow handles the "waiting game" so your main bot stays fast and responsive. Think of it as the backstage crew that handles the heavy lifting while your main workflow performs on stage. --- 🤔 The Problem This Solves Without This Workflow: User sends message ↓ Bot calls AI API ↓ ⏳ Bot waits 2-5 minutes... (BLOCKED) ↓ ❌ Timeout errors ❌ Execution limits exceeded ❌ Users think bot is broken ❌ Can't handle multiple requests With This Workflow: User sends message ↓ Bot calls AI API ↓ ✅ Bot responds instantly: "Video generating..." ↓ 🔄 This webhook polls in background ↓ ⚡ Main bot handles other users ↓ ✅ Video ready → Auto-sends to user Result: Your bot feels instant, scales infinitely, and never times out 🚀 --- 🔁 What This Workflow Does This is a dedicated polling webhook that acts as the async job handler for AI video generation. It's the invisible worker that: 1️⃣ Receives the Job json POST /webhook/poll-video { "sessionId": "user123", "taskId": "veoabc456", "model": "veo3", "attempt": 1 } 2️⃣ Responds Instantly 200 OK - "Polling started" (Main workflow never waits!) 3️⃣ Polls in Background Wait 60s → Check status → Repeat - ⏱️ Waits 1 minute between checks (API-friendly) - 🔄 Polls up to 15 times (~15 minutes max) - 🎯 Supports Veo, Sora, and Seedance APIs 4️⃣ Detects Completion Handles multiple API response formats: javascript // Veo format { status: "completed", videoUrl: "https://..." } // Market format (Sora/Seedance) { job: { status: "success", result: { url: "..." } } } // Legacy format { data: { videourl: "..." } } (No matter how the API responds, this workflow figures it out) 5️⃣ Delivers the Video Once ready: - 📥 Downloads video from AI provider - ☁️ Uploads to your S3 storage - 💾 Restores user session from Redis - 📱 Sends Telegram preview with buttons - 🔄 Enables video extension (Veo only) - 📊 Logs metadata for analytics --- ⚙️ Technical Architecture The Flow: Main Workflow Polling Webhook │ │ ├──Trigger AI Job──────────┤ │ "Task ID: abc123" │ │ │ ├──Return Instantly │ │ "Generating..." │ │ │ ├──Handle New User │ │ ├──Wait 60s │ │ │ ├──Check Status │ │ "Processing..." │ │ │ ├──Wait 60s │ │ │ ├──Check Status │ │ "Completed!" │ │ │ ├──Download Video │ │ │ ├──Upload to S3 │ │ │ └──Send to User │ │ └──────────────────────────────────┘ "Your video is ready!" --- 🚀 Key Features ⚡ Non-Blocking Architecture - Main workflow never waits - Handle unlimited concurrent jobs - Each user gets instant responses 🔄 Intelligent Polling - Respects API rate limits (60s intervals) - Auto-retries on transient failures - Graceful timeout handling (15 attempts max) 🎯 Multi-Provider Support Handles different API formats: - Veo - record-info endpoint - Sora - Market job status - Seedance - Market job status 🛡️ Robust Error Handling javascript ✅ Missing video URL → Retry with fallback parsers ✅ API timeout → Continue polling ✅ Invalid response → Parse alternative formats ✅ Max attempts reached → Notify user gracefully 💾 Session Management - Stores state in Redis - Restores full context when video is ready - Supports video extension workflows - Maintains user preferences 📊 Production Features - Detailed logging at each step - Metadata tracking (generation time, model used, etc.) - S3 storage integration - Telegram notifications - Analytics-ready data structure --- 🧩 Integration Points Works Seamlessly With: | Use Case | How It Helps | |----------|--------------| | 🤖 Telegram Bots | Keeps bot responsive during 2-5 min video generation | | 📺 YouTube Automation | Polls video, then triggers auto-publish | | 🎬 Multi-Video Pipelines | Handles 10+ videos simultaneously |