HyperFlow Workflow Orchestrator
Visual DAG-based task execution engine with distributed worker queues.
System Overview & Motivation
HyperFlow is a visual directed acyclic graph (DAG) workflow orchestrator and distributed task scheduler. It empowers teams to visually connect data ingestion, transformation, model evaluation, and notification jobs via an interactive browser canvas, executing tasks reliably across distributed Celery worker nodes backed by Redis.
Key Architectural Highlights
- Interactive drag-and-drop canvas with real-time topological sorting and DAG cycle detection
- Distributed task execution engine leveraging Celery and Redis message queues
- Granular step-level retry policies with exponential backoff and dead-letter queues
- Live node status telemetry via WebSockets showing task completion, stdout logs, and execution duration
- RESTful workflow definition format allowing code-first or UI-first job creation
Technical Architecture & Data Pipeline
Users build workflows in a React canvas, which compiles node graphs into JSON DAG specifications. A FastAPI backend validates dependency trees, schedules tasks according to topology, and dispatches job payloads to Celery workers. Execution states are broadcast over WebSockets to provide instantaneous UI updates.
Production Challenges & Edge Cases Solved
Preventing race conditions and circular deadlocks in dynamic workflows with branch merges. Solved by implementing Kahn's topological sort algorithm on graph saves and atomic Redis transaction locks during task completion.