TensorGraph C++
Tensor compute graph with reverse-mode automatic differentiation.
System Overview & Motivation
TensorGraph C++ is a minimalist deep learning computational graph library implemented in modern C++20. Designed to explore how PyTorch and TensorFlow compute gradients under the hood, it implements reverse-mode automatic differentiation (autograd) on dynamically constructed directed acyclic graphs.
Key Architectural Highlights
- Pure C++20 implementation with custom multi-dimensional strided tensor representation
- Dynamic compute graph execution with reverse-mode automatic gradient backpropagation
- SIMD vectorization (AVX2/NEON) for accelerated float32 matrix multiplications
- Custom memory arena allocator eliminating runtime heap allocation thrashing
- Python C-API bindings allowing seamless integration with NumPy arrays
Technical Architecture & Data Pipeline
Operations on Tensor instances construct computational graph nodes recording parents and gradient backward closures. Calling `.backward()` traverses the DAG in reverse topological order, accumulating gradients at leaf nodes.
Production Challenges & Edge Cases Solved
Minimizing cache misses during non-contiguous tensor transpose and slicing. Solved by implementing optimized strided index iterators and tiled matrix multiplication algorithms.