Flokkit Flow

Getting Started

  • Installation
    • From PyPI
    • From Source
      • Using Git
      • For Development
    • Using just
    • Verify Installation
    • Optional Dependencies
    • System Requirements
    • Troubleshooting
      • Import Error
      • Type Checking
      • Performance
  • Quick Start
    • Your First Flow
    • Understanding the Components
      • 1. Flow Builder
      • 2. Source Nodes
      • 3. Transform Nodes
      • 4. Sink Nodes
    • Working with Async Functions
    • Splitting and Merging
    • Error Handling
    • Next Steps
    • Common Patterns
      • ETL Pipeline
      • Real-time Stream Processing
      • Batch Processing with Caching
  • Core Concepts
    • Flow-Based Programming (FBP)
      • Benefits of FBP
    • Nodes
      • Source Nodes
      • Transform Nodes
      • Sink Nodes
      • Special Nodes
    • Ports and Connections
      • Ports
      • Connections
    • Queues and Backpressure
      • Bounded Queues
      • Queue Strategies
    • Type Safety
    • Async Execution
      • Automatic Adaptation
      • Native Async Support
    • Graph Execution
      • DAG Verification
      • Execution Model
    • Lifecycle Management
    • Pipeline vs Server Patterns
      • Pipeline Pattern (auto_stop=True)
      • Server Pattern (auto_stop=False)
    • Caching and Memoization
    • Best Practices
    • Next Steps

User Guide

  • Basic Usage
    • Building Your First Pipeline
      • Simple Linear Pipeline
      • Adding Multiple Transforms
    • Working with Different Data Types
      • Type-Safe Pipelines
    • Filtering Data
      • Basic Filtering
      • Filtering with Side Effects
    • Observing Data with Tap
      • Logging Pipeline
      • Debugging with Tap
    • Splitting and Merging Flows
      • Fan-Out Pattern
      • Fan-In Pattern
    • Caching Expensive Operations
      • Enable Caching
      • Custom Cache Key
    • Error Handling Patterns
      • Continue on Error
      • Error Output Stream
    • Working with External Resources
      • Database Pipeline
    • Best Practices
      • 1. Keep Functions Pure
      • 2. Use Type Hints
      • 3. Handle None Values
      • 4. Batch Operations
    • Next Steps
  • Backpressure and Flow Control
    • Understanding Backpressure
      • The Problem
    • Queue Configuration
      • Setting Queue Size
      • Queue Full Strategies
        • 1. BLOCK (Default)
        • 2. DROP_NEW
        • 3. DROP_OLD
        • 4. ERROR
    • Real-World Examples
      • Rate-Limited API Calls
      • Batch Processing with Backpressure
      • Multi-Speed Pipeline
    • Monitoring Queue Health
      • Queue Metrics
      • Adaptive Queue Sizing
    • Advanced Patterns
      • Backpressure with Multiple Consumers
      • Dynamic Backpressure
      • Circuit Breaker Pattern
    • Best Practices
      • 1. Start with Small Queues
      • 2. Monitor Queue Metrics
      • 3. Use Appropriate Strategies
      • 4. Consider Batch Processing
      • 5. Plan for Bursts
    • Debugging Backpressure Issues
      • Symptoms of Problems
      • Diagnostic Tools
    • Next Steps
  • Lifecycle Management
    • Lifecycle Hooks Overview
    • Resource Management
      • Database Connections
      • File Handles
      • External Services
    • Error Handling
      • Graceful Degradation
      • Error Recovery
    • Shutdown Strategies
      • Graceful Shutdown
      • State Persistence
    • Pipeline vs Server Mode
      • Pipeline Mode (auto_stop=True)
      • Server Mode (auto_stop=False)
    • Best Practices
      • 1. Always Clean Up Resources
      • 2. Idempotent Lifecycle Methods
      • 3. Fail Fast on Startup
      • 4. Log Lifecycle Events
      • 5. Handle Partial Completion
    • Next Steps
  • Common Patterns
    • ETL Pipeline Pattern
    • Stream Processing Pattern
    • Fan-Out/Fan-In Pattern
    • Request-Reply Pattern
    • Batch Processing Pattern
    • Circuit Breaker Pattern
    • Saga Pattern
    • Content-Based Router Pattern
    • Monitoring and Observability Pattern
    • Testing Pattern
    • Best Practices Summary
    • Next Steps
  • Shell Commands
    • Overview
    • Basic Usage
      • Shell Command as Source
      • Shell Command as Transform
      • Shell Command as Sink
    • Advanced Features
      • The ShellCommand Class
      • Input Modes
      • Output Modes
      • Custom Parsers
      • Error Handling
    • Complex Pipelines
    • Best Practices
      • Security
      • Performance
      • Error Handling
    • Examples
      • Log Analysis Pipeline
      • Data ETL Pipeline
    • Integration with Middleware
    • Limitations
    • Summary
  • Middleware System
    • Overview
    • Basic Usage
    • Built-in Middleware
      • LoggingMiddleware
      • MetricsMiddleware
      • ThrottleMiddleware
      • RetryMiddleware
    • Creating Custom Middleware
    • ProcessingContext
    • Middleware Composition
    • Advanced Patterns
      • Conditional Middleware
      • Error Handling Middleware
      • State-Tracking Middleware
    • Best Practices
    • Performance Considerations
    • Examples

API Reference

  • Core API
    • Flow Builder
      • flow()
      • FlowBuilder
        • FlowBuilder.__init__()
        • FlowBuilder.source()
        • FlowBuilder.transform()
        • FlowBuilder.filter()
        • FlowBuilder.tap()
        • FlowBuilder.sink()
        • FlowBuilder.to()
        • FlowBuilder.split()
        • FlowBuilder.merge_with()
        • FlowBuilder.with_middleware()
        • FlowBuilder.build()
        • FlowBuilder.execute()
    • Core Components
      • Node Base Class
        • Node
      • Port
        • Port
      • Connection
        • Connection
      • Executable Graph
        • ExecutableGraph
    • Enumerations
      • Queue Full Strategy
        • QueueFullStrategy
      • Shutdown Mode
        • ShutdownMode
    • Middleware System
      • Base Middleware
        • Middleware
      • Processing Context
        • ProcessingContext
      • Built-in Middleware
        • LoggingMiddleware
        • MetricsMiddleware
        • ThrottleMiddleware
        • RetryMiddleware
    • Utility Functions
      • Async Utilities
        • ensure_async()
      • Async Iterator Factory
        • AsyncIteratorAdapter()
  • FlowBuilder API
    • Creating a Flow
    • FlowBuilder Methods
      • Source Methods
        • FlowBuilder.source()
      • Transform Methods
        • FlowBuilder.transform()
      • Filter Methods
        • FlowBuilder.filter()
      • Side Effects
        • FlowBuilder.tap()
      • Sink Methods
        • FlowBuilder.sink()
        • FlowBuilder.to()
      • Flow Control
        • FlowBuilder.split()
        • FlowBuilder.merge_with()
      • Middleware
        • FlowBuilder.with_middleware()
      • Execution
        • FlowBuilder.build()
        • FlowBuilder.execute()
    • Complete Example
  • Graph and Execution
    • ExecutableGraph
      • ExecutableGraph
        • ExecutableGraph.__init__()
        • ExecutableGraph.run()
        • ExecutableGraph.stop()
      • Properties
        • ExecutableGraph.name
        • ExecutableGraph.nodes
        • ExecutableGraph.connections
        • ExecutableGraph.source_nodes
      • Methods
        • ExecutableGraph.run()
        • ExecutableGraph.stop()
    • Graph Patterns
      • Pipeline Pattern
      • Server Pattern
      • Concurrent Execution
    • Error Handling
      • Node-Level Errors
      • Graph-Level Errors
    • Monitoring
      • Runtime Inspection
      • Metrics Collection
    • Best Practices
  • Middleware API
    • Core Classes
      • Middleware Base Class
        • Middleware
      • ProcessingContext
        • ProcessingContext
    • Built-in Middleware
      • LoggingMiddleware
        • LoggingMiddleware
      • MetricsMiddleware
        • MetricsMiddleware
      • ThrottleMiddleware
        • ThrottleMiddleware
      • RetryMiddleware
        • RetryMiddleware
    • Custom Middleware
      • Simple Example
      • Advanced Example
    • Middleware Composition
    • Best Practices
      • Example: Production Pipeline
  • Node Reference
    • Node Lifecycle
      • Lifecycle Methods
    • Creating Custom Nodes
      • Basic Custom Node
      • Stateful Node with Resources
    • Port Management
      • Adding Ports
      • Accessing Ports
    • Advanced Patterns
      • Multi-Input Processing
      • Windowing Node
      • Error Handling Node
    • Best Practices
  • Shell Command API
    • ShellCommand Class
      • ShellCommand
        • ShellCommand.__init__()
        • ShellCommand.run()
        • ShellCommand.stream_output()
        • ShellCommand.__call__()
    • ShellResult Class
      • ShellResult
        • ShellResult.__init__()
    • Factory Functions
      • shell_source
        • shell_source()
      • shell_transform
        • shell_transform()
      • shell_sink
        • shell_sink()

Examples

  • Examples
    • Basic Examples
    • Advanced Examples
    • Pattern Examples
    • Running Examples
    • Example Files
    • Contributing Examples
    • Example Template
Flokkit Flow
  • Search

© Copyright 2025, Flokkit Flow Contributors.

Built with Sphinx using a theme provided by Read the Docs.