Skip to main content

> ML_LIBRARY // METAFLOW_v1.0

Metaflow

Outerbounds / Netflix — Netflix's human-centric Python framework for building and scaling real-world data science workflows.

orchestrationv2.12.20Apache-2.0qualified
Official DocumentationSource Repository

Model Training

Supported
Accelerators:
CPUCUDA
Distributed Training:Yes

Model Inference

Supported
Inference Accelerators:
CPUCUDA
Deployment Targets:server

What It Does

  • +Idiomatic Python workflow orchestration: define flows as simple classes inheriting from FlowSpec
  • +Seamless scaling: execute individual steps locally on laptop or on AWS Batch/Kubernetes via simple @batch/@kubernetes decorators
  • +Automatic artifact persistence: any self.variable is automatically serialized and versioned in S3/GCS without boilerplate
  • +Resume-from-step: replay and debug failed pipelines starting exactly from the failed step with state intact

What It Does Not Do

  • -Act as a low-latency real-time inference serving engine (use BentoML/vLLM)
  • -Manage declarative GitOps Kubernetes manifests directly
  • -Train models without user code

>Suitable Work Types

  • Data science workflows where individual scientists need to scale training from local laptops to hundreds of cloud GPU workers
  • Iterative ML research requiring fast debugging with resume-from-step without re-running data ingestion
  • Production batch ML pipelines deployed to AWS Step Functions, Argo Workflows, or Airflow

>Unsuitable Work Types

  • Real-time streaming microsecond scoring APIs
  • Zero-code AutoML for business analysts
Data Residency Implications

Artifacts and metadata persist inside your own private AWS/GCP/Azure buckets and database. Zero telemetry.

Security Considerations

Apache-2.0 license. Built and battle-tested at Netflix scale.

Operational Profile & Known Limitations

Maturity:mature
Learning Curve:low
Ops Complexity:low
Cost Tier:free-oss
> Known Limitations:
  • Python-centric; not designed for non-Python data pipelines (e.g. pure dbt or Java MapReduce).

Associated Incident Patterns (Incidentpedia)

Enforce safeguards and monitoring to guard against these documented real-world failure modes:

thread-pool-exhaustion

> Primary Evidence & Benchmark Citations

Metaflow Documentationofficial-docs • >=2.11.0, <=2.12.x
2026-09-25HIGH