Greedy Function Approximation: A Gradient Boosting Machine

Framed boosting as numerical gradient descent in function space, allowing gradient boosting with arbitrary differentiable loss functions.

The algorithmic foundation of XGBoost, LightGBM, CatBoost, and modern tabular competitive machine learning.

Assumptions

• Weak base learners (decision trees) can iteratively approximate the pseudo-residuals of arbitrary loss functions

Limitations

• Sequential training is inherently difficult to parallelize compared to bagging; sensitive to noisy target labels