Linear Models

Linear models form the foundation of supervised learning by assuming linear relationships between input features and target variables. In financial services, they power credit scoring and risk assessment. In retail, they enable demand forecasting and price optimization.

Mathematical Foundation

Linear Regression

Models the relationship between features and continuous targets:

Symbol Definitions:

• [mathematical expression] = Predicted output (continuous value)
• [mathematical expression] = Weight vector (learned parameters)
• [mathematical expression] = Input feature vector
• [mathematical expression] = Bias term (intercept)
• [mathematical expression] = Weight for feature [mathematical expression]
• [mathematical expression] = Value of feature [mathematical expression]
• [mathematical expression] = Number of features

Matrix Form:

Symbol Definitions:

• [mathematical expression] = Target vector (n×1)
• [mathematical expression] = Feature matrix (n×d)
• [mathematical expression] = Vector of ones (n×1)
• [mathematical expression] = Number of training samples

Loss Function (Mean Squared Error)

Closed-Form Solution (Normal Equation):

Symbol Definitions:

• [mathematical expression] = Optimal weight vector
• [mathematical expression] = Inverse of feature covariance matrix
• [mathematical expression] = Feature-target correlation vector

Logistic Regression

Binary Classification

Models probability using logistic function:

Symbol Definitions:

• [mathematical expression] = Probability of positive class given features
• [mathematical expression] = Sigmoid/logistic function
• [mathematical expression] = Euler's number (≈ 2.718)

Log-Odds (Logit):

Symbol Definitions:

• [mathematical expression] = Odds ratio (probability of success vs. failure)
• [mathematical expression] = Natural logarithm

Loss Function (Cross-Entropy):

Financial Services Example: Credit Scoring Model

Business Context: Bank develops linear model to assess credit default risk for loan applications, balancing accuracy with regulatory interpretability requirements.

Features:

• [mathematical expression] = Annual income (normalized)
• [mathematical expression] = Credit history length (years)
• [mathematical expression] = Debt-to-income ratio
• [mathematical expression] = Number of existing credit accounts
• [mathematical expression] = Previous default indicator (0/1)

Logistic Regression Model:

Model Interpretation:

• Income coefficient (0.8): Higher income reduces default probability
• Credit history (0.5): Longer history indicates stability
• Debt ratio (-1.2): Higher debt increases risk significantly
• Previous default (2.4): Strongest predictor of future default

Decision Rule:

Business Performance:

• Accuracy: 87.3% correct predictions
• Precision: 0.92 (approved loans that don't default)
• Recall: 0.81 (actual defaults correctly identified)
• ROC AUC: 0.89 (excellent discrimination)
• Business Impact: 12M reduction in annual defaults

Regulatory Compliance: Model coefficients provide clear explanations for loan decisions, meeting Fair Lending requirements.

Retail Example: Dynamic Pricing Optimization

Business Context: E-commerce retailer uses linear regression to optimize product prices based on demand factors, competitor pricing, and inventory levels.

Demand Prediction Model:

Feature Engineering:

• [mathematical expression] = Log-transformed price (captures elasticity)
• [mathematical expression] = Seasonal index (1 = normal, >1 = high season)
• [mathematical expression] = Average competitor price difference
• [mathematical expression] = Stock level normalized by target inventory

Price Elasticity:

Optimal Pricing: Maximize revenue subject to inventory constraints:

First-Order Condition:

Optimal Price:

Multi-Product Portfolio:

Symbol Definitions:

• [mathematical expression] = Price vector for all products
• [mathematical expression] = Marginal cost for product [mathematical expression]
• [mathematical expression] = Demand function for product [mathematical expression] given all prices

Business Results:

• Revenue Increase: 18.3% vs. fixed pricing
• Margin Improvement: 4.2 percentage points
• Inventory Turnover: 25% faster clearance of slow-moving items
• Competitive Position: Maintained market share while improving profitability

Regularization Techniques

Ridge Regression (L2 Regularization)

Prevents overfitting by penalizing large weights:

Closed-Form Solution:

Symbol Definitions:

• [mathematical expression] = Regularization parameter (controls penalty strength)
• [mathematical expression] = Identity matrix
• [mathematical expression] = L2 norm squared (sum of squared weights)

Lasso Regression (L1 Regularization)

Performs feature selection by driving some weights to zero:

Symbol Definitions:

• [mathematical expression] = L1 norm (sum of absolute weights)
• [mathematical expression] = Absolute value of weight [mathematical expression]

Solution Method (Coordinate Descent):

Symbol Definitions:

• [mathematical expression] = Simple least squares coefficient for feature [mathematical expression]
• [mathematical expression] = Sign function (-1, 0, or +1)

Elastic Net

Combines L1 and L2 regularization:

Symbol Definitions:

• [mathematical expression] = L1 regularization parameter
• [mathematical expression] = L2 regularization parameter

Financial Services Example: Portfolio Risk Modeling

Business Context: Investment firm uses regularized linear regression to model portfolio risk factors and optimize asset allocation.

Factor Model:

Symbol Definitions:

• [mathematical expression] = Portfolio return
• [mathematical expression] = Alpha (excess return)
• [mathematical expression] = Factor loading for factor [mathematical expression]
• [mathematical expression] = Factor [mathematical expression] return (market, size, value, etc.)
• [mathematical expression] = Idiosyncratic error

Risk Factors:

• [mathematical expression] = Market factor (S&P 500 return)
• [mathematical expression] = Size factor (small-cap premium)
• [mathematical expression] = Value factor (value-growth spread)
• [mathematical expression] = Momentum factor (price momentum)
• [mathematical expression] = Quality factor (earnings quality)

Ridge Regression for Stability:

Portfolio Optimization:

Symbol Definitions:

• [mathematical expression] = Asset allocation weights
• [mathematical expression] = Covariance matrix (from factor model)
• [mathematical expression] = Expected return vector
• [mathematical expression] = Target return

Business Results:

• Risk Prediction Accuracy: R² = 0.84 for portfolio variance
• Factor Exposure Control: Within ±0.1 of target allocations
• Sharpe Ratio: 1.23 (vs. 0.87 benchmark)
• Maximum Drawdown: 12.4% (vs. 18.7% benchmark)

Model Selection and Validation

Cross-Validation for Regularization

K-Fold CV Error:

Symbol Definitions:

• [mathematical expression] = Optimal regularization parameter
• [mathematical expression] = Model trained without fold [mathematical expression]
• [mathematical expression] = Validation fold [mathematical expression]

Information Criteria

Akaike Information Criterion (AIC):

Bayesian Information Criterion (BIC):

Symbol Definitions:

• [mathematical expression] = Number of parameters
• [mathematical expression] = Likelihood of the model
• [mathematical expression] = Sample size

Feature Engineering for Linear Models

Polynomial Features

Create non-linear relationships:

Interaction Terms

Capture feature interactions:

Standardization

Normalize features for regularization:

Symbol Definitions:

• [mathematical expression] = Feature mean
• [mathematical expression] = Feature standard deviation

Linear models provide interpretable, efficient solutions for many supervised learning problems, offering clear insights into feature importance and model behavior while maintaining computational efficiency and regulatory compliance in financial services and retail applications.