1. Introduction 1
1.1 Regression and Model Building 1
1.2 Data Collection 5
1.3 Uses of Regression 9
1.4 Role of the Computer 10
2. Simple Linear Regression 12
2.1 Simple Linear Regression Model 12
2.2 Least-Squares Estimation of the Parameters 13
2.2.1 Estimation of ß0 and ß1 13
2.2.2 Properties of the Least-Squares Estimators and the Fitted Regression Model 18
2.2.3 Estimation of s2 20
2.2.4 Alternate Form of the Model 22
2.3 Hypothesis Testing on the Slope and Intercept 22
2.3.1 Use of t Tests 22
2.3.2 Testing Significance of Regression 24
2.3.3 Analysis of Variance 25
2.4 Interval Estimation in Simple Linear Regression 29
2.4.1 Confidence Intervals on ß0, ß1, and s2 29
2.4.2 Interval Estimation of the Mean Response 30
2.5 Prediction of New Observations 33
2.6 Coefficient of Determination 35
2.7 A Service Industry Application of Regression 37
2.8 Does Pitching Win Baseball Games? 39
2.9 Using SAS® and R for Simple Linear Regression 41
2.10 Some Considerations in the Use of Regression 44
2.11 Regression Through the Origin 46
2.12 Estimation by Maximum Likelihood 52
2.13 Case Where the Regressor x is Random 53
2.13.1 x and y Jointly Distributed 54
2.13.2 x and y Jointly Normally Distributed: Correlation Model 54
Problems 59
3. Multiple Linear Regression 69
3.1 Multiple Regression Models 69
3.2 Estimation of the Model Parameters 72
3.2.1 Least-Squares Estimation of the Regression Coefficients 72
3.2.2 Geometrical Interpretation of Least Squares 79
3.2.3 Properties of the Least-Squares Estimators 81
3.2.4 Estimation of s2 82
3.2.5 Inadequacy of Scatter Diagrams in Multiple Regression 84
3.2.6 Maximum-Likelihood Estimation 85
3.3 Hypothesis Testing in Multiple Linear Regression 86
3.3.1 Test for Significance of Regression 86
3.3.2 Tests on Individual Regression Coefficients and Subsets of Coefficients 90
3.3.3 Special Case of Orthogonal Columns in X 95
3.3.4 Testing the General Linear Hypothesis 97
3.4 Confidence Intervals in Multiple Regression 99
3.4.1 Confidence Intervals on the Regression Coefficients 100
3.4.2 CI Estimation of the Mean Response 101
3.4.3 Simultaneous Confidence Intervals on Regression Coefficients 102
3.5 Prediction of New Observations 106
3.6 A Multiple Regression Model for the Patient Satisfaction Data 106
3.7 Does Pitching and Defense Win Baseball Games? 108
3.8 Using SAS and R for Basic Multiple Linear Regression 110
3.9 Hidden Extrapolation in Multiple Regression 111
3.10 Standardized Regression Coefficients 115
3.11 Multicollinearity 121
3.12 Why Do Regression Coefficients Have the Wrong Sign? 123
Problems 125
4. Model Adequacy Checking 134
4.1 Introduction 134
4.2 Residual Analysis 135
4.2.1 Definition of Residuals 135
4.2.2 Methods for Scaling Residuals 135
4.2.3 Residual Plots 141
4.2.4 Partial Regression and Partial Residual Plots 148
4.2.5 Using Minitab®, SAS, and R for Residual Analysis 151
4.2.6 Other Residual Plotting and Analysis Methods 154
4.3 PRESS Statistic 156
4.4 Detection and Treatment of Outliers 157
4.5 Lack of Fit of the Regression Model 161
4.5.1 A Formal Test for Lack of Fit 161
4.5.2 Estimation of Pure Error from Near Neighbors 165
Problems 170
5. Transformations and Weighting To Correct Model Inadequacies 177
5.1 Introduction 177
5.2 Variance-Stabilizing Transformations 178
5.3 Transformations to Linearize the Model 182
5.4 Analytical Methods for Selecting a Transformation 188
5.4.1 Transformations on y: The Box-Cox Method 188
5.4.2 Transformations on the Regressor Variables 190
5.5 Generalized and Weighted Least Squares 194
5.5.1 Generalized Least Squares 194
5.5.2 Weighted Least Squares 196
5.5.3 Some Practical Issues 197
5.6 Regression Models with Random Effects 200
5.6.1 Subsampling 200
5.6.2 The General Situation for a Regression Model with a Single Random Effect 204
5.6.3 The Importance of the Mixed Model in Regression 208
Problems 208
6. Diagnostics For Leverage and Influence 217
6.1 Importance of Detecting Influential Observations 217
6.2 Leverage 218
6.3 Measures of Influence: Cooks D 221
6.4 Measures of Influence: DFFITS and DFBETAS 223
6.5 A Measure of Model Performance 225
6.6 Detecting Groups of Influential Observations 226
6.7 Treatment of Influential Observations 226
Problems 227
7. Polynomial Regression Models 230
7.1 Introduction 230
7.2 Polynomial Models in One Variable 230
7.2.1 Basic Principles 230
7.2.2 Piecewise Polynomial Fitting (Splines) 236
7.2.3 Polynomial and Trigonometric Terms 242
7.3 Nonparametric Regression 243
7.3.1 Kernel Regression 244
7.3.2 Locally Weighted Regression (Loess) 244
7.3.3 Final Cautions 249
7.4 Polynomial Models in Two or More Variables 249
7.5 Orthogonal Polynomials 255
Problems 261
8. Indicator Variables 268
8.1 General Concept of Indicator Variables 268
8.2 Comments on the Use of Indicator Variables 281
8.2.1 Indicator Variables versus Regression on Allocated Codes 281
8.2.2 Indicator Variables as a Substitute for a Quantitative Regressor 282
8.3 Regression Approach to Analysis of Variance 283
Problems 288
9. Multicollinearity 293
9.1 Introduction 293
9.2 Sources of Multicollinearity 294
9.3 Effects of Multicollinearity 296
9.4 Multicollinearity Diagnostics 300
9.4.1 Examination of the Correlation Matrix 300
9.4.2 Variance Inflation Factors 304
9.4.3 Eigensystem Analysis of X'X 305
9.4.4 Other Diagnostics 310
9.4.5 SAS and R Code for Generating Multicollinearity Diagnostics 311
9.5 Methods for Dealing with Multicollinearity 311
9.5.1 Collecting Additional Data 311
9.5.2 Model Respecification 312
9.5.3 Ridge Regression 312
9.5.4 Principal-Component Regression 329
9.5.5 Comparison and Evaluation of Biased Estimators 334
9.6 Using SAS to Perform Ridge and Principal-Component Regression 336
Problems 338
10. Variable Selection and Model Building 342
10.1 Introduction 342
10.1.1 Model-Building Problem 342
10.1.2 Consequences of Model Misspecification 344
10.1.3 Criteria for Evaluating Subset Regression Models 347
10.2 Computational Techniques for Variable Selection 353
10.2.1 All Possible Regressions 353
10.2.2 Stepwise Regression Methods 359
10.3 Strategy for Variable Selection and Model Building 367
10.4 Case Study: Gorman and Toman Asphalt Data Using SAS 370
Problems 383
11. Validation of Regression Models 388
11.1 Introduction 388
11.2 Validation Techniques 389
11.2.1 Analysis of Model Coefficients and Predicted Values 389
11.2.2 Collecting Fresh Data'Confirmation Runs 391
11.2.3 Data Splitting 393
11.3 Data from Planned Experiments 401
Problems 402
12. Introduction To Nonlinear Regression 405
12.1 Linear and Nonlinear Regression Models 405
12.1.1 Linear Regression Models 405
12.1.2 Nonlinear Regression Models 406
12.2 Origins of Nonlinear Models 407
12.3 Nonlinear Least Squares 411
12.4 Transformation to a Linear Model 413
12.5 Parameter Estimation in a Nonlinear System 416
12.5.1 Linearization 416
12.5.2 Other Parameter Estimation Methods 423
12.5.3 Starting Values 424
12.6 Statistical Inference in Nonlinear Regression 425
12.7 Examples of Nonlinear Regression Models 427
12.8 Using SAS and R 428
Problems 432
13. Generalized Linear Models 440
13.1 Introduction 440
13.2 Logistic Regression Models 441
13.2.1 Models with a Binary Response Variable 441
13.2.2 Estimating the Parameters in a Logistic Regression Model 442
13.2.3 Interpretation of the Parameters in a Logistic Regression Model 447
13.2.4 Statistical Inference on Model Parameters 449
13.2.5 Diagnostic Checking in Logistic Regression 459
13.2.6 Other Models for Binary Response Data 461
13.2.7 More Than Two Categorical Outcomes 461
13.3 Poisson Regression 463
13.4 The Generalized Linear Model 469
13.4.1 Link Functions and Linear Predictors 470
13.4.2 Parameter Estimation and Inference in the GLM 471
13.4.3 Prediction and Estimation with the GLM 473
13.4.4 Residual Analysis in the GLM 475
13.4.5 Using R to Perform GLM Analysis 477
13.4.6 Overdispersion 480
Problems 481
14. Regression Analysis of Time Series Data 495
14.1 Introduction to Regression Models for Time Series Data 495
14.2 Detecting Autocorrelation: The Durbin-Watson Test 496
14.3 Estimating the Parameters in Time Series Regression Models 501
Problems 517
15. Other Topics in the Use of Regression Analysis 521
15.1 Robust Regression 521
15.1.1 Need for Robust Regression 521
15.1.2 M-Estimators 524
15.1.3 Properties of Robust Estimators 531
15.2 Effect of Measurement Errors in the Regressors 532
15.2.1 Simple Linear Regression 532
15.2.2 The Berkson Model 534
15.3 Inverse Estimation'The Calibration Problem 534
15.4 Bootstrapping in Regression 538
15.4.1 Bootstrap Sampling in Regression 539
15.4.2 Bootstrap Confidence Intervals 540
15.5 Classification and Regression Trees (CART) 545
15.6 Neural Networks 547
15.7 Designed Experiments for Regression 549
Problems 557
Appendix A. Statistical Tables 561
Appendix B. Data Sets For Exercises 573
Appendix C. Supplemental Technical Material 602
C.1 Background on Basic Test Statistics 602
C.2 Background from the Theory of Linear Models 605
C.3 Important Results on SSR and SSRes 609
C.4 Gauss-Markov Theorem, Var(e) = s2I 615
C.5 Computational Aspects of Multiple Regression 617
C.6 Result on the Inverse of a Matrix 618
C.7 Development of the PRESS Statistic 619
C.8 Development of S2(i) 621
C.9 Outlier Test Based on R-Student 622
C.10 Independence of Residuals and Fitted Values 624
C.11 Gauss-Markov Theorem, Var(e) = V 625
C.12 Bias in MSRes When the Model is Underspecified 627
C.13 Computation of Influence Diagnostics 628
C.14 Generalized Linear Models 629
Appendix D. Introduction To SAS 641
D.1 Basic Data Entry 642
D.2 Creating Permanent SAS Data Sets 646
D.3 Importing Data from an EXCEL File 647
D.4 Output Command 648
D.5 Log File 648
D.6 Adding Variables to an Existing SAS Data Set 650
Appendix E. Introduction To R To Perform Linear Regression Analysis 651
E.1 Basic Background on R 651
E.2 Basic Data Entry 652
E.3 Brief Comments on Other Functionality in R 654
E.4 R Commander 655
References 656
Index 670