ANCOV and Matching with Confounded Variablesã
Suppose we are interested in the effect of some categorical independent variable upon some continuous dependent variable. We have available data on an extraneous variable that we can use for matching subjects or as a covariate in an ANCOV. If we were manipulating the independent variable, we could match subjects on the covariate and then within each block randomly assign one subject to each treatment group. If our covariate is well correlated with the dependent variable but not correlated with the independent variable, the randomized blocks design or ANCOV removes from what would otherwise be error variance the variance due to the covariate, thus increasing power. If we measure the covariate prior to administering our experimental treatment and then randomly assign subjects to treatment groups (within each block for a randomized blocks design), then any apparent correlation between covariate and independent variable is due to sampling error, and statistically removing the effect of the covariate removes only error variance.
If, however, we cannot randomly assign subjects to levels of the independent variable or if our covariate is measured after administering the treatments, then removing the effect of the covariate may also result in removing the effect of the treatment. In other words, when the independent variable and the extraneous variable are correlated (confounded), you cannot remove from the dependent variable variance due to the extraneous variable without also removing variance due to the independent variable.
Consider this case: We have a nonmanipulated dichotomous "independent variable" and continuous data on a covariate and a "dependent" or criterion variable. We shall imagine that criterion variable is score on a reading aptitude test, the covariate is number of literature courses taken, and the grouping variable is gender. Download the data file Confound.sav from my SPSS Data Page at http://core.ecu.edu/psyc/wuenschk/SPSS/SPSS-Data.htm. Bring the data into SPSS and take a look at them. I recommend that you print a copy of the data and bring the printed copy to class when we discuss them. To print the data, click “File” on the screen that shows the data in data view and select “Print.” In the “Print” window click the “Properties” button and select “Landscape” orientation. Click OK, OK.
The first three columns of scores (after the leftmost column, which has case numbers) are gender (1 is female, 2 is male), number of courses, and aptitude. We match participants on number of courses (before looking at their aptitude scores), obtaining 10 pairs of participants perfectly matched on the covariate. The 4th column of scores indicates matched pair number. Participants with a missing value code (a dot) in this column could not be matched, so they are excluded from the matched pairs analysis. Note that this excludes from the analysis the female participants with very high covariate scores (and, given a positive correlation with the criterion variable, with high aptitude as well) and the male participants with very low covariate (and criterion) scores. The last three columns of data are scores on the criterion variable for matched participants (female, male) followed by the difference score.
Now click Analyze, Correlate, Bivariate. Scoot gender, courses, and aptitude into the “Variables” box and click OK. Look at the output. Number of courses is indeed well correlated with aptitude, and the women scored higher than the men on both courses and aptitude (the negative sign of the point biserial correlation coefficients indicating that the gender 2 scores are lower than the gender 1 scores).
Now click Analyze, Compare Means, Independent Samples T Test. Scoot courses and aptitude into the “Test Variables” box and gender into the “Grouping Variable” box. Click “Define Groups” and enter the number 1 for “Group 1” and 2 for “Group 2” and then click Continue. Click OK and look at the output. The output shows us again that women score higher than men on both courses and aptitude, and gives us the means etc. Note that the analyses so far are based on all 34 cases.
Now click Analyze, Compare Means, One Sample T Test. Scoot apt1, apt2, and diff into the “Test Variable” box, leave the Test value at zero, and click OK. This is equivalent to conducting correlated t tests comparing men and women for our matched pairs. The output shows us that with the matched pairs data, men have reading aptitude (M = 42.5) significantly greater than that of women (M = 37.5). Now, can we make sense out of this? Ignoring the covariate, women had a significantly higher mean than did men, but if we “control” the covariate by matching (excluding high scores from one group and low scores from the other group), we not only remove Group 1’s superiority, but we get Group 2 having the significantly higher mean. In other words, if the two groups did not differ on the covariate, Group 2 would have the higher mean -- but the two groups do differ on the covariate, so asking if the groups would differ on reading aptitude if they did not differ on number of literature courses is somewhat absurd.
Now, let us do a quick ANCOV (analysis of covariance) using all 34 participants. Click Analyze, General Linear Model, Univariate. Scoot aptitude into the “Dependent Variables” box, gender into the “Fixed Factors” box, and courses into the “Covariates” box. Do not click OK yet. “Fixed Factors” identifies the categorical predictor variable(s), with “Fixed” meaning that we have sampled all of the values of interest for the factor(s). If we had randomly sampled values from the factor of interest, we would use the “Random Factors” box. “Covariates” identifies continuously distributed predictor variables.
Click the Model button and select the Custom model. Highlight “gender(F)” in the “Factors & Covariates” list and then click the “Build Term(s)” arrow to place “gender(F)” as the first variable in the “Model” list. Place “courses(C)” as the second variable in the model. Now, be sure that “Interaction” is showing in the box just below the Build Term(s) arrow, and click on both “gender(F)” and “courses(C)” in the Factors and Covariates box. That should result in both “gender(F)” and “courses(C)” being highlighted. With those two terms highlighted, click the Build Term(s) arrow. This will result in the third term in the model being “courses* gender,” which is an interaction term. SPSS creates it by computing for each subject the product of the numerical code for gender and the score on the courses variable. If our factor had more that two levels, the factor would be coded as a set of k-1 dummy variables (each coded 0,1), and the interaction component would consist of a set of k-1 products between the covariate and the dummy variables, where k is the number of levels of the factor.
In the “Sum of squares” box, change the type to “Type I.” Verify that the “Univariate: Model” window looks like that below and then click Continue, OK.
Look at the output. Our only interest is in the test of the interaction component, Gender x Courses. The F reported for the interaction component tests the null hypothesis that the slope for predicting aptitude from courses is the same in women as in men. This must be so if we are to do a standard ANCOV, since the ANCOV “adjusts” the criterion scores in both groups (statistically to remove the effect of the covariate on the criterion) using a slope pooled across both groups. The F is clearly nonsignificant, so we go on to do the ANCOV with the interaction term dropped from the model.
Click Analyze, General Linear Model, Univariate, Model. Remove the “courses*gender” interaction term from the Covariates box -- highlight it and click the “Build Terms” arrow. Click Continue, Options. Scoot gender into the “Display Means For” box and check the “Display Descriptive Statistics” box. Verify that the “Univariate: Options” window looks like that below, and then click Continue and then OK.
Look at the output for our ANCOV. Notice that we are still using Type I sums of squares. Type I sums of squares are sequential -- that is, the effect of the first term in the model is evaluated ignoring all of the other terms in the model. Next, the effect of the second term in the model is evaluated after removing from it any overlap between it and the first term in the model -- that is, statistically holding constant the effect of the first term. In statistical jargon, that is evaluating the effect of the second term “adjusted for” or “partialled for” the second term. If there were more than two terms, this would continue, with each effect adjusted for all effects that precede it in the model but ignoring all effects that follow it in the model. If we had selected Type III (unique) sums of square, each effect in the model would be adjusted for every other effect in the model.
With our model and Type I sums of squares, the effect of the covariate was first removed from the aptitude scores. This results in the adjusted aptitude scores of participants who took many literature courses being lowered and the adjusted aptitude scores of participants who took few literature courses being raised. Look back at the scores in columns one through three. Since the women had high covariate scores and the men had low covariate scores, this results in the adjusted mean on the criterion variable being lowered in the women and raised in the men.
The F reported for gender in this analysis tests the null hypothesis that the two adjusted means (given under “Estimated Marginal Means”) are equal in the population. After taking out the "effect" of number of literature courses, men have a mean reading aptitude that is significantly higher than that of women. Once again, statistically controlling the covariate with these confounded data has resulted not only in removing Group 1’s superiority but in producing a significant difference in the opposite direction.
Please beware the use of matching or ANCOV in circumstances like this. I have contrived these data to make a point, exaggerating the degree of confounding likely with real data, but we shall see this problem with real data too. For our contrived data, women have significantly higher reading aptitude unless we statistically remove the “effect” of taking more literature courses. Does this mean that men really have higher reading aptitude that is just masked by their not taking many literature courses? I doubt it. People generally take more courses in areas where their aptitude is high rather than low, so statistically removing the gender difference in number of literature courses taken also removes (or reduces or even reverses) the (real, unadjusted) gender difference in aptitude.
Suppose that for these contrived data Group 1 was men, Group 2 women, the covariate a measure of amount eaten daily, and the criterion body weight. Men are significantly heavier than women, but if we statistically held constant the amount eaten, women have higher weights than do men. If women ate as much as men, they would weigh more than men. So what, not eating that much is part of being a woman, women eat significantly less than men do!
Despite numerous warnings from statisticians about the use of matching and ANCOV with confounded data, psychologists persist in doing it. You be a critical reader and be aware of the severe limitations of such research when you encounter it.
Lest I have overstated the case against ANCOV and matching with covariates confounded with the independent variable, let me state that I believe such analyses can be informative when interpreted with caution and understanding. Multiple regression (which is really what we are doing here) generally involves obtaining partialled (adjusted) statistics (reflecting the contribution of each predictor variable partialled for some or all of the other predictor variables). Such analyses are especially useful with nonexperimental data, where causal attribution is slippery at best. Consider the data collected by statistics student Dechanile Johnson, and used in PSYC 6430 (first semester graduate statistics). Download the data file Weights.sav from my SPSS Data Page at http://core.ecu.edu/psyc/wuenschk/SPSS/SPSS-Data.htm. Bring the data into SPSS and take a look at them. The variables are gender, height, and weight. Use SPSS to conduct the following analyses:
Correlate each variable with each other variable.
Use t tests to compare men with women on both height and weight.
Verify that the interaction between gender and height is not significant with respect to their association with weight.
Conduct an ANCOV to compare the genders on weight, using height as the covariate. Use Type I sums of squares with the covariate entered first in the model. Obtain descriptive statistics and adjusted means (mean weight for men and for women after taking out the gender difference in heights).
Look at your output. Notice that height is well correlated with weight, and that men are significantly taller and heavier than women (point biserial correlations). The T-Test output shows us the means by gender along with associated statistics. Our General Linear Model output shows us that the slope for predicting weight from height does not differ significantly between men and women, and that men still weigh significantly more than women after adjusting for height. The men averaged 163.76 - 123.36 = 40.4 lb. heavier than the women and 70.57 - 64.89 = 5.68 inches taller. These are quite large differences, 2.5 standard deviations in the case of weight, 2.3 in the case of height. The adjusted means differ by less, by only 35.2 lb (160.8 - 125.6). Removing the effect of height did not make the weight difference nonsignificant (if it did, would we conclude that men don’t really weigh more than women?), but it did reduce the difference from 40.4 to 35.2. In other words, some part of the sex difference in weight is due to men being taller, but even if we statistically hold height constant, men are significantly heavier. Why? Well, men have stockier builds and perhaps more dense tissue (more muscle, less fat, not to mention denser crania J).
Copyright 2003, Karl L. Wuensch - All rights reserved.
ã Copyright 2003, Karl L. Wuensch - All rights reserved.
No comments:
Post a Comment