In its analysis of over 1.5 million responses collected in the Consumer Assessments of Healthcare Providers and Systems (CAHPS) survey of Medicare beneficiaries, the HCP research team has faced numerous challenges requiring development of new statistical methodology. Among the methodological advances arising from this study are the following topics:

• Estimation of an unstructured cluster-level covariance matrix. Exploratory factor analysis is commonly used to study relationships among CAHPS survey items as well as other measures of quality. These measures commonly have bounded responses and extensive structured missing data (observations missing “by design” because respondents are ineligible to answer them). To support inferences about factor structure, we have developed Bayesian multilevel models that employ variance and covariance functions to approximate the level 1 sampling variance of multivariate cluster means. These methods have numerous potential applications in quality profiling and small-area estimation for health care as well as other areas such as education, where missing data at both the individual and cluster levels is a common problem.

• Methods for optimal scoring of ordinal survey items for discrimination among units. Using a version of the multivariate Fisher discriminant, we showed how to assign responses on 0?10 scales or other ordinal scales used in CAHPS to fewer groups for reporting purposes while preserving as much information as possible about quality differences among plans. Results of these analyses also provide some insight into how respondents use these ordinal scales to report quality.

• Small-area estimation to facilitate efficient design of a survey with nonresponse follow-up. Like many large surveys, the Medicare CAHPS surveys use a mailed questionnaire with telephone follow-up of mail nonrespondents. Because of the relatively high unit cost of telephone follow-up, it could be cost-effective to use statistical methods of small-area estimation to reduce the size of the telephone follow-up sample while maintaining survey accuracy. We have developed such estimation methods, and by applying the corresponding design methods to CAHPS have demonstrated the possibility of substantial savings.