b.Reference Periods
Estimates of the incidence or frequency of uninsurance are reported typically in one of three ways: (1) the number who were uninsured at a specific point in time, (2) the number who were ever uninsured during a year, or (3) the number who were uninsured for the entire year. Point-in-time estimates are sometimes reported not for a specific point in time, such as January 1, 1999, but for any time during a year. When described in this way, estimates should be interpreted as the average number uninsured at a point in time and not the number who were ever uninsured during the year.
Estimates of the number or percentage of children who were uninsured over different periods of time are useful for different purposes. Estimates of the number of children who were ever uninsured over a year indicate how prevalent uninsurance is. Estimates of children uninsured for an entire year demonstrate the magnitude of chronic uninsurance. Estimates of children uninsured at a point in time reflect a combination of prevalence and duration in that the more time children spend in the state of uninsurance, the more closely the number uninsured at a point in time will approach the number who were ever uninsured.
Table 2 presents estimates for all three types of reference periods, based on data from the 1992 SIPP panel. While 13.1 percent of children under 19 were uninsured in September 1993, 21.7 percent of children under 19 were ever uninsured during the year while 6.3 percent were uninsured for the entire year.
Measuring uninsurance as a residual has implications for the length of time over which children are identified as uninsured. When a survey identifies the uninsured as a residual, the duration of uninsurance that is measured is generally synonymous with the reference period. That is, children for whom no insurance coverage is reported during the reference period are, by definition, uninsured for the entire period. To identify periods of uninsurance occurring within a reference period in which there were also periods of insurance coverage, it is necessary to do one of the following: (1) ask about such periods of uninsurance directly, (2) ask whether the insurance coverage extended to the entire period, or (3) break the total reference period into multiple, shorter periods, such as months and establish whether a person was insured or uninsured in each month.7
In the March CPS, respondents are asked if they were ever covered by any of several types of insurance during the previous calendar year. Respondents can indicate that they had multiple types of coverage during the year. But because the survey instrument does not ask if respondents were ever uninsured, or how long they were covered, respondents cannot report that they were covered for part of the year and uninsured for the rest.
TABLE 2. ESTIMATES OF THE PROPORTION OF CHILDREN UNDER 19 WHO WERE UNINSURED FOR DIFFERENT PERIODS OF TIME
Period
Estimate
Uninsured at a Point in Time (September 1993)
13.1%
Ever Uninsured in Year
21.7%
Uninsured Continuously throughout the Year
6.3%
In the SIPP, respondents are asked to report whether they had any of several types of insurance coverage during each of the four preceding months. The month is the reference period. To be identified as uninsured during a given month, a child must be reported as having had no coverage during the month. Thus, a child is classified as uninsured during a month only if the child was uninsured for the entire month.(8) With the SIPP data, however, we can aggregate individual months into years or even longer periods, and we can identify children who were ever uninsured during the year, where being ever uninsured means being uninsured for at least one full calendar month.
The redesigned NHIS, the CTS, and the NSAF all capture insurance status at the time of the interview--that is, literally at a point in time. Other things being equal, this approach would appear likely to yield the most error-free reports and, in addition, the least biased estimates of coverage. It also has the advantage of requiring no recall. Respondents are not asked to remember when coverage began or ended, only to indicate whether they currently have it or not.
The value of estimates for different types of reference periods depends, in part, on the accuracy with which they can be measured. If the number of children uninsured at a point in time can be measured more accurately than the number ever uninsured during a year or the number uninsured for the entire year, then there is a sense in which the point-in-time estimates are more valuable. In the next section we discuss measurement problems that affect estimates of the uninsured.
3.Sources of Error in Estimates of the Uninsured
There are a number of sources of error encountered in attempting to measure uninsurance, and these affect the comparability of estimates from different surveys. These include certain limitations inherent in measuring uninsurance as a residual, as it is usually done; the possibility that respondents may not be aware of existing coverage; the bias introduced by respondents’ imperfect recall; the sensitivity of responses to question design; and the impact of basic survey design choices.
a.Limitations Inherent in Measuring Uninsurance as a Residual
Perhaps the most significant problem with measuring uninsurance as a residual is that a small error rate in the reporting of insurance becomes a large error in the estimate of the uninsured. With the number of children insured at a point in time being eight to nine times the number without insurance, and the number ever insured during a year being 18 to 19 times the number never insured, errors in the reporting of insurance coverage are multiplied many times in their impact on estimates of the uninsured. Based on the SIPP estimates reported in Table 2, a 6 to 7 percent error in the reporting of children who ever had health insurance would double the estimated number who had no insurance. In Section 4, below, we argue that this is what accounts for the fact that the CPS estimate of the uninsured resembles an estimate of children uninsured at a point in time rather than children uninsured for the entire year, which is what the questions are designed to yield.(9)
Another implication of measuring uninsurance as a residual can be seen in the CPS estimates of the frequency of uninsurance among infants. The health insurance questions in the March CPS refer to coverage in the preceding calendar year--that is, the year ending December 31. If parents answer the CPS questions as intended, a child born after the end of the year cannot be identified as having had coverage during the previous year. With no reported coverage, such a child would be classified as uninsured. If all children born after the end of the year were classified as uninsured, this would add about one-sixth of all infants to the estimated number uninsured. Because the March CPS public use files lack a field indicating the month of birth, data users cannot identify infants born after the end of the year and cannot exclude them from their analyses. Is there any evidence that uninsurance is overstated among infants in the CPS? Table 3 addresses this question by comparing estimates of the rate of uninsurance for infants and older children, based on the March CPS and the SIPP. The CPS estimates of the proportion of infants who are uninsured are markedly higher than the SIPP estimates in both the 1993 and 1994 reference years: 11.5 versus 7.7 percent in 1993 and 17.3 versus 9.3 percent in 1994.
b.Awareness of Coverage
People may have insurance coverage without being aware that they have it. While this lack of awareness may seem improbable, both the CPS and SIPP provide direct evidence with respect to Medicaid coverage. Prior to welfare reform, families that received Aid to Families with Dependent Children (AFDC) were covered by Medicaid as well. Nevertheless, surveys that asked respondents about AFDC as well as Medicaid found that nontrivial numbers reported receiving AFDC but not being covered by Medicaid. Were such people unaware that they were covered by Medicaid, or did they know Medicaid by another name and not recognize the name(s) used in the surveys?(10)
We do not know the answer. To correct for such instances, the Census Bureau employs in both the CPS and SIPP a number of “logical imputations” or edits to reported health insurance coverage. All adult AFDC recipients and their children are assigned Medicaid coverage, for example. Of the 28.2 million people estimated to have had Medicaid coverage in 1996, based on the March 1997 CPS, 4.6 million or 16 percent had their Medicaid coverage logically imputed in this manner (Rosenbach and Lewis 1998). Most if not all of these 4.6 million would have been counted as uninsured if not for the Census Bureau’s edits. With AFDC, which accounted for half of Medicaid enrollment, being replaced by the smaller Temporary Assistance to Needy Families (TANF) program, the number of logical imputations will be reduced significantly, which could increase the number of children who in fact have Medicaid coverage but are counted in the CPS and SIPP as uninsured.(11)
Table 3. ESTIMATES OF THE PROPORTION OF CHILDREN UNINSURED BY AGE: COMPARISON OF MARCH CPS AND SIPP, SELECTED YEARS
Survey and Date
less than 1
1 to 5
6 to 14
15 to 18
Total
CPS, March 1994
11.5
11.6
13.7
19.4
14.1
CPS, March 1995
17.3
13.2
14.0
16.5
14.4
CPS, March 1996
16.7
12.7
13.7
16.1
14.0
SIPP, September 1993
7.7
10.9
13.7
16.7
13.1
SIPP, September 1994
9.3
10.5
13.1
16.3
12.7
 
SOURCE: Tabulations of public use files, CPS and SIPP.
c.Recall Bias
It is well known among experienced survey researchers that respondent recall of events in the past is imperfect and that recall error grows with the length of time between the event and the present. Error also increases with the amount of change in people’s lives. Respondents with steady employment have less difficulty recalling details of their employment than do respondents with intermittent jobs and uneven hours of work. Similarly, respondents who have had continuous health insurance coverage can more easily recall their coverage history than respondents with intermittent coverage. Obtaining accurate reports from respondents with complex histories places demands upon the designers of surveys and those who conduct the interviews. Panel surveys that ascertain health insurance coverage (and other information) with repeated interviews covering short reference periods are much more likely to obtain reliable estimates of coverage over time than one-time surveys that ask respondents to recall the details of the past year or more.
d.Sensitivity to Question Design
Even when recall is not an issue, when insurance coverage is measured “at the present time,” survey questions that appear to request more or less the same information can generate markedly different responses. This point was demonstrated in dramatic fashion when the Census Bureau introduced some experimental questions into the CPS to measure current health insurance coverage. At the end of the sequence of questions used to measure insurance coverage during the preceding year, respondents were asked:
These next questions are about your CURRENT health insurance coverage, that is, health coverage last week. (Were you/Was anyone in this household) covered by ANY type of health insurance plan last week?
Those who answered in the affirmative were asked to identify who in the household was covered and then, for each such person, by what types of plans he or she was covered. This sequence of questions, which first appeared in the March 1994 survey, yielded an uninsured rate that was about double the rate measured by the NHIS and the SIPP, and the experimental questions were discontinued with the March 1998 supplement.
Even if these questions had not followed a lengthy sequence of items asking about several sources of coverage in the preceding year, it would have been difficult to imagine that they could have generated such low estimates of coverage. That they did so despite the questions that preceded them is hard to fathom, and it underscores the point that researchers cannot simply write out a set of health insurance coverage questions and expect to obtain the true measure of uninsurance--or even a good measure of uninsurance, necessarily. It is not at all clear why this should be so. Health insurance coverage appears to be straightforward enough. Generally, people either have it or they don’t. Yet the Census Bureau’s experience sends a powerful message that questions about health insurance coverage can yield rather unanticipated results. Researchers who are fielding surveys that attempt to measure health insurance coverage would be well-advised to be wary of constructing new questions unless they can also conduct very extensive pretesting. In the absence of thorough testing, it is better to borrow from existing and thoroughly tested question sets rather than construct new questions from scratch.
e.Impact of Survey Design and Implementation
While perhaps not as important as question wording, differences in the design and implementation of surveys can have a major impact on estimates of the uninsured. These differences include the choice of universe and the level of coverage achieved, the response rate among eligible households, the use of proxy respondents, the choice of interview mode, and the use of imputation.
Universe and Coverage. Surveys may differ in the universes that they sample and in how fully they cover these universes. Typically, surveys of the U.S. resident population exclude the homeless, the institutionalized population--that is, residents of nursing homes, mental hospitals, and correctional institutions, primarily--and members of the Armed Forces living in barracks. There may be other exclusions as well. For example, household surveys do not always include Alaska and Hawaii in their sampling frames.
All surveys--even the decennial census--suffer from undercoverage; that is, parts of the universe are unintentionally excluded from representation in the sample. In a household-based or “area frame” sample, undercoverage can be attributed to three principal causes: (1) failure to identify all street addresses in the sample area, (2) failure to identify all housing units within the listed addresses, and (3) failure to identify all household members within the sampled housing units. Nonresponse, discussed below, is not undercoverage, although the absence of household listings for nonresponding households can contribute to coverage errors (in either direction). The 1990 census undercounted U.S. residents by about 1.6 percent.(12) Sample surveys have much greater undercoverage. The Census Bureau has estimated the undercoverage of the civilian noninstitutionalized population in the monthly CPS to be about 8 percent in recent years. Undercoverage varies by demographic group. For children under 15, undercoverage is closer to 7 percent than to 8 percent. But among older teens it approaches 13 percent, and for black males within this group the rate of undercoverage reaches 25 to 30 percent.
To provide at least a nominal correction for undercoverage, the Census Bureau and other agencies or organizations adjust the sample weights so that they reproduce selected population totals. These population totals or “controls” may even incorporate adjustments for the census undercount.(13) This “post-stratification,” a statistical operation that serves other purposes as well, is based on a limited set of demographic characteristics--age, sex, race and Hispanic origin, typically, and sometimes state.(14) Other characteristics measured in the surveys are affected by this post-stratification to the extent that they covary with demographic characteristics. We know, for example, that Medicaid enrollment and uninsurance vary quite substantially by age, race, and Hispanic origin, so a coverage adjustment based on these demographic characteristics will improve the estimates of Medicaid enrollment and uninsurance. To the extent that people who are missing from the sampling frame differ from the covered population even within these demographic groups, however, the coverage adjustment will compensate only partially for the effects of undercoverage on the final estimates. It is quite plausible, for example, that the Hispanic children who are missed by the CPS have an even higher rate of uninsurance than those who are interviewed. We would suggest, therefore, that survey undercoverage, even with a demographic adjustment to population totals corrected for census undercount, contributes to underestimation of uninsured children.
Response Rate. Surveys differ in the fraction of their samples that they succeed in interviewing. Federal government survey agencies appear to enjoy a premium in this regard. The Census Bureau, which conducts both the CPS and the SIPP and carries out the field operations for the NHIS, reports the highest response rates among the surveys that provide our principal measures of health insurance coverage. For the 1997 March supplement to the CPS, the Census Bureau reported a response rate of 84 percent.(15) For the first interview of the 1992 SIPP panel the Bureau achieved a response rate of 91 percent, with the cumulative response rate falling to 74 percent by the ninth interview. The 1995 NHIS response rate for households that were eligible for selection into the MEPS was 94 percent (Cohen 1997). In contrast to these , MPR obtained a 65 percent response rate for the CTS, and Westat achieved a comparable percentage for the NSAF, which includes a substantial oversampling of lower income households. For the first round of the MEPS, Westat secured an 83 percent response rate among the 94 percent of eligible households that responded to the NHIS in the second and third quarters of 1995, yielding a joint response rate of 78 percent (Cohen 1997). These response rates are based on people with whom interviews were completed, but there may have been additional nonresponse to individual items in the health insurance sequence. However, unlike more sensitive items, like those pertaining to income, health insurance questions do not appear to generate much item nonresponse.
The reported response rates also do not include undercoverage, which varies somewhat from survey to survey. Arguably, people who were omitted from the sampling frame never had an opportunity to respond and, therefore, may have less in common with those who refused to be interviewed than they do with respondents. Nevertheless, their absence from the collected data represents a potential source of bias and one for which some adjustment is desirable. Generally speaking, however, less is known about the characteristics of people omitted from the sampling frame than about those who were included in the sampling frame but could not be interviewed. Hence the adjustments for undercoverage, when they are carried out, tend to be based on more limited characteristics than the adjustments for nonresponse among sampled households.
How important is nonresponse as a source of bias in estimates of health insurance coverage? We are not aware of any information with which it is possible to address that question. Certainly the nearly 30 percent difference in response rates between the NHIS and the CTS or NSAF could have a marked impact on the estimated frequency of a characteristic (uninsurance) that occurs among less than 15 percent of all children, but we have no direct evidence that it does.
Proxy Respondents. Some members of a household may not be present when the household is interviewed. Surveys differ in whether and how readily they allow other household members to serve as “proxy” respondents. From the standpoint of data quality, the drawback of a proxy respondent is the increased likelihood that information will be misreported or that some information will not be reported at all. This is particularly true when the respondent and proxy are not members of the same family. For this reason some surveys restrict proxy respondents to family members. Ultimately, however, some responses are generally better than none, so it is rare that a survey will rule out particular types of proxy responses entirely. Rather, proxy responses may be limited to “last resort” situations--that is, as alternatives to closing out cases as unit nonrespondents. For this reason, it is important to compare not only how surveys differ with respect to their stated policies on proxy respondents but the actual frequency with which proxy respondents are used and the frequency with which household members are reported as missing.
Children represent a special case. While all the surveys we have discussed collect data on children, the surveys differ with respect to whether these children are treated as respondents per se or merely other members of the family or household, about whom information is collected only or largely indirectly. For example, both the CPS and SIPP define respondents as all household members 15 and older. Some information, such as income, is not collected for younger children at all while health insurance coverage is collected through questions that ask respondents who else in the household is included under specific plans. With this indirect approach, children are more susceptible to being missed.
Mode: Telephone Versus In-person. Surveys may be conducted largely or entirely by telephone or largely or entirely in-person.(16) There are two aspects of the survey mode that are important to recognize. The first bears on population coverage while the second pertains to how the data are collected.
Pure telephone surveys, which are limited to households with telephones, cover a biased subset of the universe that is covered by in-person surveys. Methodologies have been developed to adjust such surveys for their noncoverage of households that were without telephone service during the survey period. These methodologies use the responses from households that report having had their telephone service interrupted during some previous number of months to compensate for the exclusion of households that had no opportunity to appear in the sample. How effectively such adjustments substitute for actually including households without telephones is likely to vary across the characteristics being measured, and for this reason some telephone surveys include a complementary in-person sample to obtain responses from households without telephones.(17)
In addition to the coverage issue, distinguishing telephone from in-person interviews is important because the use of one mode versus the other can affect the way in which information is collected and the reliability with which responses are reported. Telephone surveys preclude showing a respondent any printed material during the interview (such as lists of health insurance providers), and they limit the rapport that can develop between an interviewer and a respondent. Furthermore, the longer the interview, the more difficult it is to maintain the respondent’s attention on the telephone, so data quality in long interviews may suffer. On the other hand, conducting interviews by telephone may limit interviewer bias and make respondents feel less uncomfortable about reporting personal information. Moreover, until recently, telephone interviewing allowed for the use of computer-based survey instruments that could minimize the risk of interviewer error in administering instruments with complex branching and skip patterns. For all of these reasons, survey researchers recognize that there can be “mode effects” on responses. The different modes may elicit different mean responses to the same questions, with neither mode being consistently more reliable than the other. To minimize differential mode effects when part of a telephone survey is conducted in person, survey organizations sometimes conduct the in-person interviews by cellular telephone, which field representatives loan to the respondents.
Panel surveys allow for another possibility: using a household-based sample design and conducting at least the initial interview in-person but using the telephone for subsequent interviews. Both the CPS and the SIPP have utilized this approach. In the CPS, the first and last of the eight interviews are conducted in person while the middle six are generally conducted by telephone. For any given month, then, about one-quarter of the interviews are conducted in person.(18)
The recent introduction of computer-assisted personal interviewing (CAPI) has created an important variation on the in-person mode and one with its own mode effects. In some respects, CAPI may be more like computer-assisted telephone interviewing than in-person interviewing with a paper and pencil instrument. The methodology is too new to have generated much information on its mode effects yet.
Imputation Methodology. Surveys differ in the extent to which they impute values to questions with missing responses and in the rigorousness of their imputation methodologies. For example, both the CPS and SIPP impute all missing responses, and they use methodologies that have been developed to do this very efficiently. For the SIPP imputation algorithms, over time the Census Bureau has made increasing use of the responses reported in adjacent waves of the survey. Generally, questions about health insurance coverage elicit very little nonresponse, so imputation strategies are less important than they are for more sensitive items, such as income. Nevertheless, in the March 1997 CPS, the Census Bureau imputed 10 percent of the “reported” Medicaid participants (Rosenbach and Lewis 1999).(19) In the NHIS, responses of “don’t know” are not replaced by imputed values, and in published tabulations the insurance coverage of people whose coverage cannot be determined is treated as unknown. While this may not have a large impact on the estimated rates of uninsurance among children or adults, this strategy does make it more difficult for data users to replicate published results.
4.Interpreting Uninsurance as Measured in the CPS
The estimate of uninsured children provided annually by the March supplement to the CPS has become the most widely accepted and frequently cited estimate of the uninsured. At this point, only the CPS provides annual estimates with relatively little lag, and only the CPS is able to provide state- level estimates, albeit with considerable imprecision.(20) But what, exactly, does the CPS measure? The renewed interest in the CPS as a source of state-level estimates for CHIP makes it important that we answer this question.(21) While the CPS health insurance questions ask about coverage over the course of the previous calendar year, implying that the estimate of uninsurance identifies people who had no insurance at all during that year, the magnitude of the estimate has moved researchers and policymakers to reinterpret the CPS measure of the uninsured as providing an indicator of uninsurance at a point in time.(22) How can this interpretation be reconciled with the wording of the questions themselves, and how far can we carry this interpretation in examining the time trend and other covariates of uninsurance? We consider these questions below.
a.In What Sense Does the CPS Measure Uninsurance at a Point in Time?
There is little reason to doubt that the CPS respondents are answering the health insurance questions in the manner that was intended. That is, they are attempting to report whether they ever had each type of coverage in the preceding year. We can say this, in part, because the health insurance questions appear near the end of the survey, after respondents have reported their employment status, sources and amounts of income, and other characteristics for the preceding year. By the time they get to the health insurance questions, respondents have become thoroughly familiar with the concept of “ever in the preceding year.” More importantly, however, there is empirical evidence that reported coverage is more consistent with annual coverage than with coverage at a point in time. Consider Medicaid, for example. Table 4 compares CPS and SIPP estimates of children under 19 who were reported to be covered by Medicaid in 1993 and 1994. The CPS estimates match very closely the SIPP estimates of children ever covered in a year whereas the CPS estimates exceed the SIPP estimates of children covered at a point in time by 26 to 28 percent.(23)
TABLE 4. COMPARISON OF CPS AND SIPP ESTIMATES OF CHILDREN UNDER 19 ENROLLED IN MEDICAID
 
 
 
 
CPS as Percent of SIPP
 
CPS
SIPP Annual Ever
SIPP Point in Time
Annual Ever
Point in Time
1993
17,168,000
17,800,000
13,369,000
96.4%
128.4%
1994
16,727,000
17,795,000
13,259,000
94.0%
126.2%
 
NOTES: The SIPP annual estimates refer to the federal fiscal year. The point-in-time estimates refer to September of each year. The CPS estimates refer to the calendar year. Both sets of estimates were obtained by tabulating public use data files. The CPS estimates are from the March 1994 and March 1995 surveys. The SIPP estimates are from the 1992 panel.
The SIPP estimates here actually understate what SIPP finds, as these estimates refer to the survivors of the population sampled in early 1992. SIPP also understates births. SIPP point-in-time estimates made with the calendar month weights would be higher, as the calendar month weights are controlled to the full population. Annual-ever estimates cannot be produced for the calendar month samples, however.
How, then, can the frequency with which the CPS respondents report no coverage during the year imply rates of uninsurance that are double what we would expect for children uninsured all year and about equal to what we would expect for children uninsured at a point in time? The answer, we suggest, lies in the extent to which coverage for the year is underreported. That is, CPS respondents answering questions in March either forget or otherwise fail to report health insurance coverage that they had in the previous year, and they do so with greater frequency than respondents to other surveys reporting more current coverage. Presumably, coverage that ended early in the year is more likely to be missed than coverage that ended later in the year or continued to the present. Coverage that started late in the year may be susceptible to underreporting as well, with respondents who are uncertain about the starting date having some tendency to project it into the current year. With more than 90 percent of the population having had coverage for at least some part of the year, only a small fraction--about 8 percent of those with any coverage--need to fail to report their coverage to account for the observed result.
Is it simply by chance that CPS respondents underreport their coverage in the previous year to such an extent that the number who appear to have had no coverage at all rises to the same level as independent estimates of the number who were without coverage at a point in time? Or is the phenomenon the result of a more systematic process that in some sense ensures the observed outcome? The answer is important because the more the phenomenon is due to chance, the less confident we can be that the CPS estimate of the uninsured will track the true number of uninsured children (or adults) over time. Similarly, the more the resemblance to a point-in-time estimate is due to chance, the less we can rely on the CPS estimate of the uninsured to tell us how uninsurance at a point in time varies by children’s characteristics--including state of residence.
b.Covariates of Uninsurance
Time is a critical covariate of uninsurance. The CPS measure of the uninsured is used by many policy analysts to assess the trend in uninsurance for the population as a whole and for subgroups of that population. But, in truth, how well does the CPS measure track the actual level of uninsurance? There is no definitive source on the uninsured, but both the NHIS and the SIPP provide annual estimates that can be compared with the CPS. Do these estimates show the same trends over time, even though the estimates themselves may differ? The estimates presented in Table 1 are inconclusive in this regard. The CPS time series is clearly less volatile than the NHIS time series, with the latter showing large swings between 1993 and 1994 and between 1994 and 1995. Between 1995 and 1996, the CPS uninsured rate shows an upswing that observers have interpreted as a response to the implementation of welfare reform. The NHIS estimate for 1996 predates this event, as we explained earlier. With a redesign of the survey and a revision of the health insurance questions in 1997, the continuation of the NHIS time series once the 1997 data are released will shed little if any light on the validity of the CPS series. The SIPP data are too limited to provide a useful point of comparison.(24)
Even if the CPS estimate of the uninsured were a sufficiently reliable proxy for point-in-time uninsured to provide an accurate indicator of trends, this gives us no assurance that the CPS measure can accurately reflect the relationship between point-in-time uninsurance and other covariates besides time. We have already presented evidence that for reasons related, no doubt, to the measurement of uninsurance as a residual, combined with the peculiar reference period of the survey, the CPS overstates the proportion of infants who are uninsured (see Table 3). How confident can we be that the CPS can provide adequate estimates of the relationship between children’s uninsurance and very complex variables, such as Medicaid eligibility? This is an important question but one that will require more research to answer.
As a final note, the success of verification questions in the CTS and NSAF is prompting consideration of including such questions in the SIPP and the CPS. In light of our discussion of the CPS measure, we must wonder what the impact would be of introducing a verification question into the CPS. Rather than improving the point-in-time representation of the CPS, might this not move the CPS much closer to estimating the number of people who truly were uninsured throughout the preceding year? Arguably, this would reduce the policy value of the CPS measure because uninsurance throughout the year is too limited a phenomenon to be embraced as our principal measure of uninsurance. Of course, policy analysts could choose not to use the verification question, but this would only make it that much more difficult to assert that the data being reported in the CPS provide a reliable measure of uninsurance at a point in time.
5.Conclusion
The estimates of the incidence of uninsurance among children presented in Table 1 beg the question: Which estimate is the most correct? The short answer is that we do not know. There is no agreement in the research community. Clearly, the CPS estimate has been the most widely cited, but, as we explained, its timeliness and consistency account for this more than the presumption that it is the most accurate. When the estimate from the CTS was first announced, it was greeted with skepticism. Now that the NSAF, using similar survey methods, has produced a nearly identical estimate, the CTS’ credibility has been enhanced, and the CTS number, in turn, has paved the way for broader acceptance of the NSAF estimate. Yet neither survey has addressed what was felt to be the biggest source of overestimation of the uninsured in the federal surveys: namely, the apparent, substantial underreporting of Medicaid enrollment, which we discuss in Section C.(25)
Much attention has focused on the impact of the verification questions in the CTS and NSAF. Indeed, Urban Institute researchers have indicated that without the verification question the estimate of uninsured children in the NSAF would be as high as it is in the CPS. Yet analysis of CTS data has shown that the CTS estimate would have been 2 percentage points below the CPS estimate even without the verification question (Rosenbach and Lewis 1998). With the addition of a verification question to the NHIS in 1997 and an experimental application to the SIPP questions under way, we will soon know if the presumably better reporting of coverage elicited by a verification question really does account for the lower estimates obtained by the CTS and NSAF. Our suspicion is that the verification question will not account for all or even most of the difference, which leads us to consider the differences in survey methodology detailed above. From a survey design perspective, the selection of a sample based on telephones is a very different exercise from the drawing of a household sample from a list frame (CPS and SIPP) or an area frame (NHIS). Both the CTS and NSAF include nontelephone households in their samples as well as additional corrections for potential bias. Kenney et al. (1999) have documented that the NSAF matches the income distribution and other characteristics of the population as reported by the CPS, so there are no clear differences that we can point to as evidence that the CTS and NSAF samples include too few households with uninsured children. Both surveys also sample children within households rather than collecting data on all children, but there is no evidence as yet that the sampling or subsequent weighting of these children was biased. Nevertheless, it would seem that more detailed evaluation of the potential impact of sample design on the differences between the CTS and NSAF, on the one hand, and the federal surveys, on the other, is warranted--and, indeed, necessary if we are to understand the differences that we see in Table 1.
B. SIMULATING MEDICAID ELIGIBILITY
The measurement or simulation of Medicaid eligibility among the members of a survey sample is important within the context of studying uninsurance because of the impact that Medicaid participation can have on the number of uninsured children. Because the rules for determining Medicaid eligibility are complex, however, simulating Medicaid eligibility is no small undertaking. Furthermore, what it demands from the data in order to replicate all of Medicaid’s complex provisions is more than any existing survey can provide.
1.Reasons for Simulating Eligibility
There are two principal reasons for simulating Medicaid eligibility in the context of studying children’s health insurance coverage. The first is to obtain denominators for the calculation of Medicaid participation rates--for all eligible children and for subgroups of this population. The second is to estimate how many uninsured children--and what percentage of the total--may be eligible for Medicaid but not participating.
2.Complexity of Eligibility Determination
The regulations governing eligibility for the Medicaid program are exceedingly complex. There are numerous routes by which a child may qualify for enrollment, and many of the eligibility provisions and parameters vary by state. Simply documenting the published eligibility criteria is a sizable chore, and operational aspects of the Medicaid eligibility determination--such as the definition and application of income disregards--may not be published or readily accessible. Relative to other means-tested programs, Medicaid presents a far greater challenge for simulation of program- eligibility--both in terms of the complexity of the rules and the data requirements that they generate. Even the most sophisticated simulation models of Medicaid eligibility employ many simplifications (see, for example, Giannarelli 1992). More typically, simulations of Medicaid eligibility are highly simplified. For example, the General Accounting Office has reported findings based on simulating only the federally mandated poverty-related expansions (U.S. GAO 1995). While this captures the majority of Medicaid-eligible children born after September 30, 1983 (because eligibility via cash assistance programs has lower income thresholds), it attributes no eligibility at all to older children.
The data requirements for a Medicaid eligibility simulation are substantial. Like most means-tested entitlement programs, eligibility determinations are based on monthly income, and countable income includes a number of potential disregards for which the source of income and various kinds of monthly expenditures may be relevant. Participation in certain programs makes families or children eligible for Medicaid, so data on program participation are needed. Because people who were eligible for AFDC were often eligible for Medicaid even if they did not participate in AFDC, the Medicaid eligibility determination incorporated the AFDC eligibility rules, and this has been extended in some form into the post-welfare reform era. To simulate AFDC and Medicaid eligibility it is necessary to construct several alternative family income measures, which must be compared to sets of state- specific parameters, which vary by family size. AFDC and some of the other Medicaid provisions are limited to particular types of families, creating a need for family demographic and economic data. In addition, the AFDC unit may be a subset of the entire co-resident family, and other aspects of the Medicaid eligibility determination may exclude some family members as well, so there is a need for additional family demographic data as well as the economic characteristics of family members. Furthermore, AFDC imposed a resources test and other components of the Medicaid program have resource limits as well, so a simulation must include measures of not only financial resources but vehicles as well. Finally, expenditures on health care are instrumental to the determination of eligibility under the medically needy provisions, so health care expenditure data are needed to fully simulate this component of the Medicaid program.
3.Limitations of Survey Data
A full simulation of Medicaid eligibility requires, at the minimum, data on seven basic sets of characteristics:
1. Income by source, with additional information on those expenditures that are applicable to the calculation of disregards
2. Resources--specifically, financial assets along with the number and value of non- commercial vehicles
3. Participation in certain other assistance programs
4. Age and school enrollment of children
5. Family unit membership
6. Medical expenditures
7. State of residence
We discuss the limitations of survey data with respect to each of these types of data below.
a. Income and Disregards
As we noted, Medicaid eligibility is based on monthly income, and portions of this income may be disregarded based on the amount that is attributable to earnings and the level of expenditures on such items as child care, shelter costs, and transportation to work. In simulating eligibility with survey data, it is important, therefore, to have good measures of earned and unearned income as well as the relevant types of expenditures. The importance of measuring unearned income derives not from the fact that the population targeted by Medicaid has income from many sources (although public assistance income is often importance) but that people who are in fact not eligible but rely on unearned income for support may appear to be eligible if their unearned income is not measured adequately.
The surveys that we have described provide little if any data on the expenditures that are taken into consideration in calculating disregards. SIPP is clearly the best, but it collects expenditure data only once or twice in the life of a panel, so the expenditures are not measured concurrently with income except at those one or two times. The CPS collects annual rather than monthly income. Researchers who conduct the most sophisticated simulations with the CPS construct monthly income streams to improve their eligibility simulations, but in doing so they are still not able to measure other components of eligibility (such as family composition) concurrently.
b. Resources
Data on resources, or assets, are very limited. Only the SIPP captures detailed data on asset balances--including the value of vehicles, which are a major component of the applicable asset holdings of the low income population.(26) The SIPP data are collected in two survey waves a year apart, so researchers must interpolate or extrapolate to other months. Reported assets vary so substantially between the two waves, however, that this is difficult--and it suggests low quality in the data (Czajka 1999). Vehicular assets appear to turn over very rapidly as well. Some researchers deal with the limitations of CPS data in this regard by applying an assumed rate of return to reported asset income to impute the unreported balances.
c. Participation in Assistance Programs
Receipt of AFDC and Supplemental Security Income (SSI) benefits are relevant to Medicaid eligibility determination. Prior to welfare reform, which replaced the AFDC program with TANF, AFDC participants were identified in the federal surveys, but AFDC participation was underreported by as much as 25 percent. Data on SSI participation have been collected less regularly and with lower accuracy. Prior to the 1996 panel, the SIPP did not identify individual SSI recipients within a family. Data users could employ information on disability--reported in one survey wave--to infer which child or children in an SSI family may have been the SSI recipient(s). In comparing SIPP 1992 panel estimates of SSI children with published administrative records, however, we found that the survey estimates were quite low and failed to capture a significant upward trend in SSI enrollment (Czajka 1999).

d. Age and School Enrollment of Children
Survey data on these characteristics of children are generally quite adequate for the purposes of Medicaid eligibility simulation.
e. Family Unit Membership
While the official federal poverty levels are designed to be applied to all related persons living in the same household (a “census family”), Medicaid eligibility may be based on just a subset of family members. Some family members (and their incomes) are automatically excluded when determining the eligibility of the remaining family members or the children--for example, SSI recipients and adult children of the family head. In addition, to maximize potential eligibility many states allow their caseworkers considerable latitude in defining the family unit for Medicaid income eligibility determinations (Lewis and Ellwood 1998). Including or not including one particular family member can make the difference between the remaining members being eligible or not. Therefore, the composition of the survey household, the family relationships among household members, and the income available to individual members at a point in time are needed to assign family members to Medicaid eligibility units. The SIPP data are the strongest in this regard, but the simulation of eligibility units is exceedingly complex (even when the applicable rules are well-documented, which they frequently are not). The CPS data are weak because family composition is measured at the time of the survey (March) while the income data refer to the previous calendar year. No data are collected on who was actually present in the household at any time during the previous year.
f. Medical Expenditures
Medical expenditure data are the weakest element among the data collected by the CPS, SIPP, and the NHIS. The virtual lack of information on medical expenditures makes it exceedingly difficult to develop a credible simulation of eligibility under the medically needy provisions of Medicaid. Researchers who do attempt to simulate this component of eligibility must resort to imputing medical expenditures based on other surveys--such as MEPS.
g. State of Residence
Identification of the state of residence of survey households is essential to replicating the state variation in Medicaid eligibility rules. While all but MEPS among the major surveys that we have discussed identify the state of residence of most sample members, we are aware that at least one of the surveys groups sets of states in order to protect the confidentiality of respondents--and possibly to discourage estimates for states that are not adequately represented in the sample. There are nine small states that are not individually identified in SIPP files prior to the 1996 panel. These nine states are combined into groups of two, three, and four states. In order to simulate features of the Medicaid programs for these nine states, it is necessary to assign respondents to the nine states in some manner. One must assume that other characteristics reported on the SIPP files are of limited value in predicting the actual state of residence for sample households reported in one of the three state groups or else the confidentiality of the state data would be compromised. Ultimately, therefore, the assignment of respondents to individual states must rely heavily on randomization. This implies the introduction of some additional error into Medicaid simulations, which may contribute, in turn, to mismatches between simulated eligibility and reported participation.
C. CALCULATING MEDICAID PARTICIPATION RATES
Properly calculated, a Medicaid participation rate is formed by dividing the number of participants (people enrolled) by the number of people estimated to be eligible. In the previous section we detailed many of the difficulties that are inherent in estimating the denominator. Here we discuss some complications associated with estimates of the numerator and satisfying the requirement that the people who have an opportunity to be included in the numerator be fully counted in the denominator.
1. Choice of a Numerator: Survey Versus Administrative Data
Survey estimates of participants in means-tested entitlement programs generally fall well short of the counts reported in program administrative statistics. As a rule, the survey estimates tend to run between 75 and 90 percent of the administrative counts even when the two are rendered as comparable as possible with respect to the universe that they cover. For this reason, it has become a common practice to substitute administrative counts for survey estimates of participants in calculating participation rates for food stamps and AFDC. The choice of a numerator is an issue with respect to Medicaid participation rates as well. Here we discuss a number of considerations that are relevant to using the administrative statistics in this context. The bottom line is that comparability between the administrative and survey data on participation is difficult to establish.
a. Underreporting of Medicaid and Related Program Participation
Table 5 compares CPS estimates of children under 15 who were ever enrolled in Medicaid during 1993, 1994, and 1995 with enrollment statistics reported by the Health Care Financing Administration (HCFA). While a number of caveats should be addressed in making such a comparison, as we explain in the next subsection, the figures in the table give us a rough sense of how complete the CPS reports of Medicaid enrollment appear to be. In 1994 and 1995 the CPS figures lie between 75 and 76 percent of the HCFA estimates versus 83 percent in 1993. The decline in coverage would appear to be due to the CPS’s incomplete capture of a sizable growth in enrollment between 1993 and 1994. Elsewhere, with a more detailed comparison we estimated that the SIPP captured between 85 and 87 percent of Medicaid enrollment among children in FY93 and FY94 (Czajka 1999). The apparent implication, then, is that participation rates will be understated by 13 to 25 percent if we rely exclusively on survey estimates of participation.
TABLE 5. COMPARISON OF CPS ESTIMATES AND ADMINISTRATIVE COUNTS OF CHILDREN UNDER 15 ENROLLED IN MEDICAID
Year
CPS: Ever Enrolled in Calendar Year
HCFA Statistics: Ever Enrolled in Fiscal Year
CPS Estimate as Percent of HCFA
1993
15,165,000
18,348,000
82.7%
1994
14,545,000
19,227,000
75.6%
1995
14,685,000
19,444,000
75.5%
 
SOURCE: March Current Population Survey, 1994 to 1996, and HCFA Medicaid enrollment statistics, FY93 to FY95.
b. Issues in Comparing Survey and Administrative Estimates of Medicaid Enrollment
There are several issues in comparing survey and administrative estimates of Medicaid enrollment to evaluate coverage and, ultimately, to substitute the latter for the former in estimates of participation rates. These include unduplication across states, the existence of state-only programs, the reporting of average monthly versus annual ever enrollment, the limited age detail that is available from published statistics, the inclusion of institutionalized children, concerns about the quality of state Medicaid enrollment data, and retroactive eligibility.
Unduplication Across States. The Medicaid enrollment data published by HCFA are based on reports or data files submitted by the states. While researchers have at times expressed concern about duplicate counting of enrollees within states--the classic situation involving someone who is enrolled in Medicaid at the beginning of the fiscal year, leaves the program, then re-enrolls and is assigned a new, unique identification number--within-state duplication has been reduced by administrative improvements. The same cannot be said about duplication across states. People who start the year enrolled in Medicaid in one state, then move to another state and re-enroll, will be counted-- legitimately--in both states’ ever enrollment figures. In the survey data, of course, such people will be counted only once--in the state in which they reside at the time of the interview. There are no data with which to estimate the possible magnitude of this cross-state duplication, which would require matching state administrative files at the person level or matching survey data to these same administrative data. We doubt, however, that such duplication amounts to more than a few percent of the total national caseload reported by HCFA, although this is purely speculative. About 16 percent of the total U.S. population moves in the course of a year, but only a small fraction of these moves are interstate.
State-Only Programs. A few states (New Jersey, for example) operate what are generally small programs that provide Medicaid coverage to children who do not qualify for federal matching dollars. These children are not included in the enrollment counts reported by HCFA, but they would presumably report themselves (or be reported) to a survey interviewer as covered by Medicaid. If no allowance is made for their differential inclusion in survey versus federal administrative data, their presence in the survey estimates will contribute to an overestimate of survey coverage of Medicaid enrollees.
Average Monthly versus Annual Ever Enrollment. HCFA reports annual (fiscal year) estimates of people ever enrolled in Medicaid by programmatic and demographic characteristics for each state and for the nation as a whole (an aggregate of the state numbers, which may include some duplication). For all people in each state (that is, with no further breakdown), HCFA also reports the number enrolled for all 12 months, the number enrolled for less than 12 months, and the total person-months of enrollment among the latter. With these data is possible to calculate the average monthly enrollment--but only for all enrollees. Children and adults cannot be separated. Thus, the most readily available Medicaid administrative data on enrolled children can be used to evaluate only one type of survey estimate of Medicaid coverage: the number of children ever enrolled during a fiscal year. To evaluate survey estimates of Medicaid enrollment at a point in time requires that the researcher make some assumption about how the relationship between ever enrollment in a year and enrollment at a point in time differs between children and adults.
The states can and do produce their own estimates of Medicaid enrollment. They can produce estimates of the number of people enrolled each month by demographic and programmatic characteristics. Such data are not compiled nationally, however. To obtain monthly enrollment estimates, the researcher would have to request these from every state. In practice, then, it is not possible to compare survey and administrative estimates of Medicaid enrollment at a point in time with the same precision that can be done with estimates of enrollment ever in a year.
Age Detail. HCFA reports Medicaid enrollees under 21 by the following age groups: infant (under 1), 1 to 5, 6 to 14, and 15 to 20. These age categories do not map exactly to children as commonly defined from survey data: all people under 18 or all people under 19. Because Medicaid enrollment declines over ages 15 to 20, allotting two-thirds of the reported number of enrollees in this age group to the ages 15 to 18 yields too few children. To obtain better administrative estimates of enrolled children, we recommend estimating from survey data the fraction of reported Medicaid enrollees 15 to 20 years of age who are 15 to 18 (or 15 to 17 if that is the needed group) and applying this fraction to the Medicaid administrative data. Another strategy, which we followed in preparing Table 5, is to base the comparison on just those ages that can be matched (that is, 0 through 14) and assume that the same rate of coverage applies to the entire population of child enrollees.(27)
We should note that there is a programmatic definition of “children” used in determining Medicaid eligibility, and that reported counts of “children” in some of the HCFA tabulations reflect this definition rather than the purely age-based definition used in survey-based research. Enrollees identified as “children” in HCFA reports are a subset of the full age group that would be defined as children in survey-based research. An individual who is under the age of 19 but responsible for a dependent child would be reported as an adult in tabulations of the basis of eligibility.
Institutionalized Children. Administrative estimates of enrollees include some who are institutionalized whereas the surveys that are used to estimate health insurance coverage exclude people in institutions from their sampling frames. Published HCFA tabulations do not report institutionalized enrollees by age, so it is not possible to exclude institutionalized children from the administrative counts of Medicaid enrollees--except crudely. This is not a large population, but other things being equal, failing to make some adjustment for its differential treatment in the two estimates will contribute to an underestimate of the survey coverage of Medicaid enrollment.
Quality of State Medicaid Enrollment Data. Researchers have raised concern about the quality of state Medicaid enrollment data. As we noted above, one area of concern was the potential multiple counting of individuals who left the program and re-entered within the same fiscal year, but the widespread use of unique “lifetime” identifiers is eliminating this problem. Indeed, analysis of case record data provides indirect support for this assertion in the form of frequent, identifiable instances of the same individuals exiting and then re-entering Medicaid within the year (Ellwood and Lewis 1999). At the same time, however, the state statistics reported by HCFA each year are accompanied by extensive caveats that point out omissions, inconsistencies, and other errors. At a minimum, users of the published enrollment data need to be aware that the data have known imperfections that may require some form of correction before they are used.
Retroactive Eligibility. Under certain circumstances, a Medicaid enrollee’s eligibility may be applied retroactively to cover medical costs that were incurred prior to official enrollment. Survey respondents interviewed just prior to their enrollment may correctly report their status as not covered, but the administrative statistics may later change this status. As a result, the administrative statistics would tend to run slightly higher than the reports obtained from surveys even if the latter were correct at the time they were recorded.
2. Incomplete Simulation of Eligibility
Because of the aforementioned data limitations, together with the complexity of the rules, simulations of Medicaid eligibility will almost invariably be incomplete, and even very good simulations may exclude as much as one-fifth of the eligible population. This may provide the strongest argument against substituting administrative estimates for survey estimates of participants in order to calculate participation rates. If the deficiencies of the eligibility simulation can be matched to the eligibility categories reported in the Medicaid statistics, however, it may be possible to construct an administrative count of participants that is reasonably consistent with the eligibility simulation. For example, we have noted that the medically needy component of Medicaid is the most difficult to simulate, and many analysts make little attempt to do so. Medically needy children under 21, where “children” are defined by the nature of their eligibility rather than their age, are reported in the annual statistics released by HCFA, and therefore they could be subtracted from an administrative count of participants to yield a numerator that could be used to calculate a Medicaid participation rate that excluded the medically needy from both the numerator and denominator.
3. Simulated Eligibles Reporting Coverage Other Than Medicaid
In our research with SIPP data we found that 18 percent of the children we simulated to be eligible for Medicaid reported having some form of insurance coverage other than Medicaid (Czajka 1999). This other coverage could represent Medicaid being misreported as something else, or it could represent genuinely different coverage. In the former case, there are clear implications for the calculation of Medicaid participation rates. Indeed the misreported coverage would account for part of the Medicaid undercount. If there were a way to resolve this with the survey data and determine how many of the children who reported other coverage may have actually been covered by Medicaid, then the quality of a survey-based participation rate could be improved. To the extent that it is not possible to discern the amount of misreporting and in so doing correct the survey data, the argument for considering administrative data for the numerator is strengthened.
The possibility that much if not most of the reported other coverage is truly something other than Medicaid suggests another strategy--perhaps one that is best viewed as a complementary strategy rather than an alternative one. Nonparticipation in Medicaid by eligible children who have other coverage carries very different policy implications than nonparticipation by those who are uninsured. Medicaid participation rates calculated for just those children who would otherwise be uninsured may provide a more meaningful indication of the success of Medicaid outreach than participation rates that count eligible children with other coverage as eligible but not participating. Even without adjusting for Medicaid underreporting, we found that the participation rate among eligible children with no other coverage was 79 percent, compared with 65 percent for all eligible children (Czajka 1999).
4. Seemingly Ineligible Participants
Extensive research on simulating eligibility for food stamps and AFDC has identified a perplexing and thus far inescapable problem: nontrivial numbers of those who report participation are simulated to be ineligible.(28) The existence of such people may reflect the incompleteness or inaccuracy of the simulation model. This is particularly true of Medicaid, with its complex eligibility determination and the difficulties analysts face in documenting current state policies. Adding to the complexity are provisions that allow certain classes of participants to maintain their eligibility, once established, despite changes in income that would render other participants ineligible. Such provisions apply to pregnant women and infants and to families receiving transitional coverage after losing AFDC benefits because of increased earnings. The fraction of participants who experience extended eligibility is likely to increase as the Balanced Budget Act of 1997 gives states the option to guarantee coverage to children for up to 12 months after enrollment, regardless of changes in family income (Lewis and Ellwood 1998).
The appearance of simulated ineligible participants may also be due to misreported participation, incorrect edits or imputations, errors in the actual eligibility determination, or the failure of participants to report changes in their circumstances. These different explanations for the phenomenon of seemingly ineligible participants carry different implications for how such cases should be handled in calculating participation rates (Vaughan 1992). When the errors are due to the simulation, there is little question that such participants should be included in participation rates providing that the total number of simulated eligibles can be corrected to offset any bias.(29) At the same time, however, people who participate when in fact ineligible or who are incorrectly identified as participants should be excluded from participation rates. Simply adding them to the denominator to offset their inclusion in the numerator gives them an implied participation rate of 100 percent, which is difficult to interpret. Clearly, it would be desirable to know more about the seemingly ineligible participants, but “they” have proven difficult to understand. Limiting participation rates to simulated eligibles (which would also imply removing ineligible participants from administrative statistics if the latter are used to form the numerator) makes it unnecessary to address what may be unresolvable issues in the definition of the denominator, and it maintains the concept of a participation rate as the number of eligible participants divided by the total number of eligibles.
5. Population Undercoverage
Some of the apparent underreporting of Medicaid enrollment may be due to survey undercoverage-