The missing row still has a cause.

Picture a one-page summary from a product team's quarterly customer survey. It is a fictional company, but the sheet is familiar. Forms returned: 1,000. Customers who said they completed the task they came to do: 900. Success rate: 90 percent. Underneath, in pencil, the product manager has written the sentence that will go into the board deck: 90 percent of customers succeeded. Clipped to the back is a second sheet from the system log. It lists 500 more customers who were eligible for the survey in the same quarter and never completed it.

Every number on the first sheet is correct. The division is right, and nobody has invented a form. What is not yet justified is the label. The sheet describes observed records: the forms that came back. The pencil sentence describes a target population: all the customers the company serves. The two can have the same average. They can also differ by a wide margin, and nothing on the first sheet says which.

My argument is that a database records the people and events that passed through its collection process, and that before treating its averages as a description of anyone larger, you have to explain who could enter, who did enter, and why the others are absent. More rows reduce some kinds of uncertainty. They do not, by themselves, repair selective observation. The missing row is not a blank. It is the trace of a process, and the process has a cause.

Three groups that are easy to confuse

Survey researchers separate three groups that managers usually run together. The target population is the group a statement is about: every customer who tried to use the product this quarter, say. The collection frame is the group the measurement process could reach: the customers to whom the survey was shown. The observed records are the people in the frame who actually answered.

Each step can lose people, and each loss has a mechanism. Suppose the survey appears inside the product after a customer logs in. Then a customer who could not log in never sees a question. That customer belongs to the target population and is absent from the frame, before response is even possible. The 500 names on the second sheet are the easier case, because the log knows they exist. The customers who never reached the survey at all may not appear on either sheet.

Figure 1 assumes the easy case: the company knows there were 1,500 eligible customers, so the empty slots can be drawn. In many real systems the size of the unobserved group is itself unknown. An application that records only successful sessions cannot count the failed ones. A feedback form that only customers who stay can reach has no row for the customers who left.

Missing for a reason

The tempting assumption is that the missing customers are like the ones who answered. Sometimes they are. The problem is that the assumption is rarely examined, and the reasons for missing data often point somewhere.

A customer whose login failed may be less likely to answer a survey about success, because the failure is the reason the survey never appeared, or because a frustrated customer does not want to spend another minute with the company. That would make the observed 90 percent too high. But the opposite is also plausible. A customer who succeeded quickly may close the window without a thought, while a customer who struggled and then succeeded may want to say so. That would push the observed rate the other way, or nowhere at all.

Neither story is evidence. The direction and the size of the difference between respondents and nonrespondents are empirical questions, and the survey alone cannot answer them. What the survey can do is make the question visible, by recording who was eligible, who was asked, and who answered, so that someone can go and look.

What the response rate does not tell you

The natural instinct is to judge the evidence by its response rate. One thousand out of 1,500 is two thirds, which sounds respectable. The research on nonresponse says that the instinct is not wrong but is incomplete.

The U.S. Census Bureau's analysis of nonresponse bias in the 2020 calendar year of its Survey of Income and Program Participation makes the point plainly in its introduction. Falling response rates can signal bias, the authors write, but “there is not always a direct link between response rates and nonresponse bias.” Different statistics from the same survey can be biased by different amounts, depending on how each one relates to a household's likelihood of responding. A low response rate may produce serious bias in some estimates and little in others, and a high response rate does not reduce bias for a variable that has nothing to do with who responds.¹

For that claim the report cites Robert Groves and Emilia Peytcheva's 2008 meta-analysis in Public Opinion Quarterly. They gathered 59 methodological studies that had been designed to estimate the size of nonresponse bias, using designs such as sampling frames rich in known variables, administrative records matched to sample members, and follow-up of people who responded late or not at all. Their question was under which circumstances nonresponse rates and nonresponse bias move together, and under which they do not.² I have read the abstract, not the full paper, and I will not attach a pooled number to it. The question itself is the lesson: the rate describes how many people are missing, and the bias depends on how they differ.

The Census report puts the second part in a formula a manager can use. For an average, the bias of the respondents' average equals the share of the sample that did not respond, multiplied by the difference between the respondents' average and the nonrespondents' average.¹ Both parts matter. A large nonresponse share with no difference produces no bias. A small nonresponse share with a large difference can produce a lot. The response rate supplies the first part. The second part is exactly what the missing people would have told you.

Work the bounds

The opening example is small enough to work by hand. The 1,000 respondents include 900 successes. The 500 missing customers are known to exist, and their outcomes are unknown. Each of them either succeeded or did not.

If all 500 failed, the true success rate among the 1,500 eligible customers is 900 out of 1,500: 60 percent. If all 500 succeeded, it is 1,400 out of 1,500: about 93.3 percent. Any answer between those two is consistent with every form that came back. The board deck's 90 percent sits near the top of that range, and is exactly right only if the missing customers succeeded at the same rate as the ones who answered.

A range from 60 to 93 percent is not an estimate that either extreme happened. It is a statement of what the observations cannot establish on their own. It is also a useful one, because it tells you how much the decision depends on the missing customers. If the product team would act the same way at 60 percent as at 93, the gap does not matter much. If the board would cancel a support investment at 90 and fund it at 60, the gap is the whole decision.

Move the slider to zero and the board deck overstates success by 30 points. Move it to 100 and the deck understates it by about three. The asymmetry is not a law of nature. It comes from the observed rate sitting near the top of the scale, so there is more room below it than above. A survey with an observed rate of 50 percent would have a symmetric range, and one near zero would lean the other way.

Adjusting for what you can see

The standard response to nonresponse is to weight: give more influence to respondents who resemble the people who did not answer. It often helps, and it is worth seeing what it requires.

The Census Bureau's survey began with a sample of addresses drawn from its own master list. Of roughly 49,000 eligible households in the 2020 calendar year, about 22,000 were interviewed, a weighted response rate of 45.0 percent. Because the sample came from a known frame, the Bureau knew something about every household it missed: its region, whether it was in a city, and the characteristics of its neighborhood from other Census data. That is what let it compare the two groups. Nonrespondents were more likely to live in the Northeast or in urban areas, to have Black householders and to rent, and more likely to live in neighborhoods with more languages spoken, lower high school completion and higher use of public assistance.¹

The adjustment for nonresponse narrowed several of those differences. For the share of households in the Northeast, the relative difference between respondents and the full sample shrank from about minus 6.8 percent to about minus 1.5 percent once the adjustment was applied.¹ That is the method working where it can: on variables the Bureau could see for respondents and nonrespondents alike.

The same report also compares its final estimates with outside benchmarks. Median household income came out 4.28 percent below the figure from the Current Population Survey, poverty rates came out higher, and participation in programs such as food assistance came out lower than official enrollment counts.¹ Those gaps have several possible causes, and the report does not attribute all of them to nonresponse. But they show the limit of weighting. It can correct for the differences the adjustment variables capture. It assumes that, within each group those variables define, the people who answered are like the people who did not. When the reason for missing is something the frame does not record, the weights cannot see it.

A corporate survey usually has a thinner frame than the Census Bureau's. It may know a customer's plan, region and tenure, and nothing about whether the customer's login failed on the day the survey ran. Weighting by plan and region may then adjust for everything except the cause.

Draw an observation map

The practical remedy is to write down how the data came to exist, before anyone reports an average from it. I call this an observation map. It has six entries.

First, the eligible population: who the statement is meant to describe, and how many of them there are, if that is known. Second, the collection channel: where and when people could become a record, such as an in-product prompt after login or an email after a support ticket. Third, the exclusion rules: who could not reach the channel, and who was filtered out afterward. Fourth, response status: for each eligible person, whether they were asked and whether they answered. Fifth, what is known about the people who did not answer, from the frame, the logs or other systems. Sixth, the follow-up design: how a sample of nonrespondents will be reached by another route, such as a phone call to customers whose logins failed.

Where someone proposes weighting, the map adds a seventh line: the assumption that the adjustment variables are sufficient for this comparison. Writing that line down does not make it true. It makes it a claim someone can test.

Anthropic's June 26 Economic Index report shows what a stated frame looks like in public. The report links about 9,700 survey respondents, drawn at random from Claude users, to their usage. It then says plainly that the survey is not representative of the general population, that there may be selection in who completes it, that infrequent users are filtered out, and that the survey disproportionately reaches people who use AI more.³ Those sentences do not weaken the report. They tell a reader which population its findings describe, and they stop anyone from quoting its averages as a portrait of every worker.

Objections, considered

We have millions of records. Size helps with one kind of uncertainty: the random variation that comes from observing a sample rather than everyone. Under suitable assumptions, more rows make that variation small. Size does nothing for the other kind. A million records from customers who could log in contain no information about the customers who could not, and the average of a million biased rows is a precise estimate of the wrong number.

Weighting fixes representation. It can, when the adjustment model captures the differences that matter. The Census example shows it narrowing real gaps. Unknown mechanisms and unmeasured attributes remain, and the honest report says which assumption the weights rest on.

We have to decide with imperfect evidence. Of course. Perfect observation is not a prerequisite for action. The remedy is a bounded decision: act on what holds across the plausible range, run a targeted follow-up where the range is too wide, and show the sensitivity analysis alongside the headline.

The nonrespondents may well be similar. They may. That is a reasonable hypothesis, and in many surveys it turns out roughly true. But it is a hypothesis about the missing outcomes, and the response rate alone neither proves nor disproves it. The way to support it is to observe some of the missing people.

None of this makes a dataset with missing rows useless, or a high response rate worthless. A high rate narrows the bounds, and Figure 2's range would shrink if more customers had answered. The point is narrower: how much a missing row matters depends on the statistic, and on the process that made it missing.

Before trusting the average, ask who had a chance to become a row.

Before you report the average

• Who cannot enter the data at all? Name the people the collection channel cannot reach, and estimate how many there are.
• Who was asked and did not answer? Keep a response status for every eligible person, not just a table of the forms returned.
• Which nonrespondents can be reached another way? Contact a sample independently, through a channel that does not depend on the failure you are worried about.
• How far does the decision move? Work the bounds and a few plausible values for the missing outcomes, and say whether the decision changes.
• What does the weighting assume? If weights are used, write down which variables they rest on and what they cannot see.

Back to the summary sheet. The forms still say 1,000, the successes still say 900, and the division still gives 90 percent. Only the pencil sentence has changed. It now reads: 90 percent of the 1,000 customers who answered said they succeeded; 500 eligible customers did not answer, and among all 1,500 the rate is somewhere between 60 and 93 percent. Below it is a note: call 50 of the 500 next week. That is a sentence a board can act on, because it says who the number is about.