|
Size: 5224
Comment:
|
← Revision 19 as of 2026-02-06 20:41:41 ⇥
Size: 2977
Comment: Rewrite
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 3: | Line 3: |
| Survey disposition is the description of how a respondent interacted with a survey instrument. | A '''survey disposition''' encodes the outcome of an interview. |
| Line 10: | Line 10: |
| == Description == | |
| Line 11: | Line 12: |
| == Non-Response Bias == | A survey disposition encodes the most important information about how an individual interacted with a survey instrument. The major categories are: |
| Line 13: | Line 14: |
| Non-response bias is introduced when an individual's likelihood to respond to a survey instrument is correlated to a key measurement. | 1. Eligible respondents 2. Eligible nonrespondents 3. Sample members with unknown eligibility 4. Ineligible sample members |
| Line 15: | Line 19: |
| Note that non-response bias is not the same as ''low'' response rates. Bias can be introduced if a subgroup's responsivity ''increases''. | There are then subcategories with finer detail, for example the reason a sample member is ineligible. |
| Line 21: | Line 25: |
| == AAPOR Disposition Codes == | == Coding == |
| Line 23: | Line 27: |
| The rough rubric for dispositioning are: | AAPOR has published guidance to standardize the encoding of dispositions. |
| Line 25: | Line 29: |
| 1. Eligible Respondent 2. Eligible Non-Respondent 3. Unknown Eligibility 4. Ineligible |
Major categories are an integer part, while minor categories are a decimal part. These codes can be truncated at any level to collapse subcategories into parent categories. For example, codes 4.11 and 4.12 may convey different survey items whereby sample members have indicated their ineligibility. If any analyst only need to know which cases self-identified as ineligible through any survey item, the codes can be truncated to the tenths decimal place; 4.11 and 4.12 become 4.1. |
| Line 30: | Line 31: |
| ---- | When working with a sample of specifically-named individuals, there is generally a presumption of eligibility. It follows that: * An explicit refusal to participate in a survey, either as declining to give consent or as requesting to be removed from a contact list, is generally coded as eligible nonresponse. * An incomplete survey response, either due to item nonresponse or due to technical issues, is generally coded as eligible nonresponse. |
| Line 32: | Line 35: |
| The exception is when contact is attempted but not necessarily successful. This leaves open the possibility that the specifically-named individual has e.g. moved, and is no longer member to the eligible population. Mailed surveys will often have many subcategories of unknown eligibility, corresponding to the various '''nixie codes''' returned by the postal service: * Undeliverable as addresses * Refused by addressee * Vacant * Deceased * and so on |
|
| Line 33: | Line 42: |
| Similarly, a survey link that is shared by email can lead to a variety of email bouncebacks. | |
| Line 34: | Line 44: |
| == Web Surveys == | If responses are collected from a nonprobability panel, individuals often must be considered as having unknown eligibility until they provide survey responses indicating otherwise. |
| Line 36: | Line 46: |
| The common codes for web surveys are: | Nonprobability panel surveys also often incorporate live or rebalancing quotas which cause 'surplus' respondents to be terminated as 'over-quota'. In most cases these are treated as eligible nonrespondents. |
| Line 38: | Line 48: |
| ||'''Code'''||'''Explanation'''|| ||1.1 ||Complete|| ||1.2 ||Partial with sufficient information (< NN% unanswered)|| ||2.111 ||Explicit Refusal|| ||2.112 ||Implicit Refusal (i.e. logged in but did not respond)|| ||2.12 ||Partial with insufficient information (> NN% unanswered)|| ||2.3 ||Data quality issues|| ||3.19 ||Unknown eligibility (i.e. did not log in)|| ||4.11 ||Screened Out of Sample - Reason #1|| ||4.12 ||Screened Out of Sample - Reason #2|| ||4.8 ||Quota Filled|| The difference between treating low-quality data as an eligible non-respondent or ineligible is murky at best. ---- == Paper Surveys == Paper surveys are in many ways simpler. Responses are generally higher quality. Whereas web interviews are often shown interactively, paper questionnaire are seen entirely and at once. Questions cannot be conditionally skipped on a paper questionnaire. The common codes for paper surveys are: ||'''Code'''||'''Explanation'''|| ||1.1 ||Complete|| ||1.2 ||Partial with sufficient information (< NN% unanswered)|| ||2.111 ||Unknown Individual Refusal|| ||2.112 ||Known Respondent Refusal|| ||2.113 ||Implicit Refusal (i.e. known eligible but returned blank)|| ||2.12 ||Partial with insufficient information (> NN% unanswered)|| ||4.11 ||Screened Out of Sample - Reason #1|| ||4.12 ||Screened Out of Sample - Reason #2|| === Completion === In a basic survey, use codes 1.1 and 2.112. {{{ if (COMP_01 eq 1) DISPO=1.1. if (COMP_01 eq 0) DISPO=2.112. }}} where ''COMP_01'' indicates completion of any single question. If a survey requires NN% completion, use codes 1.1, 1.2, and 2.12. {{{ if (STATUS eq COMPLETE and COMP_NN eq 1) DISPO=1.1. if (STATUS ne COMPLETE and COMP_NN eq 1) DISPO=1.2. if (COMP_NN eq 0) DISPO=2.12. }}} where... * ''STATUS'' indicates the respondents access to a web survey, and ''COMPLETE'' means that the respondent does not have access resulting from reaching the end of the survey. * ''COMP_NN'' indicates the completion of NN%, or a completion rate >= 0.NN. === Consent === If a survey requires consent, use codes 1.1, 2.111, and 2.112. {{{ if (COMP_01 eq 1) DISPO=1.1. if (CONSENT ne 1) DISPO=2.111. if (COMP_01 eq 0) DISPO=2.112. }}} where ''CONSENT'' is the consent question. If a survey requires consent and NN% completion, use codes 1.1, 1.2, 2.111, 2.112, and 2.12. {{{ if (STATUS eq COMPLETE and COMP_NN eq 1) DISPO=1.1. if (STATUS ne COMPLETE and COMP_NN eq 1) DISPO=1.2. if (COMP_NN eq 0) DISPO=2.12. if (CONSENT ne 1) DISPO=2.111. if (COMP_01 eq 0) DISPO=2.112. }}} === Eligibility === If a survey requires some conditions to be met, use codes 1.1, 2.112, and 4.11 (and so on). {{{ if (COMP_01 eq 1) DISPO=1.1. if (COMP_01 eq 0) DISPO=2.112. if (STATUS eq TERMINATE and CONDITION_1 eq 0) DISPO=4.11. }}} where... * ''OVERQUOTA'' means that the respondent does not have access to the web survey resulting from failure of a required condition. * ''CONDITION_1'' indicates success of the required condition. If a survey requires some conditions to be met as well as requiring consent, use codes 1.1, 2.111, 2.112, and 4.11. {{{ if (COMP_01 eq 1) DISPO=1.1. if (CONSENT ne 1) DISPO=2.111. if (COMP_01 eq 0) DISPO=2.112. if (STATUS eq TERMINATE and CONDITION_1 eq 0) DISPO=4.11. }}} If a survey requires some conditions to be met as well as NN% completion, use codes 1.1, 1.2, 2.12, 2.112, and 4.11. {{{ if (STATUS eq COMPLETE and COMP_NN eq 1) DISPO=1.1. if (STATUS ne COMPLETE and COMP_NN eq 1) DISPO=1.2. if (COMP_NN eq 0) DISPO=2.12. if (COMP_01 eq 0) DISPO=2.112. if (STATUS eq TERMINATE and CONDITION_1 eq 0) DISPO=4.11. }}} If a survey requires some conditions to be met, consent, and NN% completion, use codes 1.1, 1.2, 2.111, 2.12, 2.112, and 4.11. {{{ if (STATUS eq COMPLETE and COMP_NN eq 1) DISPO=1.1. if (STATUS ne COMPLETE and COMP_NN eq 1) DISPO=1.2. if (COMP_NN eq 0) DISPO=2.12. if (CONSENT ne 1) DISPO=2.111. if (COMP_01 eq 0) DISPO=2.112. if (STATUS eq TERMINATE and CONDITION_1 eq 0) DISPO=4.11. }}} === Panel Quotas === If a panel survey has quotas, add code 4.8. {{{ if (STATUS eq OVERQUOTA) DISPO=4.8. }}} where ''OVERQUOTA'' means that the respondent does not have access to the web survey resulting from a closed quota. |
A multi-modal survey has to incorporate components from each of the above, including a hierarchy to reconcile contradictory dispositioning information. For example, consider an individual that was sent a survey link in both a mailed postcard and an email. If a response is collected and one of those invites is returned with information, a decision must be made with regard to which information is more accurate. |
Survey Disposition
A survey disposition encodes the outcome of an interview.
Contents
Description
A survey disposition encodes the most important information about how an individual interacted with a survey instrument. The major categories are:
- Eligible respondents
- Eligible nonrespondents
- Sample members with unknown eligibility
- Ineligible sample members
There are then subcategories with finer detail, for example the reason a sample member is ineligible.
Coding
AAPOR has published guidance to standardize the encoding of dispositions.
Major categories are an integer part, while minor categories are a decimal part. These codes can be truncated at any level to collapse subcategories into parent categories. For example, codes 4.11 and 4.12 may convey different survey items whereby sample members have indicated their ineligibility. If any analyst only need to know which cases self-identified as ineligible through any survey item, the codes can be truncated to the tenths decimal place; 4.11 and 4.12 become 4.1.
When working with a sample of specifically-named individuals, there is generally a presumption of eligibility. It follows that:
- An explicit refusal to participate in a survey, either as declining to give consent or as requesting to be removed from a contact list, is generally coded as eligible nonresponse.
- An incomplete survey response, either due to item nonresponse or due to technical issues, is generally coded as eligible nonresponse.
The exception is when contact is attempted but not necessarily successful. This leaves open the possibility that the specifically-named individual has e.g. moved, and is no longer member to the eligible population. Mailed surveys will often have many subcategories of unknown eligibility, corresponding to the various nixie codes returned by the postal service:
- Undeliverable as addresses
- Refused by addressee
- Vacant
- Deceased
- and so on
Similarly, a survey link that is shared by email can lead to a variety of email bouncebacks.
If responses are collected from a nonprobability panel, individuals often must be considered as having unknown eligibility until they provide survey responses indicating otherwise.
Nonprobability panel surveys also often incorporate live or rebalancing quotas which cause 'surplus' respondents to be terminated as 'over-quota'. In most cases these are treated as eligible nonrespondents.
A multi-modal survey has to incorporate components from each of the above, including a hierarchy to reconcile contradictory dispositioning information. For example, consider an individual that was sent a survey link in both a mailed postcard and an email. If a response is collected and one of those invites is returned with information, a decision must be made with regard to which information is more accurate.