An official website of the United States government
Office of Survey Methods Research, Bureau of Labor Statistics
Acknowledgement: The authors sincerely thank Joanne Pascale, Research Social Science Analyst at the Census Bureau, for her significant contributions to the analysis, structure and writing of this report.
Disclaimer: This report is released to inform interested parties of research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau or the Bureau of Labor Statistics.
This report presents findings from cognitive interviews that were conducted by the U.S. Census Bureau and the Bureau of Labor Statistics (BLS). Cognitive interviews were conducted in four cities to explore the feasibility of asking about sexual orientation and gender identity (SOGI) in the Current Population Survey (CPS). In the test questionnaire, we selected 46 questions from the basic CPS to both set the general context of the actual CPS, and to include items known to be challenging in some way for respondents to answer, either for themselves or others in the household, such as income, and disability. This gave a basis for comparing the difficulty and sensitivity of the SOGI items to others already in the questionnaire. Data from 132 respondents with diverse demographic characteristics were collected, in both "individual" interviews with one member of a household, and "paired" interviews with two members of a household interviewed separately. Both lesbian, gay, bisexual and transgender (LGBT) and non-LGBT respondents were included. The cognitive interviews were used to answer five research questions:
1. How difficult are the SOGI questions for respondents to understand and answer? Do respondents have the knowledge to answer for other people in their household and are they willing to provide those answers?
Overall, a majority of respondents found the SOGI questions clear and did not have difficulty answering for either themselves or for other people in their households. Respondents judged income, employment, and disability questions to be more difficult than the SOGI items for both self-response and proxy response (response for other household members). For proxy response, the date of birth item was found to be more difficult than both SOGI items, and gender identity was among the least difficult. The observed difficulty for SOGI questions tended to be related to the response categories, having a fluid identity, questioning one's identity, and/or lacking knowledge of the sexual orientation or gender identity of others in the household.
The SOGI responses collected from household members via the paired interviews matched for almost all the respondents, suggesting respondents are able and willing to provide the information about other people in their household. Additionally, data collected during a card sort exercise indicates that while many respondents categorized the SOGI questions as difficult, the income, disability, and race questions were more frequently identified as difficult.
2. How sensitive do the respondents perceive the SOGI questions to be when answering for themselves and for others in their household, and how does that sensitivity relate to willingness to answer the questions or complete the survey?
Most respondents did not find the SOGI questions sensitive, but relative to other items in the questionnaire, the sexual orientation and disability items were found to be the most sensitive items for both self and proxy response,. Respondents who found the SOGI questions sensitive to report for themselves were both LGBT and non-LGBT respondents, with the former finding their own identity personal to talk about, and the latter finding sexual orientation culturally sensitive and more private. Fewer respondents indicated that they found gender identify sensitive, for either self or proxy reporting. Those who did indicate sensitivity had similar reasons as for sexual orientation, indicating the topic is personal or culturally sensitive. A few respondents (most of whom were LGB) indicated they found SO or GI questions sensitive because they were uncomfortable with the question wording and response options.
Despite some indications of sensitivity, all respondents were willing to answer the SOGI questions for themselves and other people in their households.
3. Do difficulty and sensitivity differ based on demographics - such as geography, household structure, race and/or Hispanic ethnicity, educational attainment, sexual orientation or gender identity?
While there were not many trends by demographics, the main difference that emerged was that most of the respondents who expressed difficulty or sensitivity with the SOGI questions were lesbian, gay, or bisexual (LGB). Furthermore, the few transgender respondents in the study almost universally expressed both difficulty and sensitivity with the SOGI questions as worded.
The demographic profile of the LGBT and non-LGBT samples were different, reducing our ability to detect demographic differences in the results between those groups. For some comparisons (e.g., difficulty for proxy reporting, sensitivity for self-reporting) however, some differences by household size, age, urbanicity, educational attainment, and race were identified. These differences were not consistent across comparisons, and so are difficult to interpret.
4. Are respondents willing to answer SOGI questions for themselves and others in their household in the context of a Federal government survey on employment?
Results suggest that respondents are generally willing to answer SOGI questions for themselves and others in their household in the context of a Federal employment survey, although we note this study included only paid, volunteer respondents who were comfortable enough with the Federal government to participate. When asked directly for their thoughts on being asked SOGI questions in the context of the CPS, a majority said that they had no concerns with the idea, and no respondents refused to answer any SOGI question. A few respondents did raise issues about SOGI questions, discussing concerns over confidentiality, or mentioning that their responses could be less protected and/or used for discrimination in the current political climate. Additionally, the idea of cultural sensitivity was raised by a few respondents.
5. What feedback do respondents have on wording of SOGI questions?
While question wording was not a main focus of this study, respondents provided some valuable feedback on the wording that was tested. In general, respondents understood the SOGI questions but some respondents, particularly LGBT respondents, had difficulty with or concerns about the response options in both questions. A variety of feedback and suggestions are detailed in the report, and will be a valuable guide for future research.
Conclusion
Overall, feedback in these cognitive interviews suggests that most respondents do not find SOGI questions difficult or sensitive to report for themselves or for others in their households, and that few raised objections to these questions in the context of the CPS. However, perceptions of difficulty for self and others in the household were more frequent among LGBT respondents. These findings suggest that while collection of SOGI information on the CPS may be feasible, extensive further testing is needed. Additionally, beyond the problems identified in this study, there are a variety of issues that will need to be addressed before any implementation decisions can be made.
These interviews were just one part of a larger study on the feasibility of asking about sexual orientation and gender identity on the CPS. A decision on overall feasibility of collecting SOGI information in the CPS should consider the findings of the cognitive interviews as well as those of the focus groups with transgender respondents (Holzberg et al., 2017).
In 2015, the Office of Management and Budget (OMB) convened a working group of more than 100 representatives from 35 Federal agencies across 14 departments and 7 independent Federal agencies (Park, 2016). The purpose of the group is to share knowledge on the development and testing of questions on sexual orientation and gender identity (SOGI) in both Federal and non-Federal surveys in the United States (Federal Interagency Working Group, 2016a). The goal of including SOGI data in Federal surveys is to allow researchers to estimate the size and distribution of the sexual and gender minority populations in the U.S., and to identify disparities between people who identify as lesbian, gay, bisexual, and transgender (LGBT) and those who do not in domains such as health, crime, or employment. Currently, there are 11 Federal surveys that collect data on sexual orientation, and of these, seven also ask about gender identity (Federal Interagency Working Group, 2016a). These surveys vary on features such as question wording, mode of survey response, and population being surveyed. In terms of context (i.e., primary topic of the survey), most are health surveys; the exceptions are the National Inmate Survey (NIS), the Survey of Prison Inmates (SPI), and the National Crime Victimization Survey (NCVS). All of these surveys ask respondents to report SOGI for only themselves — not other household members.
The OMB working group has so far issued three working papers on the landscape of SOGI questions in Federal surveys: one on current SOGI measurement in Federal surveys, a second on evaluations of these Federal SOGI measures, and a third on research priorities moving forward (Federal Interagency Working Group, 2016a, 2016b, 2016c). In this latter paper, the group identified proxy reporting and question wording as primary research priorities for SOGI questions and survey context as a secondary research priority (Federal Interagency Working Group, 2016c). The working group has also recently taken an interest in asking about SOGI on the Current Population Survey (CPS), which serves as the primary source of labor force statistics for the U.S. population. The CPS is sponsored jointly by the Bureau of Labor Statistics (BLS) and the Census Bureau. The CPS differs from the surveys currently collecting SOGI information in two important ways. First, the context is employment, as opposed to health, which may affect respondents' perceptions of the relevance of SOGI questions. Second, a single household respondent answers questions about all other household members. Thus, in households with two or more members, household respondents provide self-reports for themselves, and also proxy reports for other household members. It is unknown whether respondents are able to report SOGI accurately by proxy and whether they feel comfortable doing so.
Because of these differences, the U.S. Department of Labor's Office of the Assistant Secretary for Policy sponsored research to explore the feasibility of asking about SOGI in the CPS. The overall goal was to assess the feasibility of collecting SOGI data in the CPS setting—that is, an employment survey context with proxy reporting. More specifically, we wanted to examine: (1) the sensitivity of the questions in general; (2) whether household members have the knowledge about each other with regard to SOGI questions and are willing to provide those answers, and (3) reactions to the inclusion of SOGI questions in an employment survey. To address these goals, cognitive interviews were conducted with both LGBT and non-LBGT populations, and focus groups were conducted with individuals who identified as transgender. Results of the focus groups are provided in a separate report (Holzberg et al, 2017); the purpose of this report is to document the results of the cognitive interviews.
Cognitive interviews are one-on-one sessions between a researcher and respondent. The researcher presents the respondent with the verbatim survey questions and then probes the respondent on various aspects, such as interpretation and comprehension of the question, difficulty in formulating an answer, sensitivity in providing a response, and other issues that may arise during the response process. A total of 132 cognitive interviews were conducted with roughly equal numbers of GBT and non-LGBT individuals. These groups were further divided between "individual" interviews with one member of a household (80 respondents), and "paired" interviews with two members of a household interviewed separately (52 respondents). The cognitive interviews were used to answer five research questions:
The CPS is conducted monthly by the Census Bureau on behalf of BLS with an annually-selected probability sample of about 60,000 occupied households in all 50 states and the District of Columbia. The survey excludes those living in institutions, such as prisons, long-term care facilities, and nursing homes. One person (age 15 or older), known as the "household respondent," generally responds for all household members and is usually a person who owns or rents the housing unit. Households are in the survey for four consecutive months, out of sample for eight months, and then return for another four months before leaving the sample permanently, for a total of eight interviews. In terms of content, the CPS consists of a basic monthly survey, to which a supplemental or topical module is added in most months. The basic monthly survey is divided into two parts: household and demographic information (e.g., date of birth, marital status) and labor force information (e.g., employment during the past week). Supplements cover a range of topic areas, including annual work activity and income, veteran status, school enrollment, and volunteerism, among other topics (U.S. Census Bureau, 2017a).
Most of the household and demographic questions are asked only once, in the first interview. In subsequent interviews, respondents are asked to confirm that members of the household previously reported are still living there. If someone has left the household, they are removed from the household roster; if someone has moved into the household, demographic questions are asked for that person. Respondents are also re-asked demographic questions if they previously reported they did not know the answer (but if they refused initially, the question is not re-asked). Some questions (such as educational attainment and disability) are re-asked of all respondents in select subsequent interviews regardless of prior responses because the answers could change over time. The CPS is administered through both computer-assisted personal interviewing (CAPI) and computer-assisted telephone interviewing (CATI). The majority of the first and fifth CPS interviews are conducted through personal visit, with other waves often being conducted over the telephone.
Currently, there are no questions on SOGI in the CPS. There is a question on sex, in the demographics section of the basic monthly survey. Household respondents are asked to respond for themselves and all other household members, regardless of age. The question is worded as shown below (note that for this and all items tested, responses of "don't know" and "refused" are not explicitly displayed or read to the respondent, but they do exist as response categories):
CPS Current Question on Sex
What is [NAME's] sex?
Interviewers are instructed to ask this question "only if necessary" (U.S. Census Bureau, 2017b). If SOGI questions were implemented in the CPS, gender identity questions would likely replace the current sex question, and the sexual orientation question would likely be placed in the demographic section of the basic CPS. Because of this placement at the beginning of the interview, there is concern that respondents may refuse to answer SOGI questions and break-off from the rest of the CPS.
According to a multidisciplinary expert panel convened by the Williams Institute at the University of California, Los Angeles, known as the Sexual Minority Assessment Research Team (SMART), sexual orientation has three main dimensions: sexual attraction, sexual behavior, and sexual identity (SMART, 2009). In this report, we base "sexual orientation" on only sexual identity, rather than sexual attraction or behavior. The most commonly used terms to describe different sexual orientations are lesbian, gay, bisexual, and heterosexual/straight. In general, people who self-identify as gay or lesbian are primarily attracted to and/or have sex with people of the same sex. However, the concepts of sexual identity, attraction, and behavior do not always follow these patterns, or individuals may not want to identify as gay or lesbian even if they are attracted to the same sex or only occasionally have same sex relations (Federal Interagency Working Group, 2016a). As for gender identity, SMART defines this as "A person's internal sense of gender (e.g., being a man, a woman, or genderqueer) and potential affiliation with a gender community (e.g., women, trans-women, genderqueer)"1 (Federal Interagency Working Group, 2016a). Throughout this report, we use the acronym "LGBT" to refer to sexual and gender minorities, encompassing both sexual orientation and gender identity. For sexual orientation, we use "lesbian, gay or bisexual" or "LGB" as an umbrella term to refer to anyone who self-identifies as anything other than straight. For gender identity, we use "transgender" as an umbrella term to refer to "anyone whose gender identity differs from their sex assigned at birth." (GLAAD, 2017) It is important to note that any one individual may be LGB, transgender, both LGB and transgender, or neither LGB nor transgender.
1 See glossary for definitions.
Since at least the late 2000s, researchers have been exploring alternative methods for asking about sexual orientation in surveys. In 2009, the Williams Institute issued a report from SMART on best practices for question wording on sexual orientation as:
SMART Best Practices Recommendation
Do you consider yourself to be:
In the past few years, sexual orientation questions have been added to a number of state and Federal surveys. These generally use the wording that first appeared in the National Health Interview Survey (NHIS) in 2013, or a variation of it. The NHIS question is asked of all sample adults age 18 and over, and response categories vary depending on sex:
National Health Interview Survey
Which of the following best represents how you think of yourself?
[for male respondents]
[for female respondents]
For this study, we maintained the question stem used in the 2013 NHIS for the self-response version of the question, but for the proxy-response version, we added the precursor "To the best of your knowledge," For both versions, we consolidated the response categories as shown.
CPS Test Question on Sexual Orientation
[Self-response]: Which of the following best represents how you think of yourself?
[Proxy response]: To the best of your knowledge, which of the following best represents how [NAME] thinks of themselves?
There has been limited testing of question wording on gender identity, but studies thus far have demonstrated that transgender respondents are generally able to understand and answer questions on the subject (Baker & Hughes, 2016; Lombardi & Banik, 2016; Reisner et al., 2014; Cahill et al., 2014). In 2014, the Williams Institute issued a report addressing best practices for asking about gender identity in a survey based on research by a multidisciplinary expert panel known as the Gender Identity in U.S. Surveillance (GenIUSS) Group. The group recommended using a two-question approach by first asking for a respondent's assigned sex at birth, and then asking for their current gender identity (GenIUSS group, 2014). Other organizations, such as the Center of Excellence for Transgender Health (CoE) and the World Professional Association for Transgender Health (WPATH) also recommend using a two-question approach, but they suggest asking about current gender identity before sex assigned at birth (Federal Interagency Working Group, 2016b). Of the seven Federal surveys that ask about gender identity, three of them — the National Crime Victimization Survey (NCVS), the Survey of Prison Inmates (SPI), and the National Adult Tobacco Survey (NATS) — use the two-question approach. All ask the sex-at-birth question first, and all use slightly different wording and response categories, as shown:
Question 1: Sex at Birth
NCVS and SPI: What sex were you assigned at birth, on your original birth certificate?
NATS: What sex were you at birth? [Field Interviewer Note: "Did they tell you that you were born male or female?"]
Question 2: Current Gender Identity
NCVS: Do you currently describe yourself as male, female, or transgender?
SPI: How do you describe yourself (select one)?¦
NATS: Do you currently consider yourself to be:
In terms of measuring prevalence, under a two-question approach respondents would be identifiable as transgender if they selected different options for their sex at birth and for their current gender identity. In the NCVS and SPI, those who chose "transgender" would also be added to the tally. The NCVS also includes a follow-up question that has interviewers verify they are recording the correct answer if there is a mismatch between the two questions:
[IF Q1 ≠ Q2] Just to confirm, you were assigned [male/female] at birth and now describe yourself as [male/female]. Is that correct?
The other four Federal surveys — the Behavioral Risk Factor Surveillance System (BRFSS), the National Inmate Survey (NIS), the Population Assessment of Tobacco and Health (PATH), and the Health Care Patient Survey (HCPS) — use a one-question approach that asks directly about transgender status (though the PATH survey first asks a yes/no on transgender, then a follow-up on type of transgender). The HCPS also has a follow-up question for those who identify as genderqueer or "other" to determine skip patterns for subsequent survey questions. As with the sex-at-birth question, all three surveys use different question wording for gender identity, as shown:
BRFSS: Do you consider yourself to be transgender?
NIS: Are you male, female, or transgender?
PATH:
(a) Some people describe themselves as transgender when they experience a different gender identity from their sex at birth. For example, a person born into a male body, but who feels female or lives as a woman, would be transgender. Do you consider yourself to be transgender?
[if R answers YES]
(b) Do you consider yourself to be male-to-female, female-to-male, or non-conforming?
HCPS: What is your gender?
[If age >13]:
[If Genderqueer or other]: We have entered your gender as [Genderqueer OR other]. In this interview, questions will appear based on gender. For example, we only ask questions about mammograms to females of a specific age. Since this is a research study collecting medical-related data, could you tell us your biological sex at birth?
Under the one-question approach, respondents would be identifiable as transgender only if they selected that answer category explicitly, which could risk under-reporting compared to the two-question approach (Tate et al., 2012).
For purposes of this CPS testing, we followed the recommendation of the GenIUSS report and used the two-question approach. We also adopted the approach we used for the sexual orientation question when a respondent is asked to report about another household member by adding the precursor "To the best of your knowledge,"
CPS Test Questions on Gender Identity
QUESTION 1: SEX AT BIRTH
[Self-response]: Was your sex recorded as male or female at birth?
[Proxy response]: To the best of your knowledge, was [NAME's] sex recorded as male or female at birth?
QUESTION 2: CURRENT GENDER IDENTITY
[Self-response]: Do you describe yourself as male, female, or transgender?
[Proxy response]: To the best of your knowledge, does [NAME] describe themselves as male, female, or transgender?
With regard to age, we follow suit from the sexual orientation protocol and ask the question about those 15 and older. For those under age 15, we used the CPS current question on sex shown above.
In addition to issues of question wording, the OMB working group expressed concerns about "cultural and non-binary inclusivity" 3 noting that "Some individuals who vary in age, cultural and linguistic groups, etc., may not endorse terms such as 'transgender' when responding to Federal gender identity questions because they do not identify with this term" (Federal Interagency Working Group, 2016c). To address this, we recruited on a wide range of demographic characteristics and aimed to include respondents from various minority and non-minority groups, on characteristics such as race, ethnicity, and urban or rural residency. We also recruited roughly equal numbers of LGBT and non-LGBT respondents, and we included eight respondents who identified as transgender in our study.
Another measurement concern is how surveys should use pronouns such as "he" or "she" in subsequent questions. Transgender respondents vary in their preferred pronouns; for example, some respondents prefer the gender neutral "ze" (University of California, Berkeley, 2017). Surveys collecting gender identity need to decide whether to use the preferred pronoun based on gender identity, the pronoun aligning with sex at birth, or "they" as a singular, non-gendered pronoun (Federal Interagency Working Group, 2016c). After deciding how to use pronouns, surveys then need to implement this change throughout the entire survey, a task that is operationally complex and time-consuming. In this study, the option of using the gender-neutral pronoun "themselves" was used when a pronoun was used in the current CPS demographic questions, though for most questions the household member's name was used.
Finally, the working group expressed concerns about terminology evolving over time. We acknowledge this, and below we discuss findings on the fluidity of some of these categorizations even in a static context (that is, even from day to day, self-identities may change).
2 People whose gender identity falls outside of the categories of man and woman. (GLAAD, 2017)
3 The response option "None of these" is not read by the interviewer, but is available on a flashcard that interviewers may choose to display to respondents.
Many Federal surveys use proxy response, in which one person generally responds for all eligible household members, primarily to reduce costs and nonresponse (Tamborini and Kim, 2013). It can be very time-consuming and difficult to collect survey responses when all members of the household are required to answer for themselves (Pierce et al., 1993; Park, 2015). However, proxy response involves other tradeoffs, as proxy answers may differ from those provided by other household members. Across survey topics, evidence on the quality of proxy response in surveys is mixed. Respondents may use less precise question answering strategies such as estimation when answering about other people in the household (Bickart, Blair, Menon and Sudman, 1990). Data quality can depend on the question topic and the relationship between the respondent and others in the household. For example studies have found that respondents most familiar with other household members, such as spouses, tend to be better proxies (Kojetin and Mullin, 1995; Tamborini and Kim 2013; Grieco and Armstrong, 2014; Pascale 2016). Small differences in agreement between proxy and self-response answers are more common than large differences (Mellow and Sider, 1983; Boehm, 1989; Moore, 1988; Tamborini and Kim 2013).
With regard to SOGI questions in particular, very little is known about whether respondents have the knowledge necessary to report for other household members, and whether they would be willing to report the information if they do have it. Surveys in New Zealand and the United Kingdom do not permit proxy reporting for sexual orientation due to concerns about accuracy and confidentiality (Joloza et al., 2010, Park, 2015). One of the only studies conducted on this issue employed an online nonprobability panel to test SOGI questions similar in wording to the questions we use in this study (Ortman et al., 2017). That study, while it could not determine the accuracy of the responses, found overall low rates of nonresponse for the SOGI questions. Item nonresponse for these questions was lower than for income, which is also considered to be a sensitive question. However, nonresponse to the SOGI questions was also significantly higher for proxy reports compared to self-reports. Nonresponse rates also varied by relationship categories; rates of nonresponse were higher when respondents were reporting for children of respondents (age 16 or older) and roommates than for other household members, such as spouses and unmarried partners.
As noted above, the OMB working group raised the issue of survey context — that is, asking SOGI questions in the context of an employment survey — versus the more common context of health surveys. The concern was that if respondents view SOGI questions as irrelevant to the subject matter of the survey, they may refuse to answer the questions or break-off from the survey entirely. Thus far, there is very little evidence available in this area. In one study, after SOGI questions were added to the NCVS, researchers conducted a debriefing survey with interviewers about issues they experienced when administering SOGI and other questions. A relatively small percentage of interviewers reported that at least one respondent asked why sexual orientation or gender identity was relevant to crime (Truman et al., 2017). Nonresponse to these questions was low and only a few respondents broke off from the survey after being asked these questions.
Feedback from respondents in the present study provides insight into whether SOGI questions are seen as irrelevant in surveys about topics other than health.
We conducted 132 cognitive interviews in Washington, DC; Portland, OR; Nashville, TN; and Fargo, ND. The Washington interviews were conducted from September through December 2016, while the Portland, Nashville, and Fargo interviews were held in March, April, and June 2017, respectively. These cities were selected to represent different geographic regions of the country, with the assumption that these regions would also vary on attitudes, political experiences, and other factors that would impact respondent experiences and opinions. Interviews were conducted by staff from the Census Bureau, BLS, and Community Marketing, Inc. (CMI). Each interview was allotted one hour, but many were shorter, with an average interview length of 39 minutes. All interviews were conducted in person, and respondents received $40 each to compensate them for their time. Following standard Federal research procedures, OMB provided clearance for this study prior to the start of recruitment.
In order to have a more direct measure of the accuracy of proxy reporting, we also included 52 individuals from 26 unique households for "paired interviews." Each respondent in the pair was interviewed separately, providing information both about themselves, each other and (in households with three or more members) all the other members of their household. This allowed us to directly compare survey responses between household members to gauge accuracy based on match rates.
For the research carried out in Washington, DC, recruitment methods consisted of posted flyers, advertisements through Craigslist.com, a broadcast message sent to all U.S. Census Bureau employees who work in the Suitland, Maryland headquarters, and posts on Facebook pages for LGBT groups that featured a telephone number and email address to contact a recruiter for screening.
For the other sites, the Census Bureau and BLS established a contract with CMI to handle recruiting, onsite logistics, and administration of some interviews. CMI maintains a nationwide research panel of LGBT individuals who were recruited for this study. CMI also recruited new respondents using targeted Facebook advertisements to transgender Facebook users, flyers, and Craigslist.com advertisements. Print recruiting materials featured a telephone number, while digital advertisements directed participants to a CMI intake survey first. Intake survey responses were used to evaluate whether the respondent was potentially appropriate for the study.
All prospective respondents were screened via telephone prior to being scheduled for interviews. Screening included questions on respondents' age, race, ethnicity, employment, geographic area (urban versus rural4), household composition, and LGBT status. On the latter, to identify LGBT individuals we used screener questions that differed from the SOGI questions being tested.5 Single-person households were excluded because they would not be able to inform proxy reporting. The recruitment goals were as follows:
All goals were met or reached at least ninety percent of the target rate. The next section provides details on the characteristics of those successfully recruited.
4 For the majority of respondents, this classification was based on whether respondents' zip code fell within the bounds of Census Bureau defined urbanized areas (50,000 or more people) or urban clusters (2,500-49,999 people). If not, respondents were classified as rural. Respondents' self-description of their community was used to aid classification in a few instances.
5 To screen for the DC interviews, participants were asked for their gender (male, female or transgender) and whether anyone in their household over 15, including themselves, identified as LGBT. To screen for the other interviews, participants were asked for their gender (male, female or transgender), an open-ended sexual orientation question, and whether anyone in their household over 15, including themselves, identified as LGBT.
Tables 1 and 2 below show respondent and household characteristics, respectively, by site and interview type (individual or paired). In total, 71 interviews were conducted with respondents who were LGBT or had a household member who identified as LGBT, and 61 were conducted with respondents in non-LGBT households.
Interview Type | Total | Washington, DC | Portland, OR | Nashville, TN | Fargo, ND | |||||
---|---|---|---|---|---|---|---|---|---|---|
Ind. | Pair | Ind. | Pair | Ind. | Pair | Ind. | Pair | Ind. | Pair | |
N |
80 | 52 | 16 | 4 | 17 | 14 | 23 | 12 | 24 | 22 |
Age |
||||||||||
15-19 |
1 | 3 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 1 |
20-25 |
13 | 11 | 5 | 1 | 1 | 1 | 3 | 3 | 4 | 6 |
26-35 |
21 | 12 | 3 | 1 | 8 | 3 | 4 | 1 | 6 | 7 |
36-50 |
25 | 14 | 5 | 1 | 4 | 6 | 10 | 4 | 6 | 3 |
Over 50 |
20 | 12 | 3 | 0 | 3 | 4 | 6 | 3 | 8 | 5 |
Sex |
||||||||||
Male |
38 | 28 | 9 | 2 | 6 | 8 | 12 | 8 | 11 | 10 |
Female |
37 | 21 | 4 | 1 | 10 | 6 | 11 | 4 | 12 | 10 |
Transgender/other |
5 | 3 | 3 | 1 | 1 | 0 | 0 | 0 | 1 | 2 |
LGB/non-LGB |
||||||||||
LGB(6) |
31 | 33 | 5 | 3 | 7 | 9 | 9 | 8 | 10 | 13 |
Non-LGB |
49 | 19 | 11 | 1 | 10 | 5 | 14 | 4 | 14 | 9 |
Race/ethnicity |
||||||||||
White, non-Hispanic |
44 | 35 | 5 | 3 | 11 | 9 | 11 | 6 | 17 | 17 |
Black, non-Hispanic |
13 | 7 | 5 | 0 | 1 | 0 | 5 | 5 | 2 | 2 |
Other/multi-race, |
17 | 4 | 4 | 0 | 4 | 4 | 6 | 0 | 3 | 0 |
Non-Hispanic |
||||||||||
Hispanic |
6 | 6 | 2 | 1 | 1 | 1 | 1 | 1 | 2 | 3 |
Education |
||||||||||
< Bachelor's |
43 | 34 | 7 | 1 | 12 | 9 | 11 | 9 | 13 | 15 |
Bachelor's |
24 | 9 | 4 | 2 | 4 | 2 | 8 | 2 | 8 | 3 |
> Bachelor's |
13 | 9 | 5 | 1 | 1 | 3 | 4 | 1 | 3 | 4 |
Footnotes: (6) Seven of the 8 transgender respondents also identified as LGB. The one transgender respondent who did not identify as LGB is considered to be non-LGB here. However, this person is counted as LGBT throughout the report. The number of LGBT households is 65, and the number of non-LGBT households is 67. |
Interview Type(*) | Total | Washington, DC | Portland, OR | Nashville, TN | Fargo, ND | |||||
---|---|---|---|---|---|---|---|---|---|---|
Ind. | Pair | Ind. | Pair | Ind. | Pair | Ind. | Pair | Ind. | Pair | |
N |
80 | 26 | 16 | 2 | 17 | 7 | 23 | 6 | 24 | 11 |
Household size |
||||||||||
2 household members |
28 | 13 | 1 | 0 | 6 | 4 | 9 | 4 | 12 | 5 |
3 or more |
52 | 13 | 15 | 2 | 11 | 3 | 14 | 2 | 12 | 6 |
Household Composition |
||||||||||
Any non-family member |
23 | 10 | 8 | 1 | 5 | 2 | 4 | 2 | 6 | 5 |
No non-family members |
57 | 16 | 8 | 1 | 12 | 5 | 19 | 4 | 18 | 6 |
Age |
||||||||||
Any member age 15-25 |
39 | 14 | 9 | 2 | 8 | 2 | 12 | 4 | 10 | 6 |
No members age 15-25 |
41 | 12 | 7 | 0 | 9 | 5 | 11 | 2 | 14 | 5 |
Gender Identity |
||||||||||
Any member transgender |
6 | 3 | 3 | 1 | 1 | 0 | 0 | 0 | 2 | 2 |
No members transgender |
74 | 23 | 13 | 1 | 16 | 7 | 23 | 6 | 22 | 9 |
Sexual orientation |
||||||||||
Any LGB member |
34 | 20 | 6 | 2 | 9 | 5 | 9 | 5 | 10 | 8 |
No LGB members |
46 | 6 | 10 | 0 | 8 | 2 | 14 | 1 | 14 | 3 |
Race/ethnicity |
||||||||||
Any member non-White or |
40 | 13 | 11 | 1 | 8 | 3 | 14 | 4 | 7 | 5 |
Hispanic |
||||||||||
No members non-White or |
40 | 13 | 5 | 1 | 9 | 4 | 9 | 2 | 17 | 6 |
Hispanic |
||||||||||
Geographic Area(**) |
||||||||||
Urban |
42 | 16 | 11 | 1 | 9 | 5 | 13 | 5 | 9 | 5 |
Rural |
38 | 10 | 5 | 1 | 8 | 2 | 10 | 1 | 15 | 6 |
Household Income(**) |
||||||||||
Under $50,000 |
26 | 13 | 4 | 0 | 3 | 4 | 7 | 3 | 12 | 6 |
$50,00-$99,999 |
36 | 8 | 6 | 1 | 12 | 1 | 11 | 2 | 7 | 4 |
$100,000-$149,999 |
9 | 3 | 3 | 1 | 2 | 1 | 2 | 0 | 2 | 1 |
$150,000 or more |
8 | 2 | 2 | 0 | 0 | 1 | 3 | 1 | 3 | 0 |
DK/Refused |
1 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Footnotes: (*) Paired interviews had two respondents from one household; we count each household only once in this table. (**) Based on screener. |
All respondents were provided with a consent form before beginning the interview session. They were also told that information they provided would be confidential and audio-recorded, and were notified if there were any observers. Interviewers then explained that the purpose of the study was to test new questions developed for the CPS — the primary source of labor force statistics, like employment and unemployment, in the nation. No mention was made at the start about testing of SOGI questions. Interviewers then administered several different interview protocols shown in Table 3 and explained in more detail below.
Protocol | Task |
---|---|
Standardized Questionnaire |
Questionnaire consisting of 46 CPS questions (with SOGI items embedded) administered by a researcher as if it were a standardized interview (see Table 4 for topic areas and content) |
General Debriefing |
Semi-structured protocol covering the following themes:
|
Card Sort Exercise |
Provide 15 cards, each representing a specific question, including SOGI, disability, and income. Ask respondents to sort cards twice, once for sensitivity and then for difficulty. Cards identified as sensitive or difficult were ranked from most to least sensitive/difficult. |
Card Sort Debriefing |
Semi-structured protocol covering the following themes:
|
Question Specific Probing |
Semi-structured protocol covering the following questions:
|
Paired Interview Match Rate |
Calculate the frequency of matching answers between paired household members. |
Paired Interview Debriefing |
Semi-structured protocol covering the following themes:
|
Context Debriefing |
Semi-structured protocol covering the following themes:
|
Researchers administered 46 questions from five different questionnaire topic areas of the basic CPS interview (with the SOGI questions embedded in the demographics section) as if it were a regular interview. See Table 4 for a list of topic areas and question content. Employment and unemployment questions were included in the Standardized Questionnaire in order to convey the context of the CPS as a survey on the labor force. Disability and income questions were also included since these are generally more difficult and/or sensitive than other CPS questions. This gave us the chance to evaluate the relative difficulty or sensitivity of SOGI questions by comparing them to disability and income questions. No probes were asked during administration of the CPS questions, but a respondent's spontaneous, volunteered feedback was captured by the interviewer.
During personal interviews in the actual CPS, "show cards" are used for certain questions (relationship, educational attainment, race and income). Show cards include the full text of the question and response options and are presented to the respondent as the questions are asked. We used show cards for the questions that use them in actual CPS interviews, and we also included a show card for the sexual orientation question due to the length and complexity of the response options. A show card was not used for the gender identity question because it was brief and had fairly simple response options.
Questionnaire Topic | Question Content |
---|---|
Household Roster |
|
Demographics |
|
Employment |
|
Disability |
|
Income |
|
Interviewers debriefed respondents about their overall thoughts on the interview, ease or difficulty of answering the questions, and whether anything stood out or was bothersome. At this point respondents were still not told that SOGI questions were of particular interest, because we wanted to see what was salient to the respondent without prompting. Since this section was open-ended, some respondents offered their reactions at the question-level (e.g., they found the family income or the gender identity question to be sensitive), while others offered their reactions at the topic-level (e.g., they noted the disability questions were difficult, but did not specify a particular disability question). Interviewers did not probe for more specific details at this point, since this would be done in later sections of the protocol.
Often, but not always, a topic area consisted of just one question, as shown in Table 5. Thus, comparison between topic areas is imprecise because topic areas with more than one question provide more opportunity for the respondent to find the questions difficult or sensitive. However, we note that the range of the number of questions per topic area is small (one to three), with the notable exception of the employment topic area, which contained 17 questions. In the debriefings, however, only three specific items (on work last week, class of worker, and employer name) within this topic area were mentioned frequently in respondents' reactions. Nevertheless, the difference in number of questions per topic area will impact interpretation of the coding results, described in Section 3.4.
Topic Area | Number of Questions |
---|---|
Date of birth, age confirmation |
2 |
Sex at birth; Gender identity (asked only about 15 and older) |
2 |
Race and ethnicity |
2 |
Educational attainment (asked only about 15 and older) |
1 |
Employment (work-related activities in the past week) (asked only about 15 and older) |
17 |
Military service (asked only about 15 and older) |
1 |
Relationship to householder, unmarried partner/marital status (asked only about 15 and older) |
3 |
Family income |
1 |
Disability |
3 |
Respondents were presented with index cards listing 15 of the 46 administered CPS questions and asked to complete two sorting tasks for themselves — first sorting cards into two piles that were either "sensitive" or "not sensitive," then ranking the cards in the sensitive pile from most to least sensitive. A second Card Sort task was then administered, with respondents sorting cards into "difficult" and "not difficult" groups, ranking the difficult cards by most to least difficult. Thus, the same question could be categorized as both difficult and sensitive.
After both Card Sort Exercises, respondents were asked debriefing probes about why they found the questions sensitive/difficult when answering for themselves. They also were asked if they thought other members in their household would sort the cards differently, and if so, why. Finally, they were asked if they would sort the cards differently based on answering for others in the household instead of themselves, and if so, why. For these probes, respondents were not asked to re-sort, or move the cards, though a few did so.
Using a semi-structured protocol, respondents were asked about a subset of specific questions from the Standardized Questionnaire: disability,7 sex at birth, gender identity, income, and sexual orientation. For each of these, respondents were asked about their reaction to the question, the meaning of the question and/or specific terms within the question (such as "transgender"), their certainty and comfort in answering for themselves and for other household members, and whether they thought these other household members would be certain or comfortable answering for themselves and others.
7 Just one of the three disability questions from the Standardized Questionnaire was asked about during the Question-Specific Probing section. This question asked "Because of a physical, mental, or emotional condition, does anyone have serious difficulty concentrating, remembering, or making decisions?"
In the 26 households where two respondents ("Person A" and "Person B") were sampled, each was administered the full Standardized Questionnaire protocol. Thus both respondents were asked the same questions about each other, the household as a whole, and, in households with three or more people, about other household members.
A debriefing was conducted among each person in the pair to explore both SOGI and non-SOGI items from two perspectives: Person A's reactions to being asked to provide information about Person B, and also Person A's reactions to Person B providing information about them. During their individual interviews, both respondents in the pair were asked how comfortable they felt answering questions about the other person in the pair, and how accurate they thought their own answers would be about the other person. Respondents were also asked how they felt about the other person answering survey questions on their behalf, and how accurate they thought the person would be. Both in the pair were also asked whether they had any concerns about the other person answering on their behalf, and whether they thought their answers to any particular questions might not match.
Interviewers explained that the Federal government was considering adding SOGI questions to an employment survey. Respondents were asked what they thought about this idea generally, why they thought the government would be interested in collecting SOGI in an employment survey, and how they and other household members would respond if asked to provide this information in a government survey about employment.
The interview protocols produced both qualitative and quantitative data. The analysis of each is described below.
Six different protocols produced qualitative data: spontaneous comments made during administration of the Standardized Questionnaire, responses to Question Specific Probing, and comments from four Debriefings (General, Card Sort, Paired Interview, and Context). These data were analyzed for evidence of recurring themes and patterns. We also developed a coding scheme to flag the data as sensitive or difficult, and we coded whether the difficulty/sensitivity was reported or indicated in relation to a self-report or to a report about someone else. The coding schemes are described below. Note that in some instances, the difficulty/sensitivity was directly reported and other times the difficulty/sensitivity was indirectly indicated by the nature of respondents comments during debriefings. For convenience we use the term "reported" interchangeably with "indicated" but we note that "reported" does not necessarily mean a direct report.
For reliability, two staff members, working independently, coded the data from each of the protocols (shown in Table 3). Once the independent coding was completed, final codes were assigned through adjudication, either by a third staff member or by a discussion among the coders.
Results from each of the six protocols are shown in multiple tables below, specific to each section (e.g.: Difficulty for Self-Reporting). In addition to the individual protocol results, these tables show an "Interview Protocol Summary" column that is an indicator of difficulty/sensitivity in one or more of the six individual qualitative protocols. For the sake of simplicity, as we discuss results we draw primarily from the Interview Protocol Summary column. There are two caveats to keep in mind with this approach. The first, mentioned earlier, is the differing numbers of questions in each topic; for example, 17 employment questions were included in the test questionnaire, versus one income question. This discrepancy creates unequal opportunities for respondents to have difficulty or sensitivity within a topic area.
The second caveat is the design of the Question Specific probing protocol. Based on their expected level of difficulty and/or sensitivity, only four topics were selected for this protocol (SO, GI, income and disability). These topics received probes that other topics did not receive, again creating additional opportunities for respondents to have difficulty within a topic area. Thus, while the figures from the Interview Protocol Summary column represent an imperfect metric, it is convenient for making multiple comparisons, and results from individual protocols are maintained in the main results tables along with the summary column.
In general, across the six protocols, questions were coded as difficult if there was any evidence that respondents thought that they or others (within or outside the household) would be unable to answer the question, because either they lacked the relevant knowledge or they did not understand the question and/or certain terms in the question. Questions were coded as sensitive if there was any evidence that respondents thought they or others would be unwilling to answer them or if the respondent had a negative emotional reaction to the question. The evidence used for the coding was generally verbal — that is, respondents' responses to a debriefing probe or a spontaneous comment made during the Standardized Questionnaire.
In the Standardized Questionnaire protocol, respondents were not asked directly about the difficulty or sensitivity of any questions, but interviewers were instructed to make question specific notes for the SOGI questions as they were administering the questionnaire, including observations of verbal and non-verbal reactions from respondents. Interviewers were also instructed to make notes on demographics, employment, disability, and income topics. Coding of difficulty and sensitivity was based on respondents' verbal reactions while they were answering the questions. For example, a question was coded as difficult if respondents explicitly commented that it was hard for them to understand or answer, asked for clarification, answered "don't know," or voiced uncertainty about their answer. For sensitivity, if respondents volunteered that a question was upsetting or personal, or refused to answer, the question was coded as sensitive. In addition to volunteered comments like this, questions were flagged as difficult or sensitive based on interviewer notes about nonverbal behavior. For example, if an interviewer noted that a respondent was visibly agitated or upset when a particular question was asked, this would be flagged as a sensitive question.
A key goal at the beginning of the project was to identify and compare issues for self-reporting versus proxy reporting for others within the household. However, although we did not directly ask for respondents' opinions about people outside of the household, many respondents offered comments about this during administration of one or more protocols. Respondents frequently volunteered that they expected that a given question would be sensitive or difficult for members of a particular group (e.g., people who identify as LGBT), even if no one from this group lived in their household. For example, some respondents said the sexual orientation question was not sensitive for themselves or anyone in their non-LGBT household, but they thought the question would be sensitive to someone who is gay or lesbian. Respondents also said that unspecified "others" (not in the household) would, hypothetically, find a question difficult or sensitive, without specifying whether those others belong to a particular group. We coded these mentions of others as non-household members and distinguish them from proxy (i.e., within-household) responses. Ultimately we developed three categories to describe who the comment on difficulty/sensitivity referred to: self, proxy (within household), and non-household members.
For three of the Debriefings (General, Paired Interview, and Context) and the Question Specific Probing, coding of self, proxy and non-household was fairly straightforward and was based on how the respondent characterized the difficulty/sensitivity. Coding of the Standardized Questionnaire protocol was based on who the question referred to; respondents were asked separate questions for themselves and other household members.8 If issues arose when the respondent was answering standardized questions for themselves, the difficulty/sensitivity was coded under self-response; if the issues arose when answering questions for others the difficulty/sensitivity was assigned as either proxy or non-household member. For the Card Sort Debriefing, respondents' comments on their own rationale for their sorting and ranking were coded as related to self-response unless the respondent explicitly mentioned that their rationale was in relation to other people. When debriefed on whether other household members would sort the cards the same way, these comments were coded as related to proxy reporting.
The disability and family income questions were asked at the household level (i.e., "Because of a physical, mental, or emotional condition, does anyone have difficulty [insert task]?"). Thus, sometimes it was not clear whether the respondent found the question difficult/sensitive for themselves, or for other household members. For difficulty, we coded comments as pertaining to proxy response by default unless the respondent (1) expressed comprehension issues with the question or certain terms in the question, or (2) was explicit that their reaction was based on their own status, and not the status of anyone else. For example, if a respondent said the income question was difficult because they work multiple jobs and had to add up the income across jobs, responses were coded as difficult for self-response. But, if a respondent said they were unsure of their roommates' income amount, they were coded as difficulty related to proxy response. With regard to sensitivity, we coded comments as pertaining to self-response by default unless respondents were explicit that their reaction was based on the income or disability status of someone else, and not themselves. For example, in response to a disability question, if the respondent disclosed that a household family member has a physical or mental condition, and that this is upsetting to talk about, this would be coded as pertaining to a proxy response.
8 For several demographic questions, including the target SOGI items, questions were asked about only those 15 and older.
The outcome of a Card Sort Exercise is the list of cards included in each pile and the ranking each card was given (e.g., most and least difficult of the difficult questions). Interviewers captured both these data elements, and they were analyzed to identify the questions most and least often considered sensitive and/or difficult.
The answers from Person A and Person B were evaluated as a "match" or a "mismatch." If the answers given by each person in the pair were not the same, it was coded as a mismatch. For questions that had multiple response categories (e.g., marital status with categories of single, divorced, etc., or income which had 16 response categories) if the exact same category was not chosen by both respondents in the pair, it was considered a mismatch. The match rate was interpreted as a measure of accuracy.
The unique outcomes from these paired interviews were:
Table 6 provides a summary of results for difficulty across question topic areas from the six qualitative protocols. Each cell shows the number of respondents who expressed difficulty with each topic area and reporting type (self, proxy, or non-household). For example, during the Standardized Questionnaire protocol, two respondents expressed difficulty with self-reporting for the sexual orientation question, three expressed difficulty with proxy-reporting for someone within the household, and one indicated that "other people" would have difficulty. Note that a given respondent could report difficulty for multiple types of reports; that is, a respondent could find a question difficult to report for themselves, and also for someone else in the household. Also, note that the summary column is an indication of difficulty from one or more individual protocols.
Table 7 provides results on the demographic characteristics of those reporting difficulty for the sexual orientation (SO) and/or the gender identity (GI) items.9 Results are shown for each type of report (self, proxy or non-household members), and for LGBT versus non-LGBT respondents.
We first discuss results about difficulty in self-reporting, concentrating on these four areas: the sexual orientation question, the gender identity question, demographic differences of respondents reporting difficulty with the SOGI questions, and differences in difficulty between SOGI and non-SOGI questions. We then discuss results in the same four areas for proxy reporting. Finally, we discuss respondents' perceptions of non-household members' reactions to SOGI questions.
Interview Protocol Summary | Standardized Questionnaire | General Debriefing | Card Sort Debriefing | Question Specific Probing | Context Debriefing | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Self | Proxy | Non-HH | Self | Proxy | Non-HH | Self | Proxy | Non-HH | Self | Proxy | Non-HH | Self | Proxy | Non-HH | Self | Proxy | Non-HH | |
Sexual orientation |
14 | 20 | 6 | 2 | 3 | 1 | 4 | 0 | 0 | 7 | 7 | 0 | 10 | 14 | 5 | 0 | 1 | 0 |
Gender Identity |
11 | 6 | 14 | 3 | 0 | 0 | 1 | 0 | 3 | 8 | 4 | 0 | 9 | 3 | 11 | 0 | 1 | 0 |
Income |
31 | 85 | 1 | 8 | 10 | 0 | 2 | 3 | 1 | 14 | 43 | 0 | 22 | 75 | 0 | - | - | - |
Disability |
24 | 24 | 6 | 0 | 3 | 0 | 2 | 2 | 0 | 11 | 16 | 1 | 20 | 11 | 6 | - | - | - |
DOB/Age |
0 | 28 | 0 | 0 | 23 | 0 | 0 | 5 | 0 | 0 | 5 | 0 | - | - | - | - | - | - |
Race or ethnicity |
14 | 8 | 2 | 2 | 7 | 1 | 1 | 1 | 1 | 12 | 5 | 0 | - | - | - | - | - | - |
Education |
4 | 13 | 0 | 4 | 10 | 0 | 0 | 0 | 0 | 3 | 6 | 0 | - | - | - | - | - | - |
Employment |
24 | 21 | 0 | 13 | 14 | 0 | 6 | 2 | 0 | 5 | 9 | 0 | - | - | - | - | - | - |
Military |
1 | 2 | 0 | 1 | 2 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | - | - | - | - | - | - |
Relationship & marital status |
12 | 3 | 0 | 3 | 3 | 0 | 0 | 0 | 0 | 9 | 2 | 0 | - | - | - | - | - | - |
|
9 Results for SO and GI items were analyzed separately, but as the base number of respondents who had difficulty was low (21), and the demographic profiles of respondents who had difficulty with the SO item was similar to those who had difficulty with the GI item, results were combined into the summary Table 7. The difficulty results for each question are shown in Appendix A.
All Respondents(10) | LGBT Respondents | Non-LGBT Respondents | |||||||
---|---|---|---|---|---|---|---|---|---|
Self | Proxy | Non-HH | Self | Proxy | Non-HH | Self | Proxy | Non-HH | |
Total |
21 | 24 | 19 | 19 | 18 | 16 | 2 | 6 | 3 |
Site |
|||||||||
Washington, DC |
3 | 6 | 4 | 3 | 5 | 4 | 0 | 1 | 0 |
Portland, OR |
6 | 5 | 6 | 5 | 3 | 5 | 1 | 2 | 1 |
Nashville, TN |
3 | 3 | 2 | 3 | 2 | 2 | 0 | 1 | 0 |
Fargo, ND |
9 | 10 | 7 | 8 | 8 | 5 | 1 | 2 | 2 |
Household Size |
|||||||||
2 household members |
9 | 5 | 12 | 9 | 5 | 11 | 0 | 0 | 1 |
3 or more |
12 | 19 | 7 | 10 | 13 | 5 | 2 | 6 | 2 |
Household Composition |
|||||||||
Family household |
11 | 13 | 13 | 11 | 9 | 11 | 0 | 4 | 2 |
Non-Family household |
10 | 11 | 6 | 8 | 9 | 5 | 2 | 2 | 1 |
Geographic Area |
|||||||||
Urban |
15 | 17 | 10 | 14 | 13 | 10 | 1 | 4 | 0 |
Rural |
6 | 7 | 9 | 5 | 5 | 6 | 1 | 2 | 3 |
Age |
|||||||||
Under 30 |
12 | 9 | 7 | 11 | 9 | 6 | 1 | 0 | 1 |
30 and older |
9 | 15 | 12 | 8 | 9 | 10 | 1 | 6 | 2 |
Education |
|||||||||
Less than Bachelor's |
12 | 12 | 11 | 11 | 8 | 9 | 1 | 4 | 2 |
Bachelor's or higher |
9 | 12 | 8 | 8 | 10 | 7 | 1 | 2 | 1 |
Race/ethnicity |
|||||||||
White, Non-Hispanic |
14 | 12 | 15 | 13 | 10 | 13 | 1 | 2 | 2 |
Non-White or Hispanic |
7 | 12 | 4 | 6 | 8 | 3 | 1 | 4 | 1 |
Footnotes: (10) A total of 132 respondents were interviewed; 65 of the 132 respondents were LGBT, and 67 were non-LGBT. |
CPS Test Question on Sexual Orientation
[Self-response]: Which of the following best represents how you think of yourself?
[Proxy response]: To the best of your knowledge, which of the following best represents how [NAME] thinks of themselves?
There was little evidence from the qualitative protocols of respondents having any difficulty answering the SO question for themselves (see Table 6). Of 132 respondents, only 14 had any difficulty in any of the qualitative protocols. All respondents answered the question in the Standardized Questionnaire protocol, with only two indicating difficulty during that protocol. Most of the difficulty indicated during the Question Specific Probing protocol (10), followed by the Card Sort protocol (7).
When we asked what the question was asking in their own words during Question Specific Probing, all respondents were able to give an answer that indicated understanding of the question. Typical responses were:
In terms of choosing an answer, most indicated it was an easy question to answer.
Some respondents noted that it was an expected question:
However one respondent noted, "These are not so much difficult as unexpected." They caused her to hesitate and think about the answer, but she could provide an answer without a problem.
Most of the difficulty issues indicated throughout the interview protocols were expressed by LGBT individuals; 13 of the 14 respondents who indicated difficulty with the SO question were LGB and four were transgender. This amounted to half of the transgender respondents (four of eight) 11, and 13 of the 64 LGB respondents expressing difficulty. For the most part the difficulty had to do with the response categories.
One main reason for the difficulty was among those who felt sexual orientation is an inherently fluid concept. For example, one said they were "Uncertain about where I fit in the choices, I know my identity but it can be fluid but primarily I am gay." A related issue came from a respondent who, in the end, chose "bisexual" but said they are still questioning their orientation: "No one has asked me, still figuring out identity." Another issue was respondents saying the response categories were too limited. Three transgender respondents had a hard time reconciling ideas of straight or gay with gender fluidity, and they all chose "something else" as the category:
With regard to response categories being limited, some would have preferred something that fit them better than the catch-all "something else," such as "queer," "pansexual," or "asexual."
In six cases, respondents had trouble selecting from among the LGB categories (gay/lesbian, bisexual or "something else"). However, the categories they said they would choose would be one of the three LGB categories, and not the straight/heterosexual category, so the difficulty would not result in a misclassification as long as the intent was to distinguish LGBT and non-LGBT respondents.
11 To preserve confidentiality, we did not present the transgender-only results in a table.
CPS Test Questions on Gender Identity
QUESTION 1: SEX AT BIRTH
[Self-response]: Was your sex recorded as male or female at birth?
[Proxy response]: To the best of your knowledge, was [NAME's] sex recorded as male or female at birth?
QUESTION 2: CURRENT GENDER IDENTITY
[Self-response]: Do you describe yourself as male, female, or transgender?
[Proxy response]: To the best of your knowledge, does [NAME] describe themselves as male, female, or transgender?
Gender identity (GI) results were similar to those for the SO question. Only 11 of 132 respondents reported difficulty in any of the qualitative protocols when answering for themselves. All but one respondent answered the question for themselves during the Standardized Questionnaire, and only three indicted difficulty. Again, most difficulty was reported during the Question Specific Probing (9), followed by the Card Sort Debriefing (8).
When asked about ease/difficulty throughout the interview protocols, respondents tended to say things like, "Not much, these questions seem pretty easy to be answered, just straight up: male or female?" A few respondents expressed surprise at the addition of "at birth" to the sex question, and some also found gender identify question surprising, but they did not find them difficult, just unexpected: "I haven't seen that much before in surveys or questions."
When asked to define transgender in their own words, three respondents conflated transgender with sexual orientation (e.g., "If you feel comfortable with your sexuality."). But more typical responses were:
Those who did indicate difficulty were dominated by LGBT respondents; only one respondent was non LGBT (see Appendix Table A2), and seven of the 11 were transgender.12 One respondent found it difficult because they were currently questioning their own gender identity, but most of the difficulty expressed had to do with response options rather than an understanding of the concepts. Respondents either did not see themselves in the list of response categories offered (e.g., because they were gender-fluid), and/or they wanted to choose more than one category. Some respondents did not like the limitation of choosing only one category because "transgender" was excluded from "male" and "female." Others noted that male and female were biological concepts of sex and not exclusive from gender identities. Typical responses:
Difficulty did not result in unwillingness to answer. Respondents expressed that surveys were a place where they would be more willing to answer this question, even if they did not like the wording. One respondent, after thinking about the issues at length, concluded that there may not be a better a way and asking was more important than having a perfect question.
12 Unlike LGBT, where we had roughly equal numbers of respondents in the two categories (64 LBG and 68 non-LGB), we had a total of only eight transgender respondents in sample. Thus we cannot attempt to compare transgender and non-transgender respondents, but we note where all/most transgender respondents had similar comments.
In total, 21 of 132 respondents expressed difficulty for themselves with the SO and/or GI questions (Table 7). For the most part, the demographic profile of those who did and did not have difficulty with the SO and/or GI question was similar.
Nevertheless, some demographic differences were notable, even when considering the demographic distribution within the samples. Among the 21 respondents indicating difficulty, most were LGBT (19), White/Non-Hispanic (14), and/or urban (15). Furthermore, seven of the eight transgender respondents in the sample found the GI question to be difficult, and four also found the SO question difficult (data not shown).13
13Unlike LGB, where we had roughly equal numbers of respondents in the two categories (64 LBG and 68 non-LGB), we had a total of only eight transgender respondents in sample. Thus we cannot attempt to compare transgender and non-transgender respondents, but we note where all/most transgender respondents had similar comments.
Despite the caveats noted in Section 3.4.1 on design differences by protocols, and thus potential for some CPS questions to provide more opportunities for respondents to identify difficulty concerns, we did some high-level comparisons to understand the relative difficulty of the SOGI questions compared to the non-SOGI questions.
Results from the qualitative protocols show that across topics, the income item stood out as the most difficult, with 31 respondents expressing difficulty at some point across all protocols (see Table 6). Disability and employment questions were next (24 each). Race/ethnicity, sexual orientation, relationship/marital status and gender identify were next and fairly close, with between 11 and 14 respondents expressing difficulty across these topics.
For questions other those about SOGI, respondents who reported difficulty most often said that they had to think about before answering.
For some questions (such as educational attainment, and relationship to other household members), respondents reported having difficulty with the response categories, and these required more thought.
The question topic that most resembles the SOGI questions in relation to the reason for difficulty was race/ethnicity, where some respondents did not see themselves in the categories, and some had difficulty defining or deciding on the categories that fit them:
Quantitative results from the Card Sort Exercise were generally consistent with results from the qualitative protocols. Respondents put an average of 1.05 of the 15 cards into the difficult pile, with a zero card median. Sixty-eight respondents said that none of the questions provided on the cards were difficult, and an additional 29 and 12 respondents put only one or two cards in the difficult pile, respectively. Only 11 of the 132 respondents put four or more cards in the difficult pile. This indicates that most of the questions were not seen as difficult.
As shown in Table 8, the card most frequently sorted into the difficult pile was the question about income, with 46 respondents considering it to be difficult to answer. This was followed by two disability items, and then race and sexual orientation. Patterns from the card sort ranking task are similar. The item ranked most difficult for self-response by far was income (32). Race, and the questions about disability trailed far behind, with very few respondents ranking them as the most difficult to answer question.
Question | N Sorted as Difficult | N Ranked as Most Difficult | N Ranked as Second-Most Difficult | N Ranked as Third-Most Difficult |
---|---|---|---|---|
Income |
46 | 32 | 7 | 3 |
Disability (concentrating) |
19 | 5 | 11 | 2 |
Disability (errands) |
16 | 4 | 2 | 7 |
Race |
13 | 6 | 5 | 1 |
Sexual orientation |
8 | 3 | 4 | 1 |
Name of employer |
7 | 2 | 2 | 1 |
Marital status |
7 | 2 | 2 | 1 |
Gender identity |
6 | 4 | 0 | 1 |
Education |
6 | 2 | 1 | 1 |
Worked last week |
6 | 2 | 0 | 3 |
Sex at birth |
3 | 0 | 1 | 2 |
Ethnicity |
2 | 2 | 0 | 0 |
Date of birth |
1 | 0 | 1 | 0 |
Unmarried partner in household |
0 | 0 | 0 | 0 |
Armed forces |
0 | 0 | 0 | 0 |
When asked about SO for others in the household, only 20 of 132 respondents expressed any difficulty, all but one provided answers to the question, and only three respondents indicated some difficulty during the Standardized Questionnaire protocol. Reasons for the difficulty echoed the issues found for self-reporting in terms of choosing from among the response categories. Typical responses were:
about the identity of the younger people in the household (generally teenagers). These respondents thought they knew the answers, but were reluctant to rule out the possibility that the person had not decided on a sexual orientation:
One respondent who reported difficulty was a teenager and did not know their parents' sexual orientation because they had not talked to them about it. Other respondents thought that non-relatives or roommates who lived in their household would not have the knowledge to answer the sexual orientation about another household member: "The roommate does not know she is bisexual. He would probably say 'straight.'" A few respondents also speculated that other household members would have difficulty understanding the terminology; this was generally in reference to older household members: "She would not relate to the language used in the SOGI questions. Concepts would be foreign to her."
Qualitative results indicate that only six of 132 respondents had any difficulty reporting GI for other household members. No respondent indicated difficulty with proxy reporting during the Standardized Questionnaire protocol, and all respondents answered the GI question for all of their household members age 15 and up.
Three of the six who had difficulty had older household members who they thought would have trouble answering this question about themselves and about others because of the language in the questions. For example: "The older people in the house might be thrown off by the transgender questions." One of these respondents remarked that the older people in her household would have difficulty with the question, and also said that older people in general would have trouble with this question. Another respondent said their older mother would be confused by the terms used in this question.
Two other respondents said they lacked the knowledge about their roommates. One respondent thought he could answer for his roommate but was not 100 percent sure because they had not talked about it. The other found it difficult because she had new roommates that had moved in that week: "Think I know the answer but I don't know the roommates very well. They could be transgender, but I don't think so."
One respondent felt the questions would be difficult for other household members to answer because either the response options were not inclusive enough, or because they could not select more than one option. Two transgender respondents expressed uncertainty over whether other household members would identify their transgender identity in the same way they did: "I don't know that they would answer that I am transgender or male."14
14 Either response option would have correctly identified the respondent as transgender based on the assigned sex of female at birth.
As shown in Table 7, total of 24 respondents found SO and/or GI questions difficult when asked about someone in the household, and most often the difficulty was with SO (20) rather than GI (6).15 These 24 respondents were most often LGBT (18), from households with 3 or more people (19), and/or urban (17). Of the eight transgender respondents, three indicated difficulty with the SO and/or GI questions.
15 As the demographic profiles of those who indicated difficulty with the SO and GI questions were similar, results are combined in Table 7. Question specific tables are included in Appendix A.
Despite the caveats noted in Section 3.4.1 on design differences by protocols, and thus potential for some CPS questions to provide more opportunities for respondents to identify difficulty concerns, we did some high-level comparisons to understand the relative difficulty of the SOGI questions compared to the non-SOGI questions.
Findings from the qualitative protocols indicate that by far the most difficult topic area for proxy reporting was income, with 85 respondents reporting difficulty (see Table 6).16 Next was date of birth (28), followed by disability (24). The employment and SO questions were similar (21 and 20, respectively). Next were educational attainment and race, and GI was low with only six respondents reporting difficulty.
Across questions/topics, the most frequent reason for difficulty was response categories. For example, as with the SOGI questions, respondents had difficulty choosing among answer categories for the race question for other household members:
For disability, the difficulty was more about the question's intended meaning: "More explanation and examples, if you want to get into the natty gritty… Nowadays so many people have mental and emotional difficulties… Give examples, or define it a little bit more."
Lack of knowledge was sometimes a reason for difficulty, but answers of "don't know" for at least one other household member were mainly concentrated in the date of birth item (16). Education was next (5), followed by race/Hispanic origin (4). Only one respondent reported, "don't know" to the sexual orientation question for at least one other household member, and for gender identity there were no answers of "don't know."
The Card Sort Debriefing provided another angle on this same issue, because respondents were asked if other household members would sort and rank the cards differently than they did. About half of respondents (71 of 132) said other household members would identify only a subset of the cards they selected as difficult, or none at all. Some respondents (25) said that other household members would choose the same cards. The remainder (36) said other household members would select different or additional cards. Of these, the questions that respondents thought others would have difficulty with but they themselves did not have difficulty with were most often questions on income, disability, and sexual orientation. However, each were mentioned eight or fewer times in total across the 36 respondents.
The quantitative results on match rate complement the qualitative findings. The income item had the highest number of mismatched answers between pairs of respondents17 (18 of 26) (See Table 9). Educational attainment was the next most-often mismatched item (13). Contrary to some expectations, only two mismatched answers to each of these questions were between non-relatives; most of the mismatches were among family members (data not shown). Two of these family pairs were between parents and their teenage children. SOGI questions ranked toward the bottom in terms of mismatches; SO had five, and GI had the lowest number of mismatches with only two.
Question Topic | Total Mismatched Responses (n=26 pairs) |
---|---|
Income |
18 |
Education |
13 |
Employment |
8 |
Disability |
6 |
Age/birthdate |
6 |
Sexual orientation |
5 |
Race/ethnicity |
5 |
Marital status |
4 |
Gender Identity |
2 |
For the SO question in particular, two of five mismatched cases were opposite-sex partners where females identified themselves as "bisexual" or "something else" and males identified their spouse/partner as straight. In both cases, the females noted that their spouse/partner would report them (the females) as non-LGB. Also in both cases, the male partner at some point during the debriefing protocols said that their female partner might not identify herself as straight. In three cases, members of the pair answered the SO question about another household member differently. In two cases, both members of the pair identified a different household member as LGB but did not match on categories ("bisexual" versus "something else"). In one case, a parent and teenager were paired and were asked about another teenager in the household. The parent identified the other teenager as "bisexual," while the teenager in the pair identified the sibling as "straight". The teenager also answered "don't know" about the parent's sexual orientation.
For the GI question, all respondents matched on sex at birth and GI for the other person in the pair. Two pairs mismatched when reporting GI for other members of the household. One pair was reporting for a roommate.18 The second pair mismatched on GI categories for a third household member.19
16 As noted in Section 3.4.1.2, income and disability questions were asked at a household level. Unless respondents who had difficulty were explicit that the problem was with self-reporting, the difficulty was coded as related to proxy reporting.
17 The number of response categories, and therefore opportunities for mismatching, varied by question. The fact that income had the most mismatches may be an artifact of the fact that it had the most number of categories.
18 One person in the pair reported them as male/transgender and the other reported them as male/male but noted during the Paired Debriefing that they were non-binary or genderqueer and there was not a category for that in the question. If there had been an "other" or "non-binary" response option, they would have selected that.
19 One person in the pair listed them as male/female and the other as male/transgender. Despite the mismatch, the person would have been correctly classified as transgender.