Questions and Answers on the National Youth Tobacco Survey: How We Collect and Analyze Data to Understand Youth Tobacco Use
Find out how FDA and CDC collect and analyze National Youth Tobacco Survey (NYTS) data to understand trends in youth tobacco use.
A Conversation with Karen Cullen, Ph.D., A Scientist at the FDA’s Center for Tobacco Products Who Leads the National Youth Tobacco Survey.
What is the purpose of the National Youth Tobacco Survey (NYTS)?
The NYTS is a large survey conducted each year that focuses exclusively on tobacco use among youth. FDA collaborates with the Centers for Disease Control and Prevention (CDC) to administer the survey to middle and high school students each year. We use it to monitor a variety of aspects of tobacco use, including trends in current tobacco use, exposure to tobacco marketing, susceptibility to use, use of flavored tobacco products, etc. We’re also able to learn whether there are differences in tobacco product use by different characteristics. In addition, the data collected each year are useful in tobacco prevention and control programs. Since it’s a large, nationally representative, repeated, cross-sectional survey, it’s a very valuable source of data.
What are some limitations you face and how do you overcome the limitations?
When the 2018 data were first reported we saw a huge increase in youth e-cigarette use from the previous year. One of the first questions we were asked was which brands of e-cigarettes youth were using. Unfortunately, although we’ve asked about brands in past surveys, we weren’t able to answer that question directly because we hadn’t asked youth which e-cigarette brands they used in the 2018 survey. Based on reports from schools and parents about the use of JUUL by youth, as well retail data showing surges in JUUL sales during the same time period, it is likely that the increase was a result of JUUL and JUUL-like devices on the market. We recognized the continued importance of asking about the brand of e-cigarettes that youth were using, so we added that question to the 2019 survey.
We’re also limited in how many questions we can ask, because the questionnaire needs to be completed in a single class period at school, which is about 40-minutes. For the 2019 survey, we moved from asking these questions on a paper-and-pencil scantron test to electronic data collection on tablets. With the paper and pencil, each question was asked to every student. So if we asked you about your cigarette use and you’ve never smoked a cigarette, you still had to go through all of the other questions about cigarettes and indicate that you never used them. In this new format, you skip those questions since they aren’t relevant to your situation. In addition to reducing the respondent burden, particularly among non-users of these products, we’ve been able to ask additional questions since transitioning to an electronic survey, without needing more time from the students to complete the survey.
Another change to the 2019 survey is that we’ve been able to include images of tobacco products before we ask about their use. For example, when we ask about cigars or e-cigarettes, we include non-branded pictures of these products in the questionnaire. This helps youth identify their tobacco product use more accurately, particularly given that many e-cigarettes come in a variety of different sizes and shapes, including those that look like USB flash drive. So those are big changes to the NYTS in 2019. We’re really excited about these improvements to our data collection, and we’re eager to see the results.
What is the most rewarding part of working on the NYTS?
Personally for me, it was seeing our Notes from the Field come out in November 2018, which showed the unprecedented surge in e-cigarette use during 2017-2018, and knowing that those data had a direct impact on the Commissioner’s Statement and the regulatory actions FDA announced. It’s rare that one gets to see their work have such an immediate and direct impact on policy that can improve the lives of so many. So that is what is personally rewarding for me. This is also great for the survey itself because it shows the importance of the work we put into it and its value. The NYTS provides a wealth of data that allows us to understand current youth tobacco product use in a larger historical context that can help inform policy decisions. This strengthens our justification to conduct the survey each year.
CDC conducted the survey from 1999-2009 on a routine, but not annual, basis. When CTP was established, we were able to help support the survey so that it could be conducted annually, which it has been since 2011. We work closely with CDC on every step of the process. I’m really proud of the work that our NYTS team members at FDA and CDC have done.
What is the most challenging part of working on the NYTS?
Whenever you are conducting an annual survey, you always have at least three years of surveys active at one time. You have last year’s survey—data that have been recently collected and are currently being analyzed. You have this year’s survey which is actively out in the field collecting data. And you have next year’s survey (or even the following year’s survey) that you’re planning for. It can be a juggling act.
Right now, we are thinking about the 2020 survey and the 2021 survey and what should be asked on those. Because there is a process that has to happen before the NYTS can go out into the field, there is a long time between when the questions are finalized and when students participate. Because the tobacco marketplace is constantly evolving, it can be challenging to keep the survey measures up to date. For instance, the language included in the 2018 survey that described e-cigarettes did not include JUUL in the list of example brands, because when the survey was finalized and at the Office of Management and Budget (OMB) for review, JUUL had not yet impacted the marketplace. The language around use of products evolves as well, such as use of the word “juuling” as a verb to describe e-cigarette use. It is difficult to keep the measures current while following the OMB review schedule.
A researcher might call a product an electronic nicotine delivery system and a kid might call it a vape or a JUUL. How do you communicate so kids will know what type of tobacco product you’re asking them about?
We provide a brief description of each tobacco product class (e.g., cigarettes, e-cigarettes) before we ask youth any questions about these individual products. For a product like an e-cigarette, we first briefly describe how the product works. Then we include examples of common brands, using sales data to identify the top brands within each tobacco product category. We are able to cover the vast majority of the e-cigarette market with these examples. Additionally, while the survey consistently uses “e-cigarette” to describe this product, we also provide examples of other names for e-cigarettes that kids may be familiar with. Beginning in 2019, we also include a variety of images for each individual tobacco product, which may help orient youth to this larger class of products. In the 2019 NYTS questionnaire the description is, “The next several questions are about electronic cigarettes or e-cigarettes. Some brand examples include JUUL, Vuse, MarkTen, and blu. E-cigarettes are battery powered devices that usually contain a nicotine-based liquid that is vaporized and inhaled. You may also know them as e-cigs, vape-pens, e-hookahs, vapes, or mods.”
We’re comfortable using the term e-cigarette because of the work done by other scientists here at CTP, CDC, and elsewhere in the scientific community. Their research shows that even though people might use one term when referring to their device, people generally understand that the use of the term e-cigarettes includes their specific device.
How have survey questions evolved over time to reflect the changing tobacco product market?
Well, aside from updating brand examples, we have made modifications to the survey every year. FDA and CDC staff work closely to update questions, ask new ones and remove ones that might need re-working. For example, we added a question in the 2019 survey that is important to us from a regulatory perspective: do youth perceive reduced nicotine cigarettes as being less harmful or just as harmful as regular cigarettes?
As the tobacco market has changed, we’ve adapted and expanded the questions that we ask. When e-cigarettes first came on the market, we asked about them as part of a general question on use of other tobacco products. In 2014, when e-cigarettes first became the most commonly used tobacco products among youth, we decided to remove e-cigarettes from that general question and ask about its use separately. We devoted a section of the questionnaire to e-cigarette use, so we were able to ask about ever use of e-cigarettes, curiosity about using them, susceptibility to using them, and current use of e-cigarettes. As their popularity has increased, we have increased the number of questions that we ask about them in the survey.
Can you say a little bit in plain language about how we get a nationally representative survey of kids using tobacco?
People read these estimates of youth tobacco use and might wonder how we get there. It’s a multi-step process. It can be complicated, but it’s easier to grasp if you think about it one segment at a time. First, you break the country down into different segments. And then within those segments, you sample schools and within those schools you sample the students. By doing it in a systematic way like this, you know the probability of selection at each level. You use those probabilities to figure out what the probability of selecting each student was. A sampling weight is assigned to each student—in other words, each student represents a number of other students. Conducting surveys this way allows us to get nationally representative data without having to survey every middle and high school student in the country.
Why do we ask youth about their tobacco use within the last 30 days? Do we think kids might not remember accurately if we ask about a longer timeframe?
In the NYTS, the question is, “During the past 30 days, on how many days did you use [that particular tobacco product].” There could be recall issues, even when asking about behavior within the past 30 days. Asking the question this way gives youth the ability to recall their recent tobacco use behavior while minimizing recall bias.
But if you asked about behavior in the past year, would the data have an increased likelihood of being incorrect?
It could—it depends on what’s being asked about. How often that behavior occurred and how memorable it is can play a role in recollection. For example, if it’s a really infrequent thing, like how many vacations did you go on in the past year? You’re probably really likely to remember and answer accurately. But if I asked, how many times did you go to Starbucks in the last year? Or how many times did you use social media in the past year? Those might be harder for you to answer accurately, depending on how often that event occurred. So asking about behaviors in the past 30 days provides enough time for the event to have happened while minimizing issues of recall.
I just wanted to touch on some of the other surveys that also measure youth tobacco use. Do you think it is helpful when they come up with similar results even though they have different survey methods? What if the results are different?
About a month after the 2018 NYTS data came out, data from the Monitoring the Future Survey came out. This survey is also conducted in schools, and it collected data from 8th, 10th, and 12th graders across the country. We were very eager to know: did they also see a huge increase in e-cigarette use from 2017 to 2018? And they did. It’s always nice to have validation that others see what you see, especially when big decisions are made from your results.
But having different results isn’t always bad. The differences could be because of differences in how the surveys are conducted. NYTS is asked of youth in schools, while other surveys are conducted at home. Previous research shows that household-based surveys find lower reported use of tobacco among youth compared to school-based surveys. Some youth who use tobacco may not feel comfortable responding to tobacco-use questions if their parents are close by. So we think there may be underreporting of socially undesirable behavior, such as tobacco use, in those surveys—"I’m not going to say I’m using cigarettes, not when my mom is sitting on the couch right next to me.”
Do we ever have different research priorities than CDC?
CDC and FDA do have different research priorities, so certain data are more valuable to one agency than another. But that’s because the two agencies have complementary, yet distinct, roles in addressing tobacco product use. FDA is involved in the regulation of tobacco products, while CDC works closely with states, communities, and partners to develop, distribute, and implement evidence-based tobacco control strategies.
For example, recently there has been a lot of interest in the use of marijuana in tobacco products, especially in e-cigarettes. Since FDA’s regulatory authority doesn’t include marijuana, there isn’t much we can do with that information, so it is more useful to CDC. Additionally, CDC is very interested in secondhand smoke exposure and household exposure, as several states and communities are seeking these data to inform smoke-free policies that prohibit e-cigarette use in public areas, such as worksites, restaurants, and bars. However, indoor smoking rules do not fall under FDA’s authority. There are also topics on the NYTS that are more relevant to our mission as a regulatory agency so we may work on those analyses separately. But we have many similar priorities, and we work on most analyses jointly.
We have a great team across CDC and FDA. I lived in Atlanta and worked at CDC before I came to FDA. Having that commonality has made the relationship that much stronger. When I’ve traveled to Atlanta to meet with them, it’s nice because I know the area and have mutual friends with their staff. We work well together because we talk every two weeks on the phone and see each other at conferences. Our missions are very similar and aligned for the vast majority of things related to tobacco. Even though we have distinct but complementary roles, all of us want to reduce the death and disease toll from tobacco use. It’s a really productive partnership.