Oftentimes when we share data from the youth survey we have people who question the accuracy of the data (hence the title of this blog). Which is great! It is awesome that people are thinking critically about the information they see. This is something I do on a daily basis.
We have answers to your questions!
Due to the nature of data collection, at CACTC, we could never say that data we collect for the youth survey is a complete and accurate representation of ALL 7th-12th graders in Cortland County. Kids might be absent that day, or choose to not take the survey etc.
However, it does provide an excellent (valid and reliable) point in time snapshot of over 1,800 Cortland County students in all five school districts that can be used to help drive decision making and evaluation of youth programs in our community. We are very lucky to have this resource available to us so we aren’t relying on anecdotes and guessing to make decisions.
There are several ways we ensure the data we collect and present is as representative as possible (and also help answer some commonly asked questions):
1. There are Built In Validity Checks
Before we get the survey data files, all of the surveys are sent out to the company Pride Surveys for initial analysis. They then throw out all the surveys (it’s only a small percentage!) that are determined to be invalid (aka they think a kid may have been dishonest). There are actually 5 different checks for validity that can get a survey thrown out. One example is if a student selects the highest possible usage for all types of drugs on the survey.
2. It is Anonymous
Sometimes people wonder if kids are scared to tell the full truth because teachers and parents may find out and they could get in trouble. However, the youth survey is completely anonymous which makes is less likely that students will underreport or provide dishonest answers. The survey is also administered online, but our survey company does not track IP addresses so confidentiality is still maintained.
3. We Look at Trends
Another way we can tell that most students are not lying (or not remembering correctly) is we look at yearly trends. For example, over the past five years reported 30 days use of alcohol has fluctuated between 23% and 21% with little noted changes. If all the kids were lying, it’d have to be a coordinated effort of hundreds of students over multiple years for us to get similar responses year after year.
With these built in honesty checks, we are able to provide schools and the community with a great idea of what is happening in the student body that is valid and reliable.
We are not relying on anecdotes or guessing what is happening. There are always limitations to survey data and we are continuously working to fill in these gaps to provide the most accurate data possible.
References:
1.Pride Surveys. New York State Partnership for Success Student Survey: Cortland County Grades 7-12. 2020.
