So you’ve read the Don’t Ask Don’t Tell survey (which can be downloaded here), and you’ve felt vaguely negative about it but didn’t have the words to explain why. It’s alright; I felt the same way! Luckily, I had a great Research Methods teacher (holler Professor Kiter-Edwards) and a text book by one Earl Babbie that is honestly one of the most enjoyable text books I’ve ever had to read.
We’re going to focus on a few of the biggest problems going on with the survey because we’re smart people, obviously, but we don’t want to be showoffs.
Full disclaimer: I’m only 3/4 of the way through my degree in sociology, which means that I’m ignorant of a full quarter of the information that I should know. Maybe you know more than me. If you do: spread the wealth! Let’s talk about what works and what doesn’t.
Issue Number One:
WTF is the research question?
Surveys can be done for three main reasons: exploration, description, or explanation. If you’re exploring, you’re saying “we’re not really sure what’s going on so we’re going to find out some stuff so that we’re not bumbling fools.” If you’re describing, you’re asking what. And if you’re explaining, your research question is going to be about why. According to Geoff Morrell, pentagon spokesman extraordinaire, they are “not playing games here, we’re trying to figure out what the attitudes of our force are, what the potential problems are with repeal.” In other words, they’re exploring.
Exploratory research is usually done with subjects that no one knows anything about. If you want to know how many times a year the average person eats Nutella or if aliens might exist, you would do exploratory research. The problem with this approach to the DADT problem is that exploratory research doesn’t really provide any answers. It’s meant to act as a primary indication of how things might be and to be a stepping stone for further research, but the issue here is that the military seems to be viewing it as comprehensive.
What’s most problematic is that there doesn’t appear to be any sense of direction. Are they planning on using this information to stage more in-depth research about why people feel the way they feel, or are they just going to throw out the House’s bill if enough people say that they wouldn’t want a gay tent-mate?
The survey asks how people feel about gay shower buddies and lesbian neighbors but neglects to ask why people feel the way they do. This isn’t a survey flaw as much as it is an indication of the general attitude of the country. To probe into why people are homophobic would be to admit that there’s nothing wrong with gay people and that’s something that the government’s not ready to do.
It’s important to point out that while we’re all quick to find fault with the survey, it could help us in the end. If the results are supportive, it just might mean that all that exploration was worthwhile. In defense of the idea that gays serving openly in the army ain’t no thang, I’d like to present you with a choice quotation from the beacon of truth, “Given its fundamental nature, exploratory research often concludes that a perceived problem does not actually exist.” Bam, roasted.
Glaring Issue Number Two:
Validity and reliability
Let’s talk about validity and reliability and the difference between the two. Validity means that the results accurately describe what is being studied, while reliability means that the measures used will give consistent results in different populations and at different times. To illustrate:
Are you still with me? Great! Now let’s talk about the types of validity. Did you know that this is on the first page of results that you get when typing “face validity” into Google images?
Anyway, face validity means that the instrument used (which in this case is the survey) makes sense to the population to whom it is given. What does this mean to you? Another vocab word that I’m going to throw at you is content validity, which means that the survey is a good measure of the concept being studied. Keep these two guys in mind because we’re going to come back to them when we get to survey design.
My major complaint in this department is with reliability. One of the ways to guarantee reliable responses is to used established measures — instruments that have already been used. I know it sounds old fashioned and conservative to want them to use a tried and true method, but seriously can I get a Guttman scale up in here?
A Guttman scale is a series of questions that works on the idea that if you identify positively with items that come later on the list, you should logically be copacetic with those that came before. One of the most famous examples of a Guttman scale is the Bogardus social distance scale, which goes like this:
Please place a check by all the statement with which you agree:
[ ] It is alright for people to immigrate to this country.
[ ] It is alright for people to immigrate to this state.
[ ] It is alright for people to immigrate to this city.
[ ] It is alright for people to immigrate and live on this street.
[ ] It is alright for people to immigrate and live next door.
Guttman scales are great because they help researchers compiling the data to quickly find the relevant data. If someone says that they’re comfortable with immigrants living on their street but uncomfortable with immigrants living in their state, you know that they’re either not paying attention, not making sense, or insane — you can throw their response to this question out. Another perk is that they’re straightforward and don’t leave a lot of room for interpretation for the researcher.
Por ejemplo, here’s a question from the actual survey:
But what do those answers mean? If someone says they’re going to take no action, does it mean that they’re down with gays or that they’re just too afraid to do something? Maybe they discuss how they expect others to behave with every new service member or always shower by himself. If they talk to a mentor, does that mean that they’re simple struggling with accepting their new gay comrade and trying to work to accept them or does it mean that they’re trying to pray away the gay together? The only thing we really learn from these answers is who is a tattletale, who keeps to herself, and who’s bossy.
This survey could have had a similar scale that went something like:
[ ] It is alright for gays and lesbians to serve in the military
[ ] It is alright for gays and lesbian to serve in my branch
[ ] It is alright for gays and lesbians to serve in my division
[ ] It is alright for gays and lesbians to share my tent or room
[ ] …
But instead we have all this nonsense about talking to chaplains if you have to poo near a man who you think might be gay jumbled up with other questions about fulfilling a mission during combat. My head is going to explode, so let’s just move along to the next section, shall we?
So Shiny It’s Blinding Us Issue(s) Number Three:
Survey construction
There’s a lot going on here so let’s look at it little bits at a time.
+ Choose appropriate questions
Know what one the most unclear type of survey question is? Close-ended questions. Wanna take a stab at what type of question appears most often on the survey? If you guessed close-ended questions, you’re already better at social research than most of the people at Westat, that research firm that’s being paid $4.5 million to conduct the survey.
Close-ended questions are items that already have a group of answers that you have to choose from. Besides forcing the researcher to decide what the data means (like we saw in the bathroom question), close-ended questions are often difficult to word so that the options are exhaustive and mutually exclusive. The exhaustion issue is taken care of in some questions with the addition of a “something else” option along with a box to fill in, but mutual exclusivity doesn’t appear to be at the top of their list of important things. They’re a big fan of the “check-up-to-three”-style of response, which isn’t necessarily a fallacy but does reduce the reliability of the survey.
+ Respondents must be willing to answer
In the U.S., we tend to be bigger fans of rationality, which is often equated with being moderate. If someone feels like they might be in a minority when selecting a response, they may choose fitting in over honesty. Intern Emily wondered exactly how the survey was being administered, which is a totally valid concern. The pressure to remain in the majority might be even stronger if you’re filling out a survey while others are around.
+ Questions should be relevant
Beside giving me titles for all these headings, Mr. Babbie says “when attitudes are requested on a topic that few respondents have thought about or really care about, the results are not likely to be useful.” This goes back to the question of how much troops really care if they’re serving with gays and lesbians. If you’re in a combat zone, how often are you thinking about whether the girl next to you likes girls or boys. If the answer is a lot, and you’re a guy, please stop watching bad porn and get your head in the game. You’re at war.
+ Short items are best
There are a lot of things that this survey is, but short is not one of them. Just for fun, let’s say that there’s nothing wrong with the questions, the survey’s just too lengthy. Potential respondents might be daunted by the 32 page document and decide not to take it at all. Here’s an idea, how about about combining these three questions into one question with three parts?:
+ Avoid biased items
Remember five minutes ago when we discussed validity? It’s time to use your knowledge of face validity to look at the language used.
What a bad idea. If you haven’t yet, just read this study and take a moment to think about how you might think people would respond differently to words like “assistance to poor” and “welfare,” or “dealing with drug addiction” and “rehab.” As far as biased language goes, there’s also the issue of wording used that suggests that queerness is something that must be tolerated, which Riese talked about a few days ago. It’s worth nothing that according to the Riddle homophobia scale, another famous sociological scale, the survey’s attitude toward gays and lesbians falls under the “homophobic” category.
And so there you have it. I feel happy that I’ve put my college education to good use before even graduating, which I hear is rare these days. So what happens now? Towleroad has a great round-up of some of the reactions to the survey, including the Pentagon defending it, and Nate Silver of fivethirtyeight.com saying that that “parts of it are completely useless.”
Let’s end on this note from Rob Smith, a gay Iraq War Veteran, mocking the survey:
“The way some of the survey questions are structured is enough to make one think that its creators are as obsessed with gay sexuality as those who practice it regularly. In fact, the survey really hit the nail on the head with the whole shower thing. I wasn’t able to shower for the first three weeks of my tour in Iraq, and what do you think I was looking forward to the most when I finally got the opportunity to take one? Was it perhaps the opportunity to remove the thick film of gruel that encased my skin no matter how many times I wiped myself down with the wet naps provided with our meals? If you thought that, you were wrong. It was obviously the opportunity to sneak a peek at other soldiers in the showers, soldiers who were equally if not more as disgusting as me at that point. Sexy, right? I sure thought so, but imagine my SHOCK that there were private showers! In Iraq! It was almost enough to make me want to give my two weeks’ notice right then and there.”
Equality California wants you to petition Defense Secretary Robert Gates, requesting that a more fair, unbiased survey be given to servicemen and women. EC’s Government Affairs Director, Mario Guerrero, had this to say:
The survey is insulting, one-sided, and designed to illicit a negative response.
As a veteran of the U.S. Marine Corps, I know the effects of Don’t Ask, Don’t Tell all too well. My sexual orientation was not an issue for over six years until I was outed. The military’s Don’t Ask, Don’t Tell policy prevented me from joining my fellow marines in Bagdad. Unfortunately, they were not able to rely on my training and experience.
Time is of the essence. The Senate could vote on the repeal as early as this week. And a federal court challenge starts today, brought by Log Cabin Republicans.
It’s not really a petition so much as it’s a form letter, which means all you have to do is fill in some fields and hit send. Pass it on, little queerios!