Hey, I Got Polled

Looks like many are confused by recent polling over the NSA story.

One of the things that I find frustrating about the current debate over the surveillance state is the constant bickering about what the polls show. One side cites the latest Washington Post poll, showing that most people are OK with having their phone records swept up by government computers; the other waves the latest Newsweek poll, which says the people believe the information warriors have gone too far.

They aren't alone. But I did actually get a call today from Gallup / USA Today to participate in a poll on the topic. I'll share some feedback from a participant's point of view.

First, they asked for the youngest male in the home at the time. Maybe that was for this specific poll, but it bothered me that they wanted the youngest view. In any event, it just so happened to be me at the time.

I was asked if I felt the President was doing too much, enough, or not enough as regards combating terrorism. They also asked if I was aware of the NSA story and how closley I had been following same.

Honestly, I found myself trying to think as the questions quickly came and went. I had to match thoughtful opinions to pre-determined terms. For example, on privacy issues and the NSA story, was I very concerned, somewhat concerned, concerned, not much concerned, not concerned at all, etc. I felt as though there were blanks, or context to be filled in around any particular view I held, obviously, I couldn't do that.

A question over the NSA program troubled me as it dealt with the government having all of my phone call information in their hands. If I hadn't been following the story, I might have assumed they meant the content of my calls, as opposed to simply the numbers existing within a database too large to even imagine. WOuldn't most people assume a phone record is an actual call?

They also asked something to the effect of, was I worried that the government might, or could be listening in on all of my calls.

Point is, I am concerned about privacy issues involving the current program in the news but I found myself trying to think how to answer under the context, as I have no issue with the program given what little I really know about it. And I've been following it fairly closely. Does being concerned mean you think it's wrong?

I believe they also asked about trade offs in privacy for security and how I felt.

They asked political identity questions and also if I thought Senate hearings were in order. They also asked about trust and President Bush. I self identified as an Independent but mentioned I was a right wing blogger, which got a polite laugh from the youngish and pleasant sounding pollster.

I want to be very clear here, I have no issue with Gallup and couldn't begin to put together a poll, but having gone through this one, I have to say that there seems to emerge a sort of tone. All the questions individually gave me every opportunity to respond on a scale of from very good to very bad. Yet, coming away from it as if it had been a conversation, it's impossible to feel other than someone was bascially asking me if I approve of the President in a context which suggested there was reason for concern.

I AM NOT SUGGESTING THE POLL WAS BIASED. And obviously my own bias has to be taken into account, but I did feel Bush was on trial in a sense, though not because of anything specifically asked, and certainly not the pollster's tone. It's sort of an impression left over from the whole dialogue. Does that make sense?

To be candid, given that a synapsis can fire this way or that and the the pace was fine, yet quick ... ten minutes earler or later, or a randomly different thought might have drawn out slightly different answers, though I can't say for sure.

The bottom line, I just participated in a Gallup / USA Today poll on the current NSA story and I'm still as concerned by the differing poll results floating around as I was at the start.

Comments

[dnichols]

http://hotair.com/archives/vent/2006/05/12/meet-the-nsa/

[nahncee]

don't you think they asked for the youngest male as being the one most likely to be drafted and sent to war if the draft is re-instated ... and therefore perhaps the most likely to disapprove of war and fighting terrorism?

[Byron]

I have done public opinion polling on a variety of issues, some specifically political, some not. In a poll of the kind you describe, it is difficult to interpret the results if you do not include questions that specifically query the respondent's knowledge about and interest in the issue being polled. This can be as simple as asking for a self-assessment of how closely the person says he has been following the issue, and of how he would rate his own level of knowledge about it. On this particular issue, I also would want to include a question or series of questions probing the respondent's opinion of the seriousness of the terrorist threat to the U.S. I suspect that much of the response variance would be explainable by variations in perceived seriousness.

I have not seen the interview schedule for this poll, so I do not know if these sorts of questions were included.

[A E Hansen]

"First, they asked for the youngest male in the home at the time. Maybe that was for this specific poll, but it bothered me that they wanted the youngest view. In any event, it just so happened to be me at the time."

Sounds like a intra-household random selection process. When you do a telephone survey you not only select households at random, you also have to randomly select a respondent from the household.

One common method for doing this the the Troldahl-Carter-Bryant (TCB) method. Here the interviewer would ask you 'How many adult are in the household?' and the 'How many of these are men/women?' Based on this information the interviewer would ask for a randomly selected adult from the household. (Most likely the respondent would be selected by the computer system the interviewer is using.) The person in the household for the interview would be identified by 'oldest man', 'youngest woman', etc.

There are also other similar randomizations methods also.

There is another method (the name escapes me) where they simply begin by asking for the 'youngest man', 'oldest man', 'youngest woman' or 'oldest woman' (randomly selected). If no one of the selected gender is in the household, the interviewer will then ask for the older or youngest of the other gender.

In any case, there is probably nothing sinister in their asking to speak to the youngest man in the household. If you really want to know more a great book on the subject is Sampling, Selection, and Supervision by Paul J Lavrakas.

[Jay]

Welcome to the absurdly sloppy world of surveys. I've stopped agreeing to participate, because like you, I'm sensitive to the wording of questions and find myself needing answers about the words used in the question before I can answer. Don't even get me started about questions with assumptions I would challenge. So rather than frustrate both myself and the interviewer, I beg off.

I'm also pretty hard to be around when playing Trivial Pursuit. Some questions are worded so that more than one correct answer is possible. They just are.

I'm happily married, really.

[Electrolux]

Yeah, the point of asking for someone based on age and gender is to reduce selection bias.

I think this technique is a good idea, but at the same time I don't take opinion polls very seriously as a scientific tool. AFAIK there is no real attempt to estimate what systematic variations occur due to the way a question is asked. Really they ought to come up with three or four different phrasings for each question and report these results separately. This would give a decent idea of whether the systematic variation is really no larger in size than the random statistical variation (as is implicitly assumed).

[Byron]

Electrolux is right in principle. The problem is that there is no metric of semantic content that would measure how different two questions are -- except for the way people respond to them, which is what you're trying to account for in the first place.

It is beyond dispute that question wording and question ordering (which questions lead up to and set the stage for a particular question) can have effects that swamp sampling error. Techniques to mimimize sampling error are well known and understood; minimizing question bias is an art, at best, and there is lots of opportunity pushing results in one direction or another. The polling done by different committed interest groups can be a kind of natural experiment, comparing, say, the results gathered by pro- and anti-abortion groups.

When sampling error is likely, a critic can sometimes make an argument in the form of a mathematical adustment to the results, suggesting what they might have looked like minus the sampling bias. But if it's question bias, about all you can do is commission another poll using different questions. That can lead to a battle of competing biases, with readers deciding that the true state of public opinion must be somewhere in between, which is not necessarily the case.

[FEATHERHEAD]

Humph! I have to say what I always say about polls.I don't trust them, because, nobody asked me.If I were questioned about a trade-off between security and privacy,naturally, I would have to go in favor of the security.Because, let's face it, the 9/11 attackers had way too much privacy, didn't they?

[E. Walton]

From the point of view of someone that (sadly and thankfully briefly) was a telephone pollster right before the last election, I can tell you that the younger male demographic (18-25) was alway the last group to get the needed numbers from. They probably were just trying to fill their quota on that part.

I was hired by a private company and we were never even told who commissioned the polls we did. But with some polls we KNEW who had commissioned it just by the questions while others were more unbiased.

[the punisher]

First of all, I don't trust polls. They can be biased or fixed to get the desired results. Second of all, whether we like it or not, the NSA will continue to monitor all our calls, emails, faxes and even our letters, whether legal or illegally. Third of all, the bs they are saying that because of 9/11 they have to do it, they were doing these things long before 9/11, like during the cold war and it still didn'y prevent 9/11. Its to have the power of information to control, blackmail politicians/people and for marketing purposes that the NSA exist.

[SCSoxFan]

Did any the poll questions ever mention that the NSA program was being used to try to track terrorists? I saw this as the major difference in the Wash Post & Newsweek polls. The Post poll question clearly stated that the program was used to try to track terrorism. The Newsweek post merely described how the program worked, it never said what it was being used for. The only mention of terrorism was in one of the potential answers. I think most Americans, if they understand the terrorists are the target, and more acceptable of this type of program as opposed ot general government snooping. So, again, was the PURPOSE of the program described in the Gallup questions?

[Bill]

I have been a continuously registered voter for almost 40 years.

A few comments re polls:
1) The only time I was ever called by a pollster was re a specific local issue a few years back that was related to my profession -- obviously a special interest group (probably PAC related to my profession) which wanted MY personal opinion - I caught on and immediately gave them the 'professionally correct' answer and a couple months later voted THE OPPOSITE!!!!
2) First real experience with polls and the typical slimey questions asked therein was during college freshman enrollment and orientation in the mid 1960's -- 2 or 3 written polls (which were conjured up at some liberal psychobabling think tank) given to many freshmen across the country asking all kinds of aditudinal, opinion, substance use, sexual, familial and social orientation types of questions -- all of which the majority of us at that time were wise to and just put down a bunch of BS answers -- most of the crap they were asking was no one's business anyway. You know the types - every fall they come out with the "xx% of college freshmen think/do/have done/may do this or that" press releases -- usually timed by the MSM to reinforce the liberal slant to some crisis dujour.
3) A lot of these questions are carefully crafted to make the responder feel self-guilt self-doubt or anxiety that some how they "might not be fitting in". Reread the last 4 paragraghs of the above post - to see how the seeds of doubt were planted in the pollee's mind.
4) Since they only talk to a 1000 or so people in a given poll -- the pollsters certainly can record all of the phone conversations -- what a load of crap -- asking someone their opinions re gov't receiving records of called phone numbers from your number - couched in a phrasing to imply that the gov't is recording every phone call -- while the only phone call that is actually being recorded is the one from the damm pollster.