Questionnaire surveys

First published in 2015. Minor uptates in 2023.

Introduction

In research methodology it is discussed which methods work best with regard to obtaining as sound knowledge as possible about the themes that the survey is intended to elucidate.

This includes methods for selecting the sample in such a way that the results may be generalised as accurately as possible to the population being studied, how the questions should be formulated to provide the most valid information possible about the facts at issue, and how the responses can be processed, analysed and interpreted to provide as correct a picture as possible of the phenomena being examined.

In research ethics, requirements are drawn up to prevent the research subjects, the respondents, from suffering any harm. Data should be stored and processed in a manner which ensures that unauthorised persons cannot find out what individual respondents have answered. The results should be published in such a way as to preserve the respondents' anonymity (cf. Data protection) and prevent individuals from being identified, unless otherwise agreed. The respondents should have given their consent to participate in the study, based on knowledge of its purpose (informed consent). Some will take the view that the researcher has a responsibility to give participants an insight into the results of the study, in return for the time they have spent in providing information to the researcher.

Ensuring individual anonymity

In general, ensuring participant anonymity is not a difficult requirement to fulfil in the case of a quantitative study, where the aggregate of responses, and not individual respondents, is the focus of interest for the research. The results are presented in the form of tables or statistical measures which make it impossible to distinguish individual respondents. Therefore this requirement seldom comes into conflict with methodological considerations in quantitative surveys, in contrast to the situation that may arise in qualitative interviews with a small number of respondents.

If the survey is to collect and store personal data, possible consequences for privacy must be assessed. The research institutions have routines for such assessments. The data controller can (and must in certain cases, see the Personal Data Act chapter IV article 35) consult with the data protection officer at the institution. Personal data represent information that may be linked to a specific person. This may, for example, be through a name or reference number that refers to a list of names, or by combining background information. 

Group-related indiscretion

One aspect of quantitative surveys that may be problematic is what might be termed group-related indiscretion. When results are reported for sub-groups in the material, for example persons in a particular age group or voters for different parties, the findings may in some cases be perceived by members of the group as negative or as socially burdensome. The researcher will in this context be faced with a dilemma with regard to which socially sensitive topics should be published in relation to which groups. For example, if a survey reveals that the proportion of people who say that they have evaded tax is higher for men than for women, few men will probably perceive this as a burden. If the same applies to a particular party's voters, this may be used against the party in the political debate. Respondents from this party may feel that the information they have given the researcher is being misused, with the possible consequence that they and other party sympathisers refuse to participate in subsequent surveys. This represents a possible methodological problem in the form of increased selection bias.

Informed consent

How much information is required in order to regard a consent to participate in a questionnaire as informed? When, for example, a telephone interviewer calls and asks to conduct an interview, s/he will state which institute is conducting the survey, say something about the purpose and sometimes also inform the recipient of the commissioning agency (or agencies; in so-called omnibus surveys these may be so numerous that for purely practical reasons it is difficult to name them). A problem related to providing information on the purpose and the commissioning agency is the possibility that this knowledge may influence responses to the questions. This is not necessarily the result of a deliberate strategy on the part of the respondents, but may be a result of thinking more consciously about what responses are expected. If the responses are influenced in such a way as to provide a misleading picture of the respondents' attitudes and perceptions, this constitutes a methodological problem (so-called control effect).

When weighing up ethical versus methodological considerations, one possible strategy is to wait until after the interviewing is completed before providing information that it is feared may influence the respondent. Where relevant, this is combined with providing the respondent with an opportunity to withdraw consent, so that the responses are deleted. Another possibility is to restrict the type of information provided, based on a more precise assessment of the information needed by the respondents to know what they are consenting to, as well as what information the researcher believes may be important for their decision as to whether or not to participate.

Information about the survey may also be a means to counteract another methodological problem, namely biases in the results that may arise if the response rate (the proportion of the sample selected who respond to the survey) is low. The information may motivate persons who are asked to participate to say yes. This is important for the representativity of the results, that is, how correct a picture these provide of the population from which the sample is selected.

What must obviously be avoided is "tricking" respondents, through deficient or erroneous information, into participating in projects which one may be reasonably certain they would not have consented to if they had been provided with accurate information. This could be political surveys that will be used by actors to attempt to influence public opinion, such as surveys conducted prior to the Norwegian referendum on EU membership in 1994. The purpose was to discover the type of arguments that might be effective vis-á-vis voters with a different view of the EU. Since the results were not published, they only benefited one of the parties in the debate. If this had been known, the respondents might have felt that they had provided information that was misused by the "opposition." A practice of using surveys with hidden objectives and concealment of the results is ethically debatable and, if it became known, could be detrimental to surveys as an instrument for collecting information, through increased mistrust and reduced willingness to participate.

Burden on informants – alternative data sources

The burdens to which respondents are subjected are a consideration when conducting questionnaire surveys. Questionnaires that may take two to four hours to fill out are not uncommon. In addition there are now so many surveys, particularly market research telephone surveys, that the same person may risk receiving several requests in the course of one year. The result of this can be observed in the form of declining willingness to participate, with the problems this creates for the representativity of the sample. Consideration of the burden on respondents as well as the quality of the survey results therefore calls for a greater restraint when it comes to conducting surveys. This is required particularly in relation to certain highly "popular" groups, such as members of elected bodies.

It must therefore be a primary requirement to investigate whether the relevant information can be obtained by other means than through a new questionnaire survey. The increasing number of comprehensive databanks makes it likely that there may already be surveys available containing relevant material. Another possibility may be to utilise other types of data sources, such as public registers or statistics, or various types of written material.

Requirement for relevance and meaningfulness

To avoid an unnecessary burden on respondents and prevent the depletion of this important source of data, researchers must avoid conducting surveys on topics of which the respondents have no experience, and which they therefore perceive as irrelevant and meaningless. The result is that they refuse to answer or answer somewhat randomly. This is regrettably rather common today, particularly with regard to market surveys. Such misuse of the respondents' time not only makes for the collection of data which is of little value, but also for irritation on the part of the respondents and thereby the risk of reduced willingness to participate in serious questionnaire surveys on important societal issues.

References

The first of the three Norwegian books (Haraldsen,1999) is an introduction to interview surveys, the second (Hellevik, 2002) an introduction to social science methodology in general, and the third (Hellevik, 2011) a book which provides a number of examples of how interview surveys are misinterpreted and sometimes directly distorted when they are presented in the media.

Of the three English titles, the first (Fowler, 2014) is a brief, widely used introduction to survey methodology, while the second (Weisberg, 2009) emphasises the need to examine the various sources of error that may create problems in interpreting survey data in context. The third book (Mutz, 2011) gives a popular introduction to an exciting new development in survey research, in which representative population samples are combined with experiments to test the effect of various ways of asking questions.

The first book is a brief, widely used introduction to survey methodology, with a separate chapter on ethical questions. The second book also contains such a chapter, which emphasises the need to examine various sources of error that may create problems in interpreting survey data in context. The third book gives a popular introduction to an exciting new development in survey research, in which representative samples are combined with experiments to test the effect of various ways of asking questions.

This article has been translated from Norwegian by Jane Thompson, Akasie språktjenester AS.