Tools for Survey Researchers

Several instruments were developed to examine attitudes, beliefs and behaviors related to teaching, learning and assessment at both the insitutional and individual levels:

The Student Survey

The Faculty Survey

Checklist for Survey Administrators

References on Survey Research

The Undergraduate Student Survey

Student Experiences with Teaching, Learning and Assessment (SETLA)

The purpose of the Undergraduate Student Survey was to provide campus administrators with insight regarding issues that affect student learning. The questionnaire asked undergraduate students to report their exposure to teaching, learning, and assessment strategies used by faculty in the classroom, and then document the extent to which they found these methods effective. Additional items assessed how students spent their time during the 2000-2001 school year, including the number of hours they devoted to studying, attending class, using computers, and participating in activities outside of the classroom. Students also provided estimates of their personal and educational growth in areas such as general education, written and oral communication, individual and social development, and critical thinking.

The survey asked students about their perceptions of the intellectual atmosphere on campus. In addition, the students reported course-taking patterns, educational plans for the future, and whether they were satisfied with undergraduate educational experiences. The survey collected demographic information, including gender, race/ethnicity, family status, and socioeconomic status. Personal background information and pre-college educational experiences were also documented.

Ideas to Link Factors with Variables in the Survey

Take a look at the variables and factors extracted from the student survey data.

TOP

The Faculty Survey

Faculty Survey on Teaching, Learning and Assessment (FSTLA)

The purpose of the faculty survey was to examine institutional support for undergraduate teaching, learning and assessment. Survey results provided a profile of faculty members’ professional backgrounds, responsibilities, workloads, attitudes, and perceptions during the 2000-2001 academic year. Faculty members reported their perceptions of student expectations, abilities, learning styles, and preparation. The survey also documented perceptions of institutional and departmental reward systems, governance structures, and the quality of facilities and resources. Several questions captured faculty attitudes toward assessment, perceptions of the institutional climate for innovative teaching, learning, and assessment practices, as well as satisfaction with several dimensions of the academic work environment.

The Faculty Survey on Teaching, Learning and Assessment further asked faculty to report their frequency of engagement in teaching, research, and service activities. This included the extent to which they used innovative teaching and assessment practices in the classroom. Additional questions tapped faculty participation in professional development initiatives on campus as well as their involvement in academic planning and interdisciplinary activities. In addition, the survey collected demographic information such as age, gender, race/ethnicity, family status, education, and employment background.

Ideas to Link Factors with Variables in the Survey

Take a look at the variables and factors extracted from the faculty survey data.

TOP

A Checklist for Survey Administrators

Preparation

Conduct a thorough review of the literature. Search peer-reviewed journal articles, books/monographs, conference papers, and other relevant references.
Organize the literature around the topic. Design a conceptual map of the literature.
Develop research questions or objectives based upon theory
Develop a conceptual framework to guide the research, preferably a visual representation. If using a diagram, place the dependent variable on the right side and the independent variables toward the left. Insert arrows to represent the hypothesized relationships between variables.

Budget Considerations

Staff time to plan and implement the study.
Sample selection costs.
Labor and material costs for pre-testing the questionnaire and field procedures.
Supervisory costs for interviewer hiring, training, and monitoring (if needed).
Interviewer labor costs and travel expenses, including meals and lodging.
Expenses associated with redoing some interviews, or following-up on non-respondents.
Labor/material costs related to data entry.
Cost of reviewing the quality of the computerized paper questionnaires.
Cost of “cleaning” the final data set.
Labor or material costs for data analyses and report preparation
Telephone charges, postage, reproduction, and printing costs for all stages of the survey – from planning activities to distribution of the results

Questionnaire Design

Determine a timeline for administering the survey
Plan each area of the survey, including the cover letter and closing instructions
Determine what variables the survey will target (e.g., demographic, attitudinal, behavioral, factual)
Variables should cross-reference the research questions and survey items
Design the survey questions.

Define objectives. Specify the kind of answers needed to meet these objectives.
Make sure all respondents will share a common understanding of each question
Be reasonably sure that respondents will know the answers to each question.
Ask questions that respondents can answer using the terminology provided.
Ask questions that respondents are willing to answer accurately.

Decide what types of measurement scales will be included

Rating scales (e.g., strongly agree to strongly disagree)
Categorical scales (e.g., yes, no)
Rank-ordered scales (e.g., rank from highest to lowest

Field test the survey instrument.

To establish face validity.
To improve survey design.

Sampling

Determine who will be sampled and what is to be learned about the sample.
Decide how many people will be in the sample, and whether the sample will be randomly selected

Data Collection

Select a method of data collection

Surveys
Use a standardized (commercial) or locally-developed instrument
Administer on location, by mail, or on-line (web survey)
Exams (machine readable)
Qualitative approaches (focus groups, interviews, video taping, etc.)

To ensure a high response rate to mailed surveys:

Initial mailing – cover letter and questionnaire
Second mailing – second letter and questionnaire
Postcard follow-up – reminder

Data Processing – Confidentiality Issues

Use only number codes to link respondents to questionnaires
Store the name-to-code information separate from the questionnaires
Do not share respondent information with anyone outside of the survey group
Destroy questionnaires and identifying information following data entry
Omit names of survey respondents from computer files used for analysis
Present statistical tabulations using broad categories so individuals cannot be identified

Data Analyses

Analyze returns
Check for response bias – the effect of non-responses on survey estimates
Collapse terms into scales using factor analysis
Check for reliability of scales, as needed
Run multivariate statistics to answer the research questions, as needed

TOP

References on Survey Research Method

Converse, J.M., & Presser, S. (1986). Survey questions: Handcrafting the standardized questionnaire. Sage University Paper series on quantitative applications in the social sciences, 07-063. Newbury Park, CA: Sage Publications.

Creswell, J.W. (1994). Research design: Qualitative and quantitative approaches. Thousand Oaks, CA: Sage Publications.

Dillman, D. (2000). Mail and Internet surveys: The tailored design method. New York: Wiley.

Fowler, F.J. (1988). Survey research methods. Newbury Park, CA: Sage Publications.

Fowler, F.J. (1995). Improving survey questions: Design and evaluation. Thousand Oaks, CA: Sage Publications.

Kerlinger, F.N. (1979). Behavioral research: A conceptual approach. New York: Holt, Rinehard, & Winston.

Kim, J., & Mueller, C.W. (1978). Introduction to factor analysis: What it is and how to do it. Sage University Paper series on quantitative applications in the social sciences, 07-013. Newbury Park, CA: Sage Publications.

Price, J.L., & Mueller, C.W. (1986). Handbook of organizational measurement. New York: Longman.

Spector, P.E. (1992). Summated rating scale construction: An introduction. Sage University Paper series on quantitative applications in the social sciences, 07-082. Newbury Park, CA: Sage Publications.

TOP

Introduction to Tools

Conceptual Models
The Student Survey

The Faculty Survey

Checklist

References

Related Resources

Findings from use of surveys at seven institutions