LINGUISTICS 189/289 - STANFORD UNIVERSITY

Linguistics and the Teaching of English
as a Second/Foreign Language

CALL Mini-Course

Unit 2: Courseware Evaluation, Development, and Implementation

OVERVIEW    

This unit looks at the sub-field of tutorial CALL from the perspectives of both the end users (teachers and students) and developers. It introduces the term courseware, which refers to software that is used to support formal language learning. In practice, courseware has been used to refer to everything from complete software packages that can be used without a teacher to software that is just a part of a language learning course, sometimes a minor or optional supplementary part.  We will use the term interchangeably with that of tutorial software. The objective of this unit is to give you a peek at the  three dimensions of tutorial CALL--developing courseware, evaluating courseware, and implementing courseware in your classes.

As a backdrop to this, in a series of papers from 1987 to 1996, I attempted to develop a comprehensive methodological framework for CALL that integrated development, evaluation and implementation. I did not succeed (the CALL world turned out to be more complex than my original vision), but I learned a lot from the process and the product has been of use to myself and some of my colleagues. The framework expanded on an earlier one by Martin Phillips and used the Richards and Rodgers (1982) framework as an organizing scheme to characterize the apparent relationships between elements of language teaching and learning and the computer. The driving force behind it was the observation that existing schemes of instructional design and in particular evaluation did not pay sufficient attention to language learning. I will be introducing a simplified version of it here.  

ORGANIZING PRINCIPLES

Development, evaluation, and implementation are part of a logical progression in any situation that has an end product. If a company produces a computer program for balancing your checkbook, for instance, they need to 1) design it with the needs of the end users in mind, 2) evaluate it in house and encourage outsiders to review it, and 3) have a mechanism to implement it, including figuring out how to make it available and training end users in its effective operation.

CALL software is a bit different from a checkbook balancing program in that it involves a more complex view of who the evaluators and end users are. Evaluation, for instance, may be connected to the developer and be used for improving the courseware prior to release, or it may be done by an outside reviewer for a professional journal. It may be done by a teacher representing a school or institute, or by a teacher selecting for his or her own class. It may even be by a student evaluating for possible use or purchase, or to communicate impressions to other users.

Development, evaluation, and implementation are thus simultaneously part of a logical progression of a courseware project and interacting manifestations of its reality . This is true whether the project is for a CALL or for some other educational purpose. However, the specific domain of language teaching and learning imposes on these three a set of considerations that are not exactly the same as we would find in courseware for, say, history or chemistry or math. The framework that follows addresses those considerations. This is a revised and simplified form of the content in the papers listed below (see references). I will be passing out a copy of my 1996 chapter from The Power of CALL  in class, which is the most comprehensive of the papers. The others go into more depth in language teaching approaches (1987), evaluation (1988), and development (1992).

Two final notes. First, in an extensive critique of this framework in Levy (1997) argues that "Hubbard's framework for CALL materials development, which assumes that all CALL is tutorial in nature, is not generally applicable to the computer as a tool.  Similarly, the Richards and Rodgers model...only has limited application for the computer tool" (p. 211). I think there is more applicability than he suggests, but for now we'll follow Levy's view and assume this is a framework for tutorial CALL only. Second, like Richards and Rodgers' framework, but unlike most others for CALL, there is an attempt to be agnostic here with respect to what actually constitutes good language teaching through computers. I wanted a framework which could be used equally by those whose language teaching approaches might be as diverse as grammar-translation, audio-lingual, communicative, or interactionist proponents. Rather than impose my own beliefs about language and language teaching here, I wanted to produce a tool that anyone engaged in tutorial CALL could use.

FRAMEWORK FUNDAMENTALS

The three modules (development, evaluation, implementation) share core components adapted from Richards & Rodgers (1982). In each case their original components are adapted, interpreted, and supplemented to include the reality of the computer as the interface between the teacher/developer and materials and the learner. (Realistically, in any tutorial program there IS a teacher (or at least a teaching presence) in addition to the materials themselves.) The development and evaluation modules are most closely related in terms of the elements considered. Implementation feeds on the output of evaluation. However, each module can impact the others over time, as when information from evaluation and implementation is returned to developers for updates, patches, or considerations in later versions of the product.

                         CALL Framework Interrelationships

EVALUATION MODULE

Evaluation involves three kinds of considerations. A crucial aspect of evaluation is to understand what the courseware does first before attempting to judge it. The first consideration then is the operational description of the software, which essentially focuses on the procedure level elements. The design elements essentially can be subsumed under the label "learner fit." That is, based on the information from the operational description, you are looking to see how well the design elements of language difficulty, program difficulty, program content, etc. fit the students you are evaluating for. The approach elements, in this case approach-based evaluation criteria, can be subsumed under the label "teacher fit". Ultimately, then, evaluation consists of getting a clear understanding of what the software actually has in the way of material and interaction, and then judging how closely it fits with the learner's needs as determined by their profiles and learning objectives (perhaps themselves determined by a course syllabus) and your own language teaching approach. This relationship is sketched below.

                         CALL Evaluation Framework

See www.calico.org/journalarticles/Volume6/vol6-2/Hubbard.pdf for the original article on the evaluation framework. It is worth noting that a modified version of this framework is still used by the CALICO Journal for its courseware reviews.

DEVELOPMENT MODULE

Courseware development refers to the process of going from the idea of creating a piece of tutorial software through the final product. It should be informed by general principles of instructional design. However, I believe it is also critically important to recognize the pedagogical aspects specific to language learning that traditional  instructional design approaches may overlook. These are discussed in the article (Hubbard 1996) passed out in class.

In describing the development module, I review the key terms which are part of the evaluation module as well. This captures the intuitive realization that the deliberations important in deciding whether or not to use a piece of software are the same as the deliberations taken in producing the software in the first place. Like the development module, both the evaluation and implementation modules rely on versions of Richards and Rodgers' categories of approach, design, and procedure. However, the framework diverges from Richards and Rodgers in attempting to account for individual language programs rather than whole methods.

Approach. A language teaching approach is taken to include assumptions about what language is and how languages are learned. These are manifested in software by considering them in light of the realities of the computer delivery system. The result is a set of approach-based software design criteria.

Design. The design portion of the development framework includes elements such as the following: language difficulty, program difficulty, program content, language focus (also called program focus), skill  focus (also called learner focus), and learner styles supported. These interact with considerations such as the learner profiles among the intended users (age, first language, etc.) and the objectives of the syllabus and should maintain consistency with the approach-based design criteria. 

Procedure. The procedure considerations are much more specific to the computing environment. They begin with the concept of "activity type", which is the general form of the exercise, e.g. text plus comprehension questions, picture identification, text reconstruction. The presentation scheme determines how the language will be presented, including the modalities (text, audio, graphics, and video) and what the general interaction sequence will be. Elements related to the presentation scheme are the screen layout, help and control options, the form of input judging, and the feedback in response to the input. 

Courseware Package. The preceding elements are the considerations to be made in creating the software itself. The overall courseware package may also include an accompanying textbook, tutorial, documentation, record keeping, or other utilities (e.g., for authoring additional material).

IMPLEMENTATION MODULE

Implementation considerations are relevant during the evaluation process, but they become crucial when deciding how best to use software that is available. The key questions to address in implementation are the following. What is the setting in which the students will be using the software? What kinds of training or preparatory activities (if any) will there be? What kinds of follow-up activities (if any) will there be? How much control will the teacher exert, and how much control will be left to the learner? For a more detailed description of the components to consider in implementation and their interrelationships, see Hubbard (1996).

RESOURCES FOR EVALUATION

Besides the evaluation framework presented here, it is common to see evaluation checklists or other procedures. Here are a few examples.

CALICO's Online Software Review Collection
Rice University Evaluation Guide.
Guide for Using Software in the Adult ESL Classroom by Susan Gaer
A Place to Start in Selecting Software by Deborah Healey & Norm Johnson
Software Selection, Evaluation, and Use: notes from a 4-day seminar (1996) at Oregon State by Vance Stevens
Bibliography of CALL Evaluation and Evaluation Features for CALL Multimedia from the National Foreign Language Resource Center at the University of Hawaii
TESOL CALL Interest Section Software List: the largest, most up-to-date list around.

RESOURCES FOR DEVELOPMENT

Additional instructional design issues. As noted, there is a need to consider CALL software development, as well as evaluation, and implementation, as it relates to the general field of instructional design. In making that consideration, however, it is important to be aware that instructional design often takes a "training for mastery" approach that is not appropriate for many aspects of language learning. 

Some authoring options: 
HyperStudio (a HyperCard-type authoring package)
HTML and web-page authoring tools (like Microsoft FrontPage)
Hot Potatoes (a web-authoring tool for interactive exercises).
Macromedia Flash and Director

REFERENCES
Hubbard, Philip (1987) "Language Teaching Approaches, the Evaluation of CALL Software, and Design Implications" in Smith, W.F. (ed.) Modern Media in Foreign Language Education: Theory and Implementation. Lincolnwood IL: National Textbook.
Hubbard, Philip (1988) "An Integrated Framework for CALL Courseware Evaluation" CALICO Journal 6.2: 51-74.
Hubbard, Philip (1992) "A Methodological Framework for CALL Courseware Development" in Pennington, Martha and Stevens, Vance (eds.) Computers in Applied Linguistics: An International Perspective. Clevedon UK: Multilingual Matters.
Hubbard, Philip (1996) "Elements of CALL Methodology: Development, Evaluation, and Implementation" in Pennington, Martha (ed.) The Power of CALL. Houston: Athelstan.
Levy, Michael (1997). Computer-Assisted Language Learning: Context and Conceptualization. New York: Oxford.
Richards, Jack and Rodgers, Ted (1982) Method: Approach, design, and procedure. TESOL Quarterly 16.2.

Back to CALL Mini-Course