CHAPTER ONE: Differing nations of ‘grammar’ for assessment
A. Language views (General approaches)
1. Syntactocentric perspective
It is predominantly concerned with the structure of clauses and sentence. This view defines “grammar” as a systematic way of accounting for and predicting an “ideal” speaker’s or hearer’s knowledge of the language.
2. Communicative prespective
It focuses more on the overall message being communicated and the interptretations that this message might invoke. “Grammar” is treated as one of the many resources for accomplishing something with language.
B. Syntactocentric perspectives of the language
1. Traditional grammar
It is based on a set of perspective rules along with the exceptions. It is criticized for this its inability to provide descrioptions of the language that could eduquately incorporate the exceptions into the framework and for its lack of generalizability to other languages.
2. Structural grammar
It is describe the structure of the language in terms of both its morphology and its syntax, in which aech word in a given sentence is categorized according to its use and the “patterns” or “structures” are said to constitute a unique system for that language.
3. Transformational-generative grammar
It provide a “universal” description of language behaviour revealing the internal linguistic system for which all human are predisposed. Underlying properties of any language system can be uncovered by means of a detailed sentence-level analysis. This universal Grammar (UG) has been criticized for failing to account for meaning or language use in social context.
C. Corpus linguistics
1. The most common practice of compiling linguistic corpora, or large and principled collections of natural, authentic spoken and written texts. It shows how often and where a linguistic form occurs in spoken or written text.
2. It provide information on pattern of variation in language use, language change, and varieties of language. It also provides information on the different semantic functions of lexical items, distributional and frequency information on the lexico-grammatical features of the language.
3. It challenges languages teachers to rethink how they view the content of a language curriculum and the manner in which this curriculum is presented to students
Katz and Fodor (1963) found that in addition to enconding semantic features and restrictions, a word also contains a number of syntactic features including the part of speech (noun,verb, adjective), countability (singular, plural), gender (masculine, feminime), and it can mark prepositional co-occurrence restrictions such as when the word think is followed by a preposition (about, of, over) or is followed by a that-clause. Katz and Fodor called this ‘the grammatical dimension of lexis’.
D. Theories of Communicatikn
1. Systematic-functional grammar
Context and meaning take precedence over linguistic form. It typically describes features of grammatical form that are used to express meaning beyond a single, context-free utterance. Rather, grammatical form is seen as having a symbolic relationship with meaning and pragmatic use, where each influences and shapes the other within and across utterances.
2. Speech act theory
Effective communication is not simply perceived as a function of linguistic accuracy or acceptable grmmar to convey literal and intended meaning. Communication must be appropriate for the context, i.e. speakers must have both ‘linguistic competence’ and ‘communicative competence’.
Both have had a considerable impact on L2 syllabus design, teaching and testing, and are credited for shifting the emphasis of language classrooms from a from a formal grammatical focus to a communication-based one.
E. Pedagogical grammar
1. It represent as an eclectic, but principled description of the target-language forms, created for the express purpose of helping teachers understand trhe linguistic resources of communication.
2. These grammars provide information about how language is organized and offer relatively accessible ways of describing complex, linguistic phenomena for pedagogical purposes.
3. The more L2 teachers understand how the grammatical system works, the better they will be able to tailor this information to their specific instructional contexts.
4. Besides formal pedagogical grammars (and, of course, SLA theory), language teachers would be advised to consult language textbooks when put to the task of specifying grammatical content for instruction or assessment.
5. These books not only provide descriptions, albeit less comprehensive, of the target grammar, but they also inform teachers of the scope with which a grammar point might be treated at a particular proficiency level or the sequence with which grammar points might be introduced.
CHAPTER TWO: Research on L2 grammar teaching and learning
Research on L2 teaching and learning
The SLA research looking at the role of grammar instruction in SLA might be categorized into three strands. One set of studies has looked at the relationship between the acquisition of L2 grammatical knowledge and different language-teaching methods. These are referred to as the comparative methods studies. A second set of studies has examined the acquisition of L2 grammatical knowledge through what Long and Robinson (1998) call a ‘non-interventionist’ approach to instruction. These studies have examined the degree to which grammatical ability could be acquired incidentally (while doing something else) or implicitly (without awareness), and not through explicit (with awareness) grammar instruction. A third set of studies has investigated the relationship between axplicit grammar instruction and the acquisition of L2 grammatical ability. These are referred to as the interventionist studies, and area a topic of particular interest to language teachers and tetsters
Comperative methods studies
These studies were in reaction to form-focused instruction (referred to as ‘focus on forms’ by Long, 1991), which used a traditional structural syllabus of grammatical forms as the organizing principle for L2 instruction. According to Ellis (1997), form-focused instruction contrasts with meaning-focused instruction in that meaning-focused instruction emphasizes the communication of messages (i.e., the act of making a suggestion and the content of such a suggestion) while form focused instruction stresses the learning of linguistic forms. These can be further contrasted with form-and-meaning focused instruction (referred to by Long (1991) as ‘focus-on-form’), where grammar instruction occurs in a meaning-based environment and where learners strive to communicate meaning while paying attention to form.
Non-Interventionist studies
While some language educators were examining different methods of teaching grammar in the 1960s, others were feeling a growing sense of dissatisfaction with the central role of grammar in the L2 curriculum. As a result, questions regarding the centrality of grammar were again raised by a small group of L2 teachers and syllabus designers who felt that the teaching of grammar in any form simply did not produce the desired classroom results. Newmark (1966), in fact, asserted that grammatical analysis and the systematic practice of grammatical forms were actually interfering with the process of L2 learning, rather than promoting it, and if left uninterrupted, second language acquisition, similar to first language acquisition, would proceed naturally.
Incremental leaps in grammatical ability through an accumulation of grammatical forms, as presented in a traditional grammar syllabus, learners in both instructed and naturalistic settings acquired the target structures in a relatively fixed order (Ellis, 1994) regardless of when they were introduced.
Learners acquiring any individualgrammaticalfeaturesuchasnegatives,interrogatives,relative clauses, word order, or pronouns appeared to pass through a relatively fixed developmental sequence toward mastering that form (Ellis, 1994).
The empirical evidence of ordered acquisitional patterns coupled with dissatisfaction with the results obtained from grammar teaching led a few SLA researchers to call for the total abandonment of traditional grammar instruction in the L2 classroom. Drastic as this was, researchers supporting this position (e.g., Krashen, 1982; Prabhu, 1987) argued that an L2 is not actually acquired through formal instruction; rather, it is learned incidentally and implicitly through exposure to the target language, as long as the input that learners are exposed to is made comprehensible. These researchers further claimed that in input-rich settings, the learner’s attention is focused solely on meaning in natural communication, and any form of explicit error correction is harmful to the acquisitional process. Supporters of this position further maintained that grammar acquisition was impervious to form-focused instruction, since the ‘natural’ processes of acquisition were at work. In other words, learners progress toward native-like proficiency in a predetermined order, making a number of predictable interlanguage errors, regardless of any instructional intervention. Finally, some researchers (e.g., Pica, 1983) found that learners who, in fact, did receive form-focused instruction showed an order of acquisition of grammatical features similar to that seen with the naturalistic learners, lending further support to the noninterventionist position.
Empirical studies in support of non-intervention
The non-interventionist position was examined empirically by Prabhu (1987) in a project known as the Communicational Teaching Project (CTP)insouthernIndia.Thisstudysoughttodemonstratethatthedevelopment of grammatical ability could be achieved through a task-based, ratherthanaform-focused,approachtolanguageteaching,providedthat the tasks required learners to engage in meaningful communication.
The non-interventionist position can be credited with showing us (1) that learners appear to acquire different grammatical structures in a fixed ‘acquisitional order’ and the same structure in a fixed‘acquisitional sequence’, (2) that meaning-focused classrooms can promote the development of L2 fluency provided there are plenty of opportunities for meaningful communication and (3) that meaning-focused classrooms can promote the development of grammatical ability no less than traditional classrooms, although, as we will see, this may be in a dequate for promoting high levels of SLA in a timely and efficient manner.
Possible implications of fixed developmental order to language assessment
Grammar tests targeting beginning English-language learners often include questions on the articles and the third-person singular -s affix, two features considered to be ‘very challenging’ from an acquisitional perspective.
The inclusion of these items in a placement test would be highly appropriate since the goal of placement assessment is to identify a wide range of ability levels so that developmentally homogeneous groups can be formed.
Problems with the use of development sequences as a basis for assessment
First, the number of grammatical sequences that show a fixed order of acquisition is very limited, far too limited for all but the most restricted types of grammar tests.
Second, much of the research on acquisitional sequences is based on data from naturalistic settings, where students are provided with considerable exposure to the language.
Third, as the rate (not the route) of acquisition appears to be influenced by the learner’s first language and by exposure to other languages, we need to understand how these factors might impact on development rates and how we would reconcile this if we wished to test heterogeneous groups of language learners.
Finally, as the developmental levels represent an ordering of grammatical rules during acquisition, this may or may not be on the same measurement scale as accuracy scores.
Interventionist studies
Schmidt, 1983; Swain, 1991) have maintained that although some L2 learners are successful in acquiring selected linguistic features without explicit grammar instruction, the majority fail to do so.
Most language teachers would contend that explicit grammar instruction, including systematic error correction and other instructional techniques, contributes immensely to their students’ linguistic development.
Empirical studies in support of intervention
Aside from anecdotal evidence, the non-interventionist position has come under intense attack on both theoretical and empirical grounds with several SLA researchers affirming that efforts to teach L2 grammar typically results in the development of L2 grammatical ability. Hulstijn (1989) and Alanen (1995) investigated the effectiveness of L2 grammar instruction on SLA in comparison with no formal instruction. They found that when coupled with meaning-focused instruction, the formal instruction of grammar appears to be more effective than exposure to meaning or form alone. Long (1991) also argued for a focus on both meaning and form in classrooms that are organized around meaningful and sustained communicative interaction. He maintained that the focus on grammar in communicative interaction serves as an aid to clarity and precision.
Research on instructional techniques and their effects on acquisition
Form- or rule-based techniques revolve around the instruction of grammatical forms. They can involve implicit, inductive grammar teaching, where the focus is on meaning, but the goal is to attract the learner’s attention to the form without using grammatical metatalk, or linguistic terminology.
Input-based techniques deal with how input is used in grammar instruction. One such technique is input flooding, where learners are presented with large amounts of input in which the targeted feature is present. Another involves typographical input enhancement, where input is manipulated by means of capitalization, printing in boldface and so forth. Comprehension practice is an input-based technique, where learners are asked to relate grammatical form to meaning – often by means of pictures or meaning-focused questions.
Feedback-based techniques involve ways of providing negative evidence of grammar performance. For example, ‘recast’ is a feedback based technique, where an utterance containing an error is repeated without the error. Another is referred to as ‘garden path’ since learners are explicitly shown the linguistic rule and allowed to generalize with other examples; however, when the generalization does not hold (negative evidence), further instruction is provided. Finally, metalinguistic feedback involves the use of linguistic terminology to promote ‘noticing’.
A final set of instructional techniques mentioned by Norris and Ortega (2000) are practice-based techniques of grammar instruction. These involve input-processing instruction and output practice (Lee and VanPatten, 2003).
Grammar processing and second language development
In the grammar-learning process, explicit grammatical knowledge refers to a conscious knowledge of grammatical forms and their meanings. Explicit knowledge is usually accessed slowly, even when it is almost fully automatized (Ellis, 2001b). DeKeyser (1995) characterizes grammatical instruction as ‘explicit’ when it involves the explanation of a rule or the request to focus on a grammatical feature. Instruction can be explicitly deductive, where learners are given rules and asked to apply them, or explicitly inductive, where they are given samples of language from which to generate rules and make generalizations.
Implicit grammatical knowledge refers to ‘the knowledge of a language that is typically manifest in some form of naturally occurring language behavior such as conversation’ (Ellis, 2001b, p. 252). In terms of processing time, it is unconscious and is accessed quickly. DeKeyser (1995) classifies grammatical instruction as implicit when it does not involve rule presentation or a request to focus on form in the input; rather, implicit grammatical instruction involves semantic processing of the input with any degree of awareness of grammatical form.
Implications for assessing grammar
The information from these assessments would show how well students could apply the forms in contexts where fluent and spontaneous language use is not required and where time could be taken to figure out the answers. Inferences from the results of these assessments could be useful for teachers wishing to determine if their students have mastered certain grammatical forms.
To obtain information on the students’ implicit knowledge of grammatical forms, testers would need to create tasks designed to elicit the fluent and spontaneous use of grammatical forms in situations where automatic language use was required.
CHAPTER THREE: The role of grammar in models of communicative language ability
The role of grammar in models of communicative competence
Rea-Dickins’definition of grammar
Rea-Dickins (1991) defined ‘grammar’ as the single embodiment of syntax, semantics and pragmatics.
Rea-Dickins (1991) further stated that the goal of communicative grammar tests is to provide an ‘opportunity for the test-taker to create his or her own message and to produce grammatical responses as a ppropriate to a given context’ (p. 125). This underscores the notion that pragmatic appropriateness or acceptability can add a crucial dimension to communication, and must not be ignored.
Rea-Dickins’ emphasis on grammar as pragmatics correctly reminds us of the close relationship among grammar, semantics and pragmatics. She also reminds us that the distinctions between the selevels are at times fuzzy at best.
Larsen-Freeman’s definition of grammar
Drawing on several linguistic theories and influenced by language teaching pedagogy, she has also characterized grammatical knowledge along three dimensions: linguistic form, semantic meaning and pragmatic use. Form is defined as both morphology, or how words are formed, and syntactic patterns, or how words are strung together. This dimension is primarily concerned with linguistic accuracy. The meaning dimension describes the inherent or literal message conveyed by a lexical item or a lexico-grammatical feature. This dimension is mainly concerned with the meaningfulnessof an utterance. The use dimension refers to the lexico-grammatical choices a learner makes to communicate appropriately within a specific context. Pragmatic use describes whenand whyone linguistic feature is used in a given context instead of another, especially when the two choices convey a similar literal meaning. In this respect, pragmatic use is said to embody presuppositions about situational context, linguistic context, discourse context, and sociocultural context. This dimension is mainly concerned with making the right choice of forms in order to convey an appropriate message for the context.
According to Larsen-Freeman (1991), these three dimensions may be viewed as independent or interconnected. For example, a linguistic form such as the articles in English displays a syntactic, semantic and pragmatic dimension, even though, perhaps in the classroom, it might be necessary to focus more on the pragmatic aspect, which can pose the greatest challenge to learners.
What is meant by ‘grammar’for assessment purposes?
CHAPTER FOUR: Towards a definition of grammatical ability
What is meant by grammatical ability?
Grammatical ability is, then, the combination of grammatical knowledge and strategic competence; it is specifically defined as the capacity to realize grammatical knowledge accurately and meaningfully in testing or other language-use situations.
What is ‘grammatical ability’ for assessment purposes?
First, grammar encompasses grammatical form and meaning, whereas pragmatics is a separate, but related, component of language. A second is that grammatical knowledge, along with strategic competence, constitutes grammatical ability. A third is that grammatical ability involves the capacity to realize grammatical knowledge accurately and meaningfully in test-taking or other language-use contexts. The capacity to access grammatical knowledge to understand and convey meaning is related to a person’s strategic competence. It is this interaction that enables examinees to implement their grammatical ability in language use. Next, in tests and other language-use contexts, grammatical ability may interact with pragmatic ability (i.e., pragmatic knowledge and strategic competence) on the one hand, and with a host of non-linguistic factors such as the test-taker’s topical knowledge, personal attributes, affective schemata and the characteristics of the task on the other. Finally, in cases where grammatical ability is assessed by means of an interactive test task involving two or more interlocutors, the way grammatical ability is realized will be significantly impacted by both the contextual and the interpretative demands of the interaction.
Knowledge of phonological or graphological form and meaning
Knowledge of lexical form and meaning
Knowledge of lexical form enables us to understand and produce those features of words that encode grammar rather than those that reveal meaning. This includes words that mark gender (e.g., waitress), countability (e.g., people) or part of speech (e.g., relate, relation).
Knowledge of lexical meaning allows us to interpret and use words based on their literal meanings. Lexical meaning here does not encompass the suggested or implied meanings of words based on contextual, sociocultural, psychological or rhetorical associations.
Knowledge of morphosyntactic form and meaning
Knowledge of morphosyntactic form permits us to understand and produce both the morphological and syntactic forms of the language. This includes the articles, prepositions, pronouns, affixes (e.g., -est), syntactic structures, word order, simple, compound and complex sentences, mood, voice and modality.
Morphosyntactic forms carry morphosyntactic meaningswhich allow us to interpret and express meanings from inflections such as aspect and time, meanings from derivations such as negation and agency, and meanings from syntax such as those used to express attitudes (e.g., subjunctive mood) or show focus, emphasis or contrast (e.g., voice and word order).
Knowledge of cohesive form and meaning
Knowledge of cohesive form enables us to use the phonological, lexical and morphosyntactic features of the language in order to interpret and express cohesion on both the sentence and the discourse levels. Cohesive form is directly related to cohesive meaning through cohesive devices (e.g., she, this, here)which create links between cohesive forms and their referential meanings within the linguistic environment or the surrounding co-text.
Knowledge of information management form and meaning
Knowledge of information management form allows us to use linguistic forms as a resource for interpreting and expressing the information structure of discourse. Some resources that help manage the presentation of information include, for example, prosody, word order, tense-aspect and parallel structures. These forms are used to create information management meaning.
Knowledge of interactional form and meaning
Knowledge of interactional form enables us to understand and use linguistic forms as a resource for understanding and managing talk-ininteraction. These forms include discourse markers and communication management strategies. Discourse markers consist of a set of adverbs, Knowledge of interactional form and meaning
Knowledge of interactional form enables us to understand and use linguistic forms as a resource for understanding and managing talk-ininteraction. These forms include discourse markers and communication management strategies. Discourse markers consist of a set of adverbs, Conversation-management strategies include a wide range of linguistic forms that serve to facilitate smooth interaction or to repair interaction when communication breaks down.
Similar to cohesive forms and information management forms, interactional forms use phonological, lexical and morphosyntactic resources to encode interactional meaning.
CHAPTER FIVE: Designing test tasks to measure L2 grammatical ability
How does test development begin?
Every grammar-test development project begins with a desire to obtain (and often provide) information about how well a student knows grammar in order to convey meaning in some situation where the target language is used. The information obtained from this assessment then forms the basis for decision-making. Those situations in which we use the target language to communicate in real life or in which we use it for instruction or testing are referred to as the target language use(TLU) situations (Bachman and Palmer, 1996). Within these situations, the tasks or activities requiring language to achieve a communicative goal are called the target language use tasks. A TLU task is one of many languageuse tasks that test-takers might encounter in the target language use domain. It is to this domain that language testers would like to make inferences about language ability, or more specifically, about grammatical ability.
A basis for identifying and making explicit the areas of grammatical knowledge to be measured, as seen in Figure 5.1.
Example 1:Multiple-choice task
Designed to test grammatical form (morphosyntax-word order)
Directions: Circle the correct answer.
A: Can’t Tom drive us to the airport?
B: He has ____ to take us all.
(a) such small a car (c) a too small car
(b) very small a car (d) too small a car ✔ key ✔
Example 2:Multiple-choice task
Designed to test grammatical form and meaning (cohesive-ellipsis)
Directions: Circle the correct answer.
A: Will you and Ann go away this summer?
B: I imagine ____.
(a) it (c) that
(b) so ✔ (d) we’ll
Example 3:Multiple-choice task
Designed to test grammatical form and meaning (multiple areas)
Directions: Circle the correct answer.
A: Wow! You got a new hairdo. I love it!
B: Thanks, but ____________________________________
A: No, you don’t. You look great!
(a) I liked it the other way. (c) You look great.
(b) What happened to you? (d) I look ridiculous! ✔
Example 4:Multiple-choice error identification task
Designed to measure grammatical form
Directions: Circle the letter corresponding to the error.
As my car had broken down, I decided to go there by foot.
A B C D ✔
Example 5:Matching task
Designed to measure grammatical meaning (denotation)
Directions: Match the letter of the underlined word(s) with its meaning.
Write the letter on the line.
Last week while Tom and Jane were having dinner in a restaurant, thieves (a) broke down the front door of their pretty little house, went inside and (b) broke intotheir safe. Now they’re (c) broke.
___ 1. poor ___ 3. enter to steal something
___ 2. make into two or more pieces ___ 4. enter by force
Example 6:Discrimination task
Designed to measure morphosyntactic meaning
Directions: Match the sentence with the picture by writing the number in the box on the line.
___ Se la entregó a ella. [He delivered it to her.]
Example 7: Noticing task
Designed to measure grammatical meaning (morphosyntactic meaning)
Directions: Circle ‘would’ when it refers to the habitual past. Underline it when it refers to the present or future.
You know? You think you’ve got it bad. When I was a kid, we would have to walk up hill to and from school every day. We would even do it when it snowed
– winter and summer. And we would never even think of complaining. We would smile and go about our business. I wouldn’t change those days for anything. Would you now please ‘shut up’ and take out the garbage?
Example 8: Gap-filling task
Designed to measure grammatical form and meaning
Directions: Fill in the blank with an appropriate form of the verb.
In about 20 AD Apicus was well known for the cookbooks he (1)
________________ in his spare time. He was equally famous for the lavish meals he (2) ________________ for his family and guests.
(Adapted from Purpura and Pinkley, 2000)
Example 9: Short-answer task
Designed to measure grammatical form and meaning
Directions: Use the job ad to complete the application form.
Name: Job applied for:
Qualifications for job applied for:
Current job: Reason for leaving:
Example 10: Discourse completion task
Designed to measure grammatical form and meaning on the discourse level
Directions: Complete the conversation the two friends are having.
A: I can’t believe that disgusting little waiter told me ‘to get a life’ when I showed him the hair in my soup.
B: Well, if I were you, ___________________________________________!
A: Nah, I don’t want to be rude.
Example 12: Reporting task
Designed to measure grammatical form and meaning on the sentence and discourse levels Directions – Part A: Last night there was a break-in at the Santellis’. You are the detective on the case. For each piece of evidence below, make a written speculation about the burglary. Use modals whenever possible.
1. The kitchen lock was forced open and a window was broken.
2. Traces of cookies and milk were found on the kitchen counter.
3. There was a wet towel in the shower.
4. All of Mrs. Santelli’s diamonds are missing.
Directions – Part B: Based on the evidence, draw some tentative conclusions about the thief. Write a brief progress report on the situation for a new colleague on the case.
(Adapted from Purpura and Pinkley, 2000)
Example 13: Simulation task
Designed to measure grammatical form and meaning on the discourse level
Directions: Your local government has just received a large amount of money to solve one of its problems. You are on the committee to decide which one to solve. You will be given a problem to advocate for. Your job is to convince your group that the city should solve yourproblem first. You will have five minutes to plan your argument.
Once you are in your group, describe your problem to the others.When you hear all the problems,work together to decide which problem the city should solve first.Try to get your problem solved first.
(Each student is given only one role)
Person A The city is upset about pollution. There are more and more cars every year, and they are aggravating the pollution problem. The government does not want to make pollution laws because it is afraid factories will close. However, more and more people are having pollution-related health problems. The city needs money to help the factories install anti-pollution technology.
Person B The city is worried about crime. In some neighborhoods crime has increased dramatically within the last year, and people are afraid to walk in certain areas at night. More and more people are reporting street crimes. Recently thieves broke into a bank and stole millions. Violent crime is increasing too. The city needs money to hire more policemen and to install modern crime technology. Person C The schools are in desperate need of help. Classrooms are overcrowded and buildings are falling apart from lack of maintenance. New teachers do not want to begin their careers in these conditions and veteran teachers are leaving the schools to accept jobs in the suburbs, where they are paid twice as much. The schools also need funds to support ESL instruction for growing numbers of immigrant students. Every child deserves to have the opportunity for a good education.
(Adapted from Purpura and Pinkley, 1999)
CHAPTER SIX: Developing tests to measure L2 grammatical ability
What makes a grammar test ‘useful’?
The quality of reliability
When we talk about ‘reliability’ in reference to a car, we all know what that means. A car is said to be reliable if it readily starts up every time we want to use it regardless of the weather, the time of day or the user. It is also considered reliable if the brakes never fail, and the steering is consistently responsive. These mechanical functions, working together, make the car’s performance anywhere from zero to one hundred percent reliable. Similarly, the scores from tests or components of tests can also be characterized as being reliable when the tests provide the same results every time we administer them, regardless of the conditions under which they are administered. In other words, test scores should not fluctuate drastically as a result of the time of the test administration, the form of the test used (provided there exists more than one form), or the raters who might have scored the responses. This consistency of measurement is referred to as test reliability, and it ranges on a scale from zero (no consistency) to one (perfect consistency).
The quality of construct validity
The second quality that all ‘useful’ tests possess is construct validity. Bachman and Palmer (1996) define construct validity as ‘the extent to which we can interpret a given test score as an indicator of the ability(ies), or construct(s), we want to measure. Construct validity also has to do with the domain of generalization to which our score interpretations generalize’ (p. 21).
The quality of authenticity
Authenticity of content as the degree to which the topical, thematic or contextual characteristics of the test tasks match those of the TLU tasks.
The quality of interactiveness
This quality refers to the degree to which the aspects of the test-taker’s language ability we want to measure (e.g., grammatical knowledge, language knowledge) are engaged by the testtask characteristics (e.g, the input response, and relationship between the input and response) based on the test constructs. In other words, the task should engage the characteristics we want to measure (e.g., grammatical knowledge) given the test purpose, and nothing else (e.g., topical knowledge, affective schemata); otherwise, this may mask the very constructs we are trying to measure.
The quality of impact
Impact refers to the link between the inferences we make from scores and the decisions we make based on these interpretations. In terms of impact, most educators would agree that tests should promote positive test-taker experiences leading to positive attitudes (e.g., a feeling of accomplishment) and actions (e.g., studying hard). A special case of test impact is washback, which is the degree to which testing has an influence on learning and instruction. Washback can be observed in grammar assessment through the actions and attitudes that test-takers display as a result of their perceptions of the test and its influence over them.
The quality of practicality
Test practicality is not a quality of a test itself, but is a function of the extent to which we are able to balance the costs associated with designing, developing, administering, and scoring a test in light of the available resources (Bachman, personal communication, 2002).
Overview of grammar-test construction
Stage 1:Design
Purpose
The purpose statement could also include who is impacted by the decisions and whether the stakes are high or low. It could also specify how the results of the test are intended to be used. For example, in most classroom tests the results of assessment will be used to promote further learning or to inform instruction.
TLU domains and tasks
After describing the purpose, the TLU domain is identified (e.g., real-life and/or language-instructional) and the TLU task types are selected. To identify language-use tasks and the type of language needed to perform these tasks, a needs analysis must be performed. This involves the collection and analysis of information related to the students’ target-language needs. Depending on the testing situation, a needs analysis can be relatively informal or very complex.
Characteristics of test-takers
The design statement contains a detailed description of the characteristics of the test-takers, so that the population of test-takers for whom the test is intended and to whom the test scores might generalize can be made explicit. The personal attributes of test-takers which can potentially influence test results include age, native language, gender, level of language ability and so forth.
Construct(s) to be measured
The design statement also provides a theoretical definition of the construct(s) to be measured in the test. Construct definition can be based on a set of instructional objectives in a syllabus, a set of standards, a theoretical definition of the construct or some combination of them all.
Construct definition based on a syllabus (or a textbook) is useful when we want to determine to what degree students have mastered the grammar points that have been taught during a certain period.
Plan for evaluating usefulness
The test design statement also provides a description of a plan for assessing the qualities of test usefulness. From the beginning of grammar-test development, it is important to consider all six qualities of test usefulness and to determine minimum acceptable levels for each quality. Bachman and Palmer (1996) suggest that a list of questions be provided to guide test developers to evaluate test usefulness throughout the process so that feedback can be provided. In addition, test developers should consider ways of providing empirical evidence of test usefulness (see Bachman and Palmer, 1996, Chapter 7, for a detailed list of questions to elicit information on the qualities of test usefulness).
Plan for managing resources
Finally, the test design makes explicit the human, material and time resources needed and available to develop the test. In cases of limited resources, priorities should be made in light of the qualities of test usefulness.
Stage 2: Operationalization
The operationalization stage of grammar-test development describe show an entire test involving several grammar tasks is assembled, and how the individual tasks are specified, written and scored. The outcome of the operationalization phase is both a blue print for the entire test including scoring materials and a draft version of the actual test. According to Bachman and Palmer (1996), the blue print contains two parts: a description of the overall structure of the test and a set of test-tasks pecifications for each task. (In this chapter the material don’t finish yet)
CHAPTER EIGHT: Learning-oriented assessment of grammatical ability
What is learning-oriented assessment of grammar?
In the context of learning grammar, learning-oriented assessment of grammar, is believed among the educational assessment experts, that student learning would improve if assessment, curriculum and instruction were more connected.
In reaction to conventional test, many experts have been woring in new assessments techniques that better elicit students’ outcome and that better connect to classroom goals, curricula and instruction. In this process we find alternative, authentic and performance assessment, all of them seems to be the same, but they have slightly differences. According to Purpura (2004) Alternative assessment encourage assessments in which students are asked to perform or produce meaningful tasks that need both higher level thinking and real world implication. Authentic assessment requires knowledge and skills where can be observed some real life or authentic tasks, to perform these tasks students need some complex and extended production, self-assessment is an important component of these tasks. Performance assessment refers to the evaluation of outcome, which is derived from the observation of more complete tasks that implicates real life situation. Self-assessment is required by making explicit scoring in a rubric.
The objective of learning-oriented assessment of grammar is to provide information about the grammar students know, understand or are able to use in different context, and the repercussion that this information might have for grammar processing; moreover, teachers can be provided with information about what students feel about learning grammar and about themselves as learners. In terms of method, learning-oriented assessment of grammar believe that assessment must be open to all task types, and this include the use of selected-response, limited-production and complex production tasks that may not involve real-life implication. Finally, learning-oriented assessment is designed to be an integral part of instruction. It can occur at formal or informal situation, at any stage of the learning process. The data can be collected at one point in time or over a period of time.
Implementing learning-oriented assessment of grammar. Consideration from grammar testing theory.
For implementing learning-oriented assessment of grammar some implication must be consider as design and also tests developers need to plan for and specify how assessment will be used to promote further learning.
Implication for the test design:
First consideration: in the design stage of a test constructions, classrooms teacher need to specify whom we are doing the assignment for, why assessment information is needed and what kind of information (essential information to specify assessment purpose).
Second consideration: construct definition. Learning-oriented assessment is designed to make and the feedback that can make result from observation of performance.
Third consideration: the need to measure the students’ explicit as well as their implicit knowledge of grammar, but also the students’ implicit or internalized knowledge of the grammar.
Implication for operationalization:
The learning mandates will affect the specification of test task so can be better to align some characteristics with instructional goals. Learning oriented-assessment of grammar promotes the collection of data on students’ ability and methods in classroom, and also collects information about attitudes and feeling toward learning grammar. This data collection can be taken one point in time or accumulated over a period of time.
In classroom assessment design, the scoring process results in a written or oral evaluation of candidate responses. At the same time, this provides learners with summative or formative evaluationas for example feedback. Therefore, scoring process allows test-takers to discover themselves, positive and negative evidence on their grammatical ability. The information resulting from echievements test can provide much more meaningful and constructive guidance on what to notice and how to improve. Feedback and scoring method can involve students; this can develop their capacity for self-assessment, and and also develop responsibility of their own learning.
Planning for further learning: the test blueprint should include explicit informationon how the assessment plans to satisfy the learning mandate. Teachers have many options for presenting assessment results to students. They could present student with feedback, a score for each test component, scoring rubrics, narrative summary of teachers’ observation, etc.
Consideration from L2 learning theory
In implementing learning-oriented assessment of grammar, teachers need to consider how assessment relates to and can help promote grammar acquisition. This will affect not only what is and how is assessed, but also when in the lesson grammar knowledge are best assessed, and what the results mean for learners to improve.
SLA processes-briefly revisited:
Research in SLA suggest that learning an L2 involves three simultaneously process: Input processing: relates to how the learner understands the meaning of a new grammatical featuter or how form-meaning connection are made. System change: refers to how learners accommodate new grammatical forms to their interlanguage and how this change helps restructure their interlanguage. Output processing: relates to how learners access or make use of implicit grammatical knowledge to produce utterances spontaneously in real time.
Assessing for intake:
this process is describe as the first critical stage of acquisition, as the process of converting input into intake. Students are given a communicative language classroom and are encouradged to use tasks in which they must use language meaningfully and use comprehensible input as an essential component of instruction. Assessing for intake requires learners understand the target forms, but do not produce them themselves. This can be achieved by selected-response and limited-production tasks in which learners need to make form-meaning connections.
Assessing to push restructuring:
Once input has been converted into intake, the new grammatical features is ready to be accommodated into the learners’ developing linguistic system. To initiate this process, teaches provide them with tasks that enable them to use the new grammatical forms in decreasingly controlled situation. By attending to grammatical input and by getting feedback learners are able to accommodate the differences between their interlanguage and the target language.
Assessing for output processing:
Even though learners have showed explicit knowledge of form and meaning of new grammatical points, it does not mean they can access this knowledge automatically in spontaneous communication. Learners need to be able to produce unplanned, meaningful output in real time showing that the grammatical knowledge is already unconscious part of their developing system of language knowledge.
CHAPTER NINE
1. Challenge 1
a. Defining grammatical ability
At the moment of assessing grammatical form and meaning communicative language testing. It is relevant to provide teachers and learners with a more complete assessment, taking into account the grammatical ability of the takers than just providing information about the form of the meaning.
b. Theoretical challenges about the definition of grammatical knowledge
It is concerned to language educators who need to make comprehensible distinctions between the form and meaning components of grammatical knowledge in terms of the test purpose in order to integrate these distinctions in construct definition.
2. Challenge 2
a. Scoring grammatical ability
It is related to scoring form, meaning and grammar assessments and also how language teachers need to adapt their scoring procedures to reflect the two dimension of grammatical knowledge. It requires the use of measurement models to contain dichotomous and partial- credit data in analysing test scores.
In scoring extended-production tasks descriptors, rubrics must be adapted to grade performance in form and meaning more noticeably.
b. Advantages and disadvantages
“The advantages of using complex performance tasks that are highly authentic is the generalizabilityn of theinferences these tasks allow us to make about grammatical ability”. (p.259)
The disadvantages are related to the lack of accuracy with which teachers are able to infer what students or test takers know about grammar, taking into other constructs that could be intended or no measured in such tasks by raters.
3. Challenge 3
a. Assessing maenings
It is concerned about the meaning in a model of communicative language ability can be defined and assessed.
b. The assessment of meaning in terms of grammatical meaning and pragmatic meaning
- The primary goal in grammar assessment is to notice if students are able to use forms to acquire their basic point across correctly and significantly. If meaning is construct-relevant, as a result communicative meaningfulness should be scored.
- Pragmatic meaning involves an amount of implied meanings that originate from context relating to the interpersonal relationship of the interlocuters.
- The distinctions between grammatical meaning and pragmatic meanings are observable when L2 students fail at the moment of understanding how meanings could be extended or intentionally confusing.
4. Challenge 4
a. Reconsidering grammar-test tasks
It is related to the design of test tasks that are able to measure grammatical ability and provide authcentic and engage measures of grammatical performance.
To design tasks that are authentic and engaging measures of performance, it is necessary to consider to consider:
- The assessment purpose and the construct that is going to be measure.
- To consider the kinds of grammatical performance required in order to provide evidence in support of the inferences.
- After the inferences are specified, it is required to support these claims to design test tasks to measure what students know about grammar or how they are able to use grammatical resources to accomplish a wide range of activities in the target language.
5. Challenge 5
a. Assessing the development of grammatical ability
“The challenge for language testers is to design, score and interpret grammar assessments with a consideration for developmental proficiency” (Purpura, 2005; p. 273)
Ellis (2001) states that grammar scores should be calculated to provide a measure of grammatical accuracy and the underlying acquisitional development of L2 students.
- With limited or extended production tasks. Teachers can give learners credit for and feedback by judging performance on these tasks by means of analytic rating scales.
- Rating scales need to be based on construct and task based methods in which the different level of grammatical abilities can be described completely.
REFENRENCES:
Purpura. James. E. 2004. ASSESSING GRAMMAR (p. 1-251). Cambridge University Press: United Kingdom.
https://www.slideshare.net/mobile/igotamnesia/assessing-grammar-summary-ch-1-8-9
