Examining Classroom Observation Forms in the Iranian Context: A Rhetorical Analysis of Academic Commentary in Teacher Evaluation

Introduction

Whether employed as an effective procedure for pre-service and in-service teachers' professional development or an evaluation process in the English teacher education field, classroom observation/supervision has been established as a significant component in teaching practices during the last decade of the past century. It is apparently a process that pervades all educators' professional lives from the start to the end of their careers (O’Leary, 2020, p. 12). Given its multiplicity, supervision has not been subject to a hard and fast, discipline-specific definition. Among numerous definitions of supervision in terms of contexts and time, most definitions owe not to language teaching. Rather, its origin can be traced back to general education or business and industry. Supervision as teachers’ performance evaluation has originated in appraisal systems of the private sector in organizational psychology, management, and business which pursues measuring how an employee is functioning in his or her job for decades (Rodgers, 2021).

The evaluation of teaching to appraise teachers’ performance is generally conducted within educational settings exploiting different protocols. Protocols take different structures and as data from observations are used to assess performance, they are frequently combined with other kinds of data to provide a comprehensive evaluation procedure (Hora & Ferrare, 2013). Descriptive protocols are meant to describe in-action teachers’ behavior appraising quality or efficacy. In such protocols, ready-made forms are in place to delve beneath the surface to draw on the interpretations and explanations, significance and impact of classroom life, and the meanings that have sometimes been left uncovered (Wragg, 1999). The overall comments appearing at the end of the designed observation form as a form of academic commentary refer to the provision of general feedback written by supervisors. It is realized as part of supervisory discourse constituting the description of concrete behaviors in terms of judging the quality or efficacy of teaching practice. Therefore, the academic commentary can be considered as a reflection of what the observer regards as the most noteworthy or valuable activities and qualities in teaching practice.

The application of such evaluative processes for credible reasons requires extreme caution. Weade and Everston view the observer as “the first instrument of observation.” On this account, “the observer’s frame of reference will shape what gets noticed, recorded, and/or assessed during the observation process” (cited in McDonald 2017, p. 5). In fact, the weight of idiosyncrasy is considerably high in writing academic commentary as a particular genre, and the formulation and interpretation of its rhetorical features, then, warrant a thick description in order to provide insight into why such a specific review genre is written and used by the specialist community the way it is.

To conduct a thorough investigation of an unexplored genre, one may need to consider some or all of the seven steps suggested by Bhatia (1993, p. 63-84), depending on the analysis's goal, the genre's dimensions to be studied, and the reader's prior knowledge of the genre in question. An academic commentary, as a distinct review genre and specific, complex mechanism and qualitative section of structured protocol to serve the observers’ and supervisors’ discourse community, is of great importance in assessing teachers’ performance and providing insights for teacher development. The current study intends to concentrate on the description and comprehension of the macro-structure organization in the generic structure of the academic commentary written by Iranian observers in English in order to explore discursive practices.

Review of the Literature

A great deal of studies on genre analysis have been carried out to analyze the rhetorical structures of various types of texts following the publication of Swales’ Genre Analysis (1990). In the same wake, investigating academic commentary, as part of written feedback by observers, from a genre perspective can culturally illuminate rhetorical structures and contribute to our understanding of the effectiveness of its pertinent communicative functions. Thus, in order to situate our study in the literature, a brief survey of elemental issues is required which consecutively sets the scene for introducing rhetorical analysis of academic commentary.

Observers’ Roles

Since the 1980s from which articles related to observation in language teaching began to emerge, various approaches have been delineated for the roles of observers. Freeman (1982) labels three methods for giving feedback and observing teachers namely the supervisory nondirective, and alternative options. In the supervisory option, the supervisor as the expert prescribes guidance, also referred to as the classic directive model. The non-directive technique, on the other hand, involves the supervisor listening to teachers explain their job and evaluate their activities without making any judgments. The supervisor's role in the alternatives option is to recommend or assist teachers in finding, better methods to accomplish things than they now do. The distinguishing factor of power in the supervisory roles determines the degree to which the observers can exercise control in their relationship with the teachers through post-conference discussions and feedback commentaries.

One of the challenging tasks faced by educators in observation is the different lenses they see through to suggest what effective teaching looks like, and it is likely that the same criteria will be subject to various interpretations (Archer et al., 2016). The challenging task, according to Wallace (1991), is complicated depending on the supervisory roles throughout this continuum which can also be compared with classic collaborative and prescriptive approaches to teacher supervision. There is an inherent conflict in discourses that teacher observers formulate in their communication with observees ranging from a discourse attributed to facilitators of professional development to that of assessors and evaluators of teachers’ performance. However, as Murdoch states, these roles and the emerging discourses need to elude appositional and problematic ways (1998). Bailey suggests that a general trend is toward a more collegial role and away from the inspector's role, although this trend is by no means universal; the degree of this transformation varies depending on the culture (2006). There is a concern that contexts such as EAL (English as an additional language) in Iran will compound the situation when observations are considered a means of evaluation and the observer must deliver some critical feedback. In effect, non-evaluative observations could not occur frequently in education without evaluative purposes in either state-owned or private-owned sectors for a number of practical and financial reasons.

Supervisory Discourse

Drawn from the unequal power of discourse that can be face-threatening (Bailey, 2006), the role of feedback and delivery of criticism has always been of significant concern to supervisors. Wajnryb’s study (1994a) indicated that in feedback conferences, supervisors 1) are mindful of how their words affect the instructors' faces and spirits and modify their statements accordingly, taking into account the unexpected and delayed reactions of the teachers; 2) must balance the conflicting needs of critic (judging, evaluating, highlighting flaws) and helper (advising, counseling, nurturing, encouraging, directing, and assisting); 3) must walk a tightrope between communicating a precise message to the instructor and not harming them with their remarks; 4) put a lot of effort into the affective components of empathy and trust to create an environment where critical input could be favorably appreciated; 5) protect the instructional message to offset the impact of mitigation on the word choice; and 6) make reference to the complication of their roles, show their awareness of how they have changed to become more successful practitioners and confirm the potential benefits of strategic training.

Wajnryb (1994a, 1994b) examined teacher educators' supervisory discourse in an Australian setting. In conversation analyses, she examined recordings of post-observation talks. Additionally, teachers responded to a questionnaire concerning the post-observation conference in which they received critical criticism from a supervisor regarding their teaching practice. The application of mitigation devices, face-threatening acts, their suprasegmental phonemes, voice quality, and nonverbal conduct are all included in these investigations and her subsequent works (1995a, 1995b, 1998). She studied the supervisors' use of syntactic, semantic, and indirect mitigating strategies, including above-the-utterance mitigation, hypomitigation, and hypermitigation. Chamberlin (2000) investigated supervisors’ nonverbal behavior and trust correspondingly pertinent to the concerns previously noted. She found that supervisors with dominance-induced nonverbal behavior were not seen as reliable as those with affiliation-related nonverbal behavior. Moreover, in clinical supervision, Hayashi and Hayashi (2002) examined the gender and linguistic style of the discourse during meetings with preservice school teachers.

The supervisory discourse was examined by Moradi et al, (2014) in Iran using a questionnaire and semi-structured interviews exploring EFL teachers’ perceptions of supervision practice. They suggested that class observation should be held more democratically and collaboratively when working with teachers and this should be reflected in positive, encouraging, constructive feedback. Razmjoo and Rasti (2014) suggested a framework of supervision including four components after examining the supervisors’ knowledge in the Iranian Ministry of Education. This framework has obvious implications for the efficacy of supervisory feedback. They concluded that, in order to provide transformative feedback, language teacher supervisors are required to be cognizant of both contexts of situation and culture as effective supervisors. Moreover, establishing rapport and a close relationship would eliminate defensiveness and resistance on the observees’ part toward desired changes.

Likewise, in a qualitative study, a six-component framework was proposed by Agheshteh (2017) that was based on gathering and obtaining realistic data through interviews and focus groups with Iranian supervisors and teachers. This framework was designed to assist supervisors in managing the intricate and multifaceted problem of supervisory feedback. His study suggests a structure in which supervisors can take a more innovative approach, employ mitigation above-the-utterance, meet the instructors' zone of proximal development, exhibit intercultural sensitivity, evaluate the beliefs and attitudes of teachers, and hone their public relations skills.

Academic Commentary as a Review Genre

The discourse found in the academy embodies a particular way of thinking and language. Academic discourse is required for the completion of complicated social actions such as educating pupils, displaying learning, spreading ideas, and constructing knowledge which incorporates writing reports, textbooks, dissertations, research articles, conference presentations, and lectures which are central to forming knowledge and education in academia. In all these texts, instructors and learners write to a specific group of people with whom they are working (Hyland, 2009). Since an overall comment as academic commentaries are given by observers to teachers in an educational setting and can be found within the field of teacher education, it can be regarded as an example of academic discourse. It is a way of conveying judgments or recommendations from supervisors to teachers.

Considering texts as “spoken and written instances of the system” and building on Swales’s (2004, p. 12) idea of constellations of genres, Hyland (2009, p. 27) recounted samples of academic genres, as illustrated in Table 1. Although not included, academic commentary both in written and spoken forms can be considered a part of the academic genre of review from the evaluation perspective.

In this interdiscursive, intertextual world, events are constructed and negotiated through the mediation of multifaceted genre discourse. Derrida points out that there is never a genreless text. In fact, all texts partake in genre(s), but this participation by no means equates to belonging (1980). Taking this view, we can see that the structural and rhetorical organization of academic commentary has a conceptual affinity with a number of adjacent genres including book reviews, review articles, and blurbs to name a few. Resorting to Swales’ view (2004), we can perceive that a genre stratum such as academic commentary may share several features of other genre strata. Considering the fact that their focal functions, though, are distinct, we can discover some likenesses in terms of rhetorical structures, textual strategies, or rhetorical devices of these sub-genres (Sorayyaei Azar & Hashim, 2017).

Table 1. Examples of Spoken and Written Academic Genres

The evaluative and persuasive nature of these reviews and the communicative functions they serve share a number of significant features with academic commentary which can be predominantly indicated by researchers. For instance, a balanced description of both positive and negative evaluations is an exemplary representation of a book review (Sorayyaei Azar & Hashim, 2017). In fact, the review genres' features and purposes include an introduction, giving information, evaluation, persuasion, and recommendations which incorporate a continuum and variations of text types criticality dependent on the communicative goals they represent (Sorayyaei Azar & Hashim, 2017).

Quantitative and Qualitative Protocols

Quantitative and qualitative approaches to observation as methods of documentation and recording of classroom events are the core of discussion in the field. These approaches as means of data collection and the repercussions for subsequent judgments illustrate the process and quality of learning and teaching. Recording the events of the classroom from the quantitative and qualitative perspectives may be viewed as occupying two opposite extremes. Nevertheless, many do not consider them as mutually exclusive, rather they can often complement each other (Wragg, 1999). Iran Language Institute (ILI) the sole comprehensive state-owned language institute providing language education including English, Spanish, German, French, Italian, Russian, Turkish, and Arabic established in 1925 targets three age groups of children, young adults, and adults. It is actively operative across the country with 290 branches in 131 cities and towns located in 31 provinces annually educating more than a million language learners in Iran. As a large organization with numerous departments, the ILI has a Teaching Affairs Department deploying class observations as an established practice to primarily lead the teaching staff toward performing a uniform teaching methodology at an acceptable quality standard. Equipped with observation forms, supervisors enjoy both quantitative and qualitative measures in class observation from which the text formulation as part of qualitative evaluation requires professional, academic heed.

As one of the important devices for evaluating the competence of education programs and teachers in order to make decisions with significant consequences (Mashburn, 2016), classroom observation has a substantial bearing on teachers’ evaluation and professional development. An overall comment or an academic commentary as a formal, official text and an offshoot of observation based upon which high-stake decisions are made for teachers is required to exploit rhetorical features responsive to the effective communicative function of professional development and evaluation. Moreover, since a copy of the observation form is received by the observee and other copies are also sent to the director of the Instructional Affairs Department of the institute, the delivery of written feedback outlining a follow-up action plan and development is a potentially sensitive arena of criticism. Such a recording and documentation can pose thorny issues as a critical tool for gleaning evidence regarding what happens in classrooms (O’Leary, 2020).  This study, consequently, can play an essential role in increasing the participants’ awareness of the generic features of the text that might exist in the Iranian context thereby avoiding misunderstanding and frustration on the part of both teachers and observers in terms of the effectiveness and characterization of macro and micro rhetorical features.

In response to these prerequisites, the current study is intended to address the subsequent research queries:

1. What rhetorical structures (Moves and Steps) characterize the academic commentary written by Iranian observers at the ILI?
2. How do the observed teachers and language observers at the ILI view the effectiveness of the academic commentary?

Methodology

This study is two-tier by nature. Tier one is the exploration of academic commentary from a genre perspective in terms of the dynamic area of research in ESP. The adoption of the ESP approach from various others such as Australian, Romantic, Structural, Reader-response, and Cultural studies lies in the claim stated by Dudley-Evans and John that it has style and form characteristics that are recognized by the discourse community. Thus, it has often pedagogical implications in exploring conventions of certain key genres about style and order of presentations of content and rhetorical factors that affect the readership (1998). Accordingly, tier one exploits both qualitative and quantitative methods in uncovering generic structures of the text in addition to the lexical and grammatical features. Resorting to this framework, the second tier investigates the interactional relation between observers as writers of academic commentaries and teachers as observees in the ILI context through a questionnaire.

Data

The choice of data is constrained to the particular genre of academic commentary in the observation forms. This form has been specially designed to minimize the time required to complete so that the observer can follow the class activities as closely as possible. Should the activities be carried out appropriately, there is little space to be cross-checked by the observer with accompanying short notes about the whole process. Since the academic commentary is situated at the end of an observation form under its rubric “overall comment”, difficulties and uncertainties of text identification were reduced. The pedagogic orientation of this study led the researchers to establish that the quantity of samples of academic commentary in the data as a cultural study should be generically representative. Thus, our selection procedure consisted of consulting three sources including the personal archive of one of the researchers and the personal archives of twenty teachers and twenty observers upon one of the researcher’s requests. The third source was archives located in 290 offices in 31 provinces at the ILI across the country. The researchers sent emails and WhatsApp messages to the directors of target branches, explaining the study and pedagogical aims, and asked them to produce PDF files of observation forms written in the previous five years (2018-2022). In order to secure thorough representativeness and diversity, they were requested to send five observation forms written by every observer in their branch. To ensure confidentiality, we also emphasized that all observation forms should be anonymous. As a result, the numbers of anonymous observation forms gleaned from three sources of data were respectively 42, 85, and 307. At this stage, purposive sampling was practiced to include observation forms written by as many, varied observers as possible. The resulting data, then, was reduced to 200 observation forms written during the designated period. For text manipulation, PDF files were converted into text format. Data pruning was exercised throughout the texts so that extraneous items would not affect the analysis. This exhaustive process ensured the typicality and representativeness of texts from the genre perspective in which the recurrence of rhetorical and linguistic features is pivotal.

Genre Analysis

In response to the first research question, the Swales’ approach (2004) was followed to examine the recurring rhetorical patterns of structure which consist of a sequence of moves and steps standing for the overall purpose of academic commentary. In his approach despite its original one employed for describing an RA introduction section, moves are defined as “discoursal or rhetorical units performing coherent communicative functions in texts whose linguistic realizations may be very variable in length and in other ways” (2004, p. 228-229). Accordingly, a move is “a rhetorical construct, the linguistic realization of which could be as short as a clause and as long as a paragraph. The function of a move is realized by the presence of one or more specific functions, or steps” (Swales, 1990, p. 141). The identification of move boundaries was delineated by a bottom-up approach for syntactic and lexical indicators signaling content shift and topic breaks of the text. We also drew on “the qualitative content analysis” (QCA) in which both qualitative and quantitative approaches to data treatment were considered as the stepping stones in interpreting manifest and latent meaning. To this end, we adopted the “directed” QCA where analysis is carried out with a theory and/or relevant research findings as guidance for initial codes (Fuad Selvi, 2020, p. 442-443). Premised upon this methodological approach, the data manipulation techniques resorted to an inductive, comparative, recursive manner to construct our framework. Finally, there is a good reason to rely on specialist experts to endorse the analyst's results, since they have stronger intuitions and a deeper understanding of language and the typical rhetorical patterns employed in a particular text (Tarone et al., 1981). As suggested by Moreno and Swales, additional analysts (or raters) may be needed to confirm the primary findings produced by the initial researcher (2018, p. 41). To avoid subjectivity in the analysis, inter-class and intra-class correlation coefficients were calculated in SPSS (version 26) using a two-way mixed model, and absolute agreement showed that the observed α=0.99 and α=1 were respectively excellent at P<0.05 (Tables 6 and 7).

Critical Genre Analysis

The second research question concerning the effectiveness of macro and micro rhetorical features embodied in moves and steps was addressed in a questionnaire that we developed based on Likert scales. The questionnaire evaluated and elicited the effectiveness of these features in academic commentaries from both observers’ and teachers’ perspectives. In fact, the questionnaire was designed to mainly cross-check the beliefs and uptake of observers and teachers in relation to how rhetorical features are perceived by both groups.

Table 2. Inter-class Correlation Coefficient

Table 3. Intra-class Correlation Coefficient

Critical Review of Predominant Rhetorical Structures

Subsequent to finding out constituent moves/steps fulfilling various communicative functions, we investigated the effectiveness of rhetorical structures from both observers' and observees’ views through a questionnaire. The findings of the text analysis were used to develop our 15-item questionnaire. We administered the constructed questionnaire twice. For piloting, two applied linguistics academics validated it prior to distributing it to a representative sample of 35 Ph.D. applied linguists from whom only 20 members filled out this questionnaire. Following data collection, analysis, and calculating the internal consistency of the questionnaire (α = .89), the major questionnaire was sent to a wider group of teachers and observers selected by purposive sampling (Table 4). Through social media communities and channels in Telegram and WhatsApp applications, respondents were requested to complete the questionnaire and forward it to others. To enhance the rate of responses, the link to the questionnaire was also distributed to a number of ILI branch directors to promote optimum cooperation.

Table 4. Demographic Information of Questionnaire Respondents

Since confirmatory factor analysis (CFA) requires 3 or 4 indicators per function (Boomsma, 1985), three items were designed for every five recurrent theme categories. The questionnaire was administered in English and Four-point Likert scales were used to score the attitudes of the respondents (ranging from ‘strongly agree' to ‘strongly disagree') with eliminated neural or undecided options to encourage making a decision on the part of respondents (Cohen et al., 2007). The final questionnaire was completed by 188 respondents. The sample size for CFA models with three or four indications per factor should not be less than one hundred; otherwise, small effects could occasionally give a deceptive indication of statistical significance (Boomsma, 1985).

The questionnaire's internal consistency and construct validity were assessed. To endorse the scale and determine whether the outcomes of the quantitative analysis of the questionnaire accommodated researchers’ understanding of the construct in the conceptual model, the measurement model's discriminant validity, convergent validity, and composite reliability were evaluated. The sections that follow assess both the measurement and structural models.

Confirmatory Factor Analysis

The validity of the postulated model was investigated through CFA. The goal was to assess the consistency of the observers’ and observees’ views in the framework examined in the qualitative phase of the study. CFA was conducted in this study to probe the relationship between the questionnaire responses and the 5 main functions of academic commentary. Based on the findings of the qualitative phase, five Steps were characterized under three main Moves at a higher abstraction level, namely, Establishing Rapport, Presenting Evaluation, and Closing. To observe the homogenous measurement model and determine the accuracy degree of the constituents and the relationship between the higher-order and the hidden variables, the values of the factor loadings of each of the indicators conforming to the latent variable are considered to be greater than .70. Some researchers suggest eliminating indicators with values less than.40 or.32 (Fornell & Larcker, 1981). The indicators can be preserved with a small number of indicators, a loading value of .70<, and the Average Variance Extracted of .50> for each construct (Fornell & Larcker, 1981). It is essential to determine whether the indicators ensure that the construct is measured consistently and accurately. To that purpose, the rhetorical features extracted were subdivided into five latent variables, for which three questions were developed to address Salutation and Appreciating Teachers’ Authority constituting the Move of Establishing Rapport, Praising Positive Teaching Aspects and Recommendations for Improvement materializing Presenting Evaluation Move, and Expressing Good Intention as part of Closing Move.

In order to test measurement model fit, AMOS 26 was used to compute CFA. As part of CFA, factor loading was assessed for each item with no factor loading below the critical level (< 0.5). Should the value of each factor representing the construct (t-value) be positive and greater than 1.96 at .05 and greater than 2.58 at the .000 level, the indicator can precisely measure the latent variable (Fornell & Larcker, 1981). All the items showed acceptable factor loadings and the t-value relevant to each factor loading was higher than its critical value amounting to 2.58 at 0.000 level (see Table 9 in the Appendix). As a result, all of the indicators can be retained in the final analysis, for they have adequate precision in measuring the latent variables. Furthermore, the model-fit measures (CMIN/DF, TLI, CFI, GFI, RMSEA, and SRMR) were utilized to assess the model's overall goodness of fit, and all values were within acceptable levels. The five-factor model output (Salutation, Appreciating Teachers’ Authority, Praising Positive Teaching Aspects, Presenting Evaluation, and Expressing Good Intention) was appropriately fit for the data (See Tables 10 and 11 in the Appendix).

As well as CFA, composite reliability of the measurement model, discriminant validity, and convergent validity were also measured. Using composite reliability, the internal consistency reliability of the model was assessed. Acceptable reliability for composite reliability measures rests upon values greater than .70. The Dillon-Goldstein’s rho values for all related variables to AC constituents were found to be above the criterion, confirming the measurement model's reliability (Note Table 12 in the Appendix).

The average variance extracted (AVE) coefficient was determined to confirm the variables' convergent validity. In order to be considered valid, the latent constructs in the model must account for the lowest 50% of the variation from the observable variables. According to the estimated index values in Table 12A, all of the AVE values surpassed 0.50, confirming the measurement model's strong internal consistency and convergent validity.

The discriminant validity of the factors was determined by examining the cross-loadings using the Fornell-Larcker criterion test. Based on the index of Fornell-Larcker, for each latent construct the square root of the AVE should be greater than the correlations of other latent variables. The square root of AVE for each latent variable, as shown in the tables below, exceeded the maximum correlation of the latent variables, suggesting that the measurement model has adequate discriminant validity for all factors (Note Tables 12, 13, and 14 in the Appendix).

Procedure

After the data was obtained and pruned, employing word count, the total number of words and the average length of academic commentaries were calculated. Iterative strategies suggested in the grounded theory were taken as a contributory factor toward data analysis with constant comparative analysis and theoretical sampling. Fracturing and re/interpreting the data, the researchers exercised data coding in a meaningful way through simultaneous analysis of ten percent of the data amounting to 20 academic commentaries. Our interpretations were constantly reflected in memos and extended and mapped against the remaining data to construct our framework (Hadley, 2017). This secured our analysis in terms of both the inter-coding and validation process and reliable results. From the outset and as part of open coding, careful word-by-word, line-by-line coding led us toward an empirical fit and relevance. Table 2 shows the crystallization of field notes as separate themes.

Constructed codes developing into categories depict our second-stage coding (Table 6) which describes the features and properties of the categories and reunites the data we have broken during initial coding to bring the developing analysis into coherence (Charmaz, 2014). This stage elevated our analysis to the culmination of final-stage coding (Table 7).

Furthermore, the emphasis was on Paltridge's discussion of the content and function of academic commentary in which “formal aspects and structural features are to be observed in terms of content and function.” (1995, p. 397). Subsequently, every academic commentary's occurrence of moves and steps was calculated. As part of intra-rater validation, one of the researchers replicated the analysis in a one-month interval. Then, in order to measure the linear relationship between two raters, Cronbach’s alpha correlation coefficient was employed to examine whether the possible frequency differences between the two ratings are statistically significant.

Table 5. Sample Academic Commentary Representing First Stage Coding

The next step was designing and administering the questionnaire which required a rigorous process as part of devising an instrument to yield reliable and valid data. Hence, to construct the questionnaire, the researchers followed Dörnyei and Csizér’s instructional process in which six key design issues including an appropriate sampling of content, types of questionnaire items, writing items that work, the format of the questionnaire, translating the questionnaire, and piloting the questionnaire were taken into account (2012).

Not applicable in this study, translating the questionnaire was ignored. As a result, the designed questionnaire was of cross-sectional type with two independent samples of observers (N=66) and teachers (N=122). The questionnaire elicited the participants’ critical attitudes towards the effectiveness of macro and micro rhetorical features extracted in the initial phase of analysis. The questionnaires were sent to the participants with a request letter in advance to announce the goal of the study via emails and WhatsApp messages to complete. Since the questionnaires are scale-based measuring the attitudes of participants and as part of securing the internal consistency, Cronbach’s Alpha coefficient was calculated.

Table 6. Sample Academic Commentary Representing Second-Stage Coding

Results

In response to the first research question, the descriptive statistics related to the rhetorical structure explored in the academic commentaries is reported and discussed in this section. The order in which the moves are presented does not impose a canonical order. Additionally, the description of the rhetorical structure throughout the data facilitates our analysis to answer the second research question which is the critical review of text effectiveness in such a performance review genre. Each individual move and the related steps are described and exemplified by instances from the data.

The Rhetorical Analysis of Academic Commentary

The constituent moves/steps that appeared throughout the academic commentary texts can be construed as three moves and respective thirteen alternative steps. Table 8 represents the move/step structure of the academic commentaries written at the close of observation forms.

The description, exemplification, and analysis of move/step are confined to the rhetorical features that we assume have much bearing on the overall organization of our data set. The steps with minor roles were disregarded due to the scant frequency of occurrence (less than 25%).

Table 7. Sample Academic Commentary Representing Third-Stage Coding

Establishing rapport

The first move encompassing 74% of the data serves as the opening of the academic commentary which likely attempts to reduce the distance between the observer and the observee. It may promote solidarity thinning the air of authoritativeness and status of the observer. That is, it is likely to serve as a commitment on the part of the observer to establish a relationship with the teacher before embarking on presenting an evaluation and passing judgment.

This move is realized by two alternative steps of Salutation (39%), e.g. “Dear colleague, Dear Mr./MS X”, and Appreciating teachers’ authority (61.5%). Following salutation, the observer seems to be aware of the prominence of the teacher’s authority and calls to it in his/her text as a means of establishing rapport through linguistic elements such as:

[Text 125]: Thank you for having me in your class on such a short notice.

[Text 85]: Thank you very much for allowing me to attend and enjoy your class.

Table 8. Percentage and Frequency of Moves/Steps in 200 Academic Commentaries

It suggests that since attending a class for observation can be considered as an encumbrance and thus unfavorable by the teacher, recognizing and appreciating teachers’ authority for granting permission may settle the issue of an embittered, unexpected visit on the one hand and establish rapport on the other. Although it seems an obligatory move, its constitutive steps were not consistently employed. Interestingly, the observers tending to shun this move employed pronouns such as “he” or “she” instead of directly addressing the observee throughout the rest of the commentary. In fact, their texts were almost devoid of communicative and interactional elements with the reader. Since Salutation fell short of our expectation to permeate a good number of data, we decided to address it in the questionnaire in terms of how it will affect the readers.

Presenting evaluation

The second move prevalent in 97% of the data comprised evaluating the teacher’s performance materialized by eight alternative steps. The frequency of occurrence confirms that this move is a vital component of academic commentaries. It is unsurprisingly expected on the part of observers to present their ideas on how the teacher introduced the lesson in the classroom.

The first dominant step, Praising positive teaching aspects, highlights in a rather detailed fashion the positive teacher’s performance. It visibly specifies general commendation for the performance of teachers.  This sub-move is represented by an example below:

[Text 124]: The overall performance was indicative of the teacher’s determination and commitment to pursue growth through keeping the teaching process in line with the guidelines prescribed in the methodology booklet. The teacher conducted the teaching process mostly accompanied by techniques that facilitated students’ interaction. The teacher’s not being a solo-speaker and implementing good techniques to stimulate students ’communicative performance declares this session’ teaching process a success.

Second, the recommendation for improvement is suggestive of statements that serve to provide professional advice and recommendations for the improvement of teaching practice. The following excerpt sets as an instance:

Text 183]: You are kindly requested to have the students describe the scene at your discretion not always on a voluntary basis. Also try to involve more silent, reserved students. Please try to budget your time to the demands of the timing section in the teachers’ guide.

After praising positive teaching aspects (70.5%), observers seem to be mostly concerned with the core of their observation (Recommendations for improvement, 67.5.5%) which suggests strategies that can be found as assistance for better performance on the part of teachers. Since these two steps occupied the maximum number of occurrences in this move, they were included in the questionnaire.

Closing

Unexceptionally all the academic commentaries in the data witnessed the Closing move with at least one or a number of constituent steps. This move represents the conclusion of the observer’s comments. It suggests that the observers are rather cognizant of the fact that the text they are composing is communicatively required to be finalized with at least a textual ending such as a signature (100%). However, the frequency of two steps of reiterating appreciation (18%) such as “Thank you again; Thank you and good luck” and expressing good intention (25%) were rather low as signals of the finale. Note the following examples applied to realize expressing good intention:

[Text 101]: I hope to have the pleasure to visit your class soon again.

[Text 46]: Wish you all the best.

We included the Expressing good intention step in our questionnaire in order to analyze their effectiveness as parts of the academic commentary.

Discussion and Conclusion

In an attempt to analyze the academic commentaries written by observers in the Iranian EFL context and explain the constituents accordingly, we began considering the rhetorical features and exploring the distinction between different observers’ comments on teachers' performance. These elements fulfilling various communicative functions facilitated analyzing the effectiveness of the commentaries from both observers’ and teachers’ perspectives.

The result of the study shed light on the fact that both observers and observees support the general framework drawn from our analysis of the generic organization of the academic commentary. However, there are non-conformities in the observers’ approach as far as macro and micro rhetorical features are concerned. It appears that Establishing Rapport present in 74% of the data was not employed throughout all the steps of Salutation (39%) and Appreciating Teachers’ Authority (61.5%). In terms of effectiveness, the non-observance of such a move with the constituent steps is significant.

Presenting Evaluation move present in 97% of all texts is indicative of the observers’ inclination to express a form of performance evaluation. This communicative function suggests that the purpose of observation is the assessment of the teachers’ performance which is designed for quality check and controlling uniform teaching methodology and standards. However, the realization of this move through two prevalent steps of Praising Positive Teaching Aspects (70.5%) and Recommendations for Improvement (67.5%) compared with other rhetorical patterns was significant. Therefore, should observers tend to characterize performance evaluations, the precedence of Praising Positive Teaching Aspects can show his/her comprehensive outlook in that the purpose of evaluation is not highlighting teaching malpractice. Rather, this juxtaposition facilitates the perception of recommendations for improvement and furtherance of the ultimate goal of class observation. On the other hand, instead of criticizing, what may fall into effective practice is suggesting ways so that the observee realizes how to exercise and extend efforts towards the best practice through Recommendations for Improvement.

Closing tended to occur in all the data at least with a signature step. Nevertheless, our analysis illustrated how non-observance of Expressing Good Intention (25%) may affect the effectiveness of the academic commentary from both observers’ and observee’s viewpoints.

The qualitative phase of this research endorsed the panoramic understanding of the formulation of academic commentary by observers as a review genre. The qualitative analysis of the data depicted the tendency of initiation of the commentary by establishing rapport through Salutation and Appreciating Teachers' Authority, though not practiced by a considerable number of observers. Presenting Evaluation followed by the first move characterized by eight micro rhetorical features among which praising the positive teaching aspects and recommendations for improvement were more favored by observers. Still, the frequency of the other six micro rhetorical features including stating appreciation for positive teaching aspects, endorsing current teaching practice, stating criticism against the performance/situation, Stating a reason for poor performance, recommendations for further observation and evaluation, recommendations for redeeming the course can also either raise consciousness or make caveat in terms of exploitation on the part of observers. Ultimately, closing the commentary was shown to be realized by expressing good intention after the prevalent step of signature which singled out observers who were cognizant of signaling the finale of the commentary.

The result of the quantitative analysis, on the other hand, allowed cross-sectional evaluation of observers’ and teachers’ attitudes and perceptions shunning the adoption of the standard practice of assessment-based observations for quality assessment purposes which is indicative of casting a "normalizing gaze" (Foucault, 1977, p. 184). It corroborated the fact that the communicative functions of the rhetorical features were all indicators of effective supervisory comments which were found to be valid and reliable. In establishing rapport, it was found that it is significant to appreciate the interactive nature of such a genre and observe the immediate constituent elements encompassing dialogic textual features. In fact, a faceless, impersonal text communicating the message to a reader other than the observed teacher may be considered ineffective. Moreover, making the mention of teacher’s authority and his/her permission for observation can play an important role in constructing solidarity and establishing rapport. Secondly, presenting evaluation should start with pinpointing positive aspects of teaching practice. Without noting teachers’ strengths at the outset, presenting evaluation may seem merely a load of criticism leveled against their teaching performance. As a result, recommendations for improvement can be considered as a measurement of weaknesses than suggesting constructive and good teaching practices. Thirdly, observers seeking to finalize their commentary with a credible evaluation expressed their good intention to show solidarity with observed teachers. This can acknowledge 

alternative views and control the level of personality in the commentary rendering it to a convincing argument. In other words, every effective academic commentary displays the observer’s awareness of the teacher as the primary reader and its consequences.

In an effort to study observation reports, Dinkelman (2014, p. 50) identified three types of observation reports The Positive, The Minimalist, and The Reformist, although he claimed that his interest was not classification, each posing its own questions for an educational program in terms of what kinds of recommendations would help supervisors write more effective reports. Unlike such a valuable effort, though too broad of a description, and considering the impossibility of a rubric to include all professional features a teacher can demonstrate, what differentiates our study is the adoption of the rhetorical analysis with an ESP orientation to reflect distinct elements of the immediate anatomy of supervisory discourse.

Clearly, observation is an instrument that can best serve educational institutions that have made decisions not to have it used by anyone without adequate training and preparation. Furthermore, a balanced and realistic reflection of both qualitative and quantitative aspects can warrant degrees of reliability of observers’ evaluations. However, evidence from current research points to qualitative feedback as the major aspect of the observation procedure (O’Leary, 2020). Therefore, the textual part of this instrument should not be overshadowed by the recording of quantitative, numerical data. It would be necessary to set in place an ongoing specific training and guidance provided for observers which requires careful planning by qualified staff reflecting recent research.

Although the findings of this research enrich our understanding of the academic commentary, we should address the size and representation of our dataset as part of the limitations. As this study is investigating observers’ commentary as a genre, the use of specialized written academic feedback is sufficient and representative. However, the current study's conclusions, based on 200 feedback commentaries on genre patterns, cannot be utilized to establish broader generalizations regarding written academic commentary in all English contexts. Conclusively, as demonstrated in this study and earlier in the literature review section, it is certain that supervisory commentary is subsumed under the genre of review.

Further studies on supervisory discourse are required to provide insights into how to formulate effective commentary, especially from an evaluative perspective. In addition to replicating this study in other contexts, supplementary research can be conducted to reveal effective features of recommendations for improvement and discover whether the formulation of comments is in line with those of teachers’ views from different discourse perspectives such as stance and engagement (Hyland, 2004, 2009) or appraisal (Martin, 2000). Finally, a comparison can also be made to see if there is a significant difference between the commentaries structured by more and less experienced observers.