Exploring Research Designs, Purposes, and Tools of Data Collection in Applied Linguistics Mixed Methods Research

Introduction

Mixed Methods Research (MMR), the integration of quantitative and qualitative paradigms across different stages of a research design, has received considerable attention as a valuable tool for addressing complex phenomena in the social sciences (Shan, 2022). In the same manner, various researchers have mentioned the pivotal role of MMR in illuminating the complex issues related to language learning and teaching (e.g., Brown, 2014; Hashemi, 2023; Hashemi & Babaii, 2013; Riazi & Amini Farsani, 2024; Riazi & Candlin, 2014). However, to achieve the full potential of MMR in addressing such complex issues, some critics have warned against the simplistic applications of such designs (Bergman, 2008). In other words, the quantitative and qualitative strands should be integrated across all stages of a design, i.e., at the methodology level. Therefore, it is vital to ensure the quality of these research designs (Tashakkori & Teddlie, 2003), and discussion regarding such design characteristics is among the most debated topics in the MMR literature (Fabregues & Molina-Azorin, 2017). Recent reviews, such as Riazi and Amini Farsani (2024), have further emphasized the importance of quality and transparency in MMR studies, highlighting the need for rigorous research practices in the field.

Although there is no consensus on the features of a sound MMR design, methodologists try to promote high-quality designs by establishing various criteria. Proposing and appraising these guidelines helps mixed methods practitioners in a number of ways. First and foremost, it enables researchers to design and conduct MMR transparently and rigorously (Fabregues et al., 2021). In other words, these quality criteria work as a framework that aids MMR practitioners in ensuring the warrantability of the findings. Second, examining the implementation of the proposed quality criteria, using systematic reviews of MMR designs sheds light on the strengths and weaknesses of the current practices of MMR (Hashemi & Babaii, 2013). Third, it helps consolidate the point that high-quality MMR is characterized by the integration of quantitative and qualitative strands across the whole stages of a design (Soodmand Afshar & Ranjbar, 2023). Finally, it lets researchers become familiar with the characteristics of sound examples of MMR in general (Creswell & Plano Clark, 2018).

While the importance of designing high-quality MMR is widely acknowledged, these qualities can vary across different disciplines, research contexts, and geographical locations (Collins et al., 2007; Molina-Azorin, 2011). This emphasizes the need to examine the nature of MMR designs in various disciplines, including Applied Linguistics (AL), and its different research domains, such as English as a Foreign Language (EFL). Although the study of MMR designs in AL has received a fledgling attention (Hashemi, 2012; Hashemi & Babaii, 2013; Riazi & Candlin, 2014; Soodmand Afshar & Ranjbar, 2023), studies addressing the quality issues in the Iranian EFL context remain scarce (see Amini Farsani & Mohammadi, 2022). In the same manner, much of the existing literature on MMR quality focuses on studies indexed in databases like Web of Science or Scopus. However, research published in databases such as the ISC, remains largely unexplored in this regard. This study addresses this gap by providing the first systematic analysis of MMR articles published in ISC-indexed journals within the field of Applied Linguistics. As a part of a larger project, the present study aimed to examine the purposes, research designs, and tools of data collection employed in MMR studies. It should also be noted that another gap in the literature exists regarding the examination of data collection tools used in MMR within AL. This study also addresses this gap by providing the investigation of these tools, both in the broader field of AL and specifically within the Iranian EFL context.

Literature Review

MMR has been conceptualized as the integration or combination of quantitative and qualitative strands across all stages of a design. Creswell and Plano Clark (2018) argue that researchers in mixed methods designs collect and analyze quantitative and qualitative data, integrate these two forms of data and their results, put these procedures into specific designs, and organize them into specific theories and philosophies. In what follows, first, philosophical and methodological aspects of MMR are discussed, and, then, a brief review of studies examining MMR designs in AL is presented.

Philosophical and Methodological Underpinnings of MMR

The last decades of the twentieth century witnessed a paradigm war between the advocates of quantitative and qualitative methodologies with their underlying positivist/postpositivist and constructivist/interpretivist philosophical orientations, respectively. One of the main consequences of this paradigm debate was the incompatibility thesis in which it is improper to combine quantitative and qualitative methods due to the ontological, epistemological, and axiological differences in the two paradigms (Sale et al., 2002; Teddlie & Tashakkori, 2009). However, mixed methodologists, later, proposed the compatibility thesis (Howe, 1988) with various philosophical positions to show that it is possible to integrate quantitative and qualitative approaches. From a practical perspective, philosophical positions like pragmatism, tranformativism, and dialectical pluralism try to counter the incompatibility thesis by showing that mixing not only expands our understanding of the object of study (Hashemi & Babaii, 2013), but also offers multiple ways of making sense of the social world (Greene, 2007). Shan (2022) summarizes three different senses of philosophical underpinnings of MMR. Pragmatism is the weakest sense of these philosophical foundations. It only states that mixing quantitative and qualitative approaches is possible without providing a reasonable justification for this integration. The moderate sense of these philosophical foundations is transformativism in which a good reason is provided for mixing. Finally, dialectical pluralism, the strongest sense of these philosophical foundations, not only provides a good reason but also justifies that MMR should be encouraged in social sciences.

MMR is more than using at least one quantitative and one qualitative method in a single study. Principled MMR includes mixing within and across different stages of a research design (i.e., methodology). These stages are research questions, sampling procedure, data types, data collection tools, data analysis, and concluding inferences (Alise & Teddlie, 2010; Tashakkori & Creswell, 2007a; Yin, 2006). Various guidelines and frameworks have been proposed for designing high-quality MMR within and across these stages. As for the research questions, Tashakkori and Creswell (2007b) argue that in addition to the quantitative and qualitative questions, “mixed methods studies will benefit from at least one overarching mixed (integrated, hybrid) question” (p. 210). With regard to the sampling procedure, the dichotomy between probability and purposive sampling, in MMR, turns into a continuum with variants like stratified purposive sampling and purposive random sampling (Teddlie & Tashakkori, 2009). Moreover, Collins et al. (2007) proposed a framework for sampling in MMR based on the time orientation of the components (i.e., concurrent or sequential) and the relationship between qualitative and quantitative samples. The second part of their framework includes an identical sample: quantitative and qualitative samples are the same; (b) a parallel sample: the same population is used for sampling quantitative and qualitative members; (c) a nested sample: a sub-group of one strand (e.g., quantitative) is used for the sample of the other strand (e.g., qualitative); and (d) multilevel sample: different populations are used for quantitative and qualitative sampling. Concerning the design features, two major types are convergent and sequential (i.e., subdivided into exploratory and explanatory designs) (Creswell & Plano Clark, 2018). It should be noted that the initial criterion for classifying MMR designs was based on timing in which quantitative and qualitative phases of a study are either conducted simultaneously or sequentially (e.g., Creswell et al., 2003). However, recent typologies tend to highlight the intent of a study rather than timing or sequencing (e.g., Creswell & Plano Clark, 2018).

MMR Design Reviews in Applied Linguistics

In a pioneering study, Hashemi and Babaii (2013) examined the integration of quantitative and qualitative strands in the field of AL by evaluating research designs, sampling procedures, and concluding interpretations of articles published in seven leading journals. While they found that concurrent designs and sampling procedures were mostly adopted in the corpus, high degrees of integration were barely exercised in concluding interpretations and meta-inferences.

Then, in a state-of-the-art article, Riazi and Candlin (2014) explored the status of MMR studies in language learning and teaching journals. They sampled 40 articles published between 2002 and 2011 and analyzed them based on the use of mixed methods collocation, purposes, research designs, data analysis, and concluding interpretations. First, they divided their corpus into two categories of a) studies that used mixed methods collocation in their design (i.e., 18 articles) and b) the ones that used the terms quantitative and qualitative to describe their design (i.e., 22 articles). In the first category, they found that most studies
(i.e., 83%) achieved high degrees of integration. However, in the second category, almost all studies did not represent high-quality MMR.

Amini Farsani and Mohammadi (2022) took a closer look at the quality of unpublished theses that used mixed methods research in the Iranian EFL context. They addressed the formulation stage, design, sampling, and integration in these mixed methods theses. In the design stage, they found that sequential designs were mostly used in the corpus. Overall, they suggested that more attention should be paid to transparency and integration at the method, design, and interpretation levels.

In more recent studies, Soodmand Afshar and Ranjbar (2023) examined the current state of MMR in AL research. The study found that MMR articles still lack strong integration of quantitative and qualitative data, and identified challenges faced by researchers using mixed methods. Furthermore, Riazi and Amini Farsani (2024) conducted a comprehensive review of 304 MMR studies published in 20 leading AL journals (2011-2020), employing a six-pronged quality and transparency framework. Their key findings revealed a significant increase in the use of MMR in AL, with triangulation being the predominant purpose. However, they also identified critical shortcomings in transparency, particularly concerning the lack of explicit explanations of MMR purpose, design, and, most notably, sampling procedures. Furthermore, they found a preference for core designs over more complex MMR designs.

Overall, while it has been argued that design considerations are context and discipline-dependent (Creswell & Plano Clark, 2018), scant attention has been drawn to the investigation of MMR designs in the Iranian EFL context. Moreover, to the best of our knowledge, no study has examined the status of MMR designs in ISC articles in the field of AL. Thus, the following four research questions are addressed in the present study:

1. What designs are utilized in MMR articles published in Iranian AL ISC journals?
2. What data collection tools are utilized in MMR articles published in Iranian AL ISC journals?
3. What purposes are behind MMR articles published in Iranian AL ISC journals?
4. Overall, based on the qualitative and quantitative findings, what is the quality of MMR articles published in Iranian AL ISC journals?

Methodology

Design

The present study employed an exploratory sequential mixed methods design. It consisted of an initial qualitative content analysis of the corpus followed by a quantitative frequency analysis. First, in the qualitative phase, based on the various models and frameworks, designs, tools of data collection, and purposes of the identified MMR studies were investigated. Then, in the quantitative phase, the qualitative findings were further explored to determine their recurring patterns in the corpus. Therefore, as for data analysis, this study employed descriptive statistics (i.e., frequency analysis) and thematic analysis (i.e., content analysis).

Furthermore, the current study used multilayered sampling in which the journals were sampled based on nonprobability sampling, and articles were chosen based on probability sampling (Alise & Teddlie, 2010). In line with the recommendations of Fàbregues and Guetterman (2024), a journal-based search strategy, focusing on leading AL ISC journals was adopted. More specifically, in sampling the journals, outlier or extreme case purposive sampling was used to identify Iranian AL journals with the highest Citation Impacts (CI) in the ISC database, released in 2019. The purposive sampling of the most prestigious and leading Iranian AL journals is an example of outlier or extreme case sampling. In sampling the articles, systematic random sampling (Ary et al., 2019) was employed to select every second MMR article in the corpus. The overall number of high-quality MMR studies in the corpus of this study comprised 607 articles. Reviewing the literature showed that similar MMR quality reviews used approximately 200 articles for their analysis (e.g., Hashemi & Babaii, 2013; Soodmand Afshar & Ranjbar, 2023). Therefore, using systematic random sampling, 303 articles were identified as a valid sample for the final analysis. It should also be noted that the sampling typology of this study is sequential identical (Collins et al., 2007). It means that the corpus was used to investigate the same data in both quantitative and qualitative ways, sequentially.

A Standard Framework for Design Review in MMR

Informed by the linguistic, conceptual, procedural, and consequential inconsistencies in the frameworks for evaluating MMR quality in general, Tashakkori and Teddlie (2010) proposed a framework for systematically analyzing MMR designs. This model operates at three levels of input, process, and outcome. To put it simply, the input level is concerned with the formulation stage, such as research questions and data collection procedures. Next, data analysis features are addressed at the process level. Finally, the outcome level is concerned with integrated interpretations or meta-inferences. In the present study, this systematic model initially worked as a blueprint to check the validity and comprehensiveness of the criteria used for identifying high-quality MMR articles. Subsequently, the same model was applied for analyzing the components of the identified articles. However, it should be noted that not all the identified MMR studies were integrated across all of the above-mentioned levels. The term quasi-mixed studies is introduced in the literature to account for MMR studies that are not totally integrated in their conceptualization, execution (Bergamn, 2008), or inferences (Teddlie & Tashakkori, 2009). Therefore, in line with Alise and Teddlie (2010) and Hashemi and Babaii (2013), it would be better to label most MMR articles analyzed in the current research as quasi-mixed studies.

Corpus Compilation and Data Collection

As stated earlier, this study examined the current status of MMR designs in leading Iranian AL journals. An extreme case purposive sampling was used to select the most credible Iranian AL journals in the ISC database. In doing so, six Quartile 1 journals (i.e., Research in English Language Pedagogy (RELP) CI = .469, Journal of Applied Linguistics and Applied Literature: Dynamics and Advances (JALDA) CI = .429, Iranian Journal of Language Teaching Research (IJLTR) CI = .295, Journal of English Language Teaching and Learning (JELTL) CI = .229, Teaching English as a Second Language Quarterly (TESLQ) CI = .229, Journal of Language and Translation (JLT) CI = .218) and six Quartile 2 journals
(i.e., International Journal of Research in English Education (IJREE) CI = .204, Journal of Research in Applied Linguistics (RALs) CI = .2, Applied Research on English Language (AREL) CI = .188, Teaching English language (TEL) CI = .161, Journal of Language Horizon (JLH) CI = .160, Issues in Language Teaching (ILT) CI = .125) were identified.
It should be noted that the Quartile 2 Journal of Foreign Language Research (CI = .173) was eliminated from the sample because it is published in Persian, while all the others are published in English. It’s also worth mentioning that the journals’ quartiles and impacts might have gone through some modifications based on the later announcements by ISC, and the ones reported here, as stated above, are based on the report released in 2019.

All identified journals were open access, and the Portable Document Format (PDF) of their articles was downloaded between 2015 and 2022, resulting in 1965 articles. Then, the abstract and methodology sections of all of them were scrutinized to figure out if quantitative and qualitative mixing occurred in their research questions, sampling procedure, data collection procedure, types of collected data, data analysis, and concluding interpretations. Additionally, using WordSmith Tools version 8, for each journal, a specialized corpus was compiled based on keywords such as mixed methods research, triangulation, integrating methods, synthesis, mixed methodology, multimethod, combining methods, mixed methods, mixed research, quantitative, and qualitative. These keywords were selected based on reviewing the related literature (Creswell & Creswell, 2018; Hashemi & Babaii, 2013; Soodmand Afshar & Ranjbar, 2023). To ensure that all of the designs using mixed methods are included in the sample, all occurrences of the above-mentioned keywords were checked in the corpus.

In this first screening phase, 689 articles were identified. Then, in the second screening phase, a rigorous qualitative investigation of abstracts and methodologies of these 689 articles was employed to ensure that the selected articles possessed a high degree of quantitative and qualitative mixing. In this study, articles that enjoyed a high degree of integration are labeled as MMR. In line with Alise and Teddlie (2010), a high degree of integration is operationally defined as “having more than one of the categories of sampling, data collection, data type, and data analysis being a heterogeneous mixture of QUAN and QUAL methods” (p. 111). To triangulate the criteria of identifying MMR with a high degree of integration, furthermore, Tashakkori and Creswell’s (2007a) possible ways of mixing at different stages of an MMR study were also considered:

• Two types of research questions (with qualitative and quantitative approaches),
• The manner in which the research questions are developed (participatory vs. preplanned),
• Two types of sampling procedures (e.g., probability and purposive)
• Two types of data collection procedures (e.g., focus groups and surveys),
• Two types of data (e.g., numerical and textual),
• Two types of data analysis (statistical and thematic), and
• Two types of conclusions (emic and etic representations, “objective” and “subjective,” etc.) (p. 4).

Therefore, in addition to the criteria proposed by Alise and Teddlie (2010), the operational definition of a high degree of integration in this study also entails mixing at the level of research questions, the manner in which research questions are formed, and meta-inferences or general interpretations. Taken together, for a study to be qualified as an MMR, mixing should occur at more than one of the above-mentioned stages.

In this second screening phase, 82 articles were identified as not having a high degree of quantitative and qualitative mixing. Therefore, the final sample of articles was composed of 607 articles. The final point worthy of mentioning is that data collection and analysis in this study were based on both ‘manifest content’ (i.e., authors’ explicit use of the above-mentioned keywords) and ‘latent content’ (i.e., researchers’ interpretation of the content) (Alise & Teddlie, 2010).

Data Analysis

In the data analysis stage, the high-quality MMR articles were examined based on their research designs, data collection tools, and purpose. To systematically achieve this end, operational definitions were provided in a code sheet. It should be noted that despite the advances within MMR typologies since the early design reviews in AL, this study mostly adopted Hashemi (2012) and Hashemi and Babaii’s (2013) typologies to make possible valid comparisons between the previous works that have been done in the literature. In doing so, we tried to avoid linguistic inconsistency (Tashakkori & Teddlie, 2010) in MMR design reviews of the field of AL. It should be noted that, however, some violations from previous studies were inevitable, and detailed explanations are provided wherever this happened. Overall, in the current study, the code sheet included:

• Design,
• Tools of Data Collection, and
•  

Research designs were categorized based on the frameworks proposed by Creswell
et al. (2008) and Creswell and Plano Clark (2018). Creswell et al. (2008) classified MMR designs into concurrent (i.e., using quantitative and qualitative methods concurrently) and sequential (i.e., using quantitative and qualitative methods sequentially). Concurrent designs were further classified into: a) Concurrent Triangulation Designs and b) Concurrent Embedded Designs. Three types of sequential designs were: a) Sequential Explanatory Designs; b) Sequential Exploratory Designs; and c) Sequential Embedded Designs. It should be noted that, according to Creswell and Plano Clark (2018), timing (i.e., whether quantitative or qualitative phases are conducted concurrently or sequentially) is a difficult standard to apply in practice. Therefore, in this study, the primary criterion for analyzing designs is based on their intent, followed by the timing. The intent of a design in MMR includes exploring, explaining, or converging. In so doing, the validity of the identified designs was highly increased because quantitative and qualitative data may be collected at the same or multiple points of time in both concurrent and sequential designs. According to Teddlie and Tashakkori (2009), in numerous cases of concurrent designs “the two data types are collected at different times due to practical considerations (e.g., the research team cannot collect all of the data at the same time)” (p. 129). Furthermore, sometimes, the procedure explained by the researchers in the method sections does not encompass all the necessary information regarding the timing of the collected and analyzed data.

As for tools of data collection, the analysis was done based on Cohen et al. (2018), in which the methods of data collection are comprehensively tabulated and discussed. However, it must be noted that some of the categories detected in the current corpus do not exactly coincide with the ones proposed by Cohen et al. Finally, the purposes of using MMR were examined based on Greene et al.’s (1989) classification. They subdivide the purposes behind using MMR as: a) Triangulation; b) Complementarity; c) Development; d) Initiation; and
e) Expansion.

Intra- and Inter-coder Reliability

To ensure the reliability of findings, intra- and inter-coder reliability were conducted. As for intra-coder reliability, one of the researchers analyzed the corpus twice over a span of one month. The findings of Cohen’s Kappa (κ = .86) revealed a perfect agreement. With regard to inter-coder reliability, one of the researchers conducted two sessions of one-hour training for a Ph.D. candidate in Applied Linguistics to acquaint him with the models and frameworks used for data analysis. Then, 10% of the corpus was analyzed separately. Next, the coding categories put forth by one of the researchers and that Ph.D. candidate were compared to ensure intercoder reliability. The findings of Cohen’s Kappa (κ = .79) showed an acceptable agreement.

Results

The design review of MMR articles published in Iranian ISC AL journals in terms of their design, tools of data collection, and purposes is reported in this section.

Design Types

The frequencies of Iranian Applied Linguistics MMR design types are shown in Table 1. Among the 303 MMR articles of the corpus, 176 (58.0%) used sequential designs,
105 (34.6%) adopted concurrent designs, and 22 (7.2%) were designs that cannot be classified according to the typology proposed by Creswell et al. (2008). Within the sequential designs, 137 (45.2%) favored the exploratory one, the most frequently adopted design. Moreover, 33 (10.8%) and 6 (1.9%) studies were explanatory and embedded sequential, respectively. As for the concurrent designs, 68 (22.4%) employed triangulation, the second most frequently used design. Finally, 37 (12.2%) identified MMR articles utilized embedded concurrent designs.

[11] Considering the results of these two steps mentioned above (i.e., literature review and interviews), this study developed the CPQ based on the following procedures. Based on the literature and the grounded work, about 105 Likert scale items were written in English. Exploratory Sequential Design (Adel et al., 2019, p. 341)

[87] As the quantitative results display that the two groups of test takers had different perceptions and evaluations of the two oral test modes, the qualitative results provided logic for the reasons of these differences on the side of the test takers. Explanatory Sequential Design (Bijani & Khabiri, 2017, p. 39)

[213] The current research intends to develop a practical model of a FCI for a writing class and implement it for the general IELTS writing class to analyze its efficacy. Embedded Sequential Design (Sahragard et al., 2020, p. 240)

[18] The present research was both quantitative and qualitative in nature and it was, therefore, designed based on Creswell’s (2018) mixed-method convergent approach following a side-by-side comparison between quantitative and qualitative data. Triangulation Concurrent Design (Dehghan & Sorkhi, 2020, p. 356)

[175] Two parallel forms of a 30-item grammar test, including 15 production and 15 comprehension items were designed and applied as the pre-test and post-test …The main purpose of the interview was to explore students' attitudes towards the treatment they received and the type of input to which they were exposed. Embedded Concurrent Design (Adloo & Rohani, 2019, p. 10–13)

Table 1. Designs of Iranian Applied Linguistics MMR Articles

Among the 303 MMR articles of the corpus, 22 (7.2%) were labeled other, meaning that their designs consisted of complex mixing of quantitative and qualitative strands in a way not addressed in Creswell et al.’s (2008) typology. Following Morse’s (1991) notation system (i.e., + is concurrent and        is sequential), the schematic representations of some of these studies were:

[162] Quan         (Qual + Quan) (Faghihi & Anani Sarab, 2016)

[201] (Qual        Quan) + (Qual         Quan) (Alavi & Ranjbaran, 2018)

[220] Qual        Quan        (Quan + Qual) (Pakzad & Salehi, 2018)

[94] Quan + (Qual + Quan) (Masoudzadeh et al., 2020)

Tools of Data Collection

The tools used for collecting quantitative and qualitative data are depicted in Table 2. In the corpus of the present study, most of the articles collected their data using interviews
(i.e., 174, 57.4%) and questionnaires (i.e., 164, 54.1%). Next, collecting texts was the third most frequently employed tool for data collection, with a total number of 121 (39.9%). These studies were mostly corpus analysis or questionnaire development in which literature was used as a certain source for gathering data. Moreover, 50 (16.5%) articles used tests as one of their main techniques to approach their needed data. Tests were mainly used in experimental designs that paved the way for adopting embedded MMR designs appropriately. Observations, recording interactions, essays, tasks, and reflective journals also comprised 23 (7.5%), 9 (2.9%), 7 (2.3%), 6 (1.9%), and 5 (1.6%) of the total number of tools used for data collection in the corpus. Finally, 14 (4.6%) of the studies also employed tools such as think-aloud, learner diaries, and role plays.

Table 2. Tools of Data Collection in Iranian Applied Linguistics MMR Articles

Note. RIs = Recording Interactions; RJs = Reflective Journals

Purposes

Based on Greene et al.’s (1989) framework, the frequencies of purposes behind mixing methods in Iranian AL ISC articles are depicted in Table 3. First of all, it should be noted that 92 (30.3%) of the purposes identified in the corpus cannot be explained using Greene et al.
’s framework. Therefore, as shown in Table 3, they were labeled others, the most frequently used purpose. Next, 76 (25%) articles utilized triangulation as their main purpose, the second most frequently used one. Moreover, 56 (18.4%), 36 (11.8%), and 32 (10.5%) articles had development, complementarity, and expansion purposes, respectively. Finally, 11 (3.6%) studies used a combination of different purposes.

Table 3. Purposes of Iranian Applied Linguistics MMR Articles

[187] To triangulate the data and to investigate the students’ perceptions about the merits of PA in developing their writing skill, a focus group interview was conducted with five of the students via the Adobe Connect platform (AD, 2019). Triangulation (Rahimi et al., 2021, p. 207)

[9] The purpose was to have the interviewees present a retrospective reflection of their experience with the blog-mediated writing course and express their attitudes towards the course. Expansion (Fathi & Nourzadeh, 2019, p. 74)

[169] The aim of these interviews was two-fold: (a) to identify the components of performance assessment from the perspectives of beneficiaries; and (b) to develop a valid instrument for measuring teacher performance. Development (Kiany et al., 2017, p. 119–120)

[69] As a supplementary survey to FLCAS, interviews with some migrant students and their two teachers were organized to explore possible anxiety-provoking sources. Complementarity (Jianming & Chen, 2020, p. 288)

[164] This study adopted a mixed-method design. In the first place, we explored the potential macro structure of the texts under study, counted frequencies and percentages of the occurrence of the communicative moves, and ran a number of statistical tests, using statistical software package of SPSS. In the second place, we searched out possible genre(s) hidden or mixed in the main genre. Others (Jalilifar & Musavi, 2016, p. 114).

[84] In order to validate the results and to uncover the reasons for students’ perception of the usefulness of the strategies, interviews were carried out. Multipurpose (Boroushaki & Ng, 2016, p. 5)

Discussion

As for the designs, nearly two-thirds of the MMR articles in the present study utilized sequential designs. Moreover, 34 (38.6%) articles employed concurrent designs. Additionally, the designs that did not match Creswell et al.’s (2008) framework (22, 7.2%) were classified as others. Several authors mentioned concurrent designs are more frequent in the social sciences (e.g., Christ, 2007; Hashemi & Babaii, 2013). However, a thorough search of the literature showed various design reviews in which sequential designs were also prominent in the social sciences (e.g., Corr et al., 2020; Molina-Azorin, 2011; Molina-Azorin & Cameron, 2010; Powell et al., 2008). Therefore, the findings of the present study are in line with the second line of research in the social sciences. With regard to AL, Hashemi and Babaii (2013) and Soodmand Afshar and Ranjbar (2023) found that concurrent designs are more prevalent.

However, the findings of this study are in line with Amini Farsani and Mohammadi’s (2022) study in the EFL context of Iran in which sequential designs constituted a higher number of articles in the corpus. It should also be noted that in their study, explanatory designs were more dominant, while in our study exploratory designs were more frequent. Overall, according to Creswell and Creswell (2018), sequential designs are perhaps easier to conduct because “analysis proceeds independently … and data collection can be spaced out over time” (p. 304). Moreover, nearly 25% of the studies were exploratory or explanatory sequential ones in which qualitative content analysis of texts accompanied further quantitative statistical analysis of them. In these types of studies, the identical sample paves the way for collecting quantitative and qualitative data from the corpus (Hashemi & Babaii, 2013). Therefore, it may be concluded that they are easier to accomplish. However, particular mention should be made of the fact that a fair proportion of exploratory studies (i.e., 18.4%) had ‘development’ purposes in which complex data collection procedures were adopted for instrument construction, etc.

One possible reason can be raised for these discrepancies regarding designs. Tashakkori and Teddlie (2010) mentioned inconsistencies in the standards applied in the frameworks for assessing MMR quality that can obviously lead to different results. They proposed linguistic, conceptual, procedural, and consequential levels as the areas that cause inconsistency. In the present study, we have added the design level (i.e., disagreeing on how to exactly analyze the components of a design) as one of the other sources of inconsistency. For example, following Creswell and Plano Clark (2018), we have put the main emphasis on the intent of a design rather than “the vague and often confusing priority” (p. 41) or timing. As mentioned earlier, in this study, we tried our best to avoid such inconsistencies. For example, the latest terminological advances were ignored for the sake of a clear comparison within the literature. However, in important cases, such as analyzing designs, it was inevitable to embrace the latest methodological advances in the field. Overall, widely accepted quality standards should be adopted by MMR practitioners if we want to achieve more valid and comparable results.

As for tools of data collection, the results of this study are similar to those reported by Bryman (2006) and Plano Clark et al. (2008), in which questionnaires and interviews were the most widely used methods of data collection. Unfortunately, none of the MMR design reviews in AL mentioned the tools of data collection in their studies. In a fascinating introduction, Bergman (2008) mentioned that some critics labeled the popularity of mixed methods as a trend that obliges researchers to add some kind of mixing in their projects to gain better chances for publication. With regard to AL, Mirhosseini (2017) noted that one of his colleagues categorized those who adhere only to quantitative or qualitative methodologies as, “belonging to the day before yesterday or staying in yesterday” (p. 1). Bergman (2008) further warned against one of the superficial ways of executing MMR as “a vague inclusion of a few, unconnected ‘expert interviews’ within a quantitative survey design” (p. 1). In addition to this, Amini Farsani and Mohammadi (2022) reported that 65% of the researchers did not explicate their rationale for conducting MMR in their corpus. Taken together, while the integration of questionnaires and interviews can shed more light on a variety of research problems, mixing these tools (or other ones) should offer something that goes beyond what quantitative and qualitative strands can offer alone (Bergman, 2008). In other words, there should be a clear justification for using MMR.

With regard to the purposes of using MMR, findings showed that Greene et al. (1989) and Bryman’s (2006) frameworks were incapable of explaining the most frequent category of identified purposes in this study (i.e., classified as others). The qualitative analysis of this others category revealed that it mainly consisted of content analysis studies that used statistical procedures. For example, in data source [212], the qualitative findings are further explored using quantitative analysis. Therefore, we have added the exploration rationale to Bryman’s (2006) framework to take into account this type of MMR purpose in which the quantitative strand is used to explore qualitative findings. Using Riazi’s (2016) typology, most of these studies can be categorized as eclectic, in which no reference is given to the underlying principles of MMR. However, a small proportion of them can also be labeled principled eclectic, wherein researchers give more attention to design technicalities and MMR principles. For example, in data source [176], the design of the study was labeled sequential exploratory, and in data source [54], the qualitative and quantitative findings are integrated to develop a meta-inference.

 [212] This study pivots on a mixed methods design. In fact, the qualitative data coming from the visual analysis were converted to quantitative data so that the comparisons of frameworks and models within and between the textbooks became possible. (Babaii et al., 2019, p. 63)

[176] A mixed-method design was applied to serve the purpose of this study. In particular, the design of this study was sequential exploratory. (Alemi & Motamedi, 2019, p. 9)

[54] The quantitative and qualitative analyses indicated improvements both in frequency and complex phrasal construction of nominals after the application of pedagogy in comparison with the students’ prior texts. (Pineh, 2022, p. 110)      

Furthermore, in line with Riazi and Candlin (2014), triangulation purpose was one of the most frequently used purposes in this study (i.e., 25%). However, a small proportion of studies used complementarity purpose (11.8%), which is inconsistent with its higher number in Greene et al. (1989, 33%), Bryman (2006, 44.8%), and Amini Farsani and Mohammadi (2022, 37%). One possible explanation for this finding may be the challenge inherent in conducting MMR studies with complementarity purposes (Riazi, 2017). Comparing the results with Amini Farsani and Mohammadi (2022), it might be argued that the process of writing theses and articles may incline researchers to favor different types of purposes. As for initiation purposes, the findings of this study were consistent with Bryman (2006) and Amini Farsani and Mohammadi (2022), in which such a purpose was almost nonexistent. Moreover, Amini Frasani and Mohammadi (2022) reported that expansion purpose was also absent in their corpus, while 10.5% of the recognized purposes in the present study were expansion. Nonetheless, a high number of expansion purposes were identified by Bryman (2006, 31.5%) and Greene et al. (1989, 44%). Development purpose, akin to Bryman (2006), Greene et al. (1989), and Amini Farsani and Mohammadi (2022), was the third frequently used purpose in the corpus. Finally, 3.6% of articles combined different purposes and were labeled multipurpose, while such a category is almost nonexistent in other design reviews of MMR. A variety of reasons can be offered for these discrepancies, including lack of exact operational definitions, requirements of different disciplines and sub-disciplines, the type of project (thesis, dissertation, or article), and expertise of MMR practitioners in different fields.

As for the fourth research question, the results of the quantitative and qualitative analysis revealed that fully integrated and high-quality MMR is rare in AL ISC journals, corroborating the findings reported in Hashemi and Babaii (2013), Amini Farsani and Mohammadi (2022), and Soodmand Afshar and Ranjbar (2023). As for designs, the quantitative results corroborated the sequential orientation of mixed designs in the EFL context of Iran (Amini Farsani & Mohammadi, 2022) in comparison to the concurrent orientation of the international AL community (Hashemi & Babaii, 2013; Soodmand Afshar & Ranjbar, 2023). Moreover, the qualitative analysis of the designs revealed that more attention should be given to designing sound MMR.

The frequency analysis of tools of data collection revealed a similar pattern of using questionnaires and interviews in the social sciences. It can be argued that the growth of advanced and innovative MMR designs (e.g., mixed methods ethnographic research; Hashemi, 2020) may require a variety of other tools as well. Moreover, as echoed by Bergman (2008), the combination of any quantitative and qualitative tools of data collection (including questionnaires and interviews) should be based on a sound rationale, not just because MMR is a relatively novel and popular research trend. Therefore, AL MMR practitioners need to pay more attention to justifying the tools and rationales of their MMR designs.  Finally, as for purposes, similar to Bryman’s (2006) findings, researchers should be more explicit about the rationale of their mixing. Taken together, although applied linguists have increasingly become aware of the benefits of MMR (Soodmand Afshar & Hafez, 2021), principled integration of quantitative and qualitative strands needs more attention in this field. Using Riazi’s (2016) typology, the journey of MMR in AL should be continued from its current status (i.e., eclectic and principled eclectic designs) to a more desired destination (i.e., innovative designs).

Conclusion

This article argued for a more transparent and rigorous approach to MMR in AL. While MMR is experiencing growing attention, many studies lack a strong foundation in MMR principles, leading to an unprincipled combination of quantitative and qualitative paradigms within a research project. Therefore, we argue for a systemic view of MMR (see Hashemi, 2020; Maxwell & Loomis, 2003) where the whole stages of a design are interconnected and contribute to more rigorous interpretations (i.e., meta-inferences). This requires careful consideration of MMR from the formulation (i.e., conceptualization) to the conclusion
(i.e., interpretation).

Some implications can be provided for MMR researchers in AL. First, researchers should clearly justify their MMR design choices, data collection tools, and the purposes behind mixing methods. In other words, selecting any specific tool should be justified in terms of how it serves the type of mixed methods design. Furthermore, to counteract any claim regarding the superficial integration of quantitative and qualitative tools as a fad, an explanation should be provided concerning how the selected tools help investigate the research problem from a mixed perspective. It should be noted that advanced and innovative MMR designs may require a variety of other tools besides questionnaires and interviews. Moreover, MMR practitioners should explicate the purposes behind mixing methods in their studies more vividly, using typologies such as Greene et al. (1989) and Bryman (2006). Crucially, researchers should pay attention to the quality of integration, using frameworks such as the MMIQF (Fàbregues et al., 2024) to ensure that integration is conducted and reported rigorously. Future research can also address the practical impact of adopting MMR designs on educational practices in AL. Moreover, they can also address the role of intra-disciplinary variation (i.e., Second Language Acquisition, discourse analysis, English for Specific Purposes, etc.) in adopting various MMR designs. Finally, since context affects quality issues in MMR, researchers can also examine the role of culture in shaping how MMR practitioners conceptualize their designs.

The current study has a number of limitations. First, while focusing on ISC journals provides a valuable perspective on research within the Iranian EFL context, it limits the generalizability of the findings to other contexts. Future research may compare MMR practices in ISC-indexed journals with those indexed in other databases, such as Scopus or Web of Science, to identify potential variations in methodological approaches. Second, this study focused on research designs, tools of data collection, and purposes of MMR. Future research could investigate other aspects of MMR quality, such as the integration of data analysis and the quality of meta-inferences. Third, although intra- and inter-coder reliability were established (κ = .86 and κ = .79, respectively), the qualitative content analysis inherently involves a degree of subjective interpretation in coding and categorizing designs, tools, and purposes. While the high Kappa values suggest strong agreement, future studies could employ additional measures to further mitigate potential bias, such as expert review, etc.