A Study review of: 'Review of Trends from Mobile Learning Studies: A meta-analysis'

A Study review of:

Review of Trends from Mobile Learning Studies:

A meta-analysis

- Wen-Hsiung Wu, Yen-Chun Jim Wu, Chun-Yu Chen, Hao-Yun Kao, Che-Hung Lin, and Sih-Han Huang

National Kaohsiung University of Applied Sciences;

National Sun Yat-Sen University; Meiho University;

Kaohsiung Medical University; and Cheng-Shiu University – Taiwan, ROC

Introduction

Two previous literature review-based studies have provided important insights into mobile learning, but the issue still needs to be examined from other directions such as the distribution of research purposes. This study by W.-H. Wu et al, takes a meta-analysis approach to systematically reviewing the literature, thus providing a more comprehensive analysis and synthesis of 164 studies from 2003 to 2010. Major findings include that most studies of mobile learning focus on effectiveness, followed by mobile learning system design; and surveys and experiments were used as the primary research methods. Also, mobile phones and PDAs are currently the most widely used devices for mobile learning but these may subsequently be displaced by new emerging technologies. In addition, the most highly-cited articles are found to focus on mobile learning system design, followed by system effectiveness. These findings may provide insights for researchers and educators into research trends in mobile learning.

Definition

O’Malley et al (2003: p6) defined mobile learning as taking place when the learner is not at a fixed, predetermined location, or when the learner takes advantage of learning opportunities offered by mobile technologies. Kukulshka-Hulme (2005) defines mobile learning as being concerned with learner mobility in the sense that learners should be able to engage in educational activities without being tied to a tightly-delimited physical location. Thus mobile learning features learners engaged in educational activities, using technology as a mediating tool for learning via mobile devices accessing data and communicating with others through wireless technology.

Categories of research directions regarding mobile learning

Previous studies of mobile learning fall into two broad research directions:

(i)                  Evaluating the effectiveness of mobile learning; and

(ii)                Designing mobile learning systems.

Most research in the former showed positive effectiveness, but there are also several that showed neutral or negative effectiveness. So while Evans (2008), Al-Fahad (2009) and Baya’a and Daher (2009) all saw positive results from their research, others like Ketamo (2003) and Doolittle and Mariano (2008) all recorded negative results.

For the second research direction, researchers designed mobile systems to fit their courses. For example Ullrich, Shen, Tong and Tan (2010) described the mobile live video learning system (MLVLS) developed for computer sciences courses at the Shanghai Jiao Tong University, and found that mobile devices were significantly more widely used than desktop or laptop computers.

Findings from previous mobile learning reviews

Two previous literature reviews studied research trends in mobile learning. Hung and Zhang (in press) used text mining techniques to investigate research trends in 119 academic articles on mobile learning from 2003 to 2008 taken from the SCI/SSCI database. In general, they investigated publication date, publication category, taxonomy, article clusters, and country, university and journal of origin. Results showed that articles on mobile learning increased from 8 in 2003 to 36 in 2008; the most popular domains in mobile learning studies are effectiveness, evaluation, and personalized systems and studies on strategies and frameworks are more likely to be published.

Hwang and Tsai (2011) reviewed journals (BJET, C&E, ETS, ETR&D, JCAL and IETI) in the SSCI database from 2001 to 2010, selecting 154 articles on mobile and ubiquitous learning, and noting the number of articles published, research sample groups selected, research learning domains, and country of origin. Their findings included the following:

(i)                  The volume of research in mobile and ubiquitous learning greatly expanded between 2006 and 2010;

(ii)                Higher education students were the most frequent research populations, followed by elementary school students, and high school students;

(iii)               Most studies did not explicitly focus on any particular learning domain but rather investigated the motivation, perceptions and attitudes of students toward mobile and ubiquitous learning, along with course-orientation for engineering (including computers), language and art, and science; and

(iv)               Most articles were contributed from US-based authors, followed by authors in the UK and Taiwan.

The studies listed above offer syntheses crucial to understanding issues related to mobile learning, but are incomplete. For example, they fail to account for the distribution of research purposes and methods among the various articles. They also fail to highlight the types of mobile learning devices used.

In contrast, the study by W.-H. Wu et al adopts a meta-analysis method in examining these trends in mobile learning studies.

Research Method

W.-H. Wu et al conducted a systematic review from a data pool consisting of computerized bibliographic databases – Science Direct Onsite (SDOS), Wiley Interscience, SAGE Journal Online, ProQuest, ACM Digital Library, JSTOR, Elsevier Science (Elsevier)/SDOL, Informaworld and ERIC. Manual searches were also conducted for Journal of Computer Assisted Learning, Computer in Human Behaviour, British Journal of Education Technology, Journal of Educational Technology & Society, and The International review of Research in Open and Distance Learning.

Key words included: “mobile learning” or “M-learning” with “instruct”, “teach”, “context-aware”, “adaptive”, “wireless”, “situated learning”, or “activities”.

The search produced 887 results, 448 of which were duplicates found in multiple journals or databases. Two researchers then independently confirmed the inclusion/exclusion criteria for each study. The intercoder agreement rate for coding was 94.47%. Disagreements between the two coders were resolved through discussion and further review of the disputed studies. In total, 164 studies met the inclusion criteria and were included in the analyses.

The procedure was based on the rigorous protocol developed by Glass (1976), Hossler and Scalese-Love (1989) and Ke (2009). The steps for inclusion/exclusion criteria, data sources and search strategies, and data coding and analysis are discussed below.

Inclusion criteria

In order to be included in the analysis, each study had to meet the following criteria:

a.)     Must involve mobile learning as a primary condition

b.)     Must include an identifiable learner level. All learner levels are admissible

c.)     Must include mobile devices while learners are learning

d.)     Must involve education activities when implementing mobile learning

e.)     Must be a publicly available or archived periodical article

f.)      Must be published between January 2000 and December 2010

Exclusion criteria

Studies that had the following characteristics were excluded:

a.)     Mobile learning not used for educational purposes

b.)     Conference papers or book chapters are excluded

Data Coding and Analysis

Ten features related to the quality of study research methodology were coded including: (a) research purpose, (b) learner demographic (e.g. elementary, secondary, post-secondary, higher education, adult, or disabled), (c) method (e.g. survey, experiment, etc), (d) use of mobile devices, (e) discipline-orientation (e.g. humanities, social sciences, natural sciences, formal sciences, applied sciences and professional studies), (f) courses, (g) educational contexts (i.e. formal learning and informal learning), (h) learning outcome (i.e. positive, negative or neutral), and (i) article citation counts. During data analysis, low-quality studies were excluded from the synthesis. In the current analysis, a quantitative study was considered low quality and excluded if it did not depict its methodological design features such as sample size and procedure. Qualitative studies were excluded if they failed to provide a rich description such as mobile learning outcomes, or appeared to rely more on the author’s experience rather than field observations.

Results

Of the 164 studies published on mobile learning applications in educational contexts from 2003 to 2010, frequency of publication increased from low-to-mid single digits from 2003 to 2006, to low double-digits from 2007 to 2009, and then jumped to 106 in 2010.

Research Questions:

As previously stated, W.-H. Wu et al wanted to add to the base of research that had been carried out by previous research groups in this area. They therefore had a few additional research questions that separated their work from those other studies. Their questions are stated below:

Research Question 1: Major research purposes, methods and outcomes

(a) Distribution of research purposes

They classified each article into one of four categories according to its research purpose:

(1)    Evaluating the effects of mobile learning

(2)    Designing a mobile system for learning

(3)    Investigating the affective domain during mobile learning, or

(4)    Evaluating the influence of learner characteristics in the mobile learning process

Of these four, evaluating the effects of mobile learning was the most common research purpose (accounting for 58% of the articles), followed by designing a mobile system for learning (32%), investigating the affective domain during mobile learning (5%), and evaluating the influence of learner characteristics in the mobile learning process (5%).

(b) Distribution of research methods

Their classification of research methods fell into only 2 categories:

(1)    Evaluation-dominant with application-minor, or

(2)    Design-dominant with evaluation-minor

The first applies to mobile learning systems and evaluates their effectiveness, while the latter designs mobile systems and evaluates their effectiveness. Purposes 1, 3 and 4 belonged to the former, while purpose 2 belonged to the latter.

In the case of ‘evaluating the effects of mobile learning’, researchers primarily relied on surveys (26 studies), followed by experimental research methods (20) and descriptive methods (7). For ‘evaluating the influence of learner characteristics in the mobile learning process’, experimental research methods were used most often (4 studies), followed by surveys (2), descriptive methods (1), and observation (1). As for purpose 3 (investigating the affective domain during mobile learning), only two methodologies were used: surveys (6) and interviews (1). For purpose 4 (designing a mobile system for learning), surveys were the most commonly used methodology (16 studies), followed by experimental research methods (14), descriptive methods (8), case studies (2), and observation (1).

(c) Distribution of research outcomes

86% of studies included in this research reported positive research outcomes, only 4% reported neutral outcomes, and only 1% reported negative outcomes.

Research Question 2: Types of learners and types of mobile devices used to assist learners

(a) Distribution of educational contexts by mobile device

W.-H. Wu et al based their categorisation of educational context on research by Merriam, Caffarella, and Baumgartner (2007), and Cedefop (2011). These were: formal education, non-formal education and informal education. In formal education contexts, higher education institutions were found to have favoured mobile phones (34 studies), followed by PDAs (30) and laptops (7), while PDAs were more commonly used in elementary schools (18 studies). In non-formal education contexts, mobile phones were still predominant (5 studies), but the frequency of use is conspicuously lower than in formal education use in higher education institutions. Similarly, mobile phones are used in informal education (6 studies). They found that aside from mobile phones and PDAs, other devices and mobile services (eg. Mp3/mp4 players, iPods, cameras, podcasts, GPS devices, and satellite TV), are applied in all three educational contexts but with very low frequencies. For example mp3/mp4 players were found to be used in Higher education contexts (3 studies), while iPods were found being used in non-formal learning (1 study).

(b) Distribution of mobile learners by year

They gathered that mobile learning is most frequently used by higher education students (51.98%), followed by elementary school students (17.51%), adult learners (12.43%), secondary and post-secondary school students (8.47%), and disabled students (0.56%). They also found that the number of mobile learners in many contexts increased sharply after 2009.

(c) Distribution of mobile devices by year

Among the 164 studies, mobile phones were most commonly used for mobile learning (36.55%), followed by PDAs (30.96%), laptop computers (9.14%), iPods (4.06%), mp3/mp4 players (2.54%), podcasts (2.03%), and cameras (1.52%). In addition, the choice of device changed over time with the evolution of technology. For example, iPods are first used in mobile learning in 2008, while GPS is not used until 2010, indicating that with time, studies began to expand their definition of mobile devices used as teaching tools.

Research Question 3: Representation of academic disciplines and courses

(a) Distribution of mobile learning by academic disciplines and courses

This study adopted the taxonomy developed by Belcher (1994), Franklin (1999), and Wanner, Lewis, and Gregorio (1981) which identifies five major categories of academic discipline, each of which has its own sub-disciplines: humanities (including history, languages and linguistics, literature, performing arts, philosophy, religion and visual arts); social sciences (anthropology, archeology, area/regional studies, cultural and ethnic studies, economics, gender and sexual studies, geography, political science, psychology and sociology); natural sciences (space sciences, life sciences, earth sciences, chemistry and physics); formal sciences (computer science, logic, mathematics, statistics and systems science); and the professions and applied sciences (which include: agriculture, architecture and design, business, divinity, education, engineering, environmental studies and forestry, family and consumer science, health sciences, human physical performance and recreation, journalism, media studies and communication, law, library and museum studies, military sciences, public administration, social work and transportation).

Based on these classifications, the research indicated that the studies of mobile learning for educational purposes focused most frequently on applications in the professions and applied sciences (29%), followed by humanities (20%), formal sciences (16%), social sciences (4%) and natural sciences (3%). In terms of sub-disciplines, languages and linguistics courses were the most common focus (17.05%), followed by computer science (13.07%), health sciences (10.23%), environmental studies and forestry (10.23%), physics (2.27%), business (2.27%), and journalism/media studies/communication (2.27%).

Research Question 4: Analysis and distribution of highly cited articles

To identify highly cited articles, citation counts of the 164 studies were analysed in the SSCI as of February 3, 2012. Shih et al (2008) stressed that more frequently cited articles are usually those that receive greater recognition by others in related fields. These highly cited articles could raise fundamental issues for future research. The fifteen articles with the highest citation counts in different research purpose categories were selected for analysis. Five articles were categorized as “evaluating the effects of mobile learning”; eight were categorized as “designing a mobile system for learning purposes”; and one article each was categorised as “investigating the affective domain during mobile learning” and “evaluating the influence of learner characteristics on the mobile learning process”.

The citation counts for these fifteen articles ranged from 13 to 78, with the most highly cited study (78 citation) falling in the category of “designing a mobile system for learning purposes” (chen et al, 2003), and focused on developing a mobile learning system to provide scaffolding for students learning about bird-watching. The second most highly cited study (47 citations) was categorized as “evaluating the effects of mobile learning” (Evans, 2008), and investigated the effectiveness of podcasts for teaching undergraduate students. The third and fourth most highly cited studies (43 and 41 citations respectively), were both categorized as “designing a mobile system for learning purposes” (Thornton & Houser, 2005 and Zurita & Nussbaumw, 2004). Thornton & Houser created a Vidioms system using mobile phones and PDAs to assist English idiom learning, while Zurita & Nussbaumw developed a constructivist learning environment supported by handheld devices for the teaching of reading in elementary schools. W.-H. Wu et al also found one study with 19 citations (Chu, Hwang & Tsai, 2010) that, given its recent publication at the time of their study, could be expected to have potential for a high citation count in the future.

Discussion

Both Hwang and Tsai (2011) and Hung and Zhang (in press) provide valuable synthesis for studies in mobile learning. For example, the two studies showed the increasing trend and broadening distribution of countries contributing to studies in mobile learning. However their approach is still incomplete and the topic needs to be further explored from different directions. This study provides important results and new findings. For example, the research purposes of most mobile learning studies center on effectiveness, followed by mobile learning system design. Moreover, mobile phones and PDAs may be the two devices most commonly used for mobile learning, but new devices may emerge as technology advances.

These findings are further described below:

Most studies of mobile learning focus on effectiveness, followed by mobile learning system design

Of the 164 studies, 58% took evaluating the effectiveness of mobile learning as the primary research purpose. This focus on evaluation is a new finding not raised in previous literature surveys. More importantly, this result corresponds with surveys of other technology-assisted learning contexts. For example Vogel et al (2006) indicated that most studies on game-based learning focus on effectiveness. The second most frequently cited research purpose was mobile learning system design (32%), which is also a new finding. They found out that the number of studies devoted to mobile learning system design increased over time, and suggested that this may be due to rapid technology development such as new smart phones and wireless data networks combined with the willingness of researchers to trial new technologies in developing mobile learning systems.

Most mobile learning studies adopted surveys and experiments as the primary research methods

Among the 164 studies, surveys were the primary research method (50 studies), followed by experimental research methods (38) and regardless of research purpose (i.e. evaluation-dominant with application-minor or design dominant with evaluation-minor), quantitative approaches were favoured over qualitative approaches. This is a new finding which corresponds with findings in other technology-assisted learning contexts. For example Zawacki-Richter, Backer, and Vogt (2009) found that quantitative methods dominated distance education studies from 2000 to 2008, followed by qualitative methods or triangulation methods.

Most mobile learning studies feature positive outcomes

86% of the 164 mobile learning studies presented positive outcomes. This is another new finding which corresponds to findings in other technology-assisted learning contexts. For example Ke (2009) applied a meta-analysis approach to find that studies of game-based learning generally have positive outcomes.

Mobile phones and PDAs are currently the most widely used devices for mobile learning, but may be displaced by emerging technologies

In the context of mobile learning, device type has a critical impact on teaching and learning. From their research W.-H. Wu et al showed that mobile phones and PDAs account for over 75% (69/195 + 64/195) of all mobile devices used in educational contexts. Technology advances quickly and new types of mobile devices are emerging that can be applied to education. For example Martin et al (2011) used the predictions from Horizon reports from 2004 to 2010 (covering 2004 – 2014), to analyse technologies that have impacted education in the past or are likely to have an impact in the future. Horizon report 2007 suggested that the use of mobile phones in mobile learning, particularly in higher education, would expand dramatically after 2009, which corresponded the findings of this study. In addition, Horizon report 2010 predicted that future mobile devices would add functions such as mobile computing, open content, e-books, gesture-based computing, and visual data analysis.

Questions

So what has been the distribution of ‘Studies by Technology’ over the last 5 years (2010 to 2014)?

What is new, and what needs to be new?

Is there a 2010 – 2014 survey analysis by year of devices, purposes, outcomes, methods, etc?

Possible studies include:

(1.)  Development of mobile Technology use from inception till date

(2.)  Applications of mobile technology to learning over the years

(3.)  Developmental affordances of mobile technologies since the 1980s till today

(4.)  Future applications

Use of mobile devices for learning is most common in higher education followed by elementary schools

Mobile learning is most frequently used in teaching and learning contexts for higher education students (51.98%), followed by elementary school students (17.51%), which corresponds with findings from Hwang and Tsai (2011). More importantly, their study further indicates a significant jump in mobile learning activity in higher education institutions in 2006, with studies based in higher education institutions (1 in 2006, to 50 in 2010) and elementary schools, (2 in 2009, to 26 in 2010).

Mobile learning most frequently supports learning in the professions and applied sciences, the humanities and formal sciences

Studies on mobile learning in educational contexts most frequently focus on use in supporting professional subjects and applied sciences (29%), followed by the humanities (20%), and formal sciences (16%). In terms of mobile learning activity in various sub-disciplines, their findings partially support those of Hwang and Tsai (2011). For example both studies showed mobile learning was often used in computer and language courses. More importantly, the W.-H. Wu et al study found that mobile learning was often related to environmental studies, forestry and health sciences, but considerably less so in other courses such as statistics or law. However, they suggest that mobile learning can be applied to any course or subject matter, and researchers from different disciplines can collaborate to develop suitable applications for under-represented courses.

Most highly cited articles fall into the categories of mobile learning system design and followed by effectiveness

Based on the criterion count being equal to or greater than 40, three highly cited articles fall into the category of “designing a mobile system for learning purposes” and one article is categorized as “evaluating the effects of mobile learning”. This focus on highly cited articles is a new finding not addressed in previous literature surveys. More importantly, compared with the results of the distribution of mobile learning studies by research purpose stated earlier, this finding reverses the order of the first and second categories (i.e. designing a mobile system for learning now comes before evaluating the effects of mobile learning), while the order of the third and fourth categories remains unchanged.

For mobile-based technological development, they found that articles belonging to the category “designing a mobile system for learning purposes” describe mobile systems developed by researchers and educators prior to any effectiveness evaluation. These systems can present important applications in various disciplines such as bird-watching, learning of professions, applied science/environmental studies and forestry for elementary school students (Chen et al, 2003). These applications are more likely to be cited by other related studies. Also, most of the highly cited articles were published from 2003 to 2005, aside from one article published in 2008. This is probably because similar to other technology-assisted learning contexts such as the literature surveys of e-learning by Shih et al (2008), earlier articles have a longer time to be disseminated and cited in other related studies.

Conclusions

Two previous literature review-based studies on the use of mobile learning in academic contexts provided valuable insights, but failed to examine the issue from directions such as the distribution of research purposes. This study by W.-H. Wu et al conducted a systematic meta-analysis to provide more comprehensive analysis of past studies, and discusses the implications of new findings.

The study presents seven new findings:

(1.)  The research purpose of most mobile learning studies focuses on effectiveness, followed by mobile learning system design

(2.)  Surveys and experimental methods were preferred research methods, regardless of whether the research purpose focused on evaluation or design.

(3.)  Research outcomes in mobile learning studies are significantly positive

(4.)  Mobile phones and PDAs are the most commonly used devices for mobile learning, but these may be replaced in the future by new emerging technologies

(5.)  Mobile learning is most prevalent at higher education institutions, followed by elementary schools

(6.)  Mobile learning most frequently supports students in the professions and applied sciences, followed by the humanities and formal sciences

(7.)  The most highly cited articles fall into the categories of mobile learning system design, followed by effectiveness.

In summary, this study of issues in mobile learning presents findings which can help supplement linkages with previous studies and forms an important reference base for future research in mobile learning.

References and highly cited articles

Al-Fahad, F. N. (2009). Students’ attitudes and perceptions towards the effectiveness of mobile learning in King Saud University, Saudi Arabia. The Turkish Online Journal of Educational Technology, 8(2), 111 – 119.

Baya’a, N., & Daher, W. (2009). Learning Mathematics in an authentic mobile environment: the perceptions of students. International Journal of Interactive Mobile Technologies, 3, 6-14.

Chen, G. D., Chang, C. K., & Wang, C. Y. (2008) Ubiquitous learning website: Scaffold learners by mobile devices with information-aware techniques. Computers & Education, 50(1), 77-90.

Doolittle, P., & Mariano, G. (2008). Working memory capacity and mobile multimedia learning environments: individual differences in learning while mobile. Journal of Educational Multimedia and Hypermedia, 17(4), 511-530.

Evans, C. (2008). The effectiveness of m-learning in the form of podcast revision lectures in higher education. Computers & Education, 50, 491-498.

Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8

Hossler, D., & Scalese-Love, P. (1989). Grounded meta-analysis: a guide for research synthesis. Review of Higher Education, 13, 1-28

Ke, F. (2009). A qualitative meta-analysis of computer games as learning tools. In R. E. Ferdig (ed.), Handbook on research on effective electronic gaming in education (pp. 1-32). Hershey: Information Science Reference.

Ketamo, H. (2003). xTask – an adaptable learning environment. Journal of Computer Assisted Learning, 19, 360-370.

Highly Cited Articles

Chen, C. M., & Chung, C. J. (2008). Personalised mobile English vocabulary learning system based on item response theory and learning memory cycle. Computers & Education, 51(2), 624-645.

Chen, C. M., & Hsu, S. H. (2008). Personalised intelligent mobile learning system for supporting effective English learning. Educational Technology & Society, 11(3), 153-180.

Chen, Y. S., Kao, T. C., & Sheu, J. P. (2003).