A Study review of:
Review of Trends from
Mobile Learning Studies:
A meta-analysis
- Wen-Hsiung Wu, Yen-Chun Jim Wu,
Chun-Yu Chen, Hao-Yun Kao, Che-Hung Lin, and Sih-Han Huang
National Kaohsiung University of
Applied Sciences;
National Sun Yat-Sen University;
Meiho University;
Kaohsiung Medical University; and Cheng-Shiu
University – Taiwan, ROC
Introduction
Two previous literature review-based studies have provided
important insights into mobile learning, but the issue still needs to be examined
from other directions such as the distribution of research purposes. This study
by W.-H. Wu et al, takes a meta-analysis approach to systematically reviewing
the literature, thus providing a more comprehensive analysis and synthesis of
164 studies from 2003 to 2010. Major findings include that most studies of
mobile learning focus on effectiveness, followed by mobile learning system
design; and surveys and experiments were used as the primary research methods.
Also, mobile phones and PDAs are currently the most widely used devices for
mobile learning but these may subsequently be displaced by new emerging
technologies. In addition, the
most highly-cited articles are found to focus on mobile learning system design,
followed by system effectiveness. These findings may provide insights
for researchers and educators into research trends in mobile learning.
Definition
O’Malley et al (2003: p6) defined mobile learning as taking
place when the learner is not at a fixed, predetermined location, or when the
learner takes advantage of learning opportunities offered by mobile
technologies. Kukulshka-Hulme (2005) defines mobile learning as being concerned
with learner mobility in the sense that learners should be able to engage in
educational activities without being tied to a tightly-delimited physical
location. Thus mobile
learning features learners engaged in educational activities, using technology
as a mediating tool for learning via mobile devices accessing data and
communicating with others through wireless technology.
Categories of
research directions regarding mobile learning
Previous studies of mobile learning fall into two broad
research directions:
(i)
Evaluating the effectiveness of mobile learning;
and
(ii)
Designing mobile learning systems.
Most research in the former
showed positive effectiveness, but there are also several that showed neutral
or negative effectiveness. So while Evans (2008), Al-Fahad (2009) and Baya’a
and Daher (2009) all saw positive results from their research, others like
Ketamo (2003) and Doolittle and Mariano (2008) all recorded negative results.
For the second research
direction, researchers designed mobile systems to fit their courses. For
example Ullrich, Shen, Tong and Tan (2010) described the mobile live video
learning system (MLVLS) developed for computer sciences courses at the Shanghai
Jiao Tong University, and found that mobile devices were significantly more
widely used than desktop or laptop computers.
Findings from
previous mobile learning reviews
Two previous literature reviews studied research trends in
mobile learning. Hung and Zhang (in press) used text mining techniques to
investigate research trends in 119 academic articles on mobile learning from
2003 to 2008 taken from the SCI/SSCI database. In general, they investigated
publication date, publication category, taxonomy, article clusters, and
country, university and journal of origin. Results showed that articles on
mobile learning increased from 8 in 2003 to 36 in 2008; the most popular domains in mobile learning studies
are effectiveness, evaluation, and personalized systems and studies on
strategies and frameworks are more likely to be published.
Hwang and Tsai (2011) reviewed journals (BJET, C&E, ETS,
ETR&D, JCAL and IETI) in the SSCI database from 2001 to 2010, selecting 154
articles on mobile and ubiquitous learning, and noting the number of articles
published, research sample groups selected, research learning domains, and
country of origin. Their findings included the following:
(i)
The volume of research in mobile and ubiquitous
learning greatly expanded between 2006 and 2010;
(ii)
Higher education students were the most frequent research populations,
followed by elementary school students, and high school students;
(iii)
Most studies did not explicitly focus on any
particular learning domain but rather investigated the motivation, perceptions
and attitudes of students toward mobile and ubiquitous learning, along with
course-orientation for engineering (including computers), language and art, and
science; and
(iv)
Most articles were contributed from US-based
authors, followed by authors in the UK and Taiwan.
The studies listed above offer
syntheses crucial to understanding issues related to mobile learning, but are
incomplete. For example, they fail to account for the distribution of research
purposes and methods among the various articles. They also fail to highlight
the types of mobile learning devices used.
In contrast, the study by W.-H. Wu et al adopts a
meta-analysis method in examining these trends in mobile learning studies.
Research Method
W.-H. Wu et al conducted a systematic review from a data
pool consisting of computerized bibliographic databases – Science Direct Onsite
(SDOS), Wiley Interscience, SAGE Journal Online, ProQuest, ACM Digital Library,
JSTOR, Elsevier Science (Elsevier)/SDOL, Informaworld and ERIC. Manual searches
were also conducted for Journal of Computer Assisted Learning, Computer in
Human Behaviour, British Journal of Education Technology, Journal of
Educational Technology & Society, and The International review of Research
in Open and Distance Learning.
Key words included: “mobile learning” or “M-learning” with
“instruct”, “teach”, “context-aware”, “adaptive”, “wireless”, “situated
learning”, or “activities”.
The search produced 887 results, 448 of which were
duplicates found in multiple journals or databases. Two researchers then
independently confirmed the inclusion/exclusion criteria for each study. The
intercoder agreement rate for coding was 94.47%. Disagreements between the two
coders were resolved through discussion and further review of the disputed
studies. In total, 164 studies met the inclusion criteria and were included in
the analyses.
The procedure was based on the rigorous protocol developed
by Glass (1976), Hossler and Scalese-Love (1989) and Ke (2009). The steps for
inclusion/exclusion criteria, data sources and search strategies, and data
coding and analysis are discussed below.
Inclusion criteria
In order to be included in the
analysis, each study had to meet the following criteria:
a.)
Must involve mobile learning as a primary
condition
b.)
Must include an identifiable learner level. All
learner levels are admissible
c.)
Must include mobile devices while learners are
learning
d.)
Must involve education activities when
implementing mobile learning
e.)
Must be a publicly available or archived
periodical article
f.)
Must be published between January 2000 and
December 2010
Exclusion criteria
Studies that had the following
characteristics were excluded:
a.)
Mobile learning not used for educational
purposes
b.)
Conference papers or book chapters are excluded
Data Coding and
Analysis
Ten features related to the quality of study research
methodology were coded including: (a) research purpose, (b) learner demographic
(e.g. elementary, secondary, post-secondary, higher education, adult, or
disabled), (c) method (e.g. survey, experiment, etc), (d) use of mobile
devices, (e) discipline-orientation (e.g. humanities, social sciences, natural
sciences, formal sciences, applied sciences and professional studies), (f)
courses, (g) educational contexts (i.e. formal learning and informal learning),
(h) learning outcome (i.e. positive, negative or neutral), and (i) article
citation counts. During data analysis, low-quality studies were excluded from
the synthesis. In the current analysis, a quantitative study was considered low
quality and excluded if it did not depict its methodological design features
such as sample size and procedure. Qualitative studies were excluded if they
failed to provide a rich description such as mobile learning outcomes, or
appeared to rely more on the author’s experience rather than field
observations.
Results
Of the 164 studies published on mobile learning applications
in educational contexts from 2003 to 2010, frequency of publication increased
from low-to-mid single digits from 2003 to 2006, to low double-digits from 2007
to 2009, and then jumped to 106 in 2010.
Research Questions:
As previously stated, W.-H. Wu et al wanted to add to the
base of research that had been carried out by previous research groups in this
area. They therefore had a few additional research questions that separated
their work from those other studies. Their questions are stated below:
Research Question 1: Major research purposes, methods and
outcomes
(a) Distribution of research purposes
They classified each article into
one of four categories according to its research purpose:
(1)
Evaluating the effects of mobile learning
(2)
Designing a mobile system for learning
(3)
Investigating the affective domain during mobile
learning, or
(4)
Evaluating the influence of learner
characteristics in the mobile learning process
Of these four, evaluating the effects of mobile learning was
the most common research purpose (accounting for 58% of the articles), followed
by designing a mobile system for learning (32%), investigating the affective
domain during mobile learning (5%), and evaluating the influence of learner
characteristics in the mobile learning process (5%).
(b) Distribution of research methods
Their classification of research
methods fell into only 2 categories:
(1)
Evaluation-dominant with application-minor, or
(2)
Design-dominant with evaluation-minor
The first applies to mobile learning systems and evaluates
their effectiveness, while the latter designs mobile systems and evaluates
their effectiveness. Purposes 1, 3 and 4 belonged to the former, while purpose
2 belonged to the latter.
In the case of ‘evaluating the effects of mobile learning’,
researchers primarily relied on surveys (26 studies), followed by experimental
research methods (20) and descriptive methods (7). For ‘evaluating the
influence of learner characteristics in the mobile learning process’,
experimental research methods were used most often (4 studies), followed by
surveys (2), descriptive methods (1), and observation (1). As for purpose 3
(investigating the affective domain during mobile learning), only two
methodologies were used: surveys (6) and interviews (1). For purpose 4
(designing a mobile system for learning), surveys were the most commonly used
methodology (16 studies), followed by experimental research methods (14),
descriptive methods (8), case studies (2), and observation (1).
(c) Distribution of research outcomes
86% of studies included in this research reported positive
research outcomes, only 4% reported neutral outcomes, and only 1% reported
negative outcomes.
Research Question 2: Types of learners and types of mobile
devices used to assist learners
(a) Distribution of educational contexts by mobile device
W.-H. Wu et al based their categorisation of educational
context on research by Merriam, Caffarella, and Baumgartner (2007), and Cedefop
(2011). These were: formal education, non-formal education and informal
education. In formal education contexts, higher education institutions were
found to have favoured mobile phones (34 studies), followed by PDAs (30) and
laptops (7), while PDAs were more commonly used in elementary schools (18
studies). In non-formal education contexts, mobile phones were still
predominant (5 studies), but the frequency of use is conspicuously lower than
in formal education use in higher education institutions. Similarly, mobile
phones are used in informal education (6 studies). They found that aside from
mobile phones and PDAs, other devices and mobile services (eg. Mp3/mp4 players,
iPods, cameras, podcasts, GPS devices, and satellite TV), are applied in all three
educational contexts but with very low frequencies. For example mp3/mp4 players
were found to be used in Higher education contexts (3 studies), while iPods
were found being used in non-formal learning (1 study).
(b) Distribution of mobile learners by year
They gathered that mobile learning is most frequently used
by higher education students (51.98%), followed by elementary school students
(17.51%), adult learners (12.43%), secondary and post-secondary school students
(8.47%), and disabled students (0.56%). They also found that the number of
mobile learners in many contexts increased sharply after 2009.
(c) Distribution of mobile devices by year
Among the 164 studies, mobile phones were most commonly used
for mobile learning (36.55%), followed by PDAs (30.96%), laptop computers
(9.14%), iPods (4.06%), mp3/mp4 players (2.54%), podcasts (2.03%), and cameras
(1.52%). In addition, the
choice of device changed over time with the evolution of technology. For
example, iPods are first used in mobile learning in 2008, while GPS is not used
until 2010, indicating that with time, studies began to expand their definition
of mobile devices used as teaching tools.
Research Question 3: Representation of academic disciplines
and courses
(a) Distribution of mobile learning by academic disciplines
and courses
This study adopted the taxonomy developed by Belcher (1994),
Franklin (1999), and Wanner, Lewis, and Gregorio (1981) which identifies five
major categories of academic discipline, each of which has its own
sub-disciplines: humanities (including history, languages and linguistics,
literature, performing arts, philosophy, religion and visual arts); social
sciences (anthropology, archeology, area/regional studies, cultural and ethnic
studies, economics, gender and sexual studies, geography, political science,
psychology and sociology); natural sciences (space sciences, life sciences,
earth sciences, chemistry and physics); formal sciences (computer science,
logic, mathematics, statistics and systems science); and the professions and
applied sciences (which include: agriculture, architecture and design,
business, divinity, education, engineering, environmental studies and forestry,
family and consumer science, health sciences, human physical performance and
recreation, journalism, media studies and communication, law, library and
museum studies, military sciences, public administration, social work and
transportation).
Based on these classifications, the research indicated that
the studies of mobile learning for educational purposes focused most frequently
on applications in the professions and applied sciences (29%), followed by
humanities (20%), formal sciences (16%), social sciences (4%) and natural
sciences (3%). In terms of sub-disciplines, languages and linguistics courses
were the most common focus (17.05%), followed by computer science (13.07%),
health sciences (10.23%), environmental studies and forestry (10.23%), physics
(2.27%), business (2.27%), and journalism/media studies/communication (2.27%).
Research Question 4: Analysis and distribution of highly
cited articles
To identify highly cited articles, citation counts of the
164 studies were analysed in the SSCI as of February 3, 2012. Shih et al (2008)
stressed that more frequently cited articles are usually those that receive
greater recognition by others in related fields. These highly cited articles
could raise fundamental issues for future research. The fifteen articles with
the highest citation counts in different research purpose categories were
selected for analysis. Five articles were categorized as “evaluating the
effects of mobile learning”; eight were categorized as “designing a mobile
system for learning purposes”; and one article each was categorised as
“investigating the affective domain during mobile learning” and “evaluating the
influence of learner characteristics on the mobile learning process”.
The citation counts for these fifteen articles ranged from
13 to 78, with the most highly cited study (78 citation) falling in the
category of “designing a mobile system for learning purposes” (chen et al,
2003), and focused on developing a mobile learning system to provide
scaffolding for students learning about bird-watching. The second most highly
cited study (47 citations) was categorized as “evaluating the effects of mobile
learning” (Evans, 2008), and investigated the effectiveness of podcasts for
teaching undergraduate students. The third and fourth most highly cited studies
(43 and 41 citations respectively), were both categorized as “designing a
mobile system for learning purposes” (Thornton & Houser, 2005 and Zurita
& Nussbaumw, 2004). Thornton & Houser created a Vidioms system using
mobile phones and PDAs to assist English idiom learning, while Zurita &
Nussbaumw developed a constructivist learning environment supported by handheld
devices for the teaching of reading in elementary schools. W.-H. Wu et al also
found one study with 19 citations (Chu, Hwang & Tsai, 2010) that, given its
recent publication at the time of their study, could be expected to have
potential for a high citation count in the future.
Discussion
Both Hwang and Tsai (2011) and Hung and Zhang (in press)
provide valuable synthesis for studies in mobile learning. For example, the two
studies showed the increasing trend and broadening distribution of countries
contributing to studies in mobile learning. However their approach is still
incomplete and the topic needs to be further explored from different
directions. This study provides important results and new findings. For
example, the research purposes of most mobile learning studies center on
effectiveness, followed by mobile learning system design. Moreover, mobile
phones and PDAs may be the two devices most commonly used for mobile learning,
but new devices may emerge as technology advances.
These findings are further described below:
Most studies of mobile learning focus on effectiveness,
followed by mobile learning system design
Of the 164 studies, 58% took evaluating the effectiveness of
mobile learning as the primary research purpose. This focus on evaluation is a new finding not raised in
previous literature surveys. More importantly, this result corresponds
with surveys of other technology-assisted learning contexts. For example Vogel
et al (2006) indicated that most studies on game-based learning focus on
effectiveness. The second most frequently cited research purpose was mobile
learning system design (32%), which is also a new finding. They found out that the number of studies devoted to
mobile learning system design increased over time, and suggested that
this may be due to rapid technology development such as new smart phones and
wireless data networks combined with the willingness of researchers to trial
new technologies in developing mobile learning systems.
Most mobile learning studies adopted surveys and
experiments as the primary research methods
Among the 164 studies, surveys were the primary research
method (50 studies), followed by experimental research methods (38) and
regardless of research purpose (i.e. evaluation-dominant with application-minor
or design dominant with evaluation-minor), quantitative approaches were
favoured over qualitative approaches. This is a new finding which corresponds
with findings in other technology-assisted learning contexts. For example
Zawacki-Richter, Backer, and Vogt (2009) found that quantitative methods
dominated distance education studies from 2000 to 2008, followed by qualitative
methods or triangulation methods.
Most mobile learning studies feature positive outcomes
86% of the 164 mobile learning studies presented positive
outcomes. This is another new finding which corresponds to findings in other
technology-assisted learning contexts. For example Ke (2009) applied a
meta-analysis approach to find that studies of game-based learning generally
have positive outcomes.
Mobile phones and PDAs are currently the most widely used
devices for mobile learning, but may be displaced by emerging technologies
In the context of mobile learning, device type has a
critical impact on teaching and learning. From their research W.-H. Wu et al
showed that mobile phones and PDAs account for over 75% (69/195 + 64/195) of
all mobile devices used in educational contexts. Technology advances quickly
and new types of mobile devices are emerging that can be applied to education.
For example Martin et al (2011) used the predictions from Horizon reports from
2004 to 2010 (covering 2004 – 2014), to analyse technologies that have impacted
education in the past or are likely to have an impact in the future. Horizon
report 2007 suggested that the use of mobile phones in mobile learning,
particularly in higher education, would expand dramatically after 2009, which
corresponded the findings of this study. In addition, Horizon report 2010
predicted that future mobile devices would add functions such as mobile
computing, open content, e-books, gesture-based computing, and visual data
analysis.
Questions
So what has been the distribution of ‘Studies by Technology’
over the last 5 years (2010 to 2014)?
What is new, and what needs to be new?
Is there a 2010 – 2014 survey analysis by year of devices,
purposes, outcomes, methods, etc?
Possible studies include:
(1.) Development
of mobile Technology use from inception till date
(2.) Applications
of mobile technology to learning over the years
(3.) Developmental
affordances of mobile technologies since the 1980s till today
(4.) Future
applications
Use of mobile devices for learning is most common in
higher education followed by elementary schools
Mobile learning is most frequently used in teaching and
learning contexts for higher education students (51.98%), followed by
elementary school students (17.51%), which corresponds with findings from Hwang
and Tsai (2011). More importantly, their study further indicates a significant
jump in mobile learning activity in higher education institutions in 2006, with
studies based in higher education institutions (1 in 2006, to 50 in 2010) and
elementary schools, (2 in 2009, to 26 in 2010).
Mobile learning most frequently supports learning in the
professions and applied sciences, the humanities and formal sciences
Studies on mobile learning in educational contexts most
frequently focus on use in supporting professional subjects and applied
sciences (29%), followed by the humanities (20%), and formal sciences (16%). In
terms of mobile learning activity in various sub-disciplines, their findings
partially support those of Hwang and Tsai (2011). For example both studies
showed mobile learning was often used in computer and language courses. More
importantly, the W.-H. Wu et al study found that mobile learning was often
related to environmental studies, forestry and health sciences, but
considerably less so in other courses such as statistics or law. However, they
suggest that mobile learning can be applied to any course or subject matter,
and researchers from different disciplines can collaborate to develop suitable
applications for under-represented courses.
Most highly cited articles fall into the categories of
mobile learning system design and followed by effectiveness
Based on the criterion count being equal to or greater than
40, three highly cited articles fall into the category of “designing a mobile
system for learning purposes” and one article is categorized as “evaluating the
effects of mobile learning”. This focus on highly cited articles is a new
finding not addressed in previous literature surveys. More importantly,
compared with the results of the distribution of mobile learning studies by
research purpose stated earlier, this finding reverses the order of the first
and second categories (i.e. designing a mobile system for learning now comes
before evaluating the effects of mobile learning), while the order of the third
and fourth categories remains unchanged.
For mobile-based technological development, they found that
articles belonging to the category “designing a mobile system for learning
purposes” describe mobile systems developed by researchers and educators prior
to any effectiveness evaluation. These systems can present important
applications in various disciplines such as bird-watching, learning of
professions, applied science/environmental studies and forestry for elementary
school students (Chen et al, 2003). These applications are more likely to be
cited by other related studies. Also, most of the highly cited articles were
published from 2003 to 2005, aside from one article published in 2008. This is
probably because similar to other technology-assisted learning contexts such as
the literature surveys of e-learning by Shih et al (2008), earlier articles
have a longer time to be disseminated and cited in other related studies.
Conclusions
Two previous literature review-based studies on the use of
mobile learning in academic contexts provided valuable insights, but failed to
examine the issue from directions such as the distribution of research
purposes. This study by W.-H. Wu et al conducted a systematic meta-analysis to
provide more comprehensive analysis of past studies, and discusses the
implications of new findings.
The study presents seven new findings:
(1.) The
research purpose of most mobile learning studies focuses on effectiveness,
followed by mobile learning system design
(2.) Surveys
and experimental methods were preferred research methods, regardless of whether
the research purpose focused on evaluation or design.
(3.) Research
outcomes in mobile learning studies are significantly positive
(4.) Mobile
phones and PDAs are the most commonly used devices for mobile learning, but
these may be replaced in the future by new emerging technologies
(5.) Mobile
learning is most prevalent at higher education institutions, followed by
elementary schools
(6.) Mobile
learning most frequently supports students in the professions and applied
sciences, followed by the humanities and formal sciences
(7.) The
most highly cited articles fall into the categories of mobile learning system
design, followed by effectiveness.
In summary, this study of issues
in mobile learning presents findings which can help supplement linkages with
previous studies and forms an important reference base for future research in
mobile learning.
References and highly
cited articles
Al-Fahad, F. N. (2009). Students’ attitudes and perceptions
towards the effectiveness of mobile learning in King Saud University, Saudi
Arabia. The Turkish Online Journal of
Educational Technology, 8(2), 111 – 119.
Baya’a, N., & Daher, W. (2009). Learning Mathematics in
an authentic mobile environment: the perceptions of students. International Journal of Interactive Mobile
Technologies, 3, 6-14.
Chen, G. D., Chang, C. K., & Wang, C. Y. (2008)
Ubiquitous learning website: Scaffold learners by mobile devices with
information-aware techniques. Computers
& Education, 50(1), 77-90.
Doolittle, P., & Mariano, G. (2008). Working memory
capacity and mobile multimedia learning environments: individual differences in
learning while mobile. Journal of
Educational Multimedia and Hypermedia, 17(4), 511-530.
Evans, C. (2008). The effectiveness of m-learning in the
form of podcast revision lectures in higher education. Computers & Education, 50, 491-498.
Glass, G. V. (1976). Primary, secondary, and meta-analysis
of research. Educational Researcher, 5, 3-8
Hossler, D., & Scalese-Love, P. (1989). Grounded
meta-analysis: a guide for research synthesis. Review of Higher Education, 13, 1-28
Ke, F. (2009). A qualitative meta-analysis of computer games
as learning tools. In R. E. Ferdig (ed.), Handbook
on research on effective electronic gaming in education (pp. 1-32).
Hershey: Information Science Reference.
Ketamo, H. (2003). xTask – an adaptable learning
environment. Journal of Computer Assisted
Learning, 19, 360-370.
Highly Cited Articles
Chen, C. M., & Chung, C. J. (2008). Personalised mobile
English vocabulary learning system based on item response theory and learning
memory cycle. Computers & Education,
51(2), 624-645.
Chen, C. M., & Hsu, S. H. (2008). Personalised
intelligent mobile learning system for supporting effective English learning. Educational Technology & Society, 11(3),
153-180.
Chen, Y. S., Kao, T. C., & Sheu, J. P. (2003).
