|Contents | Abstract | Preface | 1 Introduction | 2 Theoretical foundation for mobile education | 3 Methodology and data collection | 4 Data analysis | 5 Conclusion | References| Print out from the forum |
Bekkestua - Norway, January 15th, 2003

3 Methodology and data collection

"In theory, there is no difference between theory and practice. But, in practice, there is."
- Jan L.A. Van de Snepscheut

This chapter discusses our choice of research design, and the process of our study. We also discuss the sources of error, validity and reliability of our results.

3.1 Research design
Not much work has been done to determine how new technologies for mobile learning influence the way distance students and teachers work. This can be attributed to the fact that the main focus has been on developing applications that are targeted towards addressing the needs of the corporate world (Sariola, Sampson, et. al., 2002).

We have chosen Delphi techniques, which is a design used by futurologists trying to predict the future. “Futurologists commonly allude two principles in their studies; The Principle of Continuity and the Principle of Analogy” (Merriam/Simpson, 1995, p.65). This means that for future prediction to be possible we assume that the present is very much like the future and that what we study will evolve the same way as it has in the past or not at all (Merriam/Simpson, 1995). There are two situations where Delphi technique has demonstrated its usefulness and these are: (1) when the subject for research is not easily precisely analyzed, but may benefit from opinions from a collection of individuals, and (2) when time- or cost- considerations makes it impossible for the subjects participating to meet face-to-face (ibid.). Both situations apply to our research.

Explorative research design is a method often used when the problem is unclear or the subject is new to researchers (Selnes, 1997). It is best suited for formulating problems, hypotheses and clarifying terms; or to give a better insight and understanding of a given area. Explorative research design forms the basis of subsequent, conclusive research design methodologies – namely descriptive or causal design.

While explorative design does have its strengths, its main weakness lies in its inability to specify relationships between variables. This owes to the lack of theory and insight into the nature of the problem. The strength is high internal validity as discussed later and “the researcher is the primary instrument for data collection and analysis” (Merriam/Simpson, 1995, p. 98). It is said that since the researcher is able to adapt and respond to external input, the human instrument is the ideal instrument for collecting and analyzing data (Merriam/Simpson, 1995). We used authorities and experts who gave their opinions to open questions. To address the problem with being able to generalize, we used authorities and experts who gave their opinions on open questions. Their responses were analyzed and structured into a list of alternatives that was then sent back to them. Using the structured list, the respondents voted on the alternatives in a quantitative way. This was done in order to gain some of the advantages of descriptive design as well as address the weaknesses with purely qualitative data and an explorative design. To further ensure the quality of our results we had a discussion on the results as a final step.

As the aim of our study focused on addressing issues in a relatively new area, we found it appropriate to use an explorative research design bordering descriptive design, as this was best suited for predicting the future (ibid.).

3.2 Research approach
Yin (1994) identifies two types of research approaches: qualitative and quantitative. A qualitative research approach is characterized by its descriptive nature that takes the form of text, in-depth interviews or the use of focus groups. Quantitative studies, on the other hand, involve the extensive use of statistical data and tools.

The use of qualitative data gives respondents a lot of room to freely express themselves. In so doing, they provide the researcher with a good overview of the problem that has to be addressed. Yin (ibid.) recommends the use of qualitative methods when examining previously unexplored areas. The problem with qualitative methods is that answers obtained from raising open questions are unstructured. This makes data analysis a difficult and time-consuming task. In addition, respondents may express their subjective views, which may be biased, hence resulting in a higher margin of error.

Given the need to delve into our research questions, we have primarily used qualitative methods in our research. As a supplement, quantitative methods have been used to provide more precision when interpreting the data collected. By triangulating across data sources, we tried to ensure that the weaknesses of one method were compensated for by the strengths of the other (Halvorsen, 1997).

3.3 Population and sample selection
We have defined our population as all mLearning authorities and experts. The term “mLearning experts or authority” encompasses all individuals who are well versed in the field of mobile learning. Our selection consists of experts who have worked on mLearning projects in Norway, Finland, Germany, Sweden, Ireland, North America and Canada. Most experts in our research were obtained by querying the World Wide Web for mLearning researches and projects. Others were contacts we had either met prior to our study, or whom had been recommended to us by acquaintances. The list of participants can be found under section 3.5.1, Table 3 1.

3.4 Research strategy
We chose to use both primary and secondary data. The primary data we sought to obtain in order to answer our research questions were based on expert opinion (Skumanich and Silbernagel, 1997). These opinions address issues on current trends and act as a basis upon which future conditions can be assessed. Skumanich and Silbernagel (ibid.) refer to this as fore sighting. Primary data were obtained using Delphi techniques.

The Delphi technique (The Consummate Design Center, 1996) is a structured process by which experts are asked a series of questions on a given set of issues. A facilitator then analyzes the comments passed by each of the experts and produces a report showing all the participants' responses. Each of the experts then makes a comparison of their responses to that on the compiled report. This forms the basis of a discussion, which can be face-to-face or remote. In the course of the discussion views expressed by different experts can be promoted, challenged or shared. Based on the views expressed in the discussion, a new round of opinions is anonymously solicited, a report is created and the cycle continues until a consensus or a stable disagreement is reached. Section 3.5 looks at how we applied Delphi techniques in the course of our study.

As a supplement to the Delphi method, we identified thwo other methods for the collection of primary data. These are: Online focus group and attending seminars on mLearning.

The focus group method "involves organized discussion with a selected group of individuals to gain information about their views and experiences of a topic. Focus group interviewing is particularly suited for obtaining several perspectives about the same topic." (Gibbs, 1997). Generally focus groups are composed of 6 to 8 participants and a moderator who ensures the correct questions are asked and facilitates the discussion process. Our focus group was conducted through a web-based asynchronous discussion forum.

We attended three seminars that touched on the topic of mLearning to varying degrees. These were organized by: The Norwegian computer association (no: Den Norske Dataforeningen), http://dataforeningen.no/publikasjoner/paa_siden/ pdf/pdf2002/51nr_u34.pdf; Ericsson in Dublin - “The Cutting Edge”, http://learning.ericsson.net/leonardo/cutting_edge/, and Research & Educational Network (REN) - “eLearning in higher education, the status in Norway”, http://www.ntc.no/cgi-bin/wbch.exe?page_id=3091&d_id=14020, where we presented our findings and conclusions at the time being. From these we gained new insights and ideas for our research as well as support for some of our assumptions.

Given the pace at which new technologies for mLearning are being developed, it was of utmost importance that secondary data used in our research was not obsolete. To alleviate this risk, we used the latest relevant secondary data that was publicly available. The sources of our secondary data are books and articles. We also actively used the Internet to search for online articles on related work. The use of secondary data provided us with background information upon which we came up with a creative strategy for collecting primary data. Secondary data gave us a better understanding of our research problem.

3.5 Primary data collection
Primary data was collected through a series of three distinct phases. Figure 3 1 below shows the dataflow throughout the entire primary data collection process.

Figure 3 1 Dataflow throughout the primary data collection process

3.5.1 Phase one – open questions
The first phase of our research lasted from October 25th to November 7th, 2002. We sent out 24 personal invitations (see appendix A) to mLearning experts (see Table 3 1 below), asking them to take part in our mLearning study and respond to the following questions:

1. Which technologies (both existent and non-existent) and / or factors do you think will have the biggest impact on the adaptation of mLearning?
2. Which properties of the mobile device do you think will have the biggest impact on how fast mLearning is adapted?
3. How will mLearning change the work situation of a student?
4. How will mLearning change the work situation of a teacher?

Table 3 1 mLearning experts invited to the study

In total, we received 12 responses within the given time limit (see appendix B for original responses). These were then compiled into a structured report from November 8th to 12th, 2002. Upon completion, the report was sent back to all participants on November 13th, 2002.
One reply came in too late to be included in phase two, but it is still documented and taken into consideration in the later phases, discussion and conclusion of our report. Other respondents had already mentioned the answer given by the respondent who answered late. We therefore allowed him to continue in the process. We came across a new expert at this point who was a key person in an mLearning project in which he was working. This led us to invite him directly to phase two of our study. We sent him the letter sent out to those who participated in phase one, as well as that sent out for phase two, so that he would know what he had missed and bring him up to speed on where we were in our research. Given the expertise of this individual, we felt that it was of interest to get his opinion on phase one and two of our study.

Upon concluding phase one, we had a total of fourteen participants. Their answers were structured into four tables. Each table gave a summary of alternative answers per research question as shown in Table 3 2 through Table 3 5 below.

Phase One, Question 1
Which technologies (both existent and non-existent) and / or factors do you think will have the biggest impact on the adaptation of mLearning?
Total number of participants = 14

1. 3G mobile telephones (devices) and PDAs with integrated communication technology.
2. Access to learning material anytime anywhere
3. Bluetooth
4. City-wide W-LAN networks
5. Common understanding of learning as a lifelong continuous activity
6. Communications cost
7. Content must be in place
8. Content on demand (libraries and such)
9. Durable power source
10. Easy to exchange material with others
11. High bandwidth wireless data transfer
12. High resolution screens
13. Intelligent agents
14. Large unfoldable touchscreens
15. Low power high capability processors
16. PDA pricing
17. Personalization of learning material (e.g. adding own notes)
18. Small-footprint technologies for interaction
19. Small-footprint technologies for media
20. Tablet PCs
21. Text input technology
22. Universal availability of mobile phones
23. Voice input (converted to text) and recognition
24. We take technology in use if we think that it will help us to fulfill these intentions (1 - convenience/rationality, 2 - expediency and 3 - immediacy)
25. Wireless internet access

Table 3 2 Alternatives received from the open questions: phase one, question 1

The alternatives in Table 3 2 indicate a tendency towards wireless data transfer as the choice of technology that will have the biggest impact on the adaptation of mLearning. This research question received the largest number of alternatives, with experts providing a wide range of technologies. The range of technologies is a clear indication of the possibilities currently available in the market, or those that are expected to emerge as important technologies in the foreseeable future. Although we felt that some alternatives did not provide answers to this particular research question, and would be best answer one of the other research questions, we chose not to make modifications to their placement. This was done to ensure data integrity, while also taking into consideration the fact that respondents may have personal reasons for their choice of alternatives.

Phase One, Question 2
Which properties of the mobile device do you think will have the biggest impact on how fast mLearning is adapted?
Total number of participants = 14
1. Audio-processing
2. Better screens
3. Compatibility PC - PDA
4. Cool features and features that gives status to the owner
5. Full size keyboard as standard
6. Input mechanisms
7. Interactivity
8. Internet access
9. PDAs ability to read standard web pages
10. Screen size
11. Storage capability
12. The connection cost of mobile services
13. The connection speed
14. The user friendliness
15. Video and TTS (Tele-Time Systems) technology
16. Wireless connectivity
Table 3 3 Alternatives received from the open questions: phase one, question 2

The list of alternative answers to the second research question was fewer than those in the previous question. The list here is more comprehensive, with not many overlapping alternatives.

Phase one, Question 3.
How will mLearning change the work situation of a student?
Total number of participants = 14

1. Higher demand on the students and their motivation
2. Easier communication with teacher and fellow students
3. Increased flexibility
4. It will make continuous learning come true
5. Learning adapted to users needs
6. Learning will take place in other environments
7. Less books
8. Less paperwork
9. Less persistent studying
10. More efficient studying habits
11. More fun
12. More just-in-time learning
13. Slowly
14. Student’s have to plan their time and take control over their own learning
15. Students gets more organized
16. Students will develop new ways of studying
17. Students will take more notes
18. Studying on the move
Table 3-4 Alternatives received from the open questions: phase one, question 3

On this question we got many different alternatives. Some saw a great deal of changes while others saw few and that it was to happen slowly (#13), if at all. The one alternative we felt was to receive many votes later on was the increased flexibility (#3), which it also did. The most surprising answer here we found to be that students would take more notes and get more organized. Is this to say that if a mobile student is not very organized it cannot be done? Or does it mean that since the device is capable of being used not only for reading and communicating with, they will also make more notes? Will the students take more notes if they have the material with them all the time? While on the subject of note taking, we found that the ability to highlight and take notes directly on the text in Microsoft? Reader was a very good way of working.

According to the results of a couple of R&D projects I have been involved in, both students and teachers appreciated the flexibility that mobile devices provide. The results were more positive than we expected. It seems that people of those project groups had found out a way, which worked pretty well, to organize their doings and to use mobile technology in an effective way. In group interviews they argued that they had fingered their devices so eagerly during the day that they had got more leisure time. They even spoke about enhanced quality of life!
- Dr. Heikki Kynäslahti; ”Nietzsche” in the forum

Phase one, Question 4.
How will mLearning change the work situation of a teacher?
Total number of participants = 14

1. Classrooms can be extended to the real world
2. Focus on one-to-one training
3. Increased flexibility
4. Less time off from work
5. More rapid answers to the students’ questions will be demanded
6. No clear separation of work and value time
7. Place will become both more and less important
8. Slowly if at all
9. Teachers need to learn how to make use of the mobile devices and how to produce content for mLearning and mobile devices
10. Teachers will not be able to escape the students
11. Teamwork will be even more important when producing content for mobile learning.
12. Traveling teachers may work on the move
Table 3 5 Alternatives received from the open questions: phase one, question 4

This question only received a total of 12 different alternatives. We wondered if we should ask for a top-5 list only, but decided to go for the top 10 here as well. Some alternatives here were similar to those in the previous question for instance, increased flexibility and the ability to work on the move. The few alternatives provided made it natural to believe that every alternative would at least get one vote in phase two of our study.

I think the move will be to continuous, on demand, and individual learning. Systems will support your development. The 'teaching' role will be replaced by learning mentors and subject-matter experts who know learning too.
- Dr. Clark Quinn; “Aristotle” from the forum

3.5.2 Phase two – top 10 lists based on alternatives from phase one
Phase two ran from November 13th to 20th, 2002. In this phase, we asked participants to create a top-10 list from the alternatives in phase one of our studies. The top-10 list would later form the center stage for an online asynchronous forum.

Table 3 6 through Table 3 9 show the final top-10 lists for each of our research questions. A full description of how these lists were created may be found in appendix D.

The column labeled “Tot.” shows the total number of points awarded for each of the specified alternatives. Hence the higher the total number of points, the higher the rating on the list.

Table 3 6 Accumulated top-10 list received from phase two, question 1

This question is the only one that all of the respondents managed to create top 10 lists for. This may be due to the high number of alternatives or the fairly similar alternatives considering wireless data transfer. The important thing to notice here is that all but one respondent had High bandwidth wireless data transfer on the listing. However, none rated it first in their top-10 list (see appendix D). Overall it was the number one technology to influence the adaptation of mLearning.

Probably the third generation mobile telephones (devices) and PDA:s with integrated communication technology. These devices will make it possible to access the Internet anywhere at any time. I also think that W-LAN and other similar future technologies will be implemented and used locally. The problem is the cost of mobile Internet services, I think there is a risk that 3G costs will be too high and thus will only be used by a small group of people. City-wide W-LAN networks might be a cheaper and faster technology.
- Tommy Strandvall; “Pythagoras” from phase one, question 1

3G mobile telephones (devices) and PDAs with integrated communication technology on the other hand were listed in first place in the top-10 list of four respondents. However, on summing up the totals for the different alternatives, its final ranking was third.

The flexibility and the equipment used for m-learning may reduce the quality of the information exchange. It may increase the number of “quick and dirty” comments…simply because you just spend a few minutes available at the bus to write a comment. Neither the setting, nor the equipment is supportive of high quality, reflective comments.
- Dr. Morten Flate Paulsen; “Descartes” in the forum

One of the problems with the small devices is that it is hard work to write new material on them. You either have to learn a new alphabet for the palm, for instance, or use a small detachable keyboard. This is perhaps why many seem to view voice input as a good way to go.

Somehow, we've got to stop simply miniaturizing things we're used to (like display monitors and keyboards) and start thinking about mobile devices as being truly different and deserving of their own unique methods of input and output. Perhaps the answer for output is in some sort of personal visual display, such as glasses. And I think input needs to be something natural, like speech, though not necessarily.
- Dr. Paul G. Shotsberger; “Plato” from phase one, question 1

Another way of thinking is to increase the size of the mobile unit and look at what is happening with Tablet PC, here the screen is larger and it is more of a PC than the small PDA’s are. This could be the new device used in the homes for reading news on the web and so on, why not use it for education as well? Broadband technologies are beginning to be common in Norwegian homes. Similar trends are seen in many western European countries. This can be attributed in part to huge advertising campaigns by internet service providers who seek to gain a large market share in the broadband sector. It also reflects on the needs of today's consumer who wants information delivered both promptly and in a cost-effective manner.

Wireless access to broadband Internet providers in private homes in combination with future types of advanced PDAs and tablet PCs will have a substantial impact on the adaptation of mLearning. This equipment will give inexpensive, immediate, and convenient access to educational services anytime anywhere in peoples homes.
- Dr. Morten Flate Paulsen; “Descartes” from phase one, question 1

Table 3 7 Accumulated top-10 list received from phase two, question 2

On this question there was only one respondent who did not have Internet access on their top-10 list. Its rating was varied but it did get first place thrice. There are many opinions on what are the most important properties of a mobile device. As the listing in Table 3 7 shows, there are very different views on what is important. Some think it is the device's abilities. Others think screen size and compatibility with the PC are important factors.

As stated in question 1 the wireless data networks are of great importance and the devices needs access to these networks.

Wirelessness is the key. The future is wireless and I see a great increase of wireless solutions in all areas of society. Much of the solutions have already been provided by PDAs and I consider that they already provide a viable learning environment. Properties that remain to be solved for mobile phones include screen size, increase in bandwidth, increased memory. Most of these are promised by the move to 2.5G and 3G but until mature solutions are generally available and accepted by society the impact of mobile learning will be reduced.
- Dr. Desmond Keegan; “Hobbes” from phase one, question 2

Most of the properties mentioned are alreday present on new mobile units found in the market. But, there are some important points from the forum on new features and better ways of presenting course content.

Dr. Georg Ströhlein states had the following contribution in the forum:

I also believe that it could really add value to mobile learning if there were document formats allowing to add personal written remarks, sketches and even spoken words (in compressed format) and easily share this stuff with co-learners or the tutor. This means, transmit only the added "bytes", not the material itself.
- Dr. Georg Ströhlein; “Kant” in the forum

The adaptation of mLearning is not only a matter of the devices and what they can provide in an educational arena, but perhaps more importantly how many people have access to these devices.

The central factor is the almost universal availability of mobile
phones: almost 1 billion available today. In addition these are technologies that people are used to carrying around with them, they are easy to use, robust and easy to maintain. They are, in addition, trusted, personal devices.
PDAs are also important wireless devices and quite suitable for
learning as the NKI part of the Ericsson mobile learning project has clearly shown. I feel, however, that they will never become universal devices; there are only 2 million of them in the world today and sales are not increasing.
- Dr. Desmond Keegan; “Hobbes” from phase one, question 1

Table 3 8 Accumulated top-10 list received from phase two, question 3

Increased flexibility received six first places in the respondents’ top-10 lists, and is thus their favorite alternative. Only one respondent did not have it on his or her list.

In my view it is the flexibility of distance education that is the most important factor for most students. While online learning via standard technology reduced the possibility to study where and when you want, as with correspondence education, mobile technology will increase the flexibility, so that students can learn where they wish, when they wish for shorter and longer study periods.
- Torstein Rekkedal; “Sartre” from phase one, question 3

All, expect one respondent, thought that Learning will take place in other environments was worth placing in the top-10 list.

Mobile learning will make students not only able to study at a distance as in distance education and e-learning but also when they are on the move away from the institution
- Dr. Desmond Keegan; “Hobbes” from phase one, question 3

…Another thing is that, at least in my alma mater, the University of Helsinki, some departments are decisively moving a part of their activities outside of the university to real contexts: forestry students go to forest, student teachers go to school. Of course they have always done so but thanks to mobile technology this kind of studying and teaching in real contexts and everyday situation works better.
…
One issue that is often discussed in the context of the use of mobile technology is fragmentation. Mobile working is often characterized by temporal briefness. It can be that due to mobile way of working, processes of studying and teaching may become less persistent. This is speculation, I do not have evidence to prove it. However, it is interesting juxtapose the two somewhat paradoxical aspects I have mentioned, ‘harmonization’ and fragmentation. It seems that if you success to put the pieces of your day, a part of the pieces being fragments of mobile working, together in a coherent way, the result may be a rather harmonic wholeness. This sounds idealistic but I see some potential in mobile working to do so
- Dr. Heikki Kynäslahti; ”Nietzsche” from phase one, question 3

Fragmentation is good, but one of the learning goals will be to improve one's own learning ability
- Dr. Clark Quinn; “Aristotle” from the forum

Table 3 9 Accumulated top-10 list received from phase two, question 4

The alternative increased flexibility was rated first by six of the fourteen respondents. Students may study at new places and the teacher may extend the classroom to the real world.

The answers here are consistent with those presented in Table 3 8 where the change in a student’s work situation is addressed.

One issue that mobile learning may help to address is that of field work where the students might be out in the field analysing geological substances and the teacher is helping them from a remote location.

I think real life and real experiences are the best teachers. If we can bring the students closer to real life situations with the help of mLearning then the learning experience can be improved a lot.
- Tommy Strandvall; “Pythagoras” in the forum

The students might have greater expectations on the speed at which teachers respond to their queries, but not necessarily. It might also give the teacher more control over their time and when they want to answer the questions.

increased flexibility which makes it possible to work not just any time, but anywhere, while traveling, in between-situations where he/she is not near a computer. It´s easier, faster and in many situations more convenient. But also time consuming and intruding
-Tove Kristiansen; “Socrates” in phase one, question 4

3.5.3 Phase three – online asynchronous forum
An online asynchronous forum constituted the third phase of our research, which lasted from November 27th to 28th, 2002. Participants were requested to give at least three contributions in light of the top-10 lists created at the end of phase two. This forum was located at http://www.diskusjon.no/mlearning. Prior to commencing this phase, usernames and passwords were sent out to all those who wished to participate. In addition, we sent them the structured top-10 list for each of the four research questions.

Given that participants’ identities were not revealed in the discussion forum, answers given are unlikely to be biased. We were permitted to reveal all the respondents’ real names. This was done where we found it appropriate. Appendix K is one such example: The names are linked to the aliases used in the forum and respondents identification numbers (r1 through r14). Asking for participation in our study, which involved answering open questions, participating in a voting session and attending a two-day asynchronous online forum was no easy task. However, we were satisfied with the responses we received.

We arranged the forum to accompany the questions posed to ensure we understood the answers given better. We also wanted to solicit different views on mLearning and maybe also create a forum that can be used outside of this particular research. If the respondents were familiar with the forum they may want to use it later on to have other discussions on mLearning. We have used some quotes from the forum to show views on the different aspects of mLearning in the data analyses chapter 4. The forum has influenced how we analyzed the top 10 list for our conclusions. The entire forum may be found in appendixes G through J. It may be analyzed separately and act as a base for future research.

The forum ran on a Microsoft? Internet Information Server version 5.0 (hereafter referred to as IIS). Our choice of web server was incidental. One of the group members had the necessary web server software. We then had a web server available on which to post online information to the respondents.

Prior to installing the forum software, a database is required. To this end, we installed MySQL. “MySQL is the world's most popular Open Source Database, designed for speed, power and precision in mission critical, heavy load use”(http://www.mysql.com/, 13th December 2002). It is a free database available at http://www.mysql.com/.

Finally, we installed the forum software. The software we used is called phpBB. It is available free of charge at http://www.phpbb.com/. “phpBB is a high powered, fully scalable, and highly customizable open-source bulletin board package. phpBB has a user-friendly interface, simple and straightforward administration panel, and helpful FAQ. Based on the powerful PHP server language and your choice of MySQL, MS-SQL, PostgreSQL or Access/ODBC database servers, phpBB is the ideal free community solution for all web sites.” (http://www.phpbb.com/, 13th December 2002).

Upon completion of the server configuration, we performed some server administration tasks. We set up user accounts using aliases we had assigned to the respondents. As previously stated, the reason behind using aliases was to preserve anonymity, which is a requirement for the proper use of Delphi techniques. Additional security settings were applied to ensure authenticity of those logging into the mLearning forum. This was all we needed to be able to set up the discussion forum at http://www.diskusjon.no/mlearning/, which was used between the 27th and 28th of November 2002.

The screenshot in Figure 3 2 below shows the page presented to respondents upon logon.

Figure 3 2 The discussion forum front page after logging in

3.6 Secondary data – studying existing literature
We queried the BIBSYS and ACM databases at the Norwegian School of Information Technology’s main library for books and articles on mLearning using the advanced search engine. The search string included the words “mLearning”, “mLearning research”, “wireless” and “education”, “wireless networks” and “education”, “distance learning” and “wireless”. Similar search strings were used when using search engines on the Internet.

The results of our queries provided us with several references to projects and books on mLearning. Of interest to us were mLearning books and articles where didactic solutions were being tested on hand-held devices. The contributions from projects such as UniWAP at the University of Helsinki, KNOWMOBILE at the University of Oslo’s medical school, the EU project “From e-learning to mLearning” and NKI’s SPICE project – to name a few – provided us with a theoretical background upon which we have formulated our research questions and obtained statistics on current trends in the use of hand-held devices in distance education.

3.7 Validity
Validity refers to the degree to which the research addressed the problem it set out to measure (Grennes, 2001). In other words, we ask ourselves whether we are in fact measuring what we intend to measure. “…threats to validity are of particular concern in the design of an experimental study because the researcher hopes to make predictions from the research results” (Merriam/Simpson, 1995, p. 58). They go on to identify internal and external threats to validity, and list them as follows:

3.7.1 Internal threats – threats to which degree we measure what we intend to measure.
The research we have done is not experimental. However, we hope to obtain valid results upon which we can draw fairly accurate predictions about mLearning’s future. Our design is explorative bordering to descriptive; therefore we have to take considerations for both designs.

1. History of events in the experiment
2. Maturation of subjects through time
3. Effect of testing upon subjects
4. Errors of measurement or observation
5. Biased selection of subjects
6. Statistical regression
   (Merriam/Simpson, 1995, p. 59)

To ensure the validity of our research we have taken the following actions: We have tried to shorten the time in which the respondents are actively taking part in the research as much as we could without risking them not having adequate time to respond within the set time limits. This was done to avoid internal threats #1 (History of events in the experiment) and #2 (Maturation of subjects through time).

We tested the questions sent out to respondents internally within the group and also involved third parties not connected to our study. Testing was done to ensure that any correspondence sent out was comprehensible and was likely to provide us with the answers we were looking for. In this manner, we addressed internal threat #3 (Effect of testing upon subjects).

In phase one we asked open questions and got many different views. These were organized in lists of alternatives for each question and sent back to respondents. We have done our best to include all opinions and views. This was done by first writing down all answers in a Microsoft? Excel spreadsheet, followed by a new spreadsheet that contained shorter structured answers, and finally a third spreadsheet with voting alternatives. Some answers overlapped, thus compelling us to generalize them before sending them out for phase two of our study.

We had a long discussion to come up with the best rating for phase two. We thought of using a rating system where all questions should get a numerical value between 1 and 5, but not everyone uses this scale the same way. Some never use the extreme values and others do not. At one point we thought of giving the respondents a fixed sum of points (e.g. 50) to distribute among the alternatives where no one alternative exceed 20 points, for instance, and all alternative should get a minimum of 1 point. This seemed as a good idea, but we were unsure if everyone would use the system the same way, as is the case with any scaling system. The problems with scales is what lead us to the “Top 10 list” approach, since everyone knows what that is, and knows that first place is more valuable than last. All our questions had more than 10 alternatives, so every respondent had to discard some of them. This was, in our opinion, a good solution since we wanted to see which alternatives stood out the most. We thus avoided internal treat #4 (Errors of measurement or observation).

When the lists from phase 2 were all returned we created a list for each of the questions to reflect the preferences of all the respondents. To avoid loosing anonymity in the online forum, we decided to hide the respondents’ true names and e-mail addresses. Instead, we issued them with aliases. The aliases we decided to use were ancient philosophers. The reason for this was to avoid using names that would give hints about a respondent’s true identity, but also make participation at the forum interesting.

When we chose the respondents we might have fallen for threat number 5 (Biased selection of subjects). This is because one of the group members works at NKI and is involved with the EU Project “The future of learning: From e-learning to m-learning”. This was the reason for being selective in our choice of participants in the study. However, several individuals recommended experts from NKI to us. We also received many hits from NKI when searching for articles using famous search engines, such as Goggle and Yahoo. The other respondents were selected on the basis of mLearning articles they had published on the Web or on-going research that they were involved with at the time.

Statistical regression, threat number 6, (“the tendency for participants scores to move closer to the mean upon retesting” Merriam/Simpson, 1995, p. 59) was not an issue since we only did the test on the respondents once.

The discussion so far is based on the assumption that we use a descriptive or experiment design. However, as previously stated, we have used a design that has its foundation in explorative but borders both descriptive and experimental design. We have taken some precautions in our choice of research design so to be able to generalize to some extent. Merriam/Simpson (ibid.) list 5 ways to ensure that we get as close as possible to reality when using an explorative design:

1. Triangulation – the use of multiple investigators, sources of data and methods to confirm the emerging findings
2. Member checks – taking data collected from study participants and your tentative interpretations of these data back to the people from whom they were derived, asking if the data “ring true”
3. Peer / college examination – asking colleges to examine your data and to comment on the plausibility of the emerging findings
4. Statement of the researcher’s experiences, assumptions, biases
5. Submersion / engagement in the research situation – collecting data over a long enough period of time to ensure an in-depth understanding of the phenomenon
   (Merriam/Simpson, 1995, p. 102)

As stated above most of these precautions have been considered as part of the descriptive design issues, the ones not yet addressed are discussed in this section.

Strategy #1 (Triangulation – the use of multiple investigators, sources of data and methods to confirm the emerging findings) – we are three researchers who have discussed all questions and answers in-depth to ensure common understanding of the questions and answers. We have used both primary data and multiple sources of data; online reports and articles, as well as research projects, books and articles in papers and magazines. To ensure that our conclusions were based on correct data and that we understood the answers the respondents gave us, we resent data we have analyzed and structured back to respondents for voting (phase two of our study). We even went further by arranging an online asynchronous forum to discuss the answers gathered from everyone and views on the results. This also addressed strategy #2 (Member checks – taking data collected from study participants and your tentative interpretations of these data back to the people from whom they were derived, asking if the data “ring true”).

Strategy #3 (Peer / college examination – asking colleagues to examine your data and to comment on the plausibility of the emerging findings) is addressed in the discussion forum where we discuss the overall results and process.

Strategy #4 (Statement of the researcher’s experiences, assumptions, biases) is addressed in chapter 2 of this report, where we discuss our assumptions.

Strategy #5 (Submersion / engagement in the research situation – collecting data over a long enough period of time to ensure an in-depth understanding of the phenomenon) is taken into consideration by reviewing research done on the field over a period of time, but ensuring that it had not lost its meaning over time.
3.7.2 External threats – threats to which degree we may generalize from the result.
The external threats in our study are of minor significance due to the fact that we use an explorative design that only borders descriptive design.

Merriam/Simpson (1995, p. 59) identify the following external threats:

1. Extent of randomization in subject selection
2. Effects of pre testing on subjects
3. Effects of the experimental setting (Hawthorne / placebo effects)
4. Effects of multiple treatments in the experiment

Because we have a small population and naturally an even smaller selection there has not been a random selection of subjects that threat #1 addresses. We deal with the authorities and experts in mLearning. Thus a randomized selection of all mLearning researchers is not necessary. We needed experts that had focus on higher learning and more demanding courses: Not kindergarten and weekend courses. This was one of the most curtailing variables and did not leave us with an abundance of people. All the experts that we felt could enlighten us and had the required prerequisites to answer our problem statement were invited to participate in our research. We may not generalize from our results statistically, but that is not our intention either. Our aim is to try and predict the future and what must be in place for mLearning to be widely adapted. Findings from qualitative research may be generalized in the sense that is called reader or user generalizability; “the extent to which findings from an investigation can be applied to other situations is determined by the people in those situations. It is not up to the researcher to speculate how the findings can be applied to other settings; it is up to the consumer of the research” (Merriam/Simpson, 1995, p. 102-103). To predict the future is what the Delphi technique is all about.

This discussion is also based on the assumption that we use a descriptive or experiment design. But, as stated previously, we use a design that has its foundation in explorative design that borders descriptive and experimental design methods.

3.8 Reliability
Reliability is a measure of the degree of precision in the results we present (Grennes, 2001). Yin (1994) attributes reliability to the extent to which similar results can be obtained if the same research is repeated by another researcher. Given the high degree of volatility in the wireless industry and the mobile learning arena, it is unlikely that research conducted in the distant future will yield similar results. However, those conducted in the short-term, should bear a reflection of the findings we have presented.

To ensure a high reliability of our study we have taken into consideration strategies developed by Merriam/Simpson (1995). They list them as follows: Triangulation and peer examination, which we have taken into consideration under internal validity in explorative research (strategy 1 and 3). Another consideration is the audit trail. “In order for an audit to take place, the investigator must describe in detail how data were collected, how categories were derived, and how decisions were made throughout the inquiry” (Merriam, 1988, p. 172). This has been done as thoroughly as possible throughout all phases of our study (see the appendix sections).

We found the Delphi method to be useful for clarifying issues when we where in doubt as well as gaining a common understanding of the problem in hand. We found the technique helpful in understanding what the respondents

To the data analysis