Real Learning in Virtual Worlds - CHAPTER 4: Results.
CHAPTER 4: Results
4.1 Introduction
In this chapter the researcher provides the results of the experiment using the methods discussed in the previous chapter. The results presented are the quantitative and the qualitative results for the virtual world learning experiment conducted in Second Life between two groups of participants the 2D group and 3D group that undertook different methods of delivery of a lecture on The Physics of Bridges.
A quantitative analysis was performed on the pre and post quiz scores of the two groups. This analysis includes the statistical test for significant difference of the pre-quiz results and the hypothesis of this experiment which measured the differences in the learning outcome between the 2D and 3D groups for Bloom’s cognitive processes of ‘remember’ and ‘understand’.
The finding for the post quiz survey Likert scale questions will be presented that measured the responses from the two groups learning experience survey.
A qualitative analysis was performed on the post-survey open questions of both groups where the data was coded into themes in order to gain a further understanding of the quantitative results and as well as the learning experiences of the two groups.
4.2 Quantitative Analysis Results: Achievement Scores
In this section the researcher provides the quantitative results for the pre and post quiz score results, the significance results for our operational hypothesis and conclude with the quantitative results of the post survey results.
4.2.1 Overview of Results
The results of the pre and post quiz totals can be seen below in the charted box plots (Figure 60). The left box plot is a traditional box plot, which provides consolidated information into a single graph.[28] The right plot is the same plot but referenced in percentiles in order to display the variance of the pre to post quiz scores. The number of questions in the pre-quiz was 8 and the post-quiz 20, each of which were evenly divided into Bloom’s cognitive process of ‘remember’ and ‘understand’.
Figure 60. Results: Pre & Post Quiz- Box Plot
4.2.2 Pre-Quiz Results
Table 9 provides the overall results of the 2D and 3D groups for the pre-quiz achievement scores. The pass rate is a measure of how many participants scored 50% or higher their quiz scores.[29] The pre-quiz was an 8 question quiz that tested the prior knowledge of a participant before the lecture.
| 2D Group | 3D Group | |||||
| Rem | Und | Total | Rem | Und | Total | |
| Pass Rate | 80% | 35% | 51% | 66% | 52% | 55% |
| Average Score | 2.44 | 1.25 | 3.69 | 2.071 | 1.60 | 3.68 |
| Median Score | 2 | 1 | 4 | 2 | 2 | 4 |
| Mode Score | 3 | 1 | 3 | 3 | 1 | 4 |
| Minimum Score | 0 | 0 | 1 | 0 | 0 | 0 |
| Maximum Score | 4 | 3 | 6 | 4 | 4 | 7 |
| Standard Deviation | 1.032 | 0.775 | 1.372 | 1.263 | 0.867 | 1.479 |
| Skewness | -0.138 | 0.261 | 0.007 | -0.195 | 0.351 | -0.188 |
| Kurtosis | -0.730 | -0.150 | -0.718 | -1.008 | 0.037 | -0.278 |
| Number of Participants | 55 | 55 | 55 | 56 | 56 | 56 |
Table 9. Pre-Quiz Descriptive Statistical Results
Figure 61 provides an inverse cumulative normal distribution graph for the total pre-quiz scores. This graph tells us what percentage (y-axis) of participants scored under a nominated score (x-axis). For example 50% of participants for both 2D and 3D scored under 4 in pre-quiz total score. As can be seen both the 2D and the 3D pre-quiz total scores were the same. For a detailed analysis of each of the Bloom’s cognitive processes for the pre-quiz see Appendix J: Pre-Quiz Score Results.
Figure 61. Results: Pre-Quiz Totals - Inverse Cumulative Normal Distribution Graph
Figure 62 provides a histogram and normal distribution curve of the total pre-quiz achievement scores. Both graphs provide frequency distributions but in different forms. The histogram provides for the number of participants (frequency y-axis) that scored between 1 and 8 (x-axis). The Gaussian distribution (or bell curve) provides the probability (y-axis) that a participant that would score between 1 and 8 (x-axis) based upon the average and standard deviation of the scores within each group. For a detailed analysis of each of the Bloom’s cognitive processes for the pre-quiz see Appendix J: Pre-Quiz Score Results.
Figure 62. Results: Pre-Quiz Totals - Histogram & Bell Curve
4.2.2.1 Pre-Quiz Significant Results
An independent t-test was performed on the pre-quiz total scores to ensure that the groups did not differ significantly in their prior knowledge of the lecture content on ‘The Physics of Bridges’, they did not (t = -0.367, df = 119, two-tailed p = 0.714, α = 0.05).
Although no significant difference was found between the two groups pre-quiz total scores, the scores for each of the Bloom’s cognitive processes of ‘remember’ and ‘understand’ did differ significantly between the groups. The 2D pre-quiz scored significantly higher than the 3D scores for the Bloom’s cognitive process of ‘remember’ (t = 1.665, df = 109, one-tailed p = 0.0494, α = 0.05). The 3D pre-quiz scored significantly higher than the 2D pre-quiz scores for the Bloom’s cognitive process of ‘understand’ (t = -3.03167, df = 109, one-tailed p = 0.0014, α = 0.05). Appendix J: Pre-Quiz Score Results provides a detailed analysis of these results.
4.2.3 Post-Quiz Results
Table 10 provides the results of the 2D and 3D groups for the post-quiz achievement scores. The post-quiz contained 20 questions of which were divided evenly into two groups of Bloom’s Factual cognitive processes of ‘remember’ and ‘understand’. The number of questions within each cognitive process was 10. As with the pre-quiz, the pass rate is a measure of how many participants scored 50% or higher on their quiz scores.
| 2D Group | 3D Group | |||||
| Rem | Und | Total | Rem | Und | Total | |
| Pass Rate | 85% | 35% | 67% | 93% | 36% | 77% |
| Average Score | 7 | 3.98 | 10.98 | 7.32 | 4.04 | 11.36 |
| Median Score | 8 | 4 | 11 | 8 | 4 | 12 |
| Mode Score | 8 | 4 | 11 | 8 | 4 | 12 |
| Minimum Score | 3 | 0 | 5 | 3 | 1 | 6 |
| Maximum Score | 10 | 8 | 17 | 10 | 8 | 17 |
| Standard Deviation | 1.846 | 1.484 | 2.468 | 1.597 | 1.464 | 2.347 |
| Skewness | -0.642 | 0.068 | 0.052 | -0.941 | 0.332 | -0.229 |
| Kurtosis | -0.729 | 0.558 | -0.152 | 0.672 | 0.010 | 0.265 |
| Number of Participants | 55 | 55 | 55 | 56 | 56 | 56 |
Table 10. Post-Quiz Descriptive Statistical Results
Figure 63 provides an inverse cumulative normal distribution graph for the total post-quiz scores. As was provided above this graph displays what percentage of participants scored under a nominated score.
Figure 63. Results: Post-Quiz Totals Inverse - Cumulative Normal Distribution Graph
Figure 64 provides a histogram and normal distribution curve of the post-quiz scores. As provided above with the pre-quiz graphs these graphs measure the frequency distribution of both the 2D and 3D groups.
Figure 64. Results: Post-Quiz Totals - Histogram & Bell Curve
4.2.3.1 Post-Quiz Significant Results
An independent t-test was performed on the post-quiz total scores of the 2D group and the 3D group showed that there was no significant difference between the results of these groups (t = -0.8212, df = 119, two-tailed p = 0.4133, α = 0.05). Appendix K: Post-Quiz Score Results provides a detailed analysis of these results.
The next section provides an analysis of the results for each of the Bloom’s cognitive process to test for signification difference between the post-quiz results for the tested hypotheses.
4.2.4 Hypotheses Results
As stated in Chapter 3 the operational hypotheses for this research study were as follows:
- (H1): That the learning outcomes for Bloom’s factual knowledge of ‘remember’ cognitive process will result in a significant difference in post-quiz scores between 2D and 3D participants.
- (H2): That the learning outcomes for Bloom’s factual knowledge of ‘understand’ cognitive process will result in a significant difference in post-quiz scores between 2D and 3D participants.
This section will discuss test results for no significant difference using the null hypothesis of H01 and H02.
4.2.4.1 Hypothesis One: Post-Quiz Remember
Figure 65 provides the histogram and the density traces graphs for the post-quiz results where 10 questions were given to both the 2D and 3D groups for the Bloom’s cognitive process of ‘remember’. As discussed in the previous section the histogram provides the frequency distribution of a participant’s scores. The density traces graph has been provided instead of the normal distribution graph as the results of these scores was not of normal distribution. The density traces graph provides alternative view of frequency that is similar to the histogram graph.
Figure 65. Results: Post-Quiz Remember - Histogram & Density Traces
Hypothesis H01
The null hypothesis tested H01:
- That the learning outcomes for Bloom’s factual knowledge of ‘remember’ cognitive process will result in no significant difference in post-quiz scores between 2D and 3D participants.
H01 was tested using the non-parametric Mann-Whitney U Test as the results for the post-quiz ‘remember’ scores did not meet the assumptions for parametric testing, which requires the scores to be normality distributed. The 3D scores failed the D’Agostino-Pearson (K2) normal distribution test (p = 0.01161 ie < 0.05) therefore the scores from this group deviate significantly from normal distribution. Appendix K: Post-Quiz Score Results provides a detailed analysis of the parametric testing results.
Formula H01
Using the following Mann-Whitney U Test formula to find U;
Where:
n1 = number of group 1 subjects
n2 = number of group 2 subjects
R1 = rank total for group which smallest rank sum
W = the critical value of U1
Results H01
The results of Mann-Whitney U Test when applied found that there was no significant difference between the 2D and 3D post-quiz ‘remember’ scores where the average ranked scores was 2D = 53.9364 and 3D = 58.0268 resulted in U = 1653.5, W = 113.5, 2 tailed p = 0.493107, thus we do not reject the null hypothesis for α = 0.05. (Note: There is a distinct “observable” difference between these two groups, just not a statistically significant difference. This is explored in the next chapter).
4.2.4.2 Hypothesis Two: Post-Quiz Understand
Figure 66 provides the histogram and normal distribution curve for Bloom’s cognitive ‘understand’ results of the 2D and 3D groups for the post-quiz achievement scores. As discussed above these graphs display the frequency distribution of both the 2D and 3D groups where 10 questions were given in the post-quiz for Bloom’s cognitive process of ‘understand’.
Figure 66. Results: Post-Quiz Understand - Histogram & Bell Curve
Hypothesis H02
The null hypothesis tested H02:
- That the learning outcomes for Bloom’s factual knowledge of ‘understand’ cognitive process will result in no significant difference in post-quiz scores between 2D and 3D participants.
H02 was tested using the parametric independent t-test of equal variance as the results met the assumptions for parametric testing. Appendix K: Post-Quiz Score Results provides a detailed analysis of the parametric testing results.
Formula H021
Using the following t-test formula to find t;
Where:
= the mean of group 1 = the mean of group 2 = number of group 1 subjects = number of group 2 subjects = the standard deviation of group 1 = the standard deviation of group 2
Results H02
The results of an independent t-test found no significant difference (t = -0.1926, df = 109, two-tailed p = 0.8477, α = 0.05) between the results of the 2D (x1 = 3.982, s1 = 1.484) and 3D (x2 = 4.036, s2 = 1.464) post-quiz ‘understand’ scores, thus we do not reject the null hypothesis.
4.2.5 Survey Results: Likert Scales
Table 11 displays the percentages of the post survey results divided into content knowledge, delivery method and technology. The content knowledge and delivery method questions were standardised into a 3 point scales for analysis.
| 2D Group | 3D Group | ||||||
| Content Knowledge | Low | Med | High | Low | Med | High | |
| 21 | My level of understanding of the topic PRIOR to subject delivery. | 89% | 9% | 2% | 91% | 5% | 4% |
| 22 | My level of understanding of the topic AFTER to subject delivery. | 22% | 51% | 27% | 23% | 50% | 27% |
| Delivery Method & Learning Experience | Positive | Neutral | Negative | Positive | Neutral | Negative | |
| 23 | Outline of subject material was clear and informative. | 98% | 2% | 0% | 100% | 0% | 0% |
| 24 | The lecture was detailed enough to provide an understanding of subject matter. | 100% | 0% | 0% | 93% | 7% | 0% |
| 28 | I found the in-world experienced offered me a better learning experience than my usual methods of learning | 74% | 22% | 4% | 73% | 25% | 2% |
| 29 | I found the subject material to be appropriate to virtual world learning | 84% | 13% | 3% | 79% | 18% | 3% |
| Technology | No | Yes | No | Yes | |||
| 26 | During the course I experienced technical difficulties with the environment | 91% | 9% | 93% | 7% | ||
Table 11. Survey Likert Scales Results
The content knowledge questions addressed the participant’s subjective impression of their knowledge before and after attending the presentation. Both groups perceived an increase in their understanding of the subject matter after the lecture. The delivery method questions measured the subjective satisfaction levels with the virtual world 2D or 3D delivery methods (as appropriate). Both 2D and 3D indicated very high levels of satisfaction. The technology question assessed if a participant had any technological constraints to their reception of the learning material. From the results presented above a few participants experienced technological problems.
4.3 Qualitative Analysis Results
4.3.1 Introduction
Qualitative analysis was performed using methods discussed in 3.9.4 Quantitative Analysis Methods section of this thesis for the 2D and 3D groups’ open question set (25, 30, 31 and 32) contained in the post survey. In this section we present a brief overview of how the analysis was performed and the major themes that emerged from the qualitative analysis results. Interpretation of these results will be discussed in the next chapter of this thesis.
4.3.2 Analysis Approach
Hermeneutic analysis of the post survey open questions was performed using an iterative approach in order to code data into contextual structures and common themes amongst 2D and 3D post survey responses. Data was first condensed into 2D and 3D categories and further into the individual question categories. Open coding uncovered general themes within each question and to further assist in this stage of coding a participant’s entire question responses were read as a whole in order to reveal the entire context of their individual responses. Axial coding was performed once a generic set of themes emerged to form relationships between the entire set of 2D and 3D group question responses. Opening coding and axial coding took several iterations before selective coding was preformed revealing 4 major themes along with sub themes that can be seen in Table 12 below. These themes along with their meaning are discussed below.
4.3.3 Themes of the Open Survey Questions
The open questions were as follows:
- DELIVERY METHOD ASSESSMENT (Q 25) General Comment:
- VIRTUAL WORLD LEARNING EXPERIENCE
- (Q 30) List 3 positive experiences you had with using this technology to learn:
- (Q 31) List 3 negative experiences you had with using this technology to learn:
- (Q 32) General Comment:
| Theme | Sub-Theme |
| Virtual World Learning | |
| Virtual Learning Campus | |
| Lecture Delivery |
|
| Survey Instrument |
Table 12. Qualitative Analysis Reuslts: Themes
The above themes were classified as follows:
- Virtual World Learning: This category included the aspects of a participant’s experience while using the virtual world as a learning platform. The types of comments contained in this category were not specific to the experiment but rather to the experience of the virtual world medium as a learning tool. The general features and characteristics of a virtual world that a participant disliked or liked about using this method of learning and their over-all impression that they had using the virtual world as a learning platform.
- Virtual Learning Campus: This category included comments about the virtual learning campus experience. These comments related specifically to the set-up and operation of the entire virtual learning environment within the virtual world.
- Lecture Delivery: This category was the major category that included comments about the lecture experience of a participant that was specific to the lecture delivery treatment they received. This category contained sub-themes as follows:
- Format: The style and layout of the presentation, how the information was presented.
- Information Content: The depth and breadth of information content presented about the topic (The Physics of Bridges).
- Learning: The aspects of obtaining new knowledge.
- Facets of 3D Learning: This theme contained only comments from the 3D group, their perception of the use of 3D models as a learning tool in delivery.
- Instruction: The method by which knowledge was transferred from the instructor to the learner, the interface between the presentation and the learner.
- Focus: The observations affecting attention and the temporal experience of a participant within the virtual world whilst they were learning.
- Navigation: Comments that related to the controlling their avatar within the lecture theater.
- Technical Constraints: Comments that related to technical constraints that a participant experienced during the lecture.
- Survey Instrument: This category included comments that related to the pre or post quiz of the participant.
Figure 67 provides a diagram of the relationship of these themes in the context of the qualitative analysis performed on the survey results. In the next chapter we will discuss the results of this qualitative analysis.
Figure 67. Qualitative Analysis: Relationship of Vitrual World Learning Themes
4.4 Summary
In this chapter we presented the quantitative and qualitative results of the research study.
A quantitative analysis was performed for both the 2D and 3D groups where the number of participants was 55 and 56 respectively. The pass rate for both the 2D and 3D groups’ pre-quiz scores was 51% and 55% respectively.
A significance test performed on the results of the total pre-quiz showed no significant difference between the scores of each group. Significance tests performed on Bloom’s cognitive processes of ‘remember’ and ‘understand’ showed a significant difference between the groups. The 2D group scored significantly higher than the 3D group for the Bloom’s cognitive process of ‘remember’ and the 3D group scored significantly higher than the 2D group for the Bloom’s cognitive process of ‘understand’.
The post-quiz pass rate for both the 2D and 3D groups’ total post-quiz score was 67% and 77% respectively. In spite of this, the results for the significance tests performed for Bloom’s cognitive process of ‘remember’ and ‘understand’ for the hypothesis showed no significance differences between the 2D and the 3D groups learning outcomes.
The post-survey results for the Likert scale questions was presented that provided the results dividend into positive, neutral and negative percentiles for both of the groups.
A qualitative analysis performed on the open-questions contained in the post survey revealed 4 major themes in the survey comments of both groups combined, these themes were:
- Virtual world learning environment,
- Virtual learning campus,
- Lecture delivery and
- Survey components of the research study.
A definition for each of these themes was provided along with a relationship diagram.
The next chapter we discuss the results presented in this chapter.
