How do we know that a teaching and learning experience has been effective, especially in these pandemic days when nearly everything is in an online format? How do we know that the learners have engaged with the material and with each other? How can we improve instruction and strategies to make the most of educational technology tools to enhance learning? These are all excellent questions and some of the answers lie in the area of learning analytics and educational technology.
When we talk about learning analytics, there are many data points that are associated with a course that comprise an ecosystem of learning analytics for that course. Not all are of equal value in answering the aforementioned questions about learning effectiveness. The end of course grade often does not tell us much, or enough, about the effectiveness of the take-away learning from a course. It certainly does provide information about performance scores on graded assessments for a course. Those assessments, likewise, may or may not tell us much about learning effectiveness.
Similarly, student-generated course evaluations tell us something about student perceptions of the course, how students thought the course went, how they perceived the effectiveness of instruction, how they assess how much and what they learned from the course, and so on. This is another source of data analytics for the course but it too has several shortcomings. First, these are self-reported data. Second, they are subjective data without an objective verification check. Third, even if we trusted these data, the scope of what they tell us is very limited.
Luckily, there are many more options available to us to provide more robust learning analytics that can be useful in improving instruction, increasing learner engagement and expanding effective learning outcomes. When I first became involved with online learning and educational technology, approximately 25 years ago, learner analytics at the course level were really limited. There was a built-in clock in the learning management system that could record the minutes that a student was logged in to the course. Later, this was refined to specific sections of a course. Once it was revealed that students could log in and walk away from their computer and still keep the meter running, a “time out” function was added that would automatically log a student out of a course after a set period of time without keyboard activity within that course.
Today, dramatic improvements have been made in this area and more are on the horizon.
Educational technology innovations facilitate the collection of more and better data than ever before. We can finely tune our analysis of student activity in a course, down to the level of each individual content item. We can track time spent on task, measure difficulty with a concept or exam question and relate an individual student’s progress across an entire class. We can analyze the class across a wide variety of demographic or psychometric variables. Even video viewing has been revolutionized through educational technology innovations. In earlier days, we simply acknowledged that a student was watching a video. Now, we can embed questions into the video viewing itself, track student engagement with the material and monitor both embedded quizzes and discussions related to the video. Even more sophisticated technology can track eye movement on web pages.
There has been equally enormous potential in efforts to apply artificial intelligence to learning. Combined with the robust data gathering and analysis mentioned above, we can do much to create a successful learning journey for all learners. For example, Penn State World Campus built a virtual assistant based on Google Cloud that uses AI to automate responses to routine student queries. Other schools have developed and use virtual agents, leveraging artificial intelligence and fine data points that are collected at each point in the student learning journey. This has opened up the possibility of more targeted and earlier interventions for student learning through the use of alerts and nudges, both human and machine. Recent studies have shown that the majority of students who received early alerts and nudges found them to be very or extremely useful.
Even more encouraging are some of the efforts taking place nationally at scale. By way of example, the Unizin Consortium, a nonprofit organization of fourteen higher education institutions, including Rutgers University, Indiana University, the University of Wisconsin, and the University of Michigan, are collaborating with data products and services, like the Unizin Data Platform (UDP), a data warehouse built on Google Cloud. These services will include data marts, real-time event processing, and APIs in service of research, learning analytics, application development, and business intelligence. The UDP allows instructors to pool data in order to improve feedback to students and improve construction of learning objects and activities. These data analytics, combined with new technology-based interventions, implemented by a number of the leading research universities in the country, holds great promise for a positive paradigm shift for learners.
These developments open up other possibilities to benefit learners across the lifespan, like using AI for content creation and working to achieve the dream of individualized instruction for all learners. In this dream, the uniqueness of each learner is not seen as a barrier, but is celebrated, supported and extended. This approach recognizes that each learner has unique cognitive development, social-emotional capacities, and personal backgrounds that they bring to each learning experience. To use technology to create a personalized learning pathway for each learner, built on a vision for success for each learner is truly exciting.
There are efforts to encourage, incentivize and move technology companies, inventors and entrepreneurs in this direction of universal success for all through individualized pathways. One example in this arena is the Learner Variability Product Certification, created and tested to identify what product design features, tools, and support would help learners and teachers clearly understand whether a specific educational technology product can meet their needs.
While learner variability is not fully defined nor is a definition universally accepted, most would agree that it includes strategies like giving learners the opportunity to choose certain aspects of their learning experience.
Educational technology products designed with learner variability in mind are more likely to support the broad range of a learner’s strengths and challenges that can vary in different contexts and that create multiple opportunities for differentiation.
So, back to my opening question. How do we know that a teaching and learning experience has been effective, especially in these pandemic days when nearly everything is in an online format? We can begin by taking advantage of all of the new educational technology developments that give us insight and information to analyze, tools to create personalized learning and a feedback loop towards continuous improvement. And we can do our part in building for a future where every learner is recognized, valued, supported and set on a path for success.