With special thanks to:
Neil Bangs, Associate Professor in Education, Academic Lead for Teaching & Learning (Academic Advising) at Middlesex University and current Chair of UKAT.
Ed Foster, Head of Student Engagement and Analytics, Nottingham Trent University
James Gray, CEO, Kortext
Alison Mullan, Interim Director of Students, Education and Academic Services, Lancaster University
Introduction
As 2023 drew to a close, I had the pleasure of spending some time with a range of colleagues from our client institutions reflecting on the year that was 2023 and looking ahead to what we might expect from 2024, particularly in the context of student engagement analytics and the use of student data more broadly. The conversations have revealed a broad range of perspectives whilst also surfacing a clear level of consistency in terms of the trends and themes that they expect to be on their agendas for the coming year.
Looking back
The rise of generative AI and universities use of data
It’s always useful to take time to look back and consider what key issues have dominated the year, using this as a guide for what to expect in the next 12 months. A year ago, generative AI particularly ChatGPT was barely on the radar. A year down the line, generative AI has moved into the mainstream – and not just in academia. Universities are grappling with the implications of this development in terms of how it will impact workflows, job roles and education itself.
As CEO of Kortext, James Gray is not only keen to understand the implications of AI on the access and use of content, but also to consider how institutions use data more broadly. It’s an area he finds fascinating, not least because of the,
Universal acceptance of the fact that universities need to collect and use data to help them run their institutions more efficiently and effectively. … The challenge … is to find ways to partner to accelerate activity and reduce risk when some of what they want to do with data is core, strategic and sensitive to their organisations and that they would absolutely want to undertake themselves. Therefore, how can a partner provide capability, expertise and speed, while enabling broader data use across the university?
The impact of TEF and how universities use data
While generative AI has taken the whole world by storm, more specifically within the world of UK Higher Education it’s been the year of TEF. For Neil Bangs at Middlesex, the TEF has brought with it a realisation that the way in which universities use data and analytics is ‘developing’ and is an area which he is ‘confident’ will develop over time. Neil continues:
In a sector that actively engages in research using both qualitative and quantitative data, there is room for improvement in systematically translating this valuable data into practical applications that genuinely benefit our students.
Ed Foster at NTU echoes Neil’s thoughts, describing the challenge facing universities seeking to deliver equality of opportunity for all student groups as a ‘genuinely wicked problem’. Ed recognises that while the sector wants data to inform their decision making, it’s not yet sophisticated at making decisions based on that data. While data can be good at helping to identify problems, any response to that problem needs to recognise that there is often a multiplicity of root causes at play and that it would be wrong to assume that the root cause itself is student-based. The need for a deep level of understanding around the subtleties associated with these factors must therefore inform any work to meaningfully and sustainably remediate the fruit of these problems at scale. Moreover, responding to new insights potentially requires significant investment in staff development for all frontline staff supporting and teaching students.
The ongoing legacy of the COVID-19 pandemic
2023 continued to be another peculiar year thanks to the ongoing legacy of COVID. Universities have continued to try to reframe and rebalance in-person provision whilst retaining the benefits that emerged whilst everyone was learning online. Getting the balance right to the satisfaction of students, staff and the regulator is, according to Alison Mullan at Lancaster University, ‘a really tough ask, one that is different for each institution’. Moreover, this rebalancing work has been exacerbated by the cost-of-living crisis, sparking internal debates about the value of in-person attendance at a fixed point in time. As Ed identifies, student finance was a challenge pre-pandemic, its effects were then masked by the pandemic lock-downs but is now ever more apparent as students appear to be changing their learning behaviours in response to having less money – a response he describes as ‘completely rational but potentially not very wise’.
Similar discussions around how to increase the pedagogic value of physical learning opportunities have been taking place at NTU. There are warnings from the lessons from the early MOOCs (Massive Open Online Courses) in 2012 where those participants who completed the course tended to be the more sophisticated learners with time on their hands. Just assuming that first year students will behave any differently to 2012 might be a mistake, even if we all want it to be true.
Looking ahead
There’s an election coming…
With a General Election looming, some aspects of HE policy in 2024 are understandably unclear. There will be a lull in terms of priorities before the election date is announced, producing a planning gap that will only end once the election result is known and the new government up and running. Even then, Alison believes that a clear steer for HE might be a while in arriving:
I’m not optimistic that higher education will be a focus in election campaigns or that it will be an emerging priority thereafter. What is top of the government’s priority list will not be the same things that we in the sector are thinking about. There is a lot of noise and distraction, but I think it’s unlikely we will see any immediate benefits unless there is a rethink of immigration policy given the negative impact of the current policy on students seeking to study in the UK.
Ongoing use and expansion of AI in the field of analytics
The generative AI revolution is not over yet. Alison believes that universities will continue to grapple with the tension between harnessing the benefits of the digital world for students and the challenges and fears that it brings in terms of how to manage it. She hopes that it will enable increased personalisation in the spheres of neurodiversity and inclusion through the provision of personalised opportunities for engagement and the balancing of different factors at the individual level which, if done in the right way, could bring significant value. Neil agrees:
We need to understand the data and how to use it to support widening participation students, to remove barriers to learning and enable student success.
Management-level insights from engagement data will help to provide a big picture overview which, when broken down by group, background and other factors will, as we will pick up later, help with addressing the perennial ‘what works?’ question.
Recognising the ongoing disruption that AI will provide is at the forefront of James’ mind too. As a third-party provider of educational technologies to universities, he is keen to partner with institutions to explore how the Kortext Common Data Platform (CDP) proposition will integrate within existing university teaching, learning and data infrastructures to support the institution’s strategic objectives and augment day-to-day operations. Over 2024, he believes that we will see more university strategies focusing on how to partner better to collect, augment and use data to inform as well as help improve specific outcomes – both for the university itself as well as for individual students. Working in partnership, as opposed to internal ‘self-build’ projects, has the potential to accelerate this strategic imperative and reduce risk while providing flexibility and optionality but still remain focused on specific outcomes – especially in the context of, say TEF, and being able to drive improved student experience and evidence this.
James’ thinking aligns with that of Neil Bangs at Middlesex. As both a subject academic and as institutional lead for Academic Advising, Neil believes that one thing we will see over the coming year is a focus on ways to improve the collection, display and use of data for staff ‘so that academics with busy schedules can use it systematically as part of their everyday practice and can see the worth of it’. The challenge, he says, is to ‘combine academic intelligence with learning analytics and linking your support systems and professional services to triangulate the insights and make really useful assumptions about where students are at and how to support them to be the best they can be.’
One of the ongoing challenges in Neil’s view is how to win the battle around hearts and minds – how to bring all academics into this world of analytics and AI in a way where they can realise the potential it can bring and to embrace, albeit cautiously, the impact on their pedagogic practice. He wonders whether there is a way to use analytics together with academic intelligence to reduce some of the daily workload for academic colleagues on those aspects of the job that need to be done, but which aren’t valued as highly as teaching or research activity.
Data consultancy will grow
Like James, Neil is also of the opinion that Universities will become more skilled at preparing for the next round of TEF submissions. The next iteration of the TEF, due in 2027, provides a four-year opportunity in which to develop the kinds of partnerships that will improve student outcomes, enhance the student experience and demonstrate a return on investment.
As an academic in the London Institute of Sport at Middlesex, it isn’t perhaps surprising that he links preparing for the TEF to the way in which athletes prepare to win gold medals. In his view, universities will develop strategies for understanding their data, telling their contextual story and then letting the data sing. The result? Increased value and emphasis on what the data is ‘saying’ and using independent analytics consultants to support the unpicking of stories behind the evidence.
The links back to Ed’s thoughts around understanding the reasons for student (dis)-engagement are clear. Poor data in one area doesn’t always have a direct causal root. Rather there is a network of roots that point again to the need for an increasingly joined-up approach to how a student experiences life whilst a student. This is more than teaching, learning and assessment as it comprises every aspect of a student’s life. In this regard, Alison wonders whether the data will be able to identify pinch-points in service demand and delivery, surface where the risks to successful student outcomes lie and enable support departments to anticipate and pre-empt support needs within differing contexts.
Improving health and wellbeing
Within the sector, more and more of the issues around student mental health and wellbeing will come to the fore, with the work being championed by Professor Edward Peck, Vice Chancellor at NTU and government student support champion, leading to increased demands for universities to demonstrate what they are doing in this area and how they are doing it. Data and engagement analytics will prove hugely important here, providing insights and understanding around student learning and engagement behaviours when the ongoing cost-of-living crisis will continue to impact student’s time on campus. Ideally those insights will inform teaching, learning and assessment design as well as inform the provision of any wraparound support services.
Alison expects the discussion around the role and relevance of wellbeing analytics to continue through an exploration of the extent to which it will be possible to meaningfully identify data reference points to understand wellbeing and to measure it. Certainly more exploration is required here, particularly when it comes to understanding the types of student and their lived experiences. With responsibility for every non-teaching aspect of the student journey, Alison is keen to ‘find a university that has cracked institution-wide case management so that you have a really good solution with appropriate levels of access so that staff can see and gain a really rich picture of how students are experiencing university.’
The appetite to redesign and refresh services to reflect what works is there, but having the hard evidence of what works to inform any redesign activity based on impact is missing. She cites the request from the government for all universities to sign up to the University Mental Health Charter from Student Minds. It’s not that universities don’t want to sign up but rather that they need to evidence a holistic, university-wide approach, something that is tricky when there are pockets of excellence and activity within wellbeing services, or personal tutoring or academic departments. She concludes that ‘demonstrating a joined-up and holistic approach in the absence of really rich data and a great CRM is a question I’ve not yet been able to answer.’
Not surprisingly, given that his VC is Professor Peck, Ed Foster also believes that the question of wellbeing analytics will continue to evolve in 2024. The way that data is used is likely to become more proactive, to encourage changes in behaviour and more autonomy from students themselves. Linked to this is, he believes, the question of ‘nudge’ behaviour, seeking to prompt students to access guidance and support independently.
Understanding ‘what works’
Although this perennial problem has occupied the minds of experienced policy makers and thought leaders within higher education for over a decade, recent comments from members of the Office for Students in England, and the introduction of initiatives like the Student Mental Health Evidence Hub available on the TASO website, make it clear that there are no easy answers to this question. The silver bullet remains, well, elusive. No surprises there then!
Ed wonders whether the hybrid learning legacy of Covid, with its potential for more personalised learning based on those digital traces arising from engagement with academically purposeful activity, will bring answers to some of these questions. Will data, for example, change the provision of learning and teaching, of pastoral support or the integration of automated systems into student support processes that nevertheless leave room for the personal? The TASO trial that NTU were involved in during 2023 identifies some challenges with automation that are not easily addressed. In their experiment, 2% of students responded to an email, whereas 40% responded to a personal phone call.
Research activity will play its part in answering the ‘what works’ question, but caution is again required to explore the impact of those subtleties and nuances that underpin the headline data. In principle, sharing and publicising engagement data should help to pinpoint the impact of outreach and support activity, but there are risks. Comparisons between universities may not be as helpful as they appear. For example, the data sources won’t necessarily be directly comparable between institutions. What does a single login to the VLE mean on different platforms? Does each platform log the interaction in the same way? If we consider that each data source may have similar discrepancies, we should be cautious simply comparing between institutions. Secondly, our data analysis may simply not be sophisticated enough to understand the impact of interventions. What works in one situation for one student cohort may not be replicable elsewhere, or may be masked by other factors (problems elsewhere in the course, socio-economic factors etc).
While there is a general move to share research data (or at least make it available for others to interrogate) there are huge sensitivities in doing so and a real risk of missing the subtleties mentioned above. Making comparisons that ignore those subtleties isn’t an insignificant risk given that it may result in comparisons around teaching quality or the student experience, with one university always coming off unfavourably to the other based solely on headline data where the complexities and subtleties are masked.
Conclusions
Taking time to reflect on the lessons of 2023 and think about what may lie ahead for the UK higher education sector in 2024 has been both interesting and useful. Evaluating the impact of outreach and support activity will likely be high on agendas over this year as institutions look to make better use of their data as part of an ongoing, continual improvement approach to support successful student outcomes. Reflecting on progress against key performance metrics or other targets, alongside having solid evidence to support future initiatives or bring an element of agility to existing ones are certainly ways in which we are working with our clients to make better use of their student engagement data.
The opportunity to discuss these questions with colleagues has provided a good intellectual challenge with which to start the new year. We plan to repeat this activity next year – so if I didn’t get chance to chat with you this time, don’t worry – there’s always next December. Want to pop something in the diary? (Only joking – I’m not that organised!)
At the start of the new year, we continue to commit to working in partnership with our clients. And whether these predictions are realised or we find ourselves facing something unforeseen, it’s sure to be exciting and we look forward to travelling it with you!
To discuss these topics further or your interest in student engagement analytics book a call.
