Developing the Technology-Integrated Assessment Framework

The purpose of this paper is to describe the development of a new framework for understanding technology-integrated assessment in higher education based on a review of the literature using the assessment design in a digital world framework (Bearman et al., 2022) as a lens. Our review (Madland et al., 2024) revealed both congruities and incongruities between the literature and the framework, leading to the need for further work to accurately conceptualize technology-integrated assessment. In this article, we contribute to the literature on technology-integrated assessment in higher education by proposing the technology-integrated assessment framework. This paper marks an important step in extending our understanding of the factors influencing instructors who integrate technology into their assessment practice and promoting ethical and equitable approaches to technology-integrated assessment in higher education.


Introduction
In 2022, a team of educational researchers in Australia published the assessment design in a digital world framework to "support the development of assessment designs relevant to a digital world" (Bearman et al., 2022, p. 2).They note the complex nature of the relationship between technology and assessment and that assessment practices have not evolved along with the development of digital ways of interacting with the world.Their framework outlines three purposes for integrating assessment and technology.First, instructors integrate technology to improve assessment practices in some way.Three areas they propose fall under this category are: assessment rationales, the level of digital enhancement, and the potential harms that may result from using technology tools.The second purpose for integrating technology and assessment is to enable instructors to build digital literacies in their learners.They conceptualize digital literacies as comprising mastery and proficiency in using digital technologies and the ability to evaluate and critique the use of digital tools.The third purpose for integrating technology and assessment is to promote uniquely human capabilities, which they describe as consisting of the ability to imagine what activities they might be required to do in the future (e.g., related to career or personal pursuits) and also what kind of person they want to be in the future.
In Madland et al. (2024), we reported findings from a review of the literature on technologyintegrated assessment published between 2016 and 2023 by instructors from a wide variety of disciplines and geographic locations.We identified seven themes in the literature, and then drew upon Bearman et al.'s (2022) framework as a lens through which to examine these themes further.During this process, we identified important gaps between the framework and the seven themes that emerged from our review.The following sections describe these gaps and our process of extending the Bearman et al. (2022) framework to align more closely with the literature and with our collective expertise in technology integration, classroom assessment and educational measurement.

Themes in the Literature
We identified seven major themes in the literature, as described briefly below.Please see Madland et al. (2024) for a full description of each theme.

•
A focus on tools and tasks.Similar to Bearman et al. (2022), we found that an instrumental approach to technology integration predominates the field.There was a wide variety of tools mentioned, including generic computer-supported collaborative learning (Biasutti, 2017) or computer-assisted assessment (Combrinck & Vollenhoven, 2020), and specific, locally developed tools (Nutbrown et al., 2016).This instrumental focus often leads to small-scale investigations of the effectiveness of single tools in enabling specific tasks like creating videos (Sargent & Lynch, 2021), or e-portfolios (Deneen et al., 2018) instead of what Dron (2022) calls the "orchestrated assembly" (p.262) of software, hardware, learner and instructor characteristics, and other contexts.

•
Instructor workload/efficiency.One theme that does not appear in the Bearman et al. (2022) framework is the importance of using technology to reduce instructor workload by increasing the efficiency of assessment tasks.Bennett et al. (2017) frame this as an issue of economy of effort, particularly in the context of large classes.

•
The purposes of assessment.This theme is in partial alignment with the Bearman et al. (2022) framework in that there is widespread recognition of the different purposes of assessment, but the framing in the literature often follows the formative/summative binary as opposed to the more nuanced framing of assessment of/for/as learning (Boud & Soler, 2016;Earl, 2013).
• Academic integrity and remote proctoring.This theme followed the trajectory of the COVID-19 pandemic and the associated restrictions on in-person gatherings.When instructors suddenly became unable to monitor exams in person, their concern about learners accessing the answers to test items increased (Gamage et al., 2022).This resulted in a significant increase in the use of tools designed to allow instructors, thirdparty vendors, or even automated systems to monitor exam-takers.Several researchers quickly published analyses of various tools, which included recommendations for their use (Hilliger et al., 2022) as well as concerns about accessibility and equity (Hussein et al., 2020).
• Assessment design.The importance of intentional assessment design was an area of significant congruence between our review of the literature and Bearman et al.'s (2022) framework.Bennett et al. (2017) found that efficiency was a primary driver of instructors use of technology for assessment and that there was a need for institutions to provide greater pedagogical support for technology-integrated assessment.
• Ethics and equity.Although our review surfaced expressed concerns about academic integrity (i.e., learners behaving ethically), there was much less concern evident on the ethical implementation of technology for assessment (i.e., instructors and institutions behaving ethically).A recurring feature of this theme is the presumption that the use of technology for assessment will necessarily lead to increased equity (Gallavan et al., 2017) despite evidence to the contrary (Aluko & Omidire, 2020;Duncan & Joyner, 2022).
• Systemic transformations of practice (e.g., COVID-19, generative artificial intelligence).The COVID-19 pandemic was a prominent feature of many articles published since 2020, but we noticed that reports of its impact were not isolated to any one of the themes listed above.Instead, the pandemic had systemic impacts that forced the transformation of assessment practice, including summative assessments and academic integrity (Hussein et al., 2020), formative assessment (Moorhouse & Kohnke, 2022), equity of access (Pires Pereira et al., 2021), and instructor workload (Celik et al., 2022).We also note that as restrictions due to the pandemic were eased, there was a dramatic increase in the availability of generative artificial intelligence (AI) models such as ChatGPT, DALL-E (OpenAI, 2023) and a growing number of other, more capable systems.Concerns about generative AI did not appear in our literature searches, but they represent another example of systemic transformations of practice.
In addition to the themes described above, we note the additional contribution by the same research team which published the assessment design in a digital world framework (Bearman, Open/Technology in Education, Society, and Scholarship Association Journal: 2024, Vol.4(1) 1-19 Nieminen, & Ajjawi, 2022).Nieminen & Carless (2022) performed a scoping review of the literature on authentic assessment, using their own framework (as published in Bearman et al. 2022) as the organizing framework.Nieminen et al. describes an additional purpose for digital assessment, which they call fostering communality.The authors note that this purpose was derived from only one study (Thompson, 2009) of an assessment designed to encourage community engagement for social good in a statistics course.Nieminen et al. note that communality is an important connection between higher education and broader issues of social justice.Additionally, there is a long history documenting the importance of community in education (Dewey, 1916;Lave & Wenger, 2003;Vygotsky, 1978), in open, online, and distance education (Cleveland-Innes & Emes, 2005;Garrison et al., 2000), and in education for social good (hooks, 1994;Horton & Freire, 1990;Lambert, 2018).
While communality was not a theme that surfaced in our review (Madland et al., 2024), in Canadian higher education, the importance of fostering communality, or community more broadly, points directly to the recommendations of the Truth and Reconciliation Commission of Canada (TRC) (The Truth and Reconciliation Commission of Canada, 2015).The mandate from the TRC includes specific recommendations for higher education institutions in Canada to improve levels of higher education achievement among Indigenous communities and also to create culturally relevant curricula.A common framework for integrating Indigenous perspectives with online learning in higher education is the five Rs of Indigenous education (Tessaro et al., 2018).This framework outlines five characteristics of Indigenous education that serve as a model to inform our vision for equitable and inclusive learning environments.The five Rs are respect, relevance, reciprocity, responsibility, and relationship.This framework was originally conceived as four Rs (Kirkness & Barnhardt, 1991), then subsequently expanded to five Rs by Harris & Wasilewski (2004), Restoule (2008), and Styres & Zinga (2013).The five Rs have become a counterpoint to the typical approaches to higher education in Canada, which has long failed Indigenous learners because, according to Tessaro et al. (2018), Indigenous learners would be required to leave behind their communities, and, therefore a large component of their identities, to participate in higher education.Indigenous learners would have to assimilate into the larger culture of higher education rather than higher education institutions adapting to the unique needs of Indigenous learners.
According to Tessaro et al. (2018), respect refers to the need for instructors to show respect for Indigenous cultural norms and values; reciprocity values mutually beneficial relationships; relevance means that the educational activities must align with the needs of the learner and their context, responsibility refers to the fact that both the instructor and learner have responsibilities to Indigenous cultures and their families and communities; and relationships are the context through which the other four Rs can be realized.

Extending Bearman et al.'s Assessment Design in a Digital World Framework
In our review we presented a comparison in Table 8 (Madland, et al., 2024), reproduced here as Alignment Between this Review and the Assessment Design in a Digital World Framework.

Developing the Technology-Integrated Assessment Framework
The incongruities between the Bearman et al. framework ( 2022) and the literature we reviewed (Madland, et al., 2024) offer robust opportunities for further consideration of the framework.As previously mentioned, the articles we reviewed reflect on-the-ground practices of higher education instructors with varying levels of grounding in the assessment literature.The following technology-integrated assessment framework represents an integration of the findings described above, as well as our own professional expertise and scholarly engagement with the literature in technology integration, classroom assessment, and psychometrics.We introduce the technology-integrated assessment framework as a new model that extends the assessment design in a digital world framework.The technology-integrated assessment framework comprises four constructs (assessment purposes, duty of care, technology acceptance, and assessment design), representing key elements to consider when designing assessments in a digital format.Each of the main constructs has 3-5 subconstructs.This is a shift from Bearman et al., whose framework is divided into three "purposes" (digital tools, digital literacies, and human capabilities), each with 2-3 subconstructs.In developing the technology-integrated assessment framework, we retained components from Bearman et al. (2022) where there was overlap with the themes uncovered in our review (assessment rationales and assessment design).We dropped some components from Bearman that were not present in the literature (SAMR and digital literacies).While digital literacy is important, we consider it an outlier, given it is typically a self-report of the knowledge and skills within a person generally.Conversely, the key elements and constructs of the technologyintegrated assessment framework are all applied to assessment or perceptions and practices around assessment and not the person's knowledge of each construct.For example, general knowledge of inclusion is not the focus of the inclusion construct, but inclusive assessment practices would be.The targets of inference are the assessment perceptions and practices and not the instructor's attributes.We revised other components that were not prominent in the literature.For example, human capabilities and potential harms from Bearman, et al.'s (2022) framework are both merged into the duty of care construct.We also added constructs from the literature which were not in the Bearman et al. framework.Technology acceptance and its subconstructs were added as an established model that includes consideration of workload and efficiency, prominent themes in the literature.Academic integrity was added as a subconstruct of assessment design.Although assessment design alone cannot solve the challenge of academic dishonesty, particularly due to contract cheating, instructors can reduce incidences of academic dishonesty through careful assessment design.Nieminen et al.'s (2022) suggestion to include fostering communality, is included in the new framework as relationships.Throughout the process of envisioning the technology-integrated assessment framework, we considered the five Rs of Indigenous education (Tessaro et al., 2018) as explained in more detail below.

Assessment Purpose
This construct mirrors that of the Bearman et al. model, which included assessment purposes as a subcomponent of a digital tools category.We elevate the purposes of assessment to a toplevel construct as we believe that technology-integrated assessment is about assessment before it is about digital tools.Prioritizing an instrumental approach to technology-integrated assessment leads to challenges associated with generalizability in the field due to over-reliance on small-scale studies examining tool use in limited contexts.An earlier review of papers published between 2004 and 2009 (Stödberg, 2012) also noted the tendency of technologyintegrated assessment practice to be instrumental in nature.In de-emphasizing instrumental approaches, we re-centre the importance of assessment and its various purposes.Assessment purposes (assessment of/for/as learning) are not often mentioned by name in this review, but there is significant awareness of formative and summative assessment rationales.This suggests that, while assessment practices are slow to change (the summative/formative dichotomy was first popularized by Bloom in 1968), they do change but the framing of assessment of/for/as learning is more recent, introduced in 2013 (Earl, 2013), and therefore, not yet as prominent.

Duty of Care
The Bearman et al. (2022) framework highlights two components related to the idea that assessment is a human-centred process, including potential harms, and human capabilities.Similar to Bearman et al., we found that these two components were not prominent in the literature on technology-integrated assessment.We argue that instructors in higher education have a responsibility to avoid behaviours or omissions that could reasonably be foreseen to cause harm to their learners (Abrahart, 2023), commonly known as a duty of care.While a legal duty of care is currently not recognized in relation to higher education in the UK (Abrahart, 2023) or in Canada (Alexander v University of Lethbridge, 2022), it is recognized at the K12 level (BC Teachers' Council, 2019) in British Columbia, and mental health concerns among learners, including high rates of depression, anxiety, and hopelessness, point to the need for instructors to consider that a majority of their learners may be experiencing distress (Nadarajah, 2021).Additionally, following Nieminen et al. (2022), we argue that assessment practices ought to foster community and that assessment practices can promote social justice.Given Canada has a mandate to support truth and reconciliation (The Truth and Reconciliation Commission of Canada, 2015), we applied a theoretical lens during the formation of the new framework to make sure all components of the five Rs of Indigenous education, relationship, respect, relevance, responsibility, and reciprocity (Tessaro et al., 2018) were present.We believed "responsibility" and "respect" was covered in the duty of care construct and its subconstructs."Relationships" was added as a subconstruct to duty of care.The remaining Rs were added to the assessment design construct.
We posit that a duty of care exists for higher education instructors in the following ways.
Instructors have an obligation: 1. to avoid bias in grade inferences and reporting (Woo et al., 2023), 2. to ensure that assessment practices are inclusive so that all learners are able to participate fully (Fawns & Nieminen, 2023;Tai et al., 2022), 3. to implement assessment practices that honour relationships at individual, classroom, and community levels (Tessaro et al., 2018), and 4. to use tools that support ethical technology practices (Digital Learning Advisory Committee, 2022).

Technology Acceptance
We initially considered instructor workload and Bearman et al.'s (2022) digital literacy as subthemes of a technology acceptance component.We posit that Bearman et al.'s framework could be enhanced by considering the Unified Theory of Acceptance and Use of Technology (UTAUT) (Venkatesh et al., 2003).There were four mentions of technology acceptance models in the literature we reviewed and the UTAUT specifically models that effort expectancy, parallel to instructor workload, is a factor in influencing technology use.As a well-established model of the factors that influence behaviour related to technology use, UTAUT seems a better fit for our framework which seeks, in part, to explain the relationship between technology and assessment behaviours.UTAUT is most commonly known in the information technology literature, but has been applied in higher education contexts as well (Almaiah et al., 2019;Birch & Irvine, 2009;Or & Chapman, 2022).For the purposes of the new model, we will omit the moderators for behaviour (e.g., voluntariness, age, gender, etc.) and include only the direct determinants.This partial UTAUT model shows only the four factors (as determinants) that contribute to user behaviour (see Figure 2).
1. Performance expectancy is the degree to which the user has confidence the technology will improve their performance on a task, 2. Effort expectancy is the degree to which the technology is easy to use, 3. Social influence is described as the degree to which other people who are important to the user want them to use the technology.
4. Facilitating conditions refer to the level of technical support available when using the technology and contribute directly to a user's behaviour when using a technological tool.

Assessment Design
The final component of the framework relates to assessment design, which includes measurement theory (whether assessment instruments are valid and reliable), academic integrity (assessment instruments should be designed to prevent or reduce opportunities for academic dishonesty), relevance (assessment instruments should connect to the lives of learners in meaningful ways), and reciprocity (assessment instruments should allow for two-way interactions between instructor and learners).Relevance and reciprocity are the final two Rs of Indigenous education.According to Tessaro et al. (2018), within the five Rs framework, relevance requires that learning go beyond books to include oral communication and community engagement as well as being in alignment with the activities and contexts that learners will encounter outside the learning environment.Reciprocity underlines the importance of learning and assessment being two-way processes where learners have meaningful choices and input into the learning environment and assessment tasks.

Visualizing the Technology-Integrated Assessment Framework
In Figure 3, we conceptualize the technology-integrated assessment framework around the four themes identified above: assessment purposes, duty of care, technology acceptance, and assessment design.

Figure 3
Technology-Integrated Assessment Framework We provide Table 2 to highlight the primary and secondary components as well as key references to the literature that supports each component.Bias Baniasadi et al., 2023;Gallavan et al., 2017;Hilliger et al., 2022;Hussein et al., 2020;Lake & Atkins, 2021;Digital Learning Advisory Committee, 2022;@ Nieminen et al., 2022;Tessaro et al., 2018;Timmis et al., 2016;Woo et al., 4 offers a proposed structure of the technology-integrated assessment framework in a theoretical conceptualization of how the four components might be related to each other.The technology-integrated assessment framework pushes against the prevailing instrumentalist view of technology-integrated assessment and refocuses technology-integrated assessment on the centrality of the purposes of assessment and the importance of humanizing assessment through the ethical integration of technology and assessment processes.

Figure 4 Theoretical Schematic of the Technology-Integrated Assessment Framework
We believe the purposes of assessment ought to be a primary consideration for instructors as they design assessment practices.Data from our literature review suggest that knowledge of the formative and summative purposes of assessment is well established, however, more contemporary conceptions of the purposes of assessment of/for/as learning may enable more nuanced approaches (Madland et al., 2024).Instructors also have a duty of care with respect to assessment.Not only should technology-integrated assessment tasks not harm learners through either systematic bias or inequitable access, they should be designed to maximize the opportunity for relational dialogue using ethical technologies.The next element, technologybased behavioural intentions, based on the UTAUT, involves considering how performance expectancy, effort expectancy, social influences, and facilitating conditions impact the behaviour of instructors in technology-integrated learning environments (Venkatesh et al., 2003).The fourth element of the framework has to do with the logistics of designing the assessment task in alignment with sound measurement theory (validity and reliability) (Massey et al., 2020), to reduce the likelihood of academic dishonesty (Dawson, 2020), and to reduce the need for intrusive surveillance systems (Gilliard & Selwyn, 2023).Further, assessment tasks should be relevant to learners' lives (past, present, and future), and should promote and be informed by reciprocal interactions between instructors and learners.This conceptualization moves away from the instrumentality evident in much of the discourse around technology-integrated assessment.By centering the purposes of assessment and human-centred approaches, technology-integrated assessment becomes less focused on the latest new tools, and more focused on how assessment ought to support learning and learners first and then the practical challenges of implementation.It also moves away from the prominent positivity bias in the literature, where it seems axiomatic that technology integration will necessarily realize net-positive outcomes.

Recommendations for Future Research and Practice
This article represents our initial development of the technology-integrated assessment framework, which we believe fills a need for advancing theoretical framing of assessment in higher education to be in line with contemporary practice and pedagogy that focuses on learners as more than just data points.The framework serves as the basis for further investigation into both the quantitative and qualitative characteristics of the framework as it relates to various stakeholder groups in higher education, such as learners, instructors, administrators, and system-level policy makers.We also hope it is used for self-reflection both at the institutional and instructor/course level and as prompts for the design or redesign of assessment strategies where necessary.The core elements should also influence curricula for professional learning for those involved in teaching and learning in higher education.
Results from our analysis and review of contemporary literature demonstrate there is need to ensure that technology-integrated assessment practice prioritizes authentic, human-centred approaches as opposed to technological solutionism.As the impacts of AI continue to unfold and whatever systemic transformations still on the horizon come along, this framework can serve as a guide to ensure our policies and practices, as they pertain to assessment of learning, have considered the core elements of assessment purpose, care for learners, understanding of technology acceptance, and sound assessment design

Table 1 (
Bearman et al. (2022) various areas of congruity and incongruity between the seven themes we identified in the literature and theBearman et al. (2022)framework.literacies, mastery and proficiency, and potential harms; highlighted yellow) and largely missing (human capabilities, future activities, future self, level of digital enhancement and evaluation and critique; highlighted purple).Below the image are three themes (dashed outlines) that were mentioned in the literature, but are not mentioned in the Bearman et al. framework.These themes are efficiency and workload, academic integrity, and community.The following discussion relates to these incongruities, which suggest that further work is required to conceptualize technology-integrated assessment more comprehensively.

Table 2
Constructs of the Technology-Integrated Assessment Framework