Design-Based Research Literature

I'm writing/revising my dissertation chapters this summer in advance of what I hope will be an October or November defense. I'm also trying to knock out a couple of journal articles simultaneously by appropriating and modifying portions from the original dissertation text. Part of my Literature Review (Chapter 2) focuses on Design Based Research methodology, an approach which I borrow elements from in my own research design. Considering that only my dissertation committee will probably read this, I figured it might be something worthwhile to put out there for others to see.

Design-Based Research

Overview of DBR

Stating that educational research can often be separated from the actual challenges faced in teaching and learning settings, the Design-Based Research Collective (2003) proposed that educational research should go beyond experimental testing of interventions. Design-based research has several distinguishing characteristics. First, learning environments and learning theories are connected and mutually dependent. Second, the development of environments and learning theories should be an iterative and cyclical process of design, implementation, analysis, and revision. Third, design research needs to lead to shareable contextualized theories. Fourth, all aspects of the design should take place in authentic settings and must be documented as thoroughly as possible. Finally, the methods used within the research design must connect the implementations to the outcomes of interest (i.e. research questions). The Design-Based Research Collective described an intention “in design-based research for using methods that link processes of enactment to outcomes has power to generate knowledge that directly applies to educational practice” (p.8). DBR has been defined not as an approach but rather a collection of approaches (Barab & Squire, 2004; Barab & Landa, 1997) that are explored, tested, refined, and balanced in authentic settings (Wiggins & McTighe, 1998). Barab and Kirshner (2001) suggested that as complexity of learning environments increases, so do the possibilities for learning that it engenders.  Design-based research recognizes this complexity while providing a framework for negotiating it. These characteristics serve as the foundation for the research design of this dissertation. 

DBR originated from the ideas of Alan Collins and Ann Brown (Cook, Means, Haertel, & Michalchik, 2003). In 1990, Alan Collins and his research group aimed to rethink how research on emerging technologies was carried out in educational settings. Their focus was to create a methodology for carrying out design experiments and create environments that allow new technologies to be integrated effectively. They highlighted a problem with contemporary research models, describing experimental designs as typically being carried out by designers who had great incentives for seeing the technology succeed (Collins, 1990; Dede, 2004; Collins, Joseph, & Bielacyzc, 2004). The proposed methodology called for the following elements: teachers as co-investigators, comparison of multiple innovations, objective evaluation, first using technologies that are likely to succeed in school settings, design expertise, variation within sites, flexible design revision, and multiple evaluation of either success or failure. Collins and his team (1990) proposed a unit that integrated a computer and video program available at the time, The Voyage of the Mimi, and then used several other computer applications for collecting and organizing data. To evaluate the design experiment, they suggested using interviews, observations, and the collection of artifacts including paper and pencil tests and teacher daily notes. They recognized that the study itself took place in one small classroom. The model for investigating the design was repeatable and through iterations could build a cumulative understanding of optimal design. The Collins methodology has been a blueprint for numerous design-based research studies (see Boling, 2008; Bower, 2011; Mortenson, 2011).

Brown (1992) described her transition from a laboratory scientist to a design scientist as she embarked on a quest to change the nature of schools from being environments of control and management to active learning communities. Her interests shifted from cognitive sciences on a theoretical level to instructional design on a practical level. Citing her own investigations into reciprocal teaching as an instructional approach (Brown & Palinscar, 1982) she stated that the applied value of her research was much more transparent when considering cognition, content, and context. Situated learning (Lave and Wegner, 1991) can only be examined with all of the relevant contexts considered over time (Barab, Hay, & Yamagata-Lynch, 2001). While it is important to isolate variables and conduct experimental studies that shed light on reasons that certain things succeed or fail in educational settings, Brown (1992) recognized consideration of all of the contextual, content-oriented, and cognitive elements need to be explored in order to link theory to practice. 

There are certainly limits and challenges associated with design-based research. Causality (Lesgold, 2003) is difficult to interpret when dealing with complex or messy settings over sustained periods of time (Design-Based Research Collective, 2003; Collins, 1997; Collins, Joseph, & Bielaczyc, 2004). This leads to questions of validity from the larger educational research community. Any findings (i.e. knowledge) must be explicitly defined as being contextualized theories of learning, instruction, and integration.

Design-based research has been subject to criticism in scholarly settings, many of which come from traditions of scientific and experimental inquiry in teaching and learning. Dede (2004) described DBR as investigating a lot of variables without looking at any one thing particularly well. While being optimistic about the potential of DBR, he was realistic about the simple fact that the type of results that DBR was producing was not was policy makers or practitioners were interested in at the time. Dede called for explicitness of both ontological and epistemological beliefs of those conducting DBR as well as intention by publishers of DBR to encourage readers to have a critical eye and understand the limits of any results or theories that emerge. 

A recent meta-study by Anderson and Shattuck (2012) examined the current trends in design-based research. The found that 75% of DBR-related published studies were coming from the US, particularly in K-12 settings. One interesting finding was that many of the publications focused on iterations of ongoing studies (see Hakkarainen, 2007), with only one study actually being completed. In the approximately 10 years that since DBR has emerged on the educational research landscape, it is still not transformative (McKenney & Reeves, 2013).  However, they conclude that recent DBR studies have at least provided evidence that it is a valid approach for examining complex learning environments (Dede, Ketelhut, Whitehouse, Breit, & McClosky, 2009). The next sections looks at a few recent DBR publications that have come out in the last 10 years from places included the United States, Australia, and Denmark.

Selected Design-Based Research Studies

Hardre, Nanny, Refai, Ling, and Slater (2010) used design-based research methodology to follow 17 K-12 teachers during a resident learning experience and subsequent classroom planning and implementation. They chose DBR because of the characteristics described in the previous section of this literature review such as the ability to account for expected and unexpected events in complex, authentic environments. Their goal was to identify factors during the intervention that contribute to teacher engagement, on-site experience, and classroom practice. This research used iterations of varying data collection methods over the course of 50 weeks. The methods included questionnaires, mentor evaluations, mentor observations, and data artifacts such as proposal documents and project implementation reports. Their approach to data analysis depended on the types of data that was being examined in relation to the research question that they were trying to address. The researchers were explicit about stating that their findings were highly contextual, though they concluded more generally that authenticity is open to interpretation both by researchers and research subjects.

Also looking at teachers in training, Boling (2008) investigated how conceptions of new technologies in literacy education changed and impacted the curriculum of a graduate level course. The researcher uniquely included her personal evolving perceptions as part of the design and data. One core assumption of this study was that the integration of emerging technologies brings about new responsibilities for teachers in understanding the relationship between technology, pedagogy, and learning. Working with 19 graduate students enrolled in a course she was teaching, she digitally recorded and collected classroom conversations, records from one on one teacher-student meetings, and classroom artifacts included discussion forum and journal entries. Using a grounded theory approach, she developed a set of common themes that arose from the data gathered from participants. One of her main findings suggested that beliefs on the role of technology in literacy education were firm, and that a graduate level course may not have been a mechanism in order to alter those beliefs. Boling highlighted a challenge for instructors in education programs to make ontological and pedagogical beliefs public.

Distance learning environments are prevalent in Australia due to the way that people are spread out across the continent. A study by Bower (2011) used an iterative design-based research approach to examine learning and teaching in synchronous web-conferencing environments. During three iterations, he looked at the collaborative competencies of 26 college-aged students studying computer programming in Australia. The competencies he identified were operational, interactional, managerial, and design.  He chose to use DBR because of its intention to study phenomena in authentic settings. In the first iteration of the study, he was only able to look at how students interacted with the basic tools in the web conferencing system. In the second iteration, he was able to document student-centered learning activities that required collaboration such as writing a computer program together. The third iteration was a refinement of the second iteration where he adjusted both the instructional design and used different aspects of the web-conferencing interface to leverage more collaborative opportunities. Throughout the research process his goal was to modify the instructional design, the use of technology, and augment the types of interactions that were documented in order to better address his research purposes. One limitation of this study is that Bower only used observations as evidence of collaborative competencies. While the results and suggested theories were interesting, the research tried to make general statements based on a highly contextual situation.  

Mortenson (2011) explored the relationship of museum design and visitor interactions, particularly looking at how design choices affected the understanding outcomes for those experiencing a particular exhibit shown first at a science learning center in Denmark and later a science institute in Belgium. Using observations, interviews with participants, and the collection of audio digital artifacts, Mortenson uncovered a range of visitor experiences so broad that she was unable to generate contextual theories. She cited the messiness of the museum setting as making it difficult to examine the intended relationship and determine what factors may have led the exhibit to be successful (and for some visitors unsuccessful). This study is an example of how the documentation of what may be perceived by some as a failed research project is in fact quite helpful in presenting the contexts for what can be challenging in a DBR study. The benefit of DBR is that a researcher can learn from this iteration and apply changes in the research design in order to eventually address the original research purpose (Zeng & Blasi, 2010).