Lars Mathiassen, eCommerce Institute, Georgia State University, Atlanta, Georgia, USA
This research has been partially sponsored by the Danish National Centre for IT Research. The author wishes to thank the software organisations, practitioners, and researchers participating in the Danish SPI project. The author also wants to thank the associated editor and the anonymous reviewers for many valuable suggestions.
Reports from a systems development research tradition in which emphasis is put on relating research activities to practice and on establishing fruitful collaboration between groups of researchers and practitioners. Describes and evaluates a specific research project in which a large group of researchers and practitioners worked together to understand, support, and improve systems development practices in four organisations over a period of three years. Uses the case to reflect on the research goals, approaches, and results involved in this tradition for researching systems development practice. Proposes collaborative practice research as a way to organise and conduct research into systems development practice based on close collaboration between researchers and practitioners. Exemplifies the use of pluralist research methodology by combining action research with experiments and conventional practice studies. Argues that collaborative practice research offers one practical way to strike a useful balance between relevance and rigour. Concludes with a discussion of the implications for the relation between research and practice within the systems development discipline and with lessons on how to design research efforts as collaborations between researchers and practitioners.
Article type: Case study, Theoretical with application in practice.
Keywords: Systems development, Research, Collaboration.
Content Indicators: Research Implications*** Practice Implications** Originality** Readability***
Information Technology & People
Volume 15 Number 4 2002 pp. 321-345
Copyright © MCB University Press ISSN 0959-3845
The Information Systems discipline has for quite some time been preoccupied with improving the ways in which we do research. This concern for research methodology has played a major role in maturing the discipline and it has resulted in a rather rich discussion of different approaches (McFarlan, 1984; Mumford et al., 1986; Boland and Hirschheim, 1987; Galliers and Land, 1987; Galliers, 1991; Cash et al., 1989; Nissen et al., 1991; Lee et al. 1997). In our efforts to become a respected research discipline we have also established an impressive portfolio of scientific journals and international conferences that serve as the primary media for publishing research findings. Our discussions of research methodology have therefore concentrated on how to support researchers in designing dedicated research activities that lead to good scientific papers.
A number of scholars within our discipline have recently made a strong plea for more relevance without abandoning rigour (Applegate, 1999). Benbasat and Zmud (1999) recommend ways to increase the relevance of our research by reconsidering topic selection, the purpose and content of the articles we write, the article's readability, and the reviewing process. Davenport and Markus (1999) suggest more radical interventions that challenge core academic values around research rigour, publication outlets and audiences, and consulting. Lyytinen (1999) supports this broader view and encourages us to rethink critically the institutional policies and incentive schemes that govern research, the organisation of research groups, the professional image of Information Systems researchers, and last but not least, the ways in which we study practice. Also Lee (1999) argues for a need to go beyond a positivistic research tradition.
This paper is about Information Systems research methodology with particular focus on the relation between relevance and rigour. The argument is based on experiences from a research tradition that for some years has studied practice in close collaborations between groups of practitioners and researchers. Such a collaborative approach raises many practical problems and conflicts, and it is not easily implemented into most institutional settings. The purpose of the present argument is therefore not to criticise well-established research traditions, nor is it to argue that practice research in general should be organised as collaborative efforts. In a more modest fashion, the paper sets out critically to rethink key issues related to researching practice within our discipline based on experiences from a particular research project in which the ambition was to emphasise relevance without abandoning rigour. The argument of the paper is hence driven by a case. But it addresses general issues related to research methodology. Hence, on one level I present experience from a particular project in which practitioners and researchers collaborated closely to produce relevant knowledge. On another level I discuss general challenges and opportunities related to collaborative research efforts.
Starting out from research methods, or from the point of view of writing scientific papers, invites us to think in terms of choosing between different research methods (see for example Galliers and Land, 1987). This viewpoint is extremely useful when one wants to understand the variety and the relative strengths and weaknesses of available research methods. But when designing and organising research projects based on collaboration with practitioners the challenge is not so much which methods to choose. Rather it is to find practical ways to combine qualitatively different research approaches to support the diverse, and partly contradictory goals involved in such an effort. In the following I present one particular way to practice pluralist methodology within Information Systems research (Mingers, 2001). I call this approach collaborative practice research and it combines action research, experiments, and conventional practice studies to strike a useful balance between relevance and rigour. Back to Tips for getting your paper right in the first place.
The argument is organized as follows. Section 2 presents and discusses the case, i.e. a particular research project in which a large group of practitioners and researchers worked together to understand, support, and improve systems development practice in four software organisations based on the so-called Software Process Improvement (SPI) paradigm (see e.g. Emam et al., 1998; Humphrey, 1989; Kuvaja et al., 1994; Paulk et al., 1993). Based on this case and on the related tradition for doing practice research, I subsequently review classical issues and state-of-the-art literature related to research goals (Section 3), research approaches (Section 4), and research results (Section 5). Each of these sections presents experiences from the case project and address general issues related to research methodology. Section 6 summarizes the discussion by examining implications for the relation between research and practice within the systems development discipline. The section also provides lessons on how to design useful collaborations between researchers and practitioners. While the issues raised are of interest to Information Systems research in general they are discussed based on experiences from the particular field of systems development research.
The SPI project was a research collaboration involving practitioners from four software organisations and researchers from universities and technology institutes (Johansen and Mathiassen, 1998; Mathiassen et al., 2001a). The collaboration ran over three years (1997-1999) with a 4 million US$ budget of which half was financed by the Danish Ministries of Commerce and Research while the other half was sponsored by the participating software organisations. The project involved more than ten researchers each spending between 25 percent and 75 percent of their time on the project and it involved three to seven practitioners from each organisation as active members of the research project.
Establishing such a research effort is not necessarily easy in any institutional context. We had good contacts to all levels of the four participating organisations, national research programs provided funding to make researchers collaborate more closely with industry, and the particular theme of the project, i.e. SPI, was known to provide attractive opportunities for software organisations. Finally, our institutional settings were positive towards collaboration with industry and we had considerable experience with action research. Under different conditions it might be difficult, or maybe even impossible, to practice similar approaches.
The underlying approach to systems development research and practice has grown out of the Scandinavian trade union research tradition (Nygaard and Bergo, 1975; Bjerknes and Bratteteig, 1995), it has later been inspired by Checkland's approach to action research (Checkland, 1981; Checkland and Scholes, 1990), and it has been adapted and developed since the early 1980s to suit the study and improvement of systems development practice. Our research focuses on how computer-based information systems are developed in practice; it emphasises the important role played by the local organisational environment; and it combines interpretive understandings of practice with normative propositions to support professional development. I have described this approach to systems development research and practice as reflective systems development (Mathiassen 1998a,b) and it has been described and evaluated by others as the professional work practices approach (Hirschheim et al., 1995a,b; Iivari and Lyytinen, 1997).
The industry related mission of the SPI project was: to systematise SPI knowledge in Danish companies, to tailor and develop further the most promising models for SPI so they apply to the Danish software industry, to develop frameworks for managing, organising, and implementing SPI activities in Danish companies, and finally to communicate and publish knowledge about SPI to Danish companies. These missions were addressed through research efforts in which the following research questions were addressed:
To meet this diverse set of industry and research related objectives the project was organised as a loosely coupled system of interacting agendas each addressed by a dedicated forum of actors. The overall organisation of the project is illustrated in Figure 1.
First, a local research group was established to study SPI in each of the four software organisations, see Figure 2. This group worked tightly together with the local management (both informally and formally through a steering committee), the local SPI group, and the ad-hoc projects that were established to implement specific improvement initiatives. Each research group included three to seven practitioners from the software organisation (normally the SPI group) and three to four external researchers. The research group met eight to ten times a year and followed the SPI initiative closely. The group supported the software organisation in adapting and using improvement approaches; it participated in some of the organisation's dedicated improvement projects; and it continuously evaluated the way in which SPI knowledge was adapted, used, and further developed within the organisation.
Second, a sequence of plenary meetings involving the four local research groups was organized to support interaction between the involved researchers and practitioners. This plenary of 25-35 persons met twice each year in two-day workshops where: experience and knowledge were exchanged across the companies, new knowledge on process improvement was presented, ideas related to specific improvement areas were discussed, and experiences were interpreted and put into perspective through general discussions of software development, software management, and organisational learning and change.
Third, a nation-wide network was formed to support SPI initiatives in other organisations. More than 50 companies and 80 individuals participated in this network that met for one-day seminars three to four times a year. The SPI project also organised or participated in a number of conferences targeting the Danish software industry.
Last, but not least, a researchers' forum was formed to stimulate publication of results and collaboration between the involved researchers. The researchers' forum met for half a day six to eight times a year: to identify emerging research themes, to plan dedicated research initiatives, to form new patterns of collaboration between researchers, to discuss research approaches, to present and evaluate preliminary results, and to discuss relevant theoretical frameworks and related research. The researchers' forum also served as a means to support close collaboration with a number of international research colleagues. Their participation was established to improve the research process by including experiences from similar research projects and knowledge from relevant reference disciplines.
From the point of view of the individual researcher this organisation constituted a collaborative space in which dedicated research initiatives and shifting patterns of collaboration took shape as the process unfolded. Typically, each researcher participated in two local research groups, they took part in all plenary meetings, and they were actively involved in the researchers' forum. When adding additional ad-hoc meetings related to preparation of seminars and workshops, planning of joint publications, and supervision of PhD thesis work each researcher would typically participate in one to three joint activities each week during the three year course of the project. The number of shared obligations, some of a rather practical nature, that follow from participating as a researcher in such a collaborative effort is therefore high. But the opportunities to create relevant research results are at the same time very good.
All types of action research serve different interests. Action research "aims to contribute both to the practical concerns of people in an immediate problematic situation and to the goals of social science by joint collaboration within a mutually acceptable ethical framework" (Rapoport, 1970). McKay and Marshall (2001) have denoted this the dual imperative of action research and they suggest explicitly organising action research efforts into a problem solving interest cycle and a research interest cycle.
In the SPI project there was an industry-related mission together with a set of research goals. There was an ambition to improve practices in each of the participating organisations and at the same time to add to the body of knowledge within the systems development profession. Collaborative practice research is in this way constantly confronted with dilemmas between practice-driven and research-driven goals and between general and specific knowledge interests. There is a risk that researchers in such situations turn into consultants (Baskerville and Wood-Harper, 1996) and it was therefore not surprising that we actively had to promote and defend our research interests in the SPI project. SPI efforts are both demanding and exciting so the researchers were often encouraged to engage themselves in the practical struggle to make things happen and succeed in the four software organisations. In response to this pressure we built a strong sub-culture around the researchers' forum to maintain critical reflection, publication, and research methodology as key issues.
Underlying the dual imperative of a collaborative practice research effort one finds a variety of research goals and activities. These goals can be expressed in terms of the types of knowledge that a research initiative intends to create to solve specific industrial problems and to add to the body of knowledge within the systems development profession. Adapting the framework offered by Vidgen and Braa (1997) we can distinguish between different types of knowledge as illustrated in Figure 3 (Mathiassen 1998a,b). The arrows inside the triangle represent distinct, and in some respects divergent, research activities through which each type of knowledge is developed. First, to develop our understanding of systems development we engage in interpretations of practice; we collect data about practice and interpret these using different concepts and frameworks; the outcome is insights into systems development practice and concepts and frameworks that can be used to reflect upon systems development practice. Second, to build new knowledge that can support practice, we design normative propositions or artefacts, e.g. guidelines, standards, methods, techniques, or tools; our concern here is to create knowledge that can be used to plan, guide, or improve practice; the outcome is some form of artefact that has been developed and tested in relation to particular systems development disciplines, e.g. requirements elicitation, prototyping, estimation, or configuration management. Third, to learn what it takes actually to improve practice we engage in different forms of social and technical intervention; in this case we focus on changing practices both to uncover the deeply rooted assumptions involved in professional practice and to learn about approaches to improvement; the outcome is in this case increased knowledge about practice, and insights into barriers to and enablers of systems development improvement efforts. Most Information Systems research restricts itself to understanding and supporting practice. A commitment to impove practice is the distinguishing feature of collaborative practice research and of action research in general (Baskerville and Wood-Harper, 1996).
The three goals are distinct and can be pursued in isolation. But that would seriously reduce the opportunities to learn about practice. The triangle symbolises that the involved activities presuppose and support each other: we reach a deeper understanding of practice as we attempt to change it; we need to understand practice to design useful propositions; and, the propositions and our interpretations of practice are ultimately tested through attempts to improve practice. This unity of the goals is a simple expression of the basic elements in organisational learning and change: to appreciate the situation, to invent new options, to change the situation. We find these elements in different forms within theories of organisational learning (e.g. Argyris and Schön, 1978; Schein, 1985), in practical approaches to organisational problem solving (e.g. Checkland, 1981; Checkland and Scholes, 1990; Davenport, 1993), and they also constitute the basic building blocks in systems development practice, i.e. analysis, construction, and implementation. Our theoretical and practical understanding of the field of study, i.e. systems development and organisational change, distinguishes all three elements and emphasises the intrinsic relations between them. It is this understanding and experience that we transfer to the approach to study systems development practices in dynamic organisational contexts.
The three distinct research goals and activities relate to different levels of research and practice. First, they apply to systems development itself. Due to the complex and dynamic nature of the discipline, systems developers need to practice reflection-in-action (Mathiassen, 1998a,b; Schön, 1983). They must interpret the situations in which they find themselves (understand); they must develop what-if scenarios, plans, and designs to reflect on opportunities for action (support); and they must enact some of these to establish and maintain satisfactory working situations and develop useful solutions (improve). Second, these goals and activities correspond closely to the main ingredients involved in dedicated improvement activities as expressed in the so-called IDEAL model (Initiate improvement effort, Diagnose current practices and form strategy for intervention, Establish specific improvement projects, Act to improve, and Learn from the initiatives) (McFeeley, 1996) for SPI. The IDEAL process represents a specific way to research systems development practices by enacting the general learning cycle expressed in the triangle: to appreciate the situation - to invent new options - to change the situation and so on. Third, the triangle expresses the types of knowledge and activities involved in collaborative research efforts targeting systems development practices. Practice, improvement of practice, and collaborative practice research are in this way similar and partly overlapping activities. They all address understanding, support, and improvement of systems development practice. But they are at the same time different. They are situated and organised differently; they intend to create different types of knowledge and results; and they address different audiences.
A closer look at a particular area of interest within the SPI project can illustrate how the different goals and activities unfold in collaborative research practice. One of the fundamental ideas in SPI is to diagnose current development practices and base decisions on new improvement initiatives on systematically collected data. In the SPI project we carried out various types of assessments to evaluate present practices against general norms (Iversen et al., 1998; Andersen et al., 2001), problem diagnosis was used to appreciate what the involved actors considered to be critical problems in present practices (Iversen et al., 2001a), defect reports were analysed to identify patterns of problematic behaviour (Vinter, 2001), and metrics programs were implemented to learn about the effect of the improvement initiatives (Iversen and Mathiassen, 2002). In relation to the particular area of diagnosing current development practices these contributions gave different priority to the three research goals. Some (Iversen et al., 1998; Iversen and Mathiassen, 2002) focused entirely on interpretations of diagnostic practices (understand) and on that basis suggested lessons to guide further SPI efforts (support). Others (Andersen et al., 2001; Iversen et al., 2001a; Vinter, 2001) started out from specific ideas on how to diagnose systems development practice (support) and these were subsequently used in diagnostic practices (improve) and evaluated on that basis (understand) to inform further SPI initiatives. This variety of research initiatives into the specific area of diagnosing systems development practice was subsequently used to develop a framework for selecting between different diagnostic strategies (Nielsen and Pries-Heje, 2001).
Turning to research approaches we find a multiplicity of general approaches to Information Systems research together with extensive discussions of their strengths and weaknesses (McFarlan, 1984; Mumford et al., 1986; Boland and Hirschheim, 1987; Galliers and Land, 1987; Galliers, 1991; Cash et al., 1989; Nissen et al., 1991; Lee et al., 1997; Myers 2002). In addition, we find a more specialised discussion of approaches to systems development research (Basili and Weiss, 1984; Nunamaker et al., 1991; Wynekoop and Russo, 1993).
The main concern in collaborative practice research is to establish well functioning relations between research and practice. This is, however, far from easy to achieve. Practitioners must, on the one hand, agree to become objects of study. Practitioners must accept having meetings tape-recorded, they must engage in critical reflections of their practices, and they must be willing to report weaknesses and failures of their efforts. Researchers must, on the other hand, commit themselves to improving practices and adopt flexible research approaches as practices change and new priorities emerge.
We seek to establish and maintain good research-practice relationships in collaborative practice research to support the creation of relevant research results. The research activities should be based on first hand information and in-depth insights about the problems, challenges, and opportunities that characterise systems development practice. At the same time, we must structure and manage the research process in ways that produce rigorous and publishable results, e.g. by collecting data systematically, by applying suitable methods of interpretation, by relating to relevant bodies of knowledge, and by ensuring the ability to reflect critically on the studied situations. Unfortunately, these two fundamental criteria do not always point in the same direction. The dilemmas related to fulfilling the two criteria can be expressed by distinguishing between three types of research approaches, see Figure 4 (Munk-Madsen, 1986; Nunamaker et al., 1991; Wynekoop and Russo, 1993; Mathiassen, 1998a,b). These approaches differ with respect to how deeply the researcher is involved in practice and to what extent the researcher maintains control over the research process. Each of the three approaches can be practised in a variety of ways and they all contribute to the building of knowledge on systems development. It is important to note that this classification of research approaches is orthogonal to other classifications like scientific and interpretive (Galliers and Land, 1987; Galliers, 1991), quantitative and qualitative (Lee et al., 1997), or positivist, interpretive, and critical (Klein and Myers, 1999). The presented classification of approaches serve the particular purpose of highlighting differences that are important when organising collaborative practice studies without paying attention to other distinguishing features (e.g. scientific or interpretive).
Action research provides direct access to practice, but it is quite difficult to control the research process. The primary focus is on the goal of improving practice (see Figure 3). The researcher is involved in practical problem situations with practitioners and the research agenda is therefore strongly dependent on how practice evolves. The research activity can focus on the systems being developed, on the development processes, or on both. The strength of this approach is the strong integration of research and practice: practitioners are involved in the research process and researchers gain first-hand experience. The most significant weakness is the limited support provided to structure the research process and findings; it is difficult to predict and control the focus of the research outcome; little support is provided on systematic data collection; and because of the emerging nature of the findings it is quite difficult to know in advance which types of data to collect. Quite a number of recent papers have discussed the use of action research within Information Systems (Baskerville and Wood-Harper, 1996; Lau, 1997; Stowell et al., 1997; Mathiassen, 1998a,b; Avison et al., 1999; Nielsen, 1999; Avison et al., 2001; Kock and Lau, 2001; McKay and Marshall, 2001). But the actual use of action research in systems development is documented in relatively few sources (e.g. Kaiser and Bostrom, 1982; Mumford, 1983; Parnas and Clements, 1986; Salaway, 1987; Knuth, 1989; Avison and Wood-Harper, 1991; Bjerknes, 1991; Chs 2, 10, 14, 18 in Mathiassen, 1998b; Mathiassen et al., 2001a; Chiasson and Dexter, 2001).
Experiments provide direct access to practices that are controlled, or partly controlled, by the researchers. The primary focus is on supporting practice by designing and evaluating different types of artefacts (see Figure 3). Experiments can either take place in realistic settings as field experiments or in laboratory environments. Quite often it can be difficult to distinguish action research from field experiments as the same research activity can be viewed from both perspectives. Viewed as an action research effort emphasis is put on creating knowledge based on problem solving and change in the client organisation. Viewed as a field experiment the emphasis is instead on designing, implementing, and evaluating artefacts (e.g. guidelines, standards, methods, techniques, or tools). The difference between these two approaches is therefore tightly related to the commitment to improving practice (action research) versus the intention to develop normative support (field experiments). A key advantage with experiments is that the research process can be designed to focus on the development of an artefact (i.e. a framework for risk management) to support a specific professional discipline (i.e. project management). The disadvantage, as compared to action research, is the weaker relation to practice. The practice in which a particular artefact is tested is controlled to serve the purpose of the field experiments. Experiments are more commonly used in systems development research than action research and a number of contributions have been reported based on this approach (e.g. Boland, 1978; Vitalari and Dickson, 1983; Vitalari, 1985; Boehm et al., 1984; Floyd, 1986; Selby et al., 1987; Guindon et al., 1987; Baskerville and Stage, 1996; Chs 3, 7 in Mathiassen, 1998b).
Practice studies cover a wide variety of approaches to study systems development without active involvement of the researchers in the practices they study. The primary focus is on understanding practice (see Figure 3). Some approaches study practice directly, e.g. field studies and case studies, whereas others are indirect based on people's opinions and beliefs, e.g. surveys or interviews. The strengths of this approach are that it focuses on practice, that it is possible to study representative instances of the practice area under study, and that the researchers have access to a vast repertoire of techniques to structure the research process and the findings. The weakness is that it separates research from practice. The researchers observe and interpret the actions and beliefs of practitioners and the practitioners do not take an active part in the research process. Most of the empirical literature on systems development is based on practice studies (e.g. Benbasat et al., 1980; Boland and Day, 1982; Markus, 1983; McKeen, 1983; White, 1984; Gould and Lewis, 1985; White and Leifer, 1986; Necco et al., 1987; Krasner et al., 1987; Elam et al., 1987; Curtis et al., 1988; Boehm and Papaccio, 1988; Aaen et al., 1992; Stolterman, 1992; Bansler and Bødker, 1993; Kozar, 1993; Madabushi et al., 1993; Waltz et al., 1993; Ciborra and Lanzara, 1994; Tan, 1994).
The SPI project was basically organised as an action research effort to interact closely with practice and to support close collaboration between practitioners and researchers (Baskerville and Wood-Harper, 1996). This basic approach was, however, complemented with experiments and with practice studies to establish a more complete and solid foundation for producing rigorous research results (Mingers, 2001). Such a combined strategy supports the variety of research goals discussed above and it compensates for the greatest weakness of action research: the limited support that it provides for structuring the research process and findings. The action research approach is, in the SPI project, implemented through the local research groups (see Figures 1 and 2). The agendas of these groups reflect the local SPI initiatives, and the groups serve as a forum for evaluating SPI practices, for experimenting with new or modified approaches, and for learning about the impact of SPI approaches on practice. Field experiments are then staged as dedicated research initiatives within this setting and focused practice studies are initiated to learn about selected SPI practices and their impact on the organisation. Some examples illustrate this combined approach.
The major challenge faced by the SPI initiative in one of the participating organisations was to motivate the systems developers, and in particular the project managers, to commit themselves to improvement efforts. Traditional SPI initiatives are based on normative models (e.g. Emam et al., 1998; Kuvaja et al., 1994; Paulk et al., 1993) but none of these models were considered useful by the SPI group or the developers. The SPI group therefore decided to use problem diagnosis techniques rather than assessments based on general models to learn what the developers considered to be key problems. This research initiative is documented in (Iversen et al., 2001a) and it is primarily based on action research as proposed by Checkland (Checkland, 1981; Checkland and Scholes, 1990).
Each improvement project that is initiated as part of an SPI initiative is facing a complex and often quite risky task, for example to develop, implement, and institutionalise processes to support subcontract management in the organisation. The SPI group in one of the organisations wanted to develop frameworks and tools that could support specific improvement projects in managing risks thereby minimising the chance of failure. A dedicated risk management tool for improvement projects was therefore developed and tested based on the experiences from that organisation in combination with insights from the SPI literature. The resulting tool is documented in (Iversen et al., 2001b) and the underlying research approach is, when focusing on the artefact being developed and tested, primarily a field experiment.
Some events during the course of the SPI project were considered so interesting that they attracted special attention. One such case was the implementation of a metrics program to help the organisation measure the effects of their improvement efforts. A focused study of this particular case was based on the "natural traces" of the SPI program such as project plans, meeting minutes, and memos. In addition to this, we tape-recorded the monthly meetings in each local research group as well as some of the working sessions and workshops to collect supplementary data to be used in dedicated and focused research initiatives. The relevant segments for the metrics program were transcribed. The case and the lessons learned are published as a case study in Iversen et al. (2001b) based on a combination of direct and indirect practice studies.
We see in these examples how a variety of research approaches are used and combined in dedicated research initiatives within the larger project. The SPI project can from this point of view be seen as a collaborative space in which specific, and dedicated research initiatives are formed to report:
These three types of research initiative are exemplified above and they were initiated based on the agendas of the local research groups (see Figure 1).
Collaborative practice studies, as I have discussed them here, are both practice- and research-driven and they serve general knowledge interest as well as knowledge interests that are specific for the participating organisations. The results of such efforts are therefore of a more diverse nature than those of conventional research projects.
There was a strong inclination in the SPI project to focus on practical results. The rationale to participate was, from the point of view of the software organisations, to engage in collaborative activities that could lead to improved systems development performance and that could stimulate learning within the organisation. It goes without saying that actual improvements in processes, infrastructure, and competencies were the key success criteria for the local SPI groups. The constant pressure to focus on practical results implies that research results tend to have secondary priority (Baskerville and Wood-Harper, 1996). The SPI project was therefore from the very start based on a shared commitment, between the involved organisations, the individual practitioners, and the researchers, to build new knowledge that could be published as a contribution to the body of knowledge on systems development. This commitment was maintained by having research issues integrated into the agendas of all groups in the project organisation (see Figure 1).
Conventional research publications played an important role in the SPI project. But one important difference in relation to more conventional research projects was that practitioners were included as an important target group for the project. A number of papers with results from the project were published at conferences or in journals mainly for practitioners (e.g. Johansen and Mathiassen, 1998; Jakobsen, 1998). Also, it was decided to publish key lessons from the project in a book called Improving Software Organisations - from Principles to Practice (Mathiassen et al., 2001a). Each chapter in the book presents lessons on SPI from the project, it is written for practitioners to read, the foundation is academically sound based on documented research results that in most cases had been published at scientific conferences or in journals, and most of the chapters are co-authored by researchers and practitioners. Papers published in IEEE Software were used as model examples for each chapter and an editor from this journal was engaged to guide the authors and facilitate the editing process.
Professional Systems Development (Andersen et al., 1986, 1990), Quality Management in Systems Development (Bang et al., 1991), and Object Oriented Analysis and Design (Mathiassen et al., 2000) are earlier examples from previous projects illustrating that publication for practitioners is given high priority in this research tradition. There is a sound rationale for pursuing such a strategy in practice research. The researchers are constantly challenged to develop and express results they believe to be useful in practice. Practitioners study the contributions, they attempt to use them, and that reveals strengths and weaknesses of the published results. The research contributions are in this way instrumental in establishing and maintaining a dialogue between research and practice that goes beyond specific research projects. Such a dialogue serves to test and further develop new knowledge, it plays a major role in developing higher education within our discipline, and it helps build an image of the researchers as actively contributing to improved professional practices. This, in turn, makes it considerably easier to establish new collaborations with practitioners.
Unfortunately, such research contributions are given quite low, and in some cases no, priority in making career decisions within the established research community. The incentive to publish for practitioners is therefore minimal, and a simple cost/benefit analysis will lead most researchers to the conclusion that it is not worthwhile becoming engaged in collaborative practice studies. In that way, we risk isolating ourselves as an appendix to the Information Systems profession instead of being a major force in strengthening its position in society. We should not, of course, stop our efforts to mature as a respected research discipline with high quality journals and research conferences. But we should take care to combine these ongoing initiatives with modified incentive schemes, intensified collaboration with practitioners, and more publications that address practical concerns. That will help us develop new knowledge that proves to be relevant and it will strengthen our position within the profession and society in general.
The SPI project gave high priority to reporting back to relevant groups of practitioners. But the project was at the same time situated in the ongoing academic efforts within Information Systems and it focused on classical research concerns. SPI was seen as an instance of technology related organisational change, and the research activities involved theoretical issues and the use of reference disciplines, e.g. organisational learning and knowledge management. These more fundamental concerns played an important role in the researchers' forum, see Figure 1. Insights from practice were constantly related to theoretical debates within relevant reference disciplines and used as inspiration for developing theoretical contributions. A few examples illustrate these activities.
The SPI literature is extensive, but rather practical with only little concern for fundamental research questions. In the SPI project we had many discussions on the identity and boundaries of the SPI approach as described in the literature. We found no explicit, shared understanding of SPI as a strategy for change in systems development organisations. Based on our practical experiences we therefore decided to survey the SPI literature to explicate important underlying assumptions and related strategies for change (Aaen et al., 2001).
There are also only few and rather weak relations between the SPI literature and relevant reference disciplines. It is therefore interesting to interpret SPI experiences using contemporary frameworks from other research areas. One example is the use of Nonaka's theory of knowledge creation (Nonaka, 1994) to understand better how tacit and explicit forms of knowledge can be combined in software organisations and how one can support interaction between individual, group, and organisational knowledge creation processes (Arent et al., 2001). Such studies serve to interpret and inform SPI practices and they provide interesting examples for the reference discipline in question.
Other theoretical contributions from the SPI project are based on subjective/argumentative approaches (Galliers and Land, 1987; Galliers, 1991). SPI is typically based on a rather narrow view of the systems development process, e.g. from when a contract is signed until a software system is delivered, and the main concern is to improve processes within the software organisation. Mathiassen et al. (2001b) reflect on the nature of well-functioning customer-supplier relations, they evaluate those SPI models that are concerned with improved customer relations, and they propose specific initiatives to improve the collaboration between customer and supplier organisations. Such initiatives were subsequently initiated in one of the participating organisations. The research was hence developed as an inspiration for innovative activities that could lead to alternative SPI strategies.
Collaborative practice research, as illustrated by the SPI project, is a way to organise and conduct practice research in close collaboration between researchers and practitioners. It has grown out of a particular tradition for studying systems development and it is an expression of ongoing debates on research methodology within the Information Systems discipline.
Collaborative practice research is based on a particular view of practice, of research, and of the relations between the two. This view is part of a research tradition that has been developed over the past 20 years. The tradition has been discussed as "the ordinary work practices approach" (Hirschheim and Klein, 1992) or "the professional work practices approach" (Hirschheim et al., 1995a,b; Iivari and Lyytinen, 1997). I have presented the underlying perspective as "Reflective Systems Development" (Mathiassen, 1998a,b) to acknowledge the relation to Schön's (1983) ideas on how professionals think in action and to stress the intrinsic relation between research and practice. The SPI project should be understood in this context and seen as an effort to further develop the tradition by addressing some of its weaknesses, e.g. a too simplistic view of what it takes to improve practice and lack of an explicit notion of what it means to change practices for the better (Hirschheim et al., 1995a,b).
Reflective Systems Development expresses a coherent view of practice and research with two different modes of inquiry, a research and a practice mode. This is expressed in Table I based on Checkland's ideas on how intellectual frameworks are used in relation to specific application areas (Checkland and Scholes, 1990, p. 283). The table explicates the different, but related purposes of research and practice, the underlying intellectual frameworks, the type of process in which they are applied, and the shared arena to which they are applied. The research activity is primarily informed by systems development practice, but supported by various reference disciplines (e.g. design theory, organisation science, management science, and philosophy). Also, because of the complex and contradictory nature of systems development practice, dialectic reflections are used to help understand the tensions and dilemmas involved in practising and improving systems development (Robey, 1995; Mathiassen, 1998a,b).
Research becomes, in this view, an activity in which practitioners (can) participate and collaborate with researchers.
The practitioner does not function as a mere user of the researcher's product. He reveals to the reflective researcher the ways of thinking that he brings to his practice, and draws on reflective research as an aid to his own reflection-in-action. Moreover, the reflective researcher cannot maintain distance from, much less superiority to, the experience of practice ... he must somehow gain an inside view of the experience of practice (Schön, 1983, p. 323).
This view of research implies and builds on a complementary view of practice in which reflection and learning are key elements. Systems developers must, in addition to mastering a repertoire of general methods and tools, know how to cope with the specific environment in which they work. Many situations involve uncertainty, instability, uniqueness, and contradictions and they require an ability to go beyond the relatively safe territory of general professional knowledge. Systems developers must open their minds and engage in reflections, dialogues, and tailored research efforts to generate the necessary insights into the situation at hand.
The SPI project was an attempt to practice and further develop Reflective Systems Development in close collaboration between researchers and practitioners. Our discussions have presented the project from a researcher's perspective and provided concepts and experiences that can be used to organise similar initiatives. The following four lessons summarise these insights and relate the findings from this particular case to other contributions to Information Systems research methodology.
Collaborative practice research offers good opportunities to strike a useful balance between relevance and rigour in Information Systems research. But the approach requires a dedicated effort involving both research work and organisational work. During a project it is necessary to spend major efforts on facilitating collaboration and on managing the multiplicity of agendas involved. It is advisable to complement the basic action research approach with well-established research disciplines to systematically collect data and to apply appropriate methods of interpretation. After the project it is advisable to return to a more traditional activity pattern, to reflect deeper on the experiences from the project, and to publish more of the insights that were gained during the project.
The SPI project was big and involved many researchers and practitioners. Collaborative practice research can, however, also be practised on a much smaller scale (see for example Pourkomeylian, 2002). The experiences from the SPI project and the related reflections and lessons are meant as an inspiration for those that have the motivation and the opportunities to engage in collaborative practice research. It is, however, important to remember that such research efforts can be difficult to create and demanding to manage. This is partly because of the diverse and contradictory nature of the interests and goals involved. But it is mainly because our institutional settings and incentive schemes do not encourage researchers to engage in close collaboration with practitioners (Lyytinen, 1999).
Table I. Reflective Systems Development as an approach to research and practice
Figure 1. Overall research organisation
Figure 2. Local research organisation
Figure 3. Research goals and activities
Figure 4. Approaches to study systems development practice
Aaen, I., Siltanen, A., Sørensen, C., Tahvanainen, V.-P., 1992, "A tale of two countries. CASE experience and expectations", Kendall, K.E.et al, The Impact of Computer Technologies on Information Systems Development, Amsterdam.
Aaen, I., Arent, J., Mathiassen, L., Nwgenyama, O., 2001, "A conceptual MAP of software process improvement", Scandinavian Journal of Information Systems, 13.
Andersen, C.V., Arent, J., Bang, S., Iversen, J., 2001, "Project aassessments", Mathiassen, L., Pries-Heje, J., Ngwenyama, O., Improving Software Organizations - From Principles to Practice, Addison-Wesley, Upper Saddle River, NJ.
Andersen, N.E., Kensing, F., Lassen, M., Lundin, J., Mathiassen, L., Munk-Madsen, A., Sørgaard, P., 1986, Professional Systems Development. Experiences, Ideas, and Action, Teknisk Forlag, Copenhagen.
Andersen, N.E., Kensing, F., Lassen, M., Lundin, J., Mathiassen, L., Munk-Madsen, A., Sørgaard, P., 1990, Professional Systems Development. Experiences, Ideas, and Action, Prentice-Hall, Englewood Cliffs, NJ.
Applegate, L.M., 1999, "Rigor and relevance in MIS research", MIS Quarterly, 23, 1.
Arent, J., Pedersen, M.H., Nørbjerg, J., 2001, "Strategies for organisational learning in SPI", Mathiassen, L., Pries-Heje, J., Ngwenyama, O., Improving Software Organizations - From Principles to Practice, Addison-Wesley, Upper Saddle River, NJ.
Argyris, C., Schön, D., 1978, Organisational Learning, Addison-Wesley, Reading, MA.
Avison, D.E., Wood-Harper, A.T., 1991, "Information systems development research. An exploration of ideas in practice", The Computer Journal, 34, 2.
Avison, D., Baskerville, R., Myers, M., 2001, "Controlling action research projects", Information Technology & People, 14, 1.
Avison, D., Lau, F.,, Myers, M., Nielsen, P.A., 1999, "Action research: making academic research relevant", Communications of the ACM, 42, 1.
Bang, S., Efsen, S., Hundborg, P., Janum, H., Mathiassen, L., Schultz, C., 1991, Quality Management in Systems Development, Teknisk Forlag, Copenhagen.
Bansler, J., Bødker, K., 1993, "A reappraisal of structured analysis. Design in an organisational context", ACM Transactions on Information Systems, 11, 2.
Basili, V.R., Weiss, D.M., 1984, "A methodology for collecting valid software engineering data", IEEE Transactions on Software Engineering, 10.
Baskerville, R.L., Stage, J., 1996, "Controlling prototype development through risk management", MIS Quarterly, 20, 4.
Baskerville, R., Wood-Harper, A.T., 1996, "A critical perspective on action research as a method for information systems research", Journal of Information Technology, 11.
Benbasat, I., Zmud, R.W., 1999, "Empirical research in information systems: the practice of relevance", MIS Quarterly, 23, 1.
Benbasat, I., Dexter, A.S., Mantha, R.W., 1980, "Impact of organisational maturity on information system skill needs", MIS Quarterly, 4, 1.
Bjerknes, G., 1991, "Dialectical reflection in information systems development", Scandinavian Journal of Information Systems, 3.
Bjerknes, G., Bratteteig, T., 1995, "User participation and democracy: a discussion of Scandinavian research on systems development", Scandinavian Journal of Information Systems, 7, 1.
Boehm, B.W., Papaccio, P.N., 1988, "Understanding and controlling software costs", IEEE Transactions on Software Engineering, 4, 10.
Boehm, B.W., Gray, T.E., Seewaldt, T., 1984, "Prototyping versus specifying. A multiproject experiment", IEEE Transactions on Software Engineering, 10, 3.
Boland, R.J., 1978, "The process and product of system design", Management Science, 24, 9.
Boland, R.J., Day, W., 1982, "The process of system design. A phenomenological approach", Ginzberg, M.et al, Proceedings of the 3rd International Conference on Information Systems, Ann Arbor, MI.
Boland, R.J., Hirschheim, R.A., Critical Issues in Information Systems Research, John Wiley, Chichester.
Cash, J.I., Benbasat, I., Kraemer, K.L., Lawrence, P.R., 1989, The Information Systems Research Challenge, 1-3, Harvard Business School, Boston, MA.
Checkland, P., 1981, Systems Thinking, Systems Practice, John Wiley, Chichester.
Checkland, P., Scholes, J., 1990, Soft Systems Methodology in Action, John Wiley, Chichester.
Chiasson, M., Dexter, A.S., 2001, "System development conflict during the use of an information systems prototyping method of action research: implications for practice and research", Information Technology & People, 14, 1.
Ciborra, C., Lanzara, G.F., 1994, "Formative contexts and information technology. Understanding the dynamics of innovation in organisations", Accounting, Management & Information Technology, 4, 2.
Curtis, B., Krasner, H., Iscoe, N., 1988, "A field study of the software design process for large systems", Communications of the ACM, 31, 11.
Davenport, T.H., 1993, Process Innovation - Reengineering Work through Information Technology, Harvard Business School Press.
Davenport, T.H., Markus, M.L., 1999, "Rigor vs relevance revisited: response to Benbasat and Zmud", MIS Quarterly, 23, 1.
Elam, J.J., Waltz, D.B., Krasner, H., Curtis, B., 1987, "A methodology for studying software design teams. An investigation of conflict behaviors in the requirements definition phase", Empirical Studies of Programmers. Second Workshop, Ablex Publishing, Norwood, NJ.
Emam, K.E., Drouin, J.-N., Melo, W., 1998, SPICE: The Theory and Practice of Software Process Improvement and Capability Determination, IEEE Computer Society.
Floyd, C., 1986, "A comparative evaluation of systems development methods", Olle, T.W.et al, Information Systems Design Methodologies. Improving the Practice, Amsterdam.
Galliers, R.D., 1991, "Choosing appropriate information systems research approaches: a revised taxonomy", Nissen, H.-E., Klein, H.K., Hirschheim, R.A., Information Systems Research: Contemporary Approaches, and Emergent Traditions, Elsevier, Amsterdam.
Galliers, R.D., Land, F.F., 1987, "Choosing appropriate information systems research methodologies", Communications of the ACM, 30, 11.
Gould, J.D., Lewis, C., 1985, "Designing for usability. Key principles and what designers think", Communications of the ACM, 28, 3.
Guindon, R., Krasner, H., Curtis, B., 1987, "Breakdowns and processes during the early activities of software design by professionals", Empirical Studies of Programmers. Second Workshop, Ablex Publishing, Norwood, NJ.
Hirschheim, R., Klein, H.K., 1992, "Paradigmatic influences of information systems development methodologies: evolution and conceptual advances", Yovits, M., Advances in Computers, 33, Academic Press, New York, NY.
Hirschheim, R., Klein, H.K., Lyytinen, K., 1995, Information Systems Development and Data Modeling. Conceptual and Philosophical Foundations, Cambridge University Press, Cambridge, MA.
Hirschheim, R., Klein, H.K., Lyytinen, K., 1995, "Exploring the intellectual structures of information systems development. A social action theoretic analysis", Accounting, Management & Information Technology, 6, 1-2.
Humphrey, W.S., 1989, Managing the Software Process, Addison-Wesley, Upper Saddle River, NJ.
Iivari, J., Lyytinen, K., 1997, "Information systems research in Scandinavia. Unity in plurality", Currie, W.et al, Rethinking Management Information Systems, Oxford University Press, London.
Iversen, J., Mathiassen, L., 2002, "Cultivating and engineering a software metrics program", Information Systems Journal.
Iversen, J., Nielsen, P.A., Nørbjerg, J., 2001, "Problem diagnosis in SPI", Mathiassen, L., Pries-Heje, J., Ngwenyama, O., Improving Software Organizations - From Principles to Practice, Addison-Wesley, Upper Saddle River, NJ.
Iversen, J., Mathiassen, L., Nielsen, P.A., 2001, "Risk management in process action teams", Mathiassen, L., Pries-Heje, J., Ngwenyama, O., Improving Software Organizations - From Principles to Practice, Addison-Wesley, Upper Saddle River, NJ.
Iversen, J., Johansen, J., Nielsen, P.A., Pries-Heje, J., 1998, "Combining quantitative and qualitative assessment methods in software process improvement", Proceedings of ECIS 98.
Jakobsen, A.B., 1998, "Tricks of bottom-up improvements", IEEE Software.
Johansen, J., Mathiassen, L., 1998, "Lessons learned in a National SPI effort", EuroSPI'98.
Kaiser, K.M., Bostrom, R.P., 1982, "Personality characteristics of MIS projects teams. An empirical study and action-research design", MIS Quarterly, 6, 4.
Klein, H.K., Myers, M.D., 1999, "A set of principles for conducting and evaluating interpretive field studies in information systems", MIS Quarterly, 23, 1.
Knuth, D., 1989, "The errors of TEX", Software - Practice & Experience, 19, 1.
Kock, N., Lau, F., 2001, "Information systems action research", Information Technology & People, 14, 1.
Kozar, K.A., 1993, "Adopting systems development methods. An exploratory study", Journal of Management Information Systems, 5, 4.
Krasner, H., Curtis, B., Iscoe, N., 1987, "Communication breakdowns and boundary spanning activities of software design by professionals", Empirical Studies of Programmers. Second Workshop, Ablex Publishing, Norwood, NJ.
Kuvaja, P.,, Similä, J., Krzanik, L., Bicego, A., Saukkonen, S., Koch, G., 1994, Software Process Assessment & Improvement - The Bootstrap Approach, Blackwell, Oxford.
Lau, F., 1997, "A review on the use of action research in information systems studies", Lee, A.S., Liebenau, J., DeGross, J.I., Information Systems and Qualitative Research, Chapman & Hall, London.
Lee, A., 1999, "Rigor and relevance in MIS research: beyond the approach of positivism alone", MIS Quarterly, 23, 1.
Lee, A.S., Liebenau, J., DeGross, J.I., 1997, Information Systems and Qualitative Research, Chapman & Hall, London.
Lyytinen, K., 1999, "Empirical research in information systems: on the relevance of practice in thinking of IS research", MIS Quarterly, 23, 1.
Madabushi, S.V.R., Jones, M.C., Price, R.L., 1993, "Systems analysis and design models revisited. A case study", Information Resources Management Journal.
Markus, M.L., 1983, "Power, politics, and MIS implementation", Communications of the ACM, 26, 7.
Mathiassen, L., 1998, "Reflective systems development", Aalborg University, Aalborg, www.cs.auc.dk/~larsm.
Mathiassen, L., 1998, "Reflective systems development", Scandinavian Journal of Information Systems, 10, 1&2.
Mathiassen, L., Pries-Heje, J., Ngwenyama, O., 2001, Improving Software Organizations - From Principles to Practice, Addison-Wesley, Upper Saddle River, NJ.
Mathiassen, L., Bjerknes, G., Kristensen, C., 2001, "Improving customer relations", Mathiassen, L., Pries-Heje, J., Ngwenyama, O., Improving Software Organizations - From Principles to Practice, Addison-Wesley, Upper Saddle River, NJ.
Mathiassen, L., Munk-Madsen, A., Nielsen, P.A., Stage, J., 2000, Object Oriented Analysis and Design, Marko, Aalborg.
McFarlan, F.W., 1984, The Information Systems Research Challenge, Harvard Business School Press, Boston, MA.
McFeeley, B., 1996, IDEAL: A User's Guide for Software Process Improvement, SEI, Pittsburgh, PA.
McKay, J., Marshall, P., 2001, "The dual imperatives of action research", Information, Technology & People, 14, 1.
McKeen, J.D., 1983, "Successful development strategies for business application systems", MIS Quarterly, 7, 3.
Mingers, J., 2001, "Combining IS research methods: towards a pluralist methodology", Information Systems Research, 12, 3.
Mumford, E., 1983, Designing Human Systems: The ETHICS Method, Manchester Business School, Manchester.
Mumford, E., Hirschheim, R.A., Fitzgerald, G., Wood-Harper, A.T., 1986, Research in Information Systems, Amsterdam.
Munk-Madsen, A., 1986, "Knowledge about systems development", MARS Report No. 13, Aarhus University, Aarhus.
Myers, M., 2002, Qualitative Research in Information Systems, www.isworld.org.
Necco, C.R., Gordon, C.L., Tsai, N.W., 1987, "Systems analysis and design: current practices", MIS Quarterly, 11, 4.
Nielsen, P.A., 1999, "Action research and the study of IT in organisations: making sense of change", Department of Computer Science, Aalborg University, Aalborg.
Nielsen, P.A., Pries-Heje, J., 2001, "A framework for selecting an assessment strategy", Mathiassen, L., Pries-Heje, J., Ngwenyama, O., Improving Software Organizations - From Principles to Practice, Addison-Wesley, Upper Saddle River, NJ.
Nissen, H.-E., Klein, H.K., Hirschheim, R.A., 1991, Information Systems Research: Contemporary Approaches & Emergent Traditions, Elsevier, Amsterdam.
Nonaka, I., 1994, "A dynamic theory of organisational knowledge creation", Organisation Science, 5, 1.
Nunamaker, J., Chen, M., Purdin, T.D.M., 1991, "Systems development in information systems research", Journal of Management Information Systems, 7, 3.
Nygaard, K., Bergo, O.T., 1975, "The trade unions. New users of research", Personnel Review, 4, 2.
Parnas, D.L., Clements, P.C., 1986, "A rational design process. How and why to fake it", IEEE Transactions on Software Engineering, 12, 2.
Paulk, M.C., Weber, C.V., Curtis, B., Chrissis, M.B., 1993, The Capability Maturity Model: Guidelines for Improving the Software Process, Addison Wesley, Upper Saddle River, NJ.
Pettigrew, A.M., 1990, "Longitudinal field research on change: theory and practice", Organisation Science, 1, 3.
Pourkomeylian, P., 2002, "Software practice improvement", The Viktoria Institute, Gothenburg University, Gothenburg.
Rapoport, R.N., 1970, "Three dilemmas in action research", Human Relations, 23, 4.
Robey, D., 1995, "Theories that explain contradiction. Accounting for the contradictory organisational consequences of information technology", DeGross, J.I.et al, Proceedings from 16th International Conference on Information Systems, Amsterdam.
Salaway, G., 1987, "An organisational learning approach to information systems development", MIS Quarterly, 11, 2.
Schein, E.K., 1985, Organisational Culture and Leadership. A Dynamic View, Jossey-Bass, San Francisco, CA.
Schön, D.A., 1983, The Reflective Practitioner. How Professionals Think in Action, Basic Books, New York, NY.
Selby, R.W., Basili, V.R., Baker, F.T., 1987, "Cleanroom software development. An empirical evaluation", IEEE Transactions on Software Engineering, 13, 9.
Stolterman, E., 1992, "How systems designers think about design and methods. Some reflections based on an interview study", Scandinavian Journal of Information Systems, 4.
Stowell, F., West, D., Stansfield, M., 1997, "Action research as a framework for IS research", Mingers, J., Stowell, F., Information Systems: An Emerging Discipline, McGraw-Hill, New York, NY.
Tan, M., 1994, "Establishing mutual understanding in systems design. An empirical study", Journal of Management Information Systems, 10, 4.
Vidgen, R., Braa, K., 1997, "Balancing interpretation and intervention in information system research: the action case approach", Lee, A.S.et al, Information Systems and Qualitative Research, Chapman & Hall, London.
Vinter, O., 2001, "From problem reports to better products", Mathiassen, L., Prees-Heje, J., Ngwenyama, O., Improving Software Organizations - From Principles to Practice, Addison-Wesley, Upper Saddle River, NJ.
Vitalari, N.P., 1985, "Knowledge as a basis for expertise in systems analysis. An empirical study", MIS Quarterly, 9, 3.
Vitalari, N.P., Dickson, G.W., 1983, "Problem solving for effective systems analysis. An experimental exploration", Communications of the ACM, 26, 11.
Waltz, D.B., Elam, J.J., Curtis, B., 1993, "Inside a software design team. Knowledge acquisition, sharing, and integration", Communications of the ACM, 36, 10.
White, K.B., 1984, "MIS project teams. An investigation of cognitive style implications", MIS Quarterly, 8, 2.
White, K.B., Leifer, R., 1986, "Information systems development success. Perspectives from project team participants", MIS Quarterly, 10, 3.
Wynekoop, J.L., Russo, N.L., 1993, "System development methodologies. Unanswered questions and the research-practice gap", DeGross, J.I., Bostrom, R.P., Robey, D., Proceedings from 14th International Conference on Information Systems, Orlando, FL.