Mark S. Ackerman
Department of Information and Computer Science
University of California, Irvine
Irvine, CA 92717
ackerman@ics.uci.edu
http://www.ics.uci.edu/CORPS/ackerman.html
A growing concern for organizations and groups has been to augment their knowledge and expertise. One such augmentation is to provide an organizational memory, some record of the organization's knowledge. However, relatively little is known about how computer systems might enhance organizational, group, or community memory.
This paper presents findings from a field study of one such organizational memory system, the Answer Garden. The paper discusses the usage data and qualitative evaluations from the field study, and then draws a set of lessons for next-generation organizational memory systems.
A recent but growing interest for organizations and groups has been to augment and manage their knowledge and expertise. With an impetus from layoffs, down-sizing, and internationalization of personnel, the search for new ways to access, maintain, and promote the organization's intellectual assets has become vital for many organizations. One possibility is to provide an organizational memory, some record of the organization's knowledge. To do this, we would like to know how much computer systems can be used to augment or supplement existing learning and knowledge mechanisms within organizations, groups, and communities. Field studies, within the context of use, are an important part of understanding how computer systems can provide these organizational mechanisms.
This paper presents findings from a field study of an organizational memory system, the Answer Garden. In short, the study found that augmenting organizational memory was possible, but it also uncovered a number of interesting issues and problematic design assumptions. This paper describes these as the basis for future work on organizational and group memories.
To provide the necessary background for the field study, the paper first outlines organizational memory and organizational information seeking as research topics. Second, I will describe Answer Garden and distinguish it from similar systems. After these preliminary (but necessary) sections, I will then present the field study and its results. The final section of the paper will draw some conclusions and lessons from the field study for Answer Garden and similar memory systems.
Stewart [27] raised the possibility that the organizational issues of finding the right expert, growing knowledge and memory, and managing intellectual property may become increasingly important. Properly managing its intellectual resources might enable the organization to function more effectively and to prosper. Simply put, the likelihood that a perceived problem will find an organizational solution increases as the organization has additional resources to find previously created solutions or to create new solutions [8].
From the macro perspective of the organization, one method of managing its intellectual resources is to augment its organizational memory. It is beyond the scope of this paper to review the literature on organizational memory. In general, however, an organization of people should retain some knowledge of its past efforts and environmental conditions. If an organization learns, then the result should be available later ([19], [10]). In this view, "organizational memory" is organizational knowledge with persistence.
A standard connotation of organizational memory, that that can be captured in a written record, is only one form of organizational memory. In Walsh and Ungson [28], organizational memory can be retained in six places: individuals, organizational culture, organizational transformations, organizational structures, organizational ecology, and external archives. (External archives are data external to the organization, not its own archives.) It should be noted that individuals are a prime location for retention of the organization's knowledge. This list, however, should be expanded to include internal information repositories such as corporate manuals, databases, filing systems, and even stories ([29], [22], [23]).
What does it mean for knowledge to persist within an organizational setting? March and Simon [21] argued that organizations are concerned with achieving their goals in a manner that minimizes the drain on limited resources. If this is true, the goal-driven nature suggests that an organizational memory mechanism that is immediately tied to the on-going processes and considerations of an organization will be most important and useful to that organization. Furthermore, one might expect that organizational memory mechanisms most often are employed for recent events and outcomes because the situational context does not need to be reconstructed and substantially reinterpreted to use the information. In other words, we want to minimize the upstream costs of an organizational memory mechanism and make the downstream payoffs clear [12].
Thus, information technology can support organizational memory in two ways, either by making recorded knowledge retrievable or by making individuals with knowledge accessible. To augment organizational memory in this manner, an information system would need to incorporate elements of information databases and communication systems. Furthermore, we might expect that such an organizational memory system would be most useful if it were centered around a current organizational activity.
Viewed from the perspective of the organizational member, rather than the organization as a whole, organizational memory is invisible or at least muted and hazy. At a micro-level, the issue is one of information seeking by some organizational member within an organizational context. As such, information seeking can be considered as the process of finding the right "piece" of organizational memory. The goal, then, is to decrease the effort involved in information seeking in cases where the required information is not known to the individual involved, or to eliminate the information seeking in cases where the organization has redundant efforts.
In his seminal study of engineers' information seeking behavior, Allen [3] found that their major source of information was direct contact and communication with colleagues. He determined that performance was related to the presence of gatekeepers, engineers who maintained contacts outside the group. Much of an organization's communication flow (for R&D engineers) is channeled through these gatekeepers, who track both the literature and the organizational members who know about various subjects.
Possibilities for computer support are offered in Gerstberger and Allen [11]. This study noted some of the reasons why engineers would not go to colleagues, and would instead use other information channels. In their study of 19 engineers, Gerstberger and Allen found that the engineers chose not to go to the channel of the highest quality for technical information, but rather to go to the channel of highest accessibility (i.e., lowest psychological cost). Allen [3] argued that the psychological cost was in the potential lack of reciprocity between giving and obtaining information and in the status implications of admitting ignorance.
As Allen pointed out, information seeking is not a straight-forward information transfer. For people with a career anchor in technical expertise [25], each information seeking interaction is a double-edged sword. It provides the possibility for consorting with experts (or relative experts) and thus gaining prestige and status. On the other side, it also provides the possibility of being seen as incompetent, a possibility either so frightening or so likely that most engineers prefer using the documentation, other technical literature, or friends. In order to facilitate information seeking in the communication system, one should decrease the status implications and need for reciprocity. Any system attempting to augment information seeking (and use the organizational members as part of an organizational memory system) will need to consider this status interaction.
Answer Garden supports organizational memory in two ways: by making recorded knowledge retrievable and by making individuals with knowledge accessible. In the standard configuration of Answer Garden, users seek answers to commonly asked questions through a set of diagnostic questions or other information retrieval mechanisms. Figure 1 shows Answer Garden in use in its X Window System environment. These diagnostic questions form a branching tree structure through a hypermedia network. Alternatively, the user may use a number of other information retrieval mechanisms, including tree diagrams (i.e., projections of the hypermedia network) or various information retrieval engines to find the nodes that may contain the answer.
In many cases, however, the information will not be in the information database. If an answer is not found or is incomplete (or if the user becomes confused or lost), the user may ask the question through the system (Figure 1 also). Answer Garden then routes the question to an appropriate human expert. In the field study described below, the message was sent to the expert anonymously to reduce status implications [26].
Figure 1: Answer Garden (screen shot)
The expert then answers the user via electronic mail, and if the question is a common one, the expert can insert the question and its answer back into the database. Thus, users are not limited to the information in the system; if the information is not present, they can tap the social network in a natural way. As a result, the construction of the information is iterative, and the corpus of information grows over time. Answer Garden is more fully described in [2] and [1].
Several important considerations behind Answer Garden include:
Answer Garden combines many different kinds of systems in a novel manner. Interestingly, the addition of various capabilities that may be found in differing computer applications leads to a very different type of application in practice.
Answer Garden differs from standard information retrieval systems (e.g., [4]) in its addition of communication capabilities. Answer Garden varies from computer-mediated communications systems such as enhanced mail [7] or bulletin board systems (e.g., [14], [15]) in its capability for retrieval and its emphasis on growing a memory. Answer Garden differs from consulting systems using expert system technology (e.g., [13], [24]) because of its reliance on human experts and its resulting non-brittleness and because of its emphasis on iterative design without knowledge engineers. Answer Garden also differs from other forms of organizational memory, such as design rationale (e.g., [9], [18]), in its emphasis on informal information and communication flows.
The Answer Garden application used in the field study was built in the Answer Garden Substrate system, a system for building organizational memory applications [1]. It may be possible to build the Answer Garden application, as described above, in other systems such as [20] or [16], but the Answer Garden Substrate has the advantages of extensibility, open architecture, and portability.
Answer Garden offers an alluring application, providing the possibility of capturing portions of an organization's memory at a relatively low cost to that organization. But, would it work in practice? I undertook a field study of Answer Garden using the sample information database about the X Window System to determine whether, in fact, it would.
I chose to study software engineers using the X Window System. I did this for a variety of reasons including the iterative nature of user interface development, the complex nature of the X Window System and its toolkits, and the presence of electronic mail and network connections in a workstation environment. Software engineering is of special interest for an examination of expertise and information seeking because there is no body of commonly accepted knowledge (aside from low-level data abstractions and algorithms which all programmers are expected to know). Furthermore, the software world is undergoing constant revolution, preventing an individual engineer from acquiring lasting expertise. Software engineers using the X Window System, then, were ideal candidates for information seeking through electronic means such as Answer Garden.
Two field sites provided almost all of the data in the field study. These two sites were a research group at MIT and a class in the Harvard extension program with a total of 59 potential users. Additional field sites included a software group at a federally-funded laboratory, a product team in a software company, an advanced development group in a large computer company, and a support group in another computer company. Two of these supplemental sites were local; therefore, I was able to obtain interview data.
The Harvard site was a Harvard extension class in two-dimensional graphics programming using the X Window System. The Harvard class had 44 students and 3 instructors. Its students were largely working software engineers, and as such, used Answer Garden in their workplaces, potentially substituting it for asking questions and seeking information from organizational colleagues. The MIT research group, with 11 members, was engaged in sponsored research, and their work flow appeared to be similar to that in a software R&D company.
There was, additionally, another set of participants in the field study: the experts who answered questions. I wished to provide the study participants with authoritative information, so that they would not need to judge whether it was accurate or not. Two colleagues and I served as the first-line of answerers. The seven experts included staff members of the X Consortium as well as consultants with more than three years worth of experience in Motif or OpenLook (popular varieties of X). Other experts volunteered, but were not needed.
The study used multiple data collection procedures including an initial questionnaire, usage data at the "mouse-stroke" level, critical incident interviews, final questionnaires, and field observations and interviews (with both users and experts). The primary methods were the usage logs and the critical incident interviews. Most critical-incident interviews, held with users shortly after their use of the system, ran between 15 and 20 minutes in duration, including time spent obtaining qualitative data concerning the interviewee's responses. For example, users often provided (and were encouraged to provide) additional information on why they rated the system as they did.[1] Because the critical incident interviews were not systematically randomized, only qualitative data from the 49 critical incident interviews will be described.
The observational data was used to supplement and bolster the qualitative evaluations and quantitative usage data. In general, a concerted effort was made to use the qualitative and quantitative data together.
In addition to the design considerations behind Answer Garden in general, several other assumptions and simplifications were made in the field study. An implicit assumption in the field study was that there were separate groups of experts and of information seekers. This is clearly not the case in organizational life, where people are arrayed along many continua of expertise over many subject domains. Other versions of Answer Garden could allow for more of a community interaction.
Additional simplifications in the field study were made about the type of information to be captured in the information database. An implicit assumption behind Answer Garden is that the information within an information database should be easily classifiable and easily broken into node-sized "chunks." Furthermore, this information database contained only static information, and did not include connections to people, software modules, or dynamic information. (The Answer Garden Substrate does support these types.)
The version that was released to the field sites was essentially the same as the version distributed on the MIT X11r5 contributed software tape. Because it was designed to be readily available, it is quite robust, and only a handful of minor bugs have been reported. However, this version did not include information retrieval engines (other than the standard, naive retrieval engine) or window stacks (to tidy up the user's screen).
During the study, Answer Garden was used 194 times by 35 users in the two major sites (mean number of uses per user = 5.54, s.d.= 7.01). Since Answer Garden was designed to be a system that would be used intermittently, I expected usage would occur only when the user had a difficult question, perhaps twice a week or even once a month for consistent users. In fact, this is what was observed. The maximum that any user used Answer Garden was 28 times over three months.
This pattern of usage did not appear to be from lack of interest. Only five of the users appear to have been just looking from curiosity; the others self-reported serious usage. Moreover, these 35 users were 59% of the 59 potential users; additionally, several potential participants reported being unable to connect to the Harvard Answer Garden for technical reasons.
Answer Garden's use was not uniform across the test period. Figure 2 shows the aggregate usage. Usage was higher initially as users experimented with the application. Usage dropped in week 11 since this was the Thanksgiving holiday. Additionally, usage dropped after week 14 when the MIT group ended their use with the end of MIT's semester.
Click here for Picture Figure 2: Aggregate usage by week
The two sites, however, showed different usage patterns. The MIT group started two weeks earlier than the Harvard group and displayed gradual increase in the number of sessions per week through week 6. The MIT group then fell into a steady usage of two to four sessions per week, with use from nine of eleven potential users. A spike during week 8 suggests that usage can increase by a factor of 100% during "normal" periods. Interviews with these MIT participants revealed no systematic pattern since usage was driven by a combination of local deadlines and steady-state work. This is the same as the demands on experts' time now; people do not ask questions of experts on a specific schedule. Such high variations in usage increase the difficulty of scheduling experts for answering questions through Answer Garden, and suggests that a workflow component is required for similar systems. As might be expected, the usage from the Harvard class was driven by class deadlines, leading to a more irregular pattern.
Even taking into account the user's location, usage varied considerably by individual. A simple diagram of the number of sessions by user (Figure 3) shows that six users were "heavy" users, accounting for 61% of the total sessions. Heavy users often kept Answer Garden running in an iconified state between uses. One heavy user reported:
You know that I find it valuable when I've always got it running. It's just one of my normal suite [of tools].
The other participants could be further divided into two groups, a finding obscured in Figure 3. An analysis of usage by person, broken down by the date of their session, shows that there were a group of users that used Answer Garden more infrequently, but over a long time period. These 12 participants were classified as "intermittents." Some of these intermittents used Answer Garden for as long, but not as intensively, as the heavy users. That this group of intermittent users existed was extremely satisfying since it indicated that some users found Answer Garden sufficiently useful to return after several weeks. These users used Answer Garden as was originally predicted, as an occasional tool for information seeking. Additionally, there was a group of seventeen people, or 49% of the participants in the study, who used the system for a short period of time; they were categorized as "tire-kickers."[2]
Figure 3: Usage by user
The qualitative responses indicated that the standard problems in introducing a new software system were present in the field study. For example, some non-users had some difficulty in connecting, while others did not want to spend the time to learn the system. However, there were some new difficulties as well. Even a heavy user spoke of the issues of finding information within Answer Garden:
There was a learning curve issue in finding out where the good information was, you know? ...The people who persevered got good at it.
Users also spoke of more subtle issues. One important issue was an ambiguity in the users' minds about what questions were appropriate:
I was often unclear as to what were legitimate Answer Garden questions.
This may have been one of the factors that separated intermittents and tire-kickers. This issue will be discussed further below.
Users asked many questions in the field study. In 53% of the 194 sessions, mail was sent for a total of 121 mail messages. (In 16 sessions, users sent multiple mail messages.) There were an additional 43 times that users opened mail windows, but did not send any message. The reasons were very obvious in the critical incident interviews. A typical user response during an interview was: "I could have used more info [in the database]."
Largely this was the result of an incongruity between the coverage in the database and the size of the topic domain. The X information database was small (98 nodes at the start of the study with approximately 225 questions and answers) compared to the number of possible questions and answers about X (at least thousands of potential questions). This suggests two possibilities: future Gardens should be restricted to smaller domains so users will be more likely to find answers to their questions already in the information database, or the initial period of activity will be largely devoted to building the domain coverage.
Because of Answer Garden's design, two groups participated in this field study, the users and the experts. In short, the qualitative evaluations from both groups showed a fairly complex set of nuanced activities. Generally, people found the system satisfactory for their use, but there were places where the system needed to provide additional support.
The 49 critical incident interviews, conducted with the participants about their use of the system on a session basis, included 30 negative statements and 23 positive statements about the system. (Several participants gave mixed responses.) Overall, the majority of numeric scores in these same critical incident interviews were favorable.
In general, the overwhelming emphasis in the performance evaluations for the sessions was that the users were satisfied as long as they got an accurate solution quickly. The source of the answer did not matter. Users rated the application more highly when an answer was already in the information database or when they got a quick response by electronic mail to their question. A typical evaluation was "It was very good. Whoever answered it was exactly right, and very quick too." Participants gave an unfavorable rating when they got no answer or a slow answer. In fact, almost all (83%) of the unfavorable responses were due to not being able to get the information quickly, either in the information database or by electronic mail.
In addition, users did not like it when they had to wade through a considerable amount of text to find that the answer was not present. If the answer was not in the information database, they wanted to know quickly so they could ask their question via electronic mail. In fact, being able to quickly determine that the information was not in the database was often seen as a positive attribute. A majority (60%) of the favorable responses in the interviews mentioned being quickly able to determine whether the information was in the database, getting a quick response, or finding the information in the database.
Users appeared to also be comfortable with information seeking through a combination of standard information retrieval system and the social network. This conclusion is based on indirect evidence, primarily the differences in comments between the first weeks of the study and later. During the first weeks of the field study, users had to be prompted to accept the use of the social network as part of the information retrieval process. Participants consistently reported not wanting to bother the experts or expressed surprise at the idea that they could ask the question of a human if the answer was not present. (They had been told how the application worked repeatedly in introductory talks and handouts, implying that they were not used to the idea of Answer Garden at first.) Several steps were taken to reinforce their understanding, including a sentence on the initial node reminding them to ask questions.
After the first month, the performance evaluations suggested that their continued usage of the application was natural. Users stopped evaluating the information in the database separately from the electronic mail response; the system evaluation was based on both together. Users did not answer that they did not ask a question through the system because it seemed strange or novel even though they were prompted for such responses. By the middle of the field study, tying the social network into the information retrieval system appeared to be normal to the users. Said one user:
I always went straight to AG at this point [after looking at the manuals and asking people nearby], instead of trying to guess someone who would likely know the answer, because I trusted that there was a mechanism in AG that would guarantee a satisfactory answer.
In addition, Answer Garden seemed to ameliorate the status implications of information seeking for many users. No one mentioned negative status implications in the critical incident interviews. This was true for both the people who sent a question and for those who did not. In fact, session evaluations were on the whole appreciative of reducing the status interactions in information seeking. When it was mentioned, the users liked being able to ask their questions anonymously. Said one interviewee: "[It's] a vehicle where you're not intimidated; you're asking anonymously and through text." In a quarter of the interviews, participants said that the application helped them ask for information that anyone would be expected to know. Another half (56%) of the interviewees said that their questions were intermediate in difficulty.
However, when asked about the information itself, users had a mixed reaction to the information gathered through Answer Garden. In nearly 20% of the interviews, participants pointed to Answer Garden as providing information they could not have received elsewhere. However, about a third of the users reported problems with the specificity of the material (either too high or too low) and the level of the explanation provided by the expert (either too high or too low). This was an indicator of another issue, the problematic dichotomy of expert and novice, as will be discussed in the Lessons section.
As mentioned, there were seven external experts in the field study, all of whom had more than two years of experience. As mentioned, many of the experts answering questions were directly responsible for parts of the X Window System or its derivative vendor releases. The design of the field study tried to keep the workload on any of these external expert to a minimum. The most questions given to an external expert was two per week.
Five of the experts expressed initial fears about being overwhelmed by the task of answering questions. Accordingly, the experts demanded and were given the right to refuse to handle questions based on their workload. This rule was invoked by some experts for almost half of the field study, suggesting that redundancy in expertise will be required for similar systems. The external experts also requested the right to not answer any question that required excessive time or was too specialized; however, this rule was invoked only once during the field study.
No expert complained about the time required for answering questions. However, the style of answering was an issue in the study. Four of the experts showed a marked formality in their responses. Their answers were longer, containing substantially more detail and more cases than would a quick electronic mail response. When interviewed, these experts reported wanting to provide a more complete answer for two reasons. First, they tried to immediately generalize from the specific situation to a general answer. Moreover, they wished this general answer to be suitable for a large range of questions and special cases instead of growing the answer iteratively over time. Second, several reported that their response served as a public "badge" for them within the organization or community. As such, they wanted to make sure that their answer was complete and accurate - not doing so might reduce their status with people who did not know them well.
This push towards formality of response goes against the users' desires to have short, easily readable answers and the system goal of capturing informal flows of information. Not publicly identifying the expert might reduce the formality and further reduce the status implications in information seeking, but users use this identification to judge the potential accuracy of the information. This dilemma may argue for the necessity of an information "editor" or "moderator" to ensure authoritativeness and consistency in the information database.
Nonetheless, three of the external experts did provide informal responses. This suggests the possibility of keeping the responses informal. Again, an editor or moderator might be valuable in some situations; he could work to keep the tone informal.
How successful were the design assumptions behind Answer Garden? The field study resulted in the following:
These findings demonstrated that Answer Garden could work in principle.
It was also hoped that such an organizational memory system as Answer Garden would result in the reduction of status implications and the need for reciprocity. The field study resulted in both a mixed achievement and some new insights about this design consideration:
However, some users still had issues concerning the status implications in information seeking through the system. Whether these users would find their concerns ameliorated over an extended period of use is unknown. It may also be possible to reduce the status implications by providing access to lower-status help desk personnel.
This reaction could indicate continued concern over the status implications in information seeking. However, the users' responses suggested that software engineers may have an additional social consideration when seeking information, a consideration that had been previously masked by the concern over status implications.
One possibility for these comments might be a respect for the experts' position. The words the participants in the field study used could indicate a respect for the tasks and duties of people above them in a technically-based hierarchy. Because software engineers recognize technical expertise as a meritorious achievement and because their authority is based on technical expertise, they may not wish to "bother" the experts.[3 ] This situation would then show characteristics of Blau's example of a medical bureaucracy with its respect for professional authority [6]. It is possible that the status implications were masking this additional concern for the experts' role and position.
If such a concern does exist, Answer Garden does little to ameliorate it. Indeed, Answer Garden may exacerbate it. At least in the field study version, Answer Garden identifies some organizational members as experts, placing them automatically above the user. This would imply that the clear-cut separation of experts and users is not only artificial, it may lead to operational difficulties.
An alternative possibility is that the concern resulted from the experts' being volunteers, rather than being organizationally responsible for these answers. In other words, users felt that answering questions required labor that was not organizationally rewarded. Shifting incentives might change the behavior and comments of the users. While I provided users with knowledge of the incentives for the experts, it may not have meant as much to the users as clear organizationally based rewards.
Further research is required to differentiate whether either respect for professional author or concern over organizational rewards causes such comments.
The design of Answer Garden in the field study assumed that the status implications and need for reciprocity were only negative and should be removed. It may be, however, that they also provide positive organizational benefits. For example, the status implications in information seeking and the need for reciprocity may serve to channel information seekers towards others at their expertise level, thus providing the seekers with answers at the right level and length of explanation.
If this is correct, then it is not the status implications and need for reciprocity that are problematic per se. The organizational problem arises when the information seeking can be considered dysfunctional [5]. For example, an organizational dysfunctionality occurs when the people at the same expertise level cannot answer a question and there is no person with greater expertise available.
The study resulted in two intriguing findings: First, some experts' perceptions of needing a public "face" in an organization or community pushed towards a formalization of their responses. Second, there are positive effects from reciprocity and status implications for the expertise network of an organization, and these effects route organizational members more effectively to the right level of expertise. Both of these findings will require some adjustment of the design assumptions behind Answer Garden and similar systems.
Some important caveats remain. I examined only the use among software engineers, and generalizations to groups with career anchors other than technical expertise are unclear. For example, there is a strong possibility that doctors or teachers would have different usage patterns. Furthermore, this study could not follow long-term results from Answer Garden usage, and many studies have suggested that the long-term effects of an information system cannot be adequately determined from observing the short-term effects (e.g., [14]). For example, organizations could choose to use Answer Garden to prevent experts from being "interrupted." If users were prevented from this, it might lead to loss of social activity by both users and experts. Poking one's head in a office door has other purposes and benefits than information seeking.
Nonetheless, within the confines of the field study, Answer Garden "worked" in general. After a short period of adjustment, users found the inclusion of the social network in the system to be natural. Experts answered questions, and the database grew. The combination of being able to find an expert and growing the information database led to an organizational memory.
This work was done while the author was at the MIT Center for Coordination Science. The work was supported, in part, by research grants from the X Consortium, Digital Equipment Corporation, the National Science Foundation (IRI-8903034), and the MIT International Financial Services Research Center.
Tom Malone provided me with invaluable guidance and assistance for the entire Answer Garden project. Wanda Orlikowski, JoAnne Yates, and Brian Pentland helped considerably in my understanding of organizational memory. Jintae Lee expanded my view of organizational memory systems. Chris Schmandt, Debby Hindus, and Jonathan Grudin provided assistance on this paper. I would also like to thank the people at the various field sites for their insights and assistance.
1. Ackerman, M. S. Answer Garden: A Tool for Growing Organizational Memory. Massachusetts Institute of Technology, Ph.D. Thesis, 1993.
2. Ackerman, M. S. and T. W. Malone. Answer Garden: A Tool for Growing Organizational Memory. Proceedings of ACM Conference on Office Information Systems, 1990: 31-39.
3. Allen, T. Managing the Flow of Technology. MIT Press, Cambridge, MA, 1977.
4. Bartschi, M. An Overview of Information Retrieval Subjects. IEEE Computer, 1985, May 1985: 67-84.
5. Blau, P. M. The Dynamics of Bureaucracy: A Study of Interpersonal Relations in Two Government Agencies. University of Chicago Press, Chicago, 1955.
6. Blau, P. M. and M. Meyer. Bureaucracy in Modern Society. Random House, New York, 1987.
7. Borenstein, N. S. Computational Mail as Network Infrastructure for Computer-Supported Cooperative Work. Proceedings of CSCW'92, 1992: 67-74.
8. Cohen, M., J. March and J. Olsen. A Garbage Can Model of Organizational Choice. Administrative Science Quarterly, 1972, 17(1): 1-25.
9. Conklin, J. and M. L. Begeman. gIBIS: A Hypertext Tool for Exploratory Policy Discussion. Proceedings of CSCW '88, 1988: 140-152.
10. Duncan, R. and A. Weiss. Organizational Learning: Implications for Organizational Design. in Research in Organizational Behavior. JAI Press, 1979.
11. Gerstberger, P. G. and T. J. Allen. Criteria Used by Research and Development Engineers in the Selection of an Information Source. Journal of Applied Psychology, 1968, 52(4): 272-279.
12. Grudin, J. Why groupware applications fail: problems in design and evaluation. Office: Technology and People, 1989, 4(3): 245-264.
13. Gwei, G. and E. Foxley. Towards a Consultative On-Line Help System. International Journal of Man-Machine Studies, 1990, 32: 363-383.
14. Hiltz, S. R. and M. Turoff. The Evolution of User Behavior in a Computerized Conferencing System. Communications of the ACM, 1981, 24(11): 739-762.
15. Horton, M. and R. Adams. Standard for Interchange of USENET Messages. RFC-1036.
16. Lai, K.-Y., T. Malone and K.-C. Yu. Object Lens: A "Spreadsheet" for Cooperative Work. ACM Transactions on Information Systems, 1988, 6(4): 332-353.
17. Lancaster, F. W. and E. G. Fayen. Information Retrieval On-Line. Melville, Los Angeles, 1973.
18. Lee, J. SIBYL: A Tool for Managing Group Decision Rationale. Proceedings of CSCW'90, 1990: 79-92.
19. Levitt, B. and J. G. March. Organizational Learning. Annual Review of Sociology 1988, 1988, 14: 319-340.
20. Lotus Development Corporation. Lotus Notes. manuals, release 2.0 and 3.0, 1991-1993.
21. March, J. G. and H. A. Simon. Organizations. Wiley, New York, 1958.
22. Morgan, H. L. and D. J. Root. A Concept of Organizational Memory. Proceedings of Office Automation Conference, 1979: 31-36.
23. Orr, J. Narratives at Work: Storytelling as Cooperative Diagnostic Activity. Proceedings of CSCW 86, 1986: 62-72.
24. Ram, S., S. Hayne and D. Carlson. Integrating Information Systems Technologies to Support Consultation in an Information Center. Information & Management, 1992, 23: 331-343.
25. Schein, E. H. Career Dynamics. Addison-Wesley, Reading, MA, 1978.
26. Sproull, L. and S. Kiesler. Connections: New Ways of Working in the Networked Organization. MIT Press, Cambridge, MA, 1991.
27. Stewart, T. A. BrainPower. Fortune. 44-60, 1991.
28. Walsh, J. P. and G. R. Ungson. Organizational Memory. The Academy of Management Review, 1991, 16(1): 57-91.
29. Yates, J. Control through Communication: The Rise of System in American Management. John Hopkins Press, Baltimore, 1989.
