David S. Stodolsky
Entropy is on the rise.

 
Home

About



Members
Join Now
Login

 
 

Consensus Journals

Abstract

Computer networks open new possibilities for scientific communication in terms of quality, efficiency, and rapidity. Consensus journals have the economy of invitational journals and the objectivity of journals based upon peer review. That is, all articles are published and the reader benefits from article selection based upon impartial refereeing. An additional benefit of consensus journals is that the negotiation process, that typically occurs prior to publication, is automated, thus saving efforts of participants.

Readers submit reviews that evaluate articles on agreed dimensions. A statistical procedure is used to identify the most knowledgeable representative of each consensus position and these persons are invited to submit articles that justify the review judgments they have submitted. A major advantage of this approach is the ability to develop reputation without article publication.

The approach includes a protection mechanism based upon pseudonyms, which substitutes for the protection of anonymity typical of scientific journals. This reduces the potential for irresponsible behavior and facilitates reputation development. The level of quality enhancement is superior to that achievable with anonymous peer review.

Eliminating the editor and the delay associated with conventional refereeing makes message quality enhancement available in message systems for educational and business environments.

____________________________________________________________________

Stodolsky, D. S. (1995). Consensus journals: Invitational journals based upon peer review. _The Information Society_, _11_(4).

This document has been prepared for electronic publication. Underscore characters indicate the start and end of italicized character sequences, and indentation of subheadings. Figures and tables assume a monospace font.

____________________________________________________________________

Invitational journals can be distinguished from typical scholarly journals by the sequence of events that results in publication of an article. The sequence of events with a typical journal starts with writing of an article. The article is then transmitted to an editor and refereed. After a successful review, often contingent upon negotiated revisions, the article is published and read. With invitational journals, however, events are reversed. The tentative decision to publish an author is made first, often based upon the reading of previous work by that author. Then negotiation between the editor and author occurs, or there is informal refereeing of a proposal, which if successful, results in writing of an article. The great advantage of this second sequence -- read, negotiate, write -- is that almost every article written gets published. The disadvantage is that selection of authors is somewhat arbitrary and there is no way an unknown author can get published. The objective of this article is to outline a method of scientific communication that has the economy of invitational journals and the objectivity of journals based upon peer review. These self-edited journals will be called _consensus journals_ in order to distinguish them from conventional invitational journals.

The terms _referee_, _review_, _reputation_, and so on, are defined with greater precision, and their referents have a slightly different form and function in the proposed consensus journal format then they do in conventional peer reviewed journals. Therefore, to minimize confusion, we will introduce this format with a brief scenario, which may give a flavor of the social dynamics and tempo made possible by the consensus journal. Elements of this new style in scientific writing are already apparent on scholarly electronic mailing lists. An example is mathematical advance, that went from promising concept to completed result in a matter of weeks, as a result of a "feeding frenzy" triggered by an informal worldwide competition (Kolata, 1990). Perhaps an example more relevant to this discussion is the volume "Scholarly journals at the crossroads: A subversive proposal for electronic publishing" (Okerson & O'Donnell, 1995), which reprints an on-line discussion.

Another potential source of confusion is the integration of elements from the fields of bibliometrics and the sociology of science into the operation of the consensus journal. While such approaches have often been used to analyze scientific journals, scientific fields, and science as a whole, they have not influenced operation of a journal, except for the rare case where an editor becomes aware of such results and implements changes to editorial procedures. Bookstein (1991) has suggested an analytical approach based upon evolutionary biology and the use of biometrics as appropriate to characterize the success of articles within the network of citations which would comprise the hyperlinks connecting articles. Such broad analyses are appropriate, since the mechanisms described can be used to determine both article quality and author competence. An author could think of a consensus journal as a highly structured forum for argumentation. A forum embedded in a pseudomized social field, that is, in an interaction space in which identities can not be discerned. So, let's begin our scenario.

Bob "The Bomb" Berdly, a referee for the electronic journal _Human Interaction Research_, awakens from a nap after a massive Sunday lunch. He reminisces of the day back in 1999 when he acquired his nickname. He had rejected an article of an _ultra-modern_ theorist by pressing a single key on his computer to indicate an incorrect conclusion. A week later, he was invited to justify his review by the Journal's consensus _server_. The review report containing the invitation showed his position was shared by many other referees. He had let the job slip until the last possible day and then typed in a paragraph from a book by an obscure independent scholar, which was not available on-line. Based upon this book's exposition of human evolution, he had parameterized a human interaction simulation programmed by an unknown anthropologist at a hinterland college in India. His real contribution had been the simulation parameters and a Uniform Resource Locator (URL) pointing to the simulation. He finished his article with the line "Kaboom!", because the simulation results had clearly shown an erroneous analogy in the target article. This last minute submission would not have been possible at all without computer networking, and certainly he could never have run the simulation without connecting directly to the computer in India, since he knew virtually nothing about programming and very little anthropology.

A day later, he got an electronically mailed letter from the Indian anthropologist, Jasud. He was asked to send a letter supporting an emergency request for more computer power. His single URL was generating so much load that network jobs were being rejected. This was causing the College a loss of revenue and damaging its reputation. A week later it became obvious that he had triggered a major controversy. The review report for his article showed a major split in the field. The ultra-modernists had apparently rejected his critique as irrelevant, but even more surprising, the only other opposition concerned the simulation model. When he examined the review report in detail, he saw that over a thousand people had offered to write a response to his article. Even more surprising were the competence ratings of the potential authors, over a hundred in each group had adjusted reputation estimates predicting a better than 90 percent chance of successful justification for their keypress reviews. Since both sides could not be correct, a lot of peoples' reputations were on the line.

The next week was like a lull before a storm, as the articles citing his were prepared. One complained that he had not done a sensitivity analysis on his parameters. This was formally correct. He prepared a follow-up article that showed that even with widely varying parameters his results were supported. Another criticism was of the validation procedures, which had been applied to the simulation code. This was also a formally correct objection, but had no obvious impact on the substance of his article. The two ultra-modernist authors attacked his application of philosophical anthropology to a concrete question related to language use. The best article concluded that the simulation results were so remote from the question addressed in the original article, that no evidence from that source could be meaningful. At this point, all interest focused on this "too remote" criticism and the question of relevance. Next week's review report for that article showed thousands more had been attracted into the controversy, this was clearly a chance for young researchers to improve their reputations' fast. However, the support for the ultra-modernist position, the predicted degree of backing a new author could expect, had dropped.

While four invitations had again been automatically issued for a response to the "too remote" article, only three were delivered. The shock was that they all opposed the "too remote" article. One of the ultra-modernists had changed his position during the week and his article also rejected the "too remote" critique. He had applied a new automated induction program to the simulation results and after several hundred thousand runs bracketed a formalized version of ultra-modern theory. While he suffered a minor reputation loss for changing his position, he would have a tremendous reputation gain if the strongly supported "too remote" article was cancelled by its author.

Sure enough, after seeing his proof, most of the ultra-modernists dropped their support for the "too remote" critique. The others, seeing that the whole ultra-modernist school could be destroyed, mobilized their resources to defuse "The Bomb". An army of graduate students was set to work, and every article, program, and data set linked to Bob's article was examined for weaknesses. This resulted in a series of articles following up on the irregularities in the validation of the simulation code. It was discovered that Jasud's adviser had signed off on his thesis without checking the validation. They were both fired and Jasud's degree was revoked. Meanwhile, after a moment of excitement, when a virus was discovered in the simulation code, the validation checked out and the "too remote" article was cancelled. Its anonymous author, and its remaining supporters, suffered a disastrous loss of reputation. Finally, the article that Bob had rejected was cancelled, with even a greater reputation loss for its author, and a leading ultra- modernist went into early retirement. Whether he had been the anonymous author of the cancelled article was never discovered. In any case, the ultra-modernist school disappeared from the Journal and Bob could relax since his "Kaboom" article had also gone _out of print_, when its target was cancelled.

Bob's reputation had increased with each ultra-modernist article cancelled, since the faulty analogy was inherent in ultra-modernist theory. After that, he could get all the money he requested. He hired Jasud as a graduate assistant at a salary double that which he had received as a faculty member in India. Whenever, Bob sent him a really hard problem to work on, which might tempt one to cut corners, he concluded his email with the line, "Remember, No Kaboom!", but with a smile. While no one could ever prove that Bob was the author of "The Bomb", he had suffered a remarkable improvement in reputation immediately after its target article was cancelled, and it was no secret that he was a referee for _Human Interaction Research_ and that he had a copy of that old book by the obscure independent scholar. Confirming such suspicions, however, would have been a minor security breach, but worse than that, a form of crass self-promotion universally frowned upon in the academic world.

His reminiscences were suddenly interrupted by a flourish of trumpets from his study. He glance at his watch, it was exactly three o'clock. That was midnight in London, and it must be his computer announcing the arrival of a Monday morning issue of _Human Interaction Research_. He rarely delayed glancing at the review reports when new issues arrived, but the flourish of trumpets was only triggered if a big reputation shift was predicted. He couldn't remember any controversial articles, but he went to investigate anyway. Even though he was very well established in the field, it was always possible to lose one's reputation as a reliable critic with on-line journals.

He found the issue routine, several articles and a few routine review reports. His automated trigger had been set off by the review report for an article on the effects of feedback with the new Mediation Technology vote collection unit, which he had submitted the week before. This had been a replication of earlier work, with a new machine, and hardly the stuff of controversy. However, the review report showed two well separated consensus positions. The first characterized his article as not very original and the second, while conceding originality, rejected the reasoning as faulty. He had been prepared for the complaint of non-originality, and had included a hyperlink to an old article in a print journal, which had never before been cited on-line, to avoid a formal error. But what could be making people consider his reasoning faulty? He looked at his article, it basically said "feedback effects confirm earlier findings", with links to the experimental data and previous articles. He summoned up the on-line articles and took the printed one off the shelf, and examined their conclusions. The feedback effects on speaker performance were entirely incompatible. How could such contradiction exist between articles? Then it dawned upon him, the printed article concerned _feedback_ from a speaker to listeners and others concerned feedback from listeners to a speaker. The exact same word used in an opposite way. Well, concluded Bob, this was not going to be any problem. In any case, he had a week to respond....

Theoretically, any reader of an article in a consensus journal can act as a referee and become an author. We have potential authors, often called _associates_, who are involved in real-time and those readers who view the journal output historically. These historical readers do not engage in knowledge creation, but treat the journal as a _book_, a source of received wisdom. These readers only see articles when they explicitly search for information. Associates, however, receive new issues immediately and must register secure pseudonyms, so they can engage in reputation development. Except in rare instances, both groups of readers can see the same articles, so we will not distinguish among them below. Therefore, the terms reader, author, and referee will refer to the same persons playing different roles in the knowledge creation cycle.

Assume, for simplicity, that referees send reviews to a consensus server, a mediator system. At a deadline, the server performs calculations and issues invitations to the referees who have been selected as new authors (Figure 1). These calculations are implicit negotiations, that is, they predict which persons would have been selected to respond to the reviewed article if referees had actually negotiated and reached a consensus. One benefit of consensus journals is that the negotiation process is automated, thus saving participant effort.

------------------------------------------------------------------ --

------ R: Review ------- M: Invitation ------ - | Read |----------->| Calc. |--------------->| Write |-----------> ------ ------- ------- R: Article | | R: Renege R = Referee Calc. = Calculate consensus | M = Mediator V

------------------------------------------------------------------ -- Figure 1. Simplified cycle of operation for a consensus journal

The simplified cycle of operation for a consensus journal shows actions in boxes and messages as arrows. In this simplified cycle, referees invited to publish (and justify the reviews they have submitted) have a choice of submitting their article by a deadline or reneging on the promise implied by their review. This simplified cycle of operation assumes, additionally, that consensus positions can be calculated and that published articles are retained indefinitely. Eliminating these assumptions requires a more articulated cycle of operation and additional message types.

Before considering a more articulated cycle of operation, however, we should to note an important feature of peer review that contributes to impartial judgment. This feature is a protection mechanism, typically anonymity, that shields referees from pressures that might be associated with evaluation of a colleague. Further, names and affiliations of authors are often hidden from referees to ensure that only article content is the basis for evaluation. Protection can alternatively be provided by a pseudonym system. This has the advantage of reducing opportunities for irresponsible behavior as compared to systems based upon anonymity (Stodolsky, 1990a). It has a further advantage of permitting reputation development through the refereeing alone, thereby making it possible to establish a reputation without contributing articles. This is advantageous, because there is no necessity to publish work which is not essential to knowledge creation. The effect is to make it easier for the reader to find truly important articles.

When there are multiple referees, it is important that their judgments are independent, so referee reports must not be made available until all have been submitted. This last requirement can be met by ensuring that reviews transmitted to the mediator are hidden until the deadline. The dynamics and implementation of protection systems are beyond the scope of this article, so only the necessity for the simultaneous release of information is addressed here.

Definition of message types

In the simplest case, messages in the consensus journal environment consist of only articles and reviews (Stodolsky, 1990a). However, considerations of effective negotiation and of storage management, suggest defining additional message types. There are five types of messages transmitted in the consensus journal environment.

_ Article

Articles, while shorter than those usually seen in conventional journals, will most often play the same role. However, it is quite possible to have an article in a consensus journal that is only a few lines long, and that can only be understood in connection with the review message it follows and its target article.

_ Review

Review messages must be distinguished from conventional reviews because they are characterized by a vector of numbers that summarize a reader's reaction to an article. If we think of articles as nodes in a graph or pages in a hypertext network, then review messages are the labels on arcs or links that connect the articles. Reviews can go beyond merely evaluating an article, by offering to provide new information that may be essential to support the target article's position. Review messages also serve as a commitment to deliver a justification of the referee's report, if invited.

_ Invitation

Invitations are public, and therefore, impossible to refuse without some loss of reputation. This makes them somewhat different than invitations from an editor of a journal. In effect, the invitation says, "We offer you storage space for an article." Also, a person may post an invitation for themselves during the negotiation stage of review, if they feel confident they can support the position claimed in their review message.

_ Cancellation

It is possible for an author to cancel an article, thereby releasing the associated storage space. The article then goes off-line (i. e., out of print) along with its reviews and the articles that were dependent upon it for their place in storage. This would typically occur during explicit negotiation, after the author had seen the article's reviews. It could, however, occur much later, when a new criticism was delivered.

_ Withdrawal

Finally, during explicit negotiations, a review may be withdrawn. This eliminates the referee from those prepared to respond to an invitation.

Cycle of operation

The sequence of events with a consensus journal is the same as with an invitational journal. The review method, however, involves the entire readership, or at least those who offer a judgment. New authors are then selected based upon the review judgements. While most articles will follow from reviews and be connected to their target articles, independent articles are also permitted. However, articles posted without consensus-based invitations are less likely to be read and cannot be assumed to have the support of others.

If we assume that a consensus journal is already functioning, we can follow the events through a cycle of operation that starts with reading of an article. While it is not essential for smaller readerships, we assume that participants exchange information electronically.

All readers are presented with the target article at the same time. A reader offers a review judgment in order to be considered for future authorship. The review message must be received before a certain deadline, say one week later. The review message consists of scores along several preselected dimensions. For instance, a scientific article is expected to be relevant, correct, and original. A more conversational approach might include the dimensions completeness, clarity, and appropriateness.

At the deadline, the mediator runs a statistical procedure to determine if there are consensus positions among the referees. The most central referee from each of these positions is invited to submit a new article. These most central referees are also considered most knowledgeable, within the framework of cultural consensus theory. D'Andrade (1987) discusses the evidence supporting this view.

Cultural consensus theory is based on the assumptions of common truth (i. e., there is a fixed answer pattern _applicable_ to all referees), of local independence (i. e., the referee-dimension response variables satisfy conditional independence), and of homogeneity of items (i.e., each respondent has a fixed _cultural competence_ over all dimensions) (Romney, Weller & Batchelder, 1986). Results can be obtained with as few as three respondents, but four are required if the significance of the results is to be calculated (i. e., a degree of freedom is then available in the statistical model) (Batchelder & Romney, 1988). A recent development in the model is the ability to identify two consensual groupings within the population of respondents (Romney, Weller & Batchelder, 1987) This is extremely helpful, since it permits a minority to publicize their viewpoint under the same conditions as a majority.

Cultural consensus theory assumes that we have no _a priori_ knowledge about referees, that is, they have no reputations. This is extremely valuable when a new topic comes up or when there are violations of the assumptions required for calculations concerning a current article based upon previous information (Stodolsky, 1984b). Given that reputations have developed and assumptions are satisfied, however, the theory requires elaboration to be applied most effectively. Cultural consensus theory provides, in effect, a crosssectional estimation of competence. That is, given a sample of responses at a given moment, relative competence is estimated. On the other hand, given a performance history, Bayesian estimation can be used to assess the relative importance of different persons' judgments. That is, there are reputations that give information about relative competence independent of the current responses. This assumes stationarity: that the same area of competence is required for valid response, and that responses are generated in the same manner (e. g., respondents continue to give honest answers). Both methods are based upon likelihood estimation; therefore, a combined theory should be achievable. The combined sources of information are likely to make achieving an implicit consensus more common.

The mediator releases a review report showing issued invitations, submitted judgments, the degree of consensus achieved, the number of consensus positions identified, competence of each referee, and so on. If a consensus has been identified, invited referees are expected to submit articles before the next publication deadline.

Negotiation must proceed explicitly if no consensus can be identified (Figure 2). In that case, referees may look at the judgments submitted and decide if their positions have sufficient support. If not, they could reconsider their review judgments, and either revise them or withdraw from the review process. The author of an article might, on the basis of these judgments, cancel an article, thus avoiding publication and potentially reputationdamaging criticism.

------------------------------------------------------------------ --

------ R: Review ------- M: Invitation ------ - | Read |----------->| Calc. |--------------->| Write |-----------> ------ ------- ------- R: Article ^ | | R: Review | | M: No consensus | R: Renege | | | | V V ---------- - R: Withdrawal | Reconsider|--------------> ---------- - | | A: Cancellation | V

Key

A = Author M = Mediator Calc. = Calculate consensus R = Referee ------------------------------------------------------------------ --

Figure 2. Cycle of operation for a consensus journal

(A referee becomes an author only after a submitted article has been published by the mediator [not shown in figure]).

Assuming that the article was not cancelled, the combined effects of withdrawal by referees with most deviant judgments and reconsideration by others, would probably lead to consensus, particularly if the requirements for consensus were successively relaxed. This assumes that revision of judgments would be in the direction of dominant viewpoints, a common finding. The result would be an invitation issued by the mediator to selected referees. Subsequent submission of articles by selected referees and their publication by the mediator would complete the cycle of operation.

The invitation report can guide negotiation when a consensus cannot be identified. Individual invitation staging could proceed along with a relaxation of requirements for consensus. For instance, if the first round of reviews did not generate a consensus, referees could issue invitations to others (Table 1). If the second round of reviews did not generate a consensus, referees could issue selfinvitations (these would be acceptances for those who had received invitations), or perhaps, direct the invitations they had already received to others. Failure on the third round would permit these previously issued invitations to serve as a coordination mechanism. That is, certain referees would have indicated a readiness to respond, and others would have rejected the option of authorship unsupported by a consensus. Thus, duplication of effort could be avoided by examining the ranking of persons in terms of the invitations received and accepted, and then responding accordingly.

------------------------------------------------------------------ --

Message Level and (Receiver):

Public Private (Readers) (Mediator)

Stage of Process:

Publication Article

Refereeing Review(1)

Invitation Report

Reconsideration Review(2), Withdraw, Cancel, Invite

Invitation Report

Reconsideration Review(3), Withdraw, Cancel, Self invitation

Invitation Report

Submit Article

------------------------------------------------------------------ --

Table 1. Message level, receiver, and type, by stage of processing (assumes no consensus reached during negotiation)

New articles are requested either by the mediator or by referees, if an author and referees follow negotiations to completion. New articles must be submitted before a deadline. At the deadline, the new articles received are distributed. This makes them available to the readership and completes the cycle of operation for a consensus journal.

Thus in the simplest case, articles are read, reviews are transmitted, invitations are issued, and new articles are submitted in a timely manner. In the extended cycle, at least a single reconsideration or negotiation stage occurs during which a target article can be cancelled by its author and during which referees can withdraw. A failure to achieve consensus leads to explicit negotiation and options such as nonconsensus invitation. The extended negotiation option makes the consensus journal more similar to a conventional invitational journal, because there is explicit negotiation prior to the writing of an article.

It is only in the case of a failure to identify a consensus that an article continues to be held _in review_. In the worst case, this in review status could last two additional publication cycles. By this definition, an article is not considered published until it has become the target of another article. That is, publication has occurred once a referee has been invited to prepare an article in response to that of the author.

The quality of articles can be assessed by the degree of consensus achieved. Readers might select only those articles resulting from a consensus-based invitation, thereby controlling the quality of articles they see.

Rules of dialogue

The rules of operation of a consensus journal can be thought of as specifying an action system, or language game, where the actions relate to the placement of articles in a network of interconnected nodes. Participants in the game try to maximize their influence. Reputation is a crucial resource in scientific argumentation (Smolensky, Fox, King & Lewis, 1988). Participants are expected to maximize this resource. While there may be other payoffs available within a given system, such as royalty payments, this discussion assumes only reputation maximization as an individual objective.

There are several opportunities for reputation enhancement in the cycle of operation. Selection as an author is a major opportunity for reputation enhancement. However, refereeing also offers significant opportunities not available with conventional journals. Referees can commit themselves to delivering a rebuttal to an article and thereby improve their reputation (assuming they make good on their commitment, given an opportunity). If an author examines the reviews an article receives and decides to cancel it before a rebuttal is written, the referees offering rebuttals would have their reputations enhanced, without any further risk or effort.

With a consensus journal, the review message can be thought of as an offer to deliver a certain type of article before the deadline. Obviously, a review message that claims a target article is erroneous, and thereby offers to deliver a rebuttal, plays a different structural role in a debate than one that criticizes an article for not being original. Thus, reviews can have a great deal of structural impact and can express a level of commitment which would not be relevant in an environment that limits referees to a gatekeeping role.

Structural Aspects

With electronically published documents, it is highly desirable to structure interconnections so that retrieval is facilitated and the relevance of statements becomes clear (Smolensky, Fox, King & Lewis, 1988). Thus, review messages can deal not only with the quality of an article, but also with its relationship to its target article. Explicit relationships among articles becomes more necessary as the size of articles decreases and number of articles increases.

With conventional journals, reviews are used to determine whether or not an article should be published. The publication decision is not dominant in computer networks, however, since distribution constraints are greatly relaxed (Quarterman, 1990, p. 259; Stodolsky, 1990b). Because of this, the period during which an article remains on-line assumes importance, because storage is limited. It is in this connection that the reviews of articles and the relations between articles become critical. In the simplest case, an article that is found incorrect by an overwhelming consensus is cancelled by its author. Failure to cancel an article will result in a continuing devaluation of the author's reputation, as more and more readers come to agree with the majority. In the case of conflicting consensus positions, a rebuttal claiming that a target article is flawed is explicitly linked to the target. Failure to rebut that claim in turn has much the same effect as an overwhelming consensus that the target article is incorrect. Most interactions, therefore, take place at the knowledge frontier, as various positions are argued. These interactions generate very _bushy_ argument trees that require sophisticated navigation strategies, if large amounts of effort are not to be expended unnecessarily (Stodolsky, 1984a). The trees are thinned in the process of argumentation. Positions that are sustained remain on- line until they are thoroughly integrated into summaries or overarching theories.

A common question about consensus journals is how quality can be ensured if all articles are published. As we have seen above, a troublesome article, one which does not yield an immediate consensus, may be held in review for a couple of publication cycles, during which time referee's reactions become apparent and the article may be cancelled. The key to understanding this feature and the overall operation of a consensus journal is that the system is structured on the basis of author reputation, not just upon the basis of article quality. Certain scientific societies already use this approach, in that publication in certain journals by society members is expedited and subject only to cursory review. The cancellation mechanism that permits articles to be taken out of print can even be used by an author after publication. In the long run, almost all articles would be cancelled. Later work always shows the errors of earlier articles, and the authors would be stupid to continue to defend faulty ones. Finally, powerful reading programs that integrate reputation information could hide faulty articles that have not been cancelled.

Let us consider the various stages at which quality control is exercised. Potential authors must register as associates prior to submitting articles. This is a first (self-) selection stage for authors. After a target article is submitted, it is distributed to associates who have a cycle time to respond with evaluations. Only those who feel confident they can critique the article respond. This is (self-) selection stage 2. A consensus calculation then results in invitations. This is selection stage 3 and may stimulate articles from persons who otherwise would not have thought of submitting them. Given a consensus is identified, we can say with some confidence that those invited are competent to comment on the target article. New articles, justifying the evaluations are either submitted or the potential authors renege after closer consideration, this is (self-) selection stage 4. If a new article's evaluations don't generate a consensus, because model assumptions are violated, the author may cancel the article, that is, it is never published -- seen by anyone other than the associates. This is evaluation stage 5 and (self-) evaluation stage 6. Troublesome articles, those that still do not generate a consensus may go around this loop again. Finally, at any time after an article is published, the author may decide that criticisms of it are too strong, and cancel the article, (self-) evaluation stage 7, at least. At each consensus calculation all evaluators receive reports placing them in a multi-dimensional competence hierarchy. This means that self- selection functions upon a highly informed basis. Persons who ignore this information damage their reputations as associates and perhaps their public reputations. Secure pseudonyms guarantee that reputations "stick" to the anonymized associates.

Consider that each article, which is normally about a page long, is the target of say thirty evaluations. A ten page article in a traditional peer reviewed journal may get three reviews. With a consensus journal, the equivalent would get three hundred reviews. (If the article was controversial, we could expect continuing evaluative activity, unless or until it was cancelled.) This is what is meant by _evaluative intensity_, and it is why article quality would be much higher in the consensus journal format. We do not consider here the systematic biases that normally are involved in article selection with traditional journals.

Citation indexes were developed to facilitate locating new articles which might correct flaws in already published material. This works far from flawlessly and it requires the reader to evaluate the validity of criticisms in the new articles. Faulty articles can be expected to be cancelled, within the framework of a consensus journal. In any case, powerful reading tools would permit the mobilization of all reputation information of the associates in determining the quality of any article. Thus, even a non-specialist could make informed, up to date judgements about article quality. In fact, the reader would typically be spared the identification of individual articles and assessment of their quality, since a request, especially by a non-specialist, would select a _thread_, and integrate articles from a certain discussion into a single high quality presentation for reading. The combination of cancellation of faulty articles and automatic utilization of reputation information, would make the apparent quality of a consensus journal much higher than any traditional format. The automated tools for authors and readers would also make the journal easier to use effectively.

Decentralization

A central mediator has been assumed in this description to simplify explanation. There is no reason why the calculations necessary to select new authors could not be performed decentrally. In fact, this would be necessary if readers preferred different methods of calculation for author selection. Then coordination in the selection of new authors would be shifted from consensus calculation to collection of invitations. Various types of voting rules could be applied. Authors receiving the most invitations would then be expected to submit articles. Thus, decentralization leads to an integration of the two types of invitations (consensus and individual) already discussed.

The task of protecting review judgments until the deadline is reached is another required function. It is necessary, for example, because analysis of earlier submitted judgments could permit a referee submitting at the last moment to simulate a competence that did not exist, thus violating assumptions of the model. Protection can, however, be achieved decentrally using cryptography, assuming a _beacon_ that emits enciphering and deciphering keys at fixed intervals (Rabin, 1983). Use of cryptography would be necessary, in any case, to ensure the authenticity of messages and security of pseudonyms.

Applicability

Many questions have been raised about the applicability of the suggested format. First, it should be kept in mind that there are a wide range of parameters that can be adjusted to fit different user groups. Most important are cycle time, degree of consensus and degree of satisfaction of various model assumptions needed to avoid explicit negotiation, and even the model used to determine consensus. The main limitation is to short, conversationally- oriented articles. However, we maintain that this format is closer to the way science is really done than traditional articles. Traditional published papers are several times removed from the actual "doing" of science. To the working scientist, the interactions mediated by a consensus journal would appear much like conversations which occur at scientific meetings or with local colleagues. These interactions are where new ideas first appear and are tested. The historical reader, on the other hand, would not notice a great change, since individual articles would be automatically assembled into presentations from discussion threads.

A related question is how laboratory science journals could work effectively with the new format. This question is usually a result of a failure to see that the new format incorporates much more of the scientific process than does the traditional journal. The consensus journal format would result in laboratories being "invited" to undertake certain investigations. This already happens in cohesive scientific fields like physics, where scientists may specialize as theoreticians. Their work often shows what experimental work must be undertaken, if the field is to progress. Experimentalists then actually perform the work and report their results.

Another objection raised is that articles would be very uniform and people would loose interest. Such perceptions are most often a result of failing to understand that there can be more than one consensus position, and that a consensus is on the level of terms, not on the level of ideas or theories. In any case, excessive uniformity might be a problem within a thread, but not in general. The level of conflict is tunable, so interest should be maintained. Also, when consensus breaks down, it is often an indication that an old line of discussion must go, and be replaced by a new one. Finally, with the cultural consensus model, only two consensus positions can be identified. Therefore, we would expect that there would be constant replacement of old threads as new consensus positions were established. Some tuning of parameters might be necessary to establish a balance between threads continuing too long, on one hand, and being replaced before adequate evaluation of claims occurred, on the other.

Yet another common objection is that the model is not applicable to an area where most readers are practitioners, while articles are typically written by researchers. First, it is likely that practitioners would be historical readers, and not participate in real-time, so the possibility of their reviews overwhelming those of the researchers could not be a problem. Even if they did choose to participate as referees, however, it is unlikely that this would cause any problems. Practitioners would have a less precise grasp of technical terms and therefore would generate inputs less central to a consensus compared to researchers. They would also be less likely to spot a methodological flaw. Therefore, a researcher would in all likelihood be invited to author. If by chance, a practitioner was to author an article, it would probably fail to generate a consensus, and if it did, it would most likely be demolished by the researchers in short order and cancelled. People who overestimate their competence damage their reputations and thereby their ability to influence future invitations. So, any undue influence by less competent readers would rapidly decrease, if it was a problem in the first place.

Bookstein (1991) suggests that the format could work well only in a relatively contained area of considerable turbulence, but also considerable altruism regarding new ideas. The format would probably work best in contained areas, such as subfields of the physical and mathematical sciences, where there is a high level of consensus on the meaning of terms. A high level of turbulence would create a degree of variation in referee reports which would make it easier to achieve a consensus through calculation, as long as it didn't get too high, that is, chaotic. However, the altruism regarding new ideas is probably not essential, because of the strong reputational gains available to those who first switch to eventually accepted positions. Also, the anonymization of inputs would reduce motivation to defend one's position dogmatically. While Bookstein's suggestion can help identify an area which might be appropriate for a trial of the new format, the tunability of the model makes its extension to other areas likely.

A transitional model

The closest thing to a consensus journal currently is _Psycoloquy_, an on-line journal with open peer commentary. The consensus journal format would substitute evaluations and calculation for the initial peer review used there to select target articles. That journal prequalifies associates who may publish commentaries on target articles, which are subject to only nominal review. This parallels the registration of potential authors in the consensus journal format.

A yet unanswered question is how the evaluative dimensions used by referees would be selected. This is covered in presentation of a transitional model in which editorial staff alone participate in generating invitations (Stodolsky, 1992). After reading a published article, potential authors submit short (e.g., single paragraph) reactions to the target article. These reactions are treated as proposals for new articles and directed to a jury of editors. The editors then independently select a subset of the proposals, thought to be mutually exclusive. The judgment of mutual exclusivity can be performed by sorting of proposals into groups. Within each group, the potential authors would be addressing the same question. Between groups there would be a significant difference in what question was being treated (or how it was to be treated). Discrepancy among groupings by editors would require a consensus set of groupings to be selected. Invitations would then be issued based upon these groupings and estimated author competence.

Given a history of operation for this journal structure based upon editorial consensus, the groupings made by editors could be used for generation of dimensions for evaluative judgments. A reliable set of such dimensions would permit more efficient sorting of proposals within the framework of an invitational journal based upon editorial consensus, and would also permit the testing of the more powerful invitational structure based upon peer consensus discussed here.

Summary

A consensus journal requires mechanisms for both coordination and protection. In the simplest case, a mediator can provide these. This assumes protected channels of communication and a trusted mediator. Coordination is necessary to identify consensus positions and avoid duplication of effort. Protection of reviews is necessary to ensure that assumptions of models for evaluating expertise are not violated. This protection allows reputation development by both authors and referees. Such reputations can then be used to ensure effective allocation of expertise. The extension of review opportunities to the entire readership vastly extends the available field of expertise. This, combined with the effective allocation of expertise and coordination that eliminates duplication of effort, provides consensus journals with a significant advantage over current mechanisms for enhancement of message quality.

References

Batchelder, W. H. & Romney, A. K. (1988). Test theory without an answer key. _Psychometrika_, _53_(1), 71-92.

Bookstein, F. L. (March 25, 1991). Can PLS be applied? Personal communication.

D'Andrade, R. G. (1987). Modal response and cultural expertise. _American Behavioral Scientist_, _31_(2), 194-202.

Kolata, G. (1990, June 26). In a frenzy, math enters age of electronic mail. _New York Times_, pp. C1, C10.

Okerson, A. S. & O'Donnell, J. J. (Eds.). (1995). Scholarly journals at the crossroads: A subversive proposal for electronic publishing. Wash. DC: Association of Research Libraries, Office of Scientific and Academic Publishing. [ISBN 0-918006-26-0. Raw source files: URL: ftp://pub/harnad/Psycoloquy/Subversive.Proposal]

Quarterman, J. S. (1990). _The matrix: Computer networks and conferencing systems worldwide_. Bedford, MA: Digital Press.

Rabin, M. (1983). Transaction protection by beacons. _Journal of Computer and Systems Science_, _27_(2), 256-267.

Romney, A. K. , Weller, S. C., & Batchelder, W. H. (1986). Culture as consensus: A theory of culture and informant accuracy. _American Anthropologist_, _88_(2), 313-338.

Romney, A. K. , Weller, S. C., & Batchelder, W. H. (1987). Recent applications of cultural consensus Theory. _American Behavioral Scientist_, _31_(2), 163-177.

Smolensky, P., Fox, B., King, R., & Lewis, C. (1988). Computer-aided reasoned discourse or, how to argue with a computer. In R. Guindon (Ed.), _Cognitive science and its applications for human-computer interaction_. Hillsdale, NJ: Lawrence Erlbaum Associates.

Stodolsky, D. (1984a). Commonalities among [sic] conferencing systems and their implication for marketing strategy. _Organisatoriske Fragmenter 1984_, _12_, 43-58.

Stodolsky, D. (1984b, December). _Self-management of criticism in dialog: Dynamic regulation through automatic mediation_. Paper presented at the symposium Communicating and Contracts between People in the Computerized Society, Gothenburg University, Sweden.

Stodolsky, D. S. (1990a). Protecting expression in teleconferencing: Pseudonym-based peer review journals. _Canadian Journal of Educational Communication_, 19, 41-51. ([1989, May 9]. _Communication Research and Theory Network [CRTNET]_, No. 175 [Semi- final draft available by electronic mail from LISTSERV@PSUVM.BITNET and COMSERVE@Vm.ecs.rpi.edu])

Stodolsky, D. S. (1990b). Archiving secure interactions. _Psychological Science_, _1_(6), 353-354. ([1990, May 25]. Comments on Gardner's Electronic Archive by Stodolsky. _Psycoloquy_, _1_[8]. [Available by anonymous FTP from PRINCETON.EDU in directory /pub/harnad])

Stodolsky, D. S. (1992). Invitational Journals based upon Editorial Consensus: A New Editorial Role in Electronic Journal Publication. Psycoloquy, 3(67). (Available by anonymous FTP from PRINCETON.EDU in directory /pub/harnad)




Last update: Wednesday, May 28, 2003 at 1:15:21 PM.