RJJR: Reviews The Journal of Journal Reviews

Rubric

PDF Version

About the Rubric

This Rubric for journal evaluation is meant to encourage context-centered reviews of journals, referencing established ethical publishing guidelines, like the Committee on Publication Ethics's Principles of Transparency and Best Practice in Scholarly Publishing. The rubric is organized into several areas of evaluation, including the transparency of the journal, the journal’s policies and observable actions, people connected to the journal, the material published by the journal, and the journal’s relationships to indexes and professional societies. The rubric, which guides authorship of Journal Reviews, Peer Reviews, and editorial review, provides a framework to encourage objective observation of available evidence and to avoid subjective value judgments about a journal. This rubric is openly available here, so that it might be properly vetted itself, and is licensed for reuse, so others may modify it for their own purposes.

General guidance

  • These questions cover a range of practice that may be unreachable for many journals. The rubric is meant to give a comprehensive set of factors to consider, not a checklist, that determines the success or failure of a journal.
  • Not all of these questions apply to every journal.
  • The absence of information is not inherently bad. 
  • Not every reviewer will be an expert in every area included in this rubric. Reviewers should be transparent when they don’t feel qualified to evaluate a particular element.
  • If you have comments or other elements outside of this rubric that you feel are relevant, add them!
  • Be clear when you’re referring to evidence vs anecdotal/personal perceptions
  • Capture a snapshot of evidentiary webpages using Internet Archive’s Wayback Machine’s Save Page Now feature as may be helpful [enter URL and click “save page”]. Use these links when providing evidentiary material.
  • If you need to provide images, please include clear and descriptive alt text.
  • The best person to make a value judgment is the author considering submission. These reviews are meant as a tool to help authors make an informed decision, not necessarily the decision itself.
  • Please share evidence to support your statements. If you are unable to provide evidence, please clearly indicate that your statement(s) are anecdotal or personal.  
    • The journal has a preference for publicly available information, when making factual statements, please provide evidence for otherwise inaccessible (internal) documentation. Wherever possible, use links to saved pages in the Internet Archive’s Wayback Machine to document the site as it appeared during your review.
  • Reviewers should work through this rubric as a starting point, then write a cohesive review based on the pertinent responses. They may not respond to each question, yet reviews should respond to each category area, at a minimum. 
  • Reviews should be written in a narrative fashion, rather than as bullet points or a set of discrete statements.

Journal Information

  • Journal Name
  • Journal URL
  • ISSN (if applicable)
  • URL for the Wayback Machine snapshot
  • Date of Snapshot

Reviewer Information

  • Your name
  • Your role (student library worker, librarian, faculty, etc.)
  • Bias Statement:
    • Please briefly describe any potential conflicts of interest or biases which may shape your review. Remember, everyone has biases, and conflicts of interest do not necessarily disqualify your review. See the Conflicts of Interest policy for more information.
  • Optional: Reviewer’s Motivation:  
    • Please provide 1-2 sentences describing how you came to review this journal?
    • Things to consider: 
      • Interested in the journal, considering submission, reviewed on behalf of someone else, have published in the journal, positive (or negative) experience with the journal, an arbitrary choice
  • Optional: Areas of Expertise
    • If you have particular expertise which is relevant to your review, briefly describe it. For example, this might be subject-specific research experience, expertise in digital accessibility, or language fluency.

Transparency of Practice

  • Does the journal’s website clearly include an aim and scope statement with a clearly defined readership? (Briefly summarize this in your review.)
  • What persistent identifiers does the journal utilize? This includes, at the journal level, ISSN; if the journal publishes both online and in print, there will be two different ISSNs) author level (ORCID, etc.) and at the article level (Digital Object Identifier DOI)?
  • Does the journal’s website clearly include the process and method of peer review used, if any?
    • Speed of process, guidelines for peer-reviewers, experience of authors, stated timeline, experience as a peer-reviewer, etc.
  • Does the journal suggest time to publication? Do they report submission, acceptance, and publication dates? If so, do those dates align with the suggested timeline?
  • Does the journal’s website clearly include who owns and/or manages the journal? This may be a publishing business, a scholarly society, an academic institution, and etc. 
  • Does the journal’s website clearly include the editorial board, including full names and affiliations?
  • Does the journal’s website clearly include copyright policy and licensing terms, including who owns the rights to articles?
  • Does the journal’s website clearly include whether the journal charges fees for processing or publication (this includes stating if they don’t charge a fee)?
    • Is the payment process clear and secure?
  • Does the journal’s website clearly include the process for identifying and dealing with issues of research misconduct: 
    • Research misconduct (e.g., plagiarism, citation manipulation, falsified data)
    • Authorship, identifying contributors (e.g., who counts as an author?)
    • Handling complaints or appeals
    • Data sharing (for reproducibility research data sharing is encouraged)
    • Post-publication discussions, revisions, or corrections
  • Does the journal's website clearly include policies relating to publication ethics, including: authorship and contribution, handling of complaints and appeals?

Behavior

  • Does the journal clearly state its policies? Does the journal list all the policies required by COPE’s core practices?
  • Are you aware of any additional information that has been emailed to authors, especially that contradicts or confirms publicly available information?
  • Do you have personal experience working with this journal, or supporting authors working with this journal? What does that experience suggest, especially if your experience contradicts or confirms the stated policies and practices of the journal?
    • How would you describe the aggregate behavior of the journal(‘s staff): is the aggregate behavior suggestive of a consistent and transparent approach to publishing? Is it courteous, helpful, aggressive, neglectful, overly solicitous,or inconsistent? 

People

  • Who sits on the journal's editorial board? Do they appear to be experts in the journal’s field? Do they represent the journal’s stated geographic scope?
    • Note: please refrain from making assumptions about people involved in the journal. This rubric item is about the information provided by the journal.
  • What do you notice about authors who publish in the journal? (E.g., multiple articles by one person in a single issue, authors mostly from one country in an "international" journal)
  • Can you figure out how to reach an actual person with question(s) about the journal? For example, can you find an editor's name with a telephone number or email address?

Equity, inclusivity, and accessibility practices

This is an emerging area of practice for most journals/publishers and may not be available. By explicitly considering it here, Reviews seeks to encourage more journals to engage in this area of practice.

  • What, if any, information does the journal provide related to diversity among their editors or authors? Without making name-derived assumptions about race or gender, what does the evidentiary record of published material suggest?
  • Does the journal publish reports on their publishing practices, evidence of anti-racist or inclusive actions undertaken, statements of principles?
  • If you have direct experience with this journal, does their behavior indicate a commitment to equitable practice?
  • If the journal charges subscription or submission fees, what clear pathways exist to request a waiver or discount?
  • What information does the journal provide regarding the accessibility of their content?

Scholarship 

Recommendation: sample articles from the last 1-2 years of publications. You are not expected to read the entirety of every article, but it may be appropriate to browse the abstract, introduction, discussion, conclusion, and/or citations. 

  • Does the content of the journal indicate quality, consistency, and materials within the scope of the journal?
  • What is the actual material they publish like? Does it appear to fit their stated aim and scope? Does it appear to be copy edited and proofread? (Copy editing is the process of making an article more readable and checking for typographical, grammatical, and formatting errors.)
  • Do you notice any strange claims or other unusual characteristics in the articles? (E.g., does anything about the articles make you think "wait, that can't be right"?)
  • Do articles include contradictory rights statements? (E.g., a policy says one entity holds the copyright, but the copyright statement in the article says something else)
  • If you have first-hand experience with this journal (e.g., as an author or a peer reviewer), what was your experience with the peer review process? Did the process feel thorough? Rushed? Superficial? Positive?

Relationships 

  • Where does the journal say it is indexed? 
    • Are they actually indexed in those places? (E.g., if the journal claims to be indexed by a ProQuest database, can you confirm that?) 
    • Are any of those places selective or curated databases? (E.g., Google Scholar automatically indexes any and all online content; selective databases have collections curated by experts)
  • Is the journal published by a professional society or other organization? Is there any other relationship with an organization? 
    • If so, do they describe that relationship? 
  • Is there an affiliation with an institution, society and/or larger publisher?
    • If yes, what affiliation(s) exist?
    • Is there an existing controversy, critique, or validation of the affiliated body that would be relevant to include?
  • Are there irregularities in geographic location or contact information that call into question the journal’s transparency or integrity?
  • Does the journal or publisher state that it belongs to an organization for scholarly publishers, such as OASPA or COPE members?
    • Can you confirm this claim of membership on the organization's website?
  • Can you find any other evidence of how the journal interacts with the academic community? (E.g., conference sponsorships, other relationships with academic organizations)

Background and History 

Reminder: The RJJR rubric emphasizes clear information over value judgements. A journal may have a haphazard appearance or irregular publishing schedule and still be a definitive publication in its field. Another journal may appear highly polished and professional but be highly criticized in its field. 

  • What does the journal's reputation appear to be? (E.g., what kinds of conversation about this journal appear in an online search? Does the journal appear repeatedly in RetractionWatch.org?) 
  • Is the journal new or longstanding? 
    • Have they published more than two issues? Is there consistency across issues?
  • How consistent is the journal’s publishing history, in both the frequency and the number of articles?
  • Can past issues be accessed from the journal or publisher’s website? Are past articles open access?
  • Has this journal previously appeared in RJJR?