tl;dr – JOSS is introducing new submission criteria whereby submissions under 1000 lines of code will automatically be flagged as potentially being out of scope, and those under 300 lines desk-rejected. This blog post describes some of the motivations behind this decision. Sometime in 2020, JOSS will publish its 1000th paper – an incredible achievement by a volunteer team of editors and reviewers. Since its inception a little over four years ago, the primary goal of JOSS has always been to provide credit for authors of research software. Quoting from the original blog post announcing JOSS: The primary purpose of a JOSS paper is to enable citation credit to be given to authors of research software. One challenge we’ve always struggled with as an editorial team is defining clear guidelines for submissions allowed (and not allowed) in JOSS. Our current submission criteria are available online and include language about software having to be a significant contribution, feature complete, and having an obvious research application. In these criteria we also explicitly exclude a category of software we generally call “minor utilities”. The challenge of defining a unit of publication credit We think of JOSS essentially granting “1 publication credit” for each software package that is reviewed and published in the journal. In empirical research, a publication is often the result of years of work. In engineering research, rarely does a paper represent less than one year of work. Other fields may vary, but let’s say that a scientific paper resulting from work measured in just months is rare or exceptional. Since the earliest days of the journal, there has been a range of views within the editorial team on what level of effort we should require from authors for a submission to be allowed in JOSS – some JOSS editors feeling that every useful piece of software should be considered, others believing that the “bar” for publishing in JOSS should be higher than it currently is. Building trust in JOSS As an editorial team we want JOSS papers to count the same as any other publication in the CV of researchers who write software. With career credit the stated primary reason for starting JOSS, if this isn’t true then the mission of the journal is at risk. In reality this means that our editorial policy requires us to balance two competing needs: Providing an excellent service to authors by offering peer-review of their software and the opportunity to receive career credit for their work. Building the trust of an existing academic culture to accept a JOSS paper as equal to any peer-reviewed journal paper. These two aspects are in tension with each other because while we would dearly love to publish any and all research software in JOSS regardless of size/scale and level of effort to implement, part of building and maintaining that trust with the community relies on us ensuring that our published authors can continue to expect JOSS papers to “count” in their future merit reviews and promotions. Updates to our submission requirements and scope-checking procedures Over the past couple of years, as the number of submissions to JOSS has grown, we’ve found that our existing submission criteria and protocol for rejecting papers as out of scope have been taking a significant fraction of our editorial team time. With a volunteer team of editors, it’s essential that we use their time carefully and an update to our procedures for handling scope assessments is long overdue. Going forward we’re going to adopt the following new process: Automatically flagging small submissions As part of the pre-review process, incoming submissions that are under 1000 lines of code1 (LOC) will be automatically flagged as potentially out of scope by the EiC on rotation. Submissions under 300 lines2 of code will be desk rejected with no further review. Mandatory “Statement of need” section in JOSS papers While this has always been preferred, a clear Statement of need section in the JOSS paper is usually extremely valuable in helping the editorial team understand the rationale for the development of the software. Gauging the scholarly content of the software as part of the review Reviewers will be asked if the software under review is a “substantial scholarly effort” and guidelines will be provided on how they can make that assessment. A streamlined editorial review process Rather than each paper that is potentially out-of-scope being discussed in a separate thread on our editorial mailing list, JOSS is going to move to a weekly review of such papers by our editorial team. Topic editors will be asked to review papers flagged as potentially out of scope in their area and help the EiC team make a decision. References Smith A. Announcing The Journal of Open Source Software - Arfon Smith. Published online May 5, 2016. Accessed July 7, 2020. https://www.arfon.org/announcing-the-journal-of-open-source-software In a high-level language such as Python or R. More verbose languages such as Java, C++, Fortran etc. will require more LOC. ↩ We realize that introducing numerical thresholds may encourage some authors to unnecessarily “pad” their submissions with additional lines of code to meet our thresholds. As reviewers are already asked to judge the standard of the implementation as part of their review we expect that situations like these will usually be flagged during the review. ↩

On March 12th of this year, we suspended new submissions to JOSS in order to reduce the load on our volunteer team of editors and reviewers. We revisited this suspension in early April and decided to continue the pause on new submissions for at least another month. Over the past two weeks, we’ve been reviewing our current status and discussing what it might look like to reopen JOSS.

Once again we’re looking to grow our editorial team at JOSS. We’re especially interested in recruiting editors with expertise in bioinformatics, material science, physics, R/statistics, and the social sciences. Since our launch in May 2016, our existing editorial team has handled over 800 submissions (830 published at the time of writing, 119 under review) and the demand from the community continues to grow.

2019 has been a big year for JOSS: This year, we’ve already published 300 papers and are on target to reach ~360 papers by the end of the calendar year. We’ve substantially improved our website and made a number of important changes to our editorial team to help us scale further.

Following our call for editors late last year, once again we’re looking to grow our editorial team. Over the past ~40 months, our existing editorial team has handled close to 800 submissions (665 accepted at the time of writing, 101 under review) and the demand from the community continues to grow. The last three months have been our busiest yet, with JOSS publishing a little over one paper every day, and we see no sign of this demand dropping.

The Journal of Open Source Software (JOSS) started with one editor-in-chief (EiC) and 10 topic editors, and in our first year, we published about 100 papers. Three years later, the rate of publication has increased to 300 papers per year. How did we scale up to this point, and how can we continue scaling the journal? Scaling to-date Today, we have one editor-in-chief (EiC), three associate editors-in-chief (AEiCs), and 23 topic editors.

The Journal of Open Source Software (JOSS) is a free, open-access online journal, with no article processing charge (APC). We are committed to operating as a free service to our community, and we do so thanks to the volunteer labor of editors and reviewers, and by taking advantage of existing infrastructure.

JOSS is an adventure in next generation publishing, made possible by the volunteer work of many people. Our editors, of course, guide the style and the content of the journal. And our reviewers make a uniquely valuable contribution, both to the software they’ve reviewed and to the broader community of open-source research software. Some reviewers have been extra generous in contributing.

JOSS is expanding its editorial board and we’re opening this opportunity to the open source research software community at large. If you think you might be interested, take a look at our editorial guide which describes the editorial workflow at JOSS and also some of the reviews for recently accepted papers. Between these two, you should be able to get a good overview of what editing for JOSS looks like.

Today we’re starting something new at JOSS: we’re collaborating with American Astronomical Society (AAS) Journals to offer software reviews for some of the papers submitted to their journals. As part of this process, AAS Publishing will make a small contribution to our parent organization NumFOCUS to support the running costs of JOSS.