Daniel S. Katz

The end of last week (26 March) marked the start of my fourth and final year as a rotator at NSF.  It’s been a really good experience so far, and I’m a bit sad to see the end coming. I’ve really enjoyed being able to work with a wonderful set of people in the Division of Advanced Cyberinfrastructure and across NSF.

One year to go at NSF

The research community is moving towards the acceptance that data and software are both essential parts of understanding and reproducing many research (e.g., science, engineering, humanities) outcomes (e.g., papers, books, presentations) and are themselves valuable outputs that can capture and explain knowledge.  However, we don’t currently have a good set of practices to support reuse and sharing of data and software.

Data and Software Management Plans must be public and should be machine-readable

*This viewpoint is published jointly on software.ac.uk, hpcnotes.com (personal blog), danielskatzblog.wordpress.com (personal blog) under a CC-BY licence. It was written by Neil Chue Hong (Software Sustainability Institute), Simon Hettrick (Software Sustainability Institute), Andrew Jones (NAG), and Daniel S. Katz (University of Chicago &amp;

The Price of Open-Source Software – A Joint Response

While preparing for a workshop that the URSSI project is hosting on software credit today, I started thinking again about a recent blog by Titus Brown, “Revisiting authorship, and JOSS software publications.” Titus says, “fundamentally, in order to nurture a diverse array of valuable scientific contributions, we need new models of publication with new models of authorship,” a statement with which I strongly agree, and in fact, part of the

Contribution vs Credit vs Authorship for software

The FAIR data principles, defined as “a set of guiding principles to make data

 F

indable,

 A

ccessible,

 I

nteroperable, and

 R

e-usable,“ came out of a meeting in Jan 2014 that “brought together 25 high level participants representing leading research infrastructures and policy institutes, publishers, semantic web specialists, innovators, computer scientists and experimental

FAIR is not fair enough

I’m currently the Steering Committee chair for the IEEE International Conference on eScience series.  (Note that I’m going to talk about IEEE in this blog, but the problem is really much more general;

Attempting conference changes: Why moving away from traditional publishers for proceedings is difficult

(please cite this post as https://doi.org/10.59350/zme5j-0jp45)  I previously wrote a blog post, “There’s no such thing as sustainable research software,” trying to be somewhat provocative. In it I wrote:  However, after thinking about this a bit more, I don’t think this means we shouldn’t try.

Probabilistic research software project sustainability

Following Titus Brown’s blog post, A framework for thinking about Open Source Sustainability?, which follows Nadia Eghbal’s, An alternate ending to the tragedy of the commons, I’ve finally started reading Elinor Ostrom’s Governing the Commons. I urge you to read Nadia’s and Titus’ blog posts.

Following Ostrom-Eghbal-Brown into thinking about open source sustainability

While I was a program director at NSF, I led the SI2 (Software Infrastructure for Sustained Innovation) program that had been created by Manish Parashar and Abani Patra, then was led by Gabrielle Allen, before it was my turn to shape it and lead it for about four years, from 2012 to 2016. This program initially funded projects that developed and maintained software, as well as projects that planned and built community institutes around software.

By Daniel S. Katz, Daina Bouquin, Neil Chue Hong    Identification of software is essential to a number of important issues, such as citation, provenance, and reproducibility. Here, we are focusing on issues related to citation. Identification can be thought of as a subset of naming. Some important questions are therefore: How do we name things? How do we know how things are named? And who gets to name things?

Towards Software Non-Creator-Instigated Identification (NCI) and Citation

Over the holidays, I saw an interesting article on F1000Research  Soergel DAW.

Thinking about Open Peer Reviews

Make-or-buy decisions are common in business and research. They often involve an analysis of the pros and cons of creating something (e.g., a physical item, software, a service) vs. acquiring it from someone else. Factors that are considered can include cost, trust in suppliers, and current internal knowledge and skills as well as desired future knowledge and skills.

Make-or-buy decisions in research and research software

The latest Research Software Alliance (ReSA) newsletter includes the fact that: The 2018 European Commission report, Turning FAIR into Reality, concludes that FAIR digital objects (including software) need to be supported by metrics, incentives, skills and FAIR services that provide persistent identifiers, metadata specifications, stewardship and repositories, actionable policies and Output Management Plans.

Software and repositories in the context of FAIR

Eric Lander, who co-chairs the President’s Council of Advisors on Science and Technology (PCAST) for the White House, gave a great talk at the 2015 National Math Festival in Washington, DC on April 16th, called “The Miracle Machine.”  The full talk is available from this page, specifically as a Word document.  Eric’s abstract was:  Many governments are currently redirecting money for basic research into more applied areas.

Eric Lander’s argument for public funding of basic research

This is an expansion of a comment on Titus Brown’s blog post “Please destroy this software after publication. kthxbye.” which talked about how much work should go into software that was used in a submitted paper. In the past, I’ve thought of software as having one of two different purposes: Some software is just written for a single research purpose – this can be quick and dirty, as long as it does the immediate job.

Scientific Software Quality versus Purpose?

Open Science is a great concept (and recently described really nicely by Titus Brown, who said in part, “scientific progress relies on the sharing of both scientific ideas and scientific methodology.”) This is a wonderful ideal, and it describes how a lot of scientists would like to act. But as we know, most scientists don’t actually act this way, or at least not as strongly as they could. Why not?

Clash of cultures: Why all science isn’t open science

I recently led the writing of a report on software sustainability and high-energy physics: https://arxiv.org/abs/2010.05102. This report effectively has three groups of authors, with each group listed in alphabetic order: Katz – Stewart, the organizers of the workshop, leaders of the analysis and summarization, and lead authors of the report; Assamagan – Sexton-Kennedy, speakers in the workshop and authors of sections of the report;

Authorship Tiers

Based on some recent experiences, I want to try to redefine cyberinfrastructure. While I think Craig Stewart’s 2010 definition is still useful, new thinking might also have benefits. Let’s start with the elements of cyberinfrastructure. At the most basic level, these elements are simply people and things.

Redefining Cyberinfrastructure

About a month ago, I attended (thanks Sloan Foundation!) the Scientific Software Registry Collaboration Workshop, which had very interesting discussions. While there, I continued thinking about how software should be cited, focusing on open source. My thinking was certainly influenced by talking with Anita Bandrowski, Bryce Mecum, Shelley Stall, Katrina Fenlon, and Carly Robinson at the workshop.

Second thoughts on Proper Citation Guidance for Software Developers

(Please cite this post as https://doi.org/10.59350/rjt2m-2hy19)  A few things recently have made me think about how we support the core work of maintaining research software and supporting its users. I was interviewed by Georgia Iacovou for her report, “The state of the open source ecosystem &amp;

Supporting the core work in research software

I am writing to point out an interesting experiment that’s started up in the ACM Transactions on Mathematical Software (TOMS) computational journal.  There’s an editorial by Mike Heroux that explains the process. In brief, TOMS has created a “Replicated Computational Results (RCR)” review process, which is really a designation saying that the computational results published in an article are replicable.

Interesting Replicable “Badge” for journal articles

This blog post is intended as companion text for a talk I gave at the September 2018 NumFOCUS Project Forum in in New York, though I also hope it stands on its own. To address software sustainability, it is important first to understand how the term sustainability is used more generally.  It’s most often used in the context of ecology, often specifically in the relationship between humans and the planet.

Fundamentals of Software Sustainability

While on a call today for the ACM Emerging Interest Group on Reproducibility and Replicability, I realized that for computational reproducibility to become pervasive, we need to solve a scalability problem. Today, a lot of computational reproducibility is checked by hand, whether this is in terms of artifact evaluation in a conference or a paper, or with post-publication checks.

Scalable computational reproducibility

(This is taken in part from another blog post.  Doug Thain asked a few of us to participate in a group blog on reproducibility, and this was my initial post on that blog.)  Given my two roles, a researcher and a funder, it’s clear to me that reproducibility in science is increasingly seen as a concern, at least a high level. And thus, making science more reproducible is a challenge that many people want to solve.

Mark Twain on Reproducibility, more or less

Since we often forget what reproducibility means, particularly in the context of repeatability and replicability, I think it is worth highlighting this slide, taken from a talk by Neil Chue Hong, “Open Software for Open Science,” Young Alliance Against Cancer, 22-23 May 2015, Copenhagen, http://dx.doi.org/10.6084/m9.figshare.1424440  {.alignnone .size-full .wp-image-337 loading=“lazy” attachment-id=“337”

Re****bility

This post is cross-posted from Upstream: https://doi.org/10.54900/zks0k-pe034  I recently attended the FORCE2024 conference at UCLA. I’m a member of the board of directors of FORCE11, the parent organization for the conference, and the co-located FORCE11 Scholarly Communications Institute (FSCI), which I was not able to attend this year but have taught at in the past.

Research Software, Citation, and FAIR at FORCE2024

(by Daniel S. Katz and Tom Honeyman)  There are a number of challenges in rewarding scholars for their work in software, including the fact that both software itself and the idea of it being of scholarly value are relatively new, particularly given the centuries of experience we have with journals and the Humboldtian university model.

Software is a process, not a series of products: the failure of traditional metrics

As some may know, the first image of a black hole was announced April 10. This quickly led to a lot of different institutions explaining how they were involved (e.g., my own University of Illinois), as well as a bunch of software projects explaining how their software was used (e.g., Matplotlib).  Those of us concerned with open source research software and its sustainability are trying to raise the profile of such software in research.

A call for funding agencies to require grantees to report on the research software they use

Research software is an essential part of most research today, as I’ve written about before. One of the important models that allows people to develop and maintain it is grant funding: someone proposes building and/or maintaining software to a funder as an effort that aligns with the funder’s mission and programs.

Evaluating research software proposals

[Note: In December 2021, I decided to write down my experiences with the founding of US-RSE. Vanessa Sochat convinced me that it would make more sense for all interested US-RSE members to write a collective post, which we have now done as https://us-rse.org/2022-02-06-a-brief-history-of-usrse/. So much of this blog post overlaps that one, but some of it doesn’t, and I’m publishing this as a record of my recollection.

A kind-of brief (and personal) early history of US-RSE

I’m really excited to have been able to share some thoughts and stories through a recent RSE Stories podcast, and after being interviewed for this by Vanessa Sochat (which was a great experience; Vanessa is a good interviewer and a good editor), I realized that I left out one story I meant to tell, and decided to write it here.

Valuing people vs hardware

(by Daniel S. Katz, Kenton McHenry, Jong S. Lee)  (in conjunction with a talk [slides, video] presented 28 September 2021 at SeptembRSE)  Over the past nine years there has been growing acceptance of the role of Research Software Engineers (RSEs) and the value they bring to projects.

Senior level RSE career paths (with an s)

I’m excited by this paper: “Recognizing the value of software: a software citation guide [version 2; peer review: 2 approved]” and want to provide some context here (also in a thread of tweets).  This is work by the Journals Task Force, which I lead with a lot of help from Shelley Stall. It’s part of the FORCE11 Software Citation Implementation Working Group, which I co-lead with Neil Chue Hong and Martin Fenner.

Progress in Software Citation in Scholarly Publishing

by Daniel S. Katz and Kenton McHenry  We typically think of Research Software Engineers (RSEs) as working to support one or more researchers, either one-on-one or through a university’s centralised RSE group. We’ll call this the traditional RSE role, while being fully aware how ironic this phrase is. Simon Hettrick has used the following image, where “Here” indicates where the RSE fits between software engineering and research.

Super RSEs: Combining research and service in three dimensions of Research Software Engineering

This post begins with an idea from a response to an NIH request for information that I wrote with Kyle E. Niemeyer and Arfon M. Smith, discussing what I then thought were the two needed roles to make software citation work: creating/publishing and citing, which I referred to as push (creating the object to be cited) and pull (actually citing it.)  Upon further thought, I now believe that there are really three relevant steps, which can

Creation, publication, and citation: Issues in software citation versus paper and data citation

Recently (19-20 September 2016), a group convened at a workshop in San Diego under the auspices of FORCE11 (and with funding from the Helmsley Trust) to work on understanding the Scholarly Commons, a scholarly communications ecosystem designed for 21st century scholarship across all scholarly fields.  Since the Scholarly Commons doesn’t yet exist, the overall goal is to define and to bring it into existence.

The Scholarly Commons: Challenges in Philosophy and Governance

In a series of tweets (starting with one from @dimazest and including some from @turingfan, @Iza_Romanowska, and @CIRCA_StAndrews), it’s become apparent that we don’t have a standard way of providing attribution (e.g., citations, acknowledgements, dependencies) within software like we do in papers, where we have standard acknowledgement and reference sections, and for at least the reference section, a standard practice

How should we add citations inside software?

I’m very pleased to see that the Netherlands group working on “Software Management Plans: Towards national guidelines” has released a draft set of guidelines for community consultation (through 20 June 2022) and is organizing a workshop on 9 June to test and discuss the guidelines with use cases from the research community: https://www.eventbrite.com/e/software-that-makes-you-proud-using-smps-tickets-341888165707.

Progress on Software Management Plans in the Netherlands

I’ve previously written about the concept of workflows, sets of independent tasks connected by data dependencies, being expressed either as data, for example, a Pegasus DAG or CWL document, or as code, for example, a Python program written in Parsl.

Using workflows expressed as code and workflows expressed as data together

I’m happy to announce that I’ve been awarded an inaugural fellowship from the Better Scientific Software (BSSw) Project, along with my great colleagues, Jeff Carver, Ivo Jimenez, and Andrew Lumsdaine.

Better Scientific Software Fellowship

This is a third (and last) in a series of short (ok, the first and second were actually short, even if this one isn’t) blogs related to talks and discussions at the 10th RDA Plenary. Software Heritage   Software Heritage (as described in a recent white paper) is a really interesting and very ambitious project that started (or at least was announced) last summer.

Software Heritage and repository metadata: a software citation solution

The new international initiative, Initiative for Open Citations (I4OC), that started this week with the aim of promoting the unrestricted availability of scholarly citation data, is a wonderful idea.

Adding citations as part of the software publication process

I’m happy to announce that I’ve started a project with rOpenSci under their recent award from the Helmsley Foundation. My work with rOpenSci will focus on sustainability of the project itself. Sustainability can be defined as having the resources to do the necessary work to continue and grow rOpenSci. This is one of the most difficult challenges for rOpenSci and for many other research software projects.

Software sustainability research with rOpenSci

Measuring software impact is important, for a variety of reasons, and a variety of people, including software creators, their employers, their funders, potential users of the software, and academics studying software and science.  This topic has been studied by many, including some nice work by Thain, Tannenbaum, and Livny about 10 years ago. What should we count?

Software metrics – what to count and how

This post is inspired by a post from C. Titus Brown, where Titus asked “What do you think about the term &amp; practice of ‘hardening software’?” and a few of us used comments to discuss the concept. What is hardening?

Hardening software vs making it sustainable

Daniel S. Katz and Rajiv Ramnath, National Science Foundation {dkatz, rramnath}@nsf.gov   (A white paper for the Computational Science &amp;

Looking at Software Sustainability and Productivity Challenges from NSF

In response to Doug Thain’s question: What currently-available tools do you recommend for enabling reproducible scientific computing?  Is there a tool that we ought to have, but do not? P.S. I am using “reproducibility” as an easy shorthand for re-usability, re-creation, verification, and related tasks that have already seen some discussion.  Please interpret the question broadly.

In response to Doug Thain’s question:  I think the most important thing we need is an electronic lab notebook that really allows us to go back and understand exactly what we did, repeat it, modify it, record it, etc.  If you accept this, it leads to a number of points: Why don’t more people (including me) do this now? What tool(s) should we use?

The Need for Notebooks

I’ve been watching and participating in FAIR work for a while, first in terms of data, and more recently in research software, workflows, and machine learning models.

FAIR is not the end goal

The problem   The following question was emailed to me by Jim Fowler:  “I’m a professor at Ohio State and I sit on the committee which is drafting a new IP policy for faculty work. I’d like the new policy to support faculty contributions to open source software projects. I am very impressed with what you’ve done to recommend open source licensing on NSF funded projects;

Open Source Software and University Intellectual Property Policies

(with contributions from Michelle Barker, Neil Chue Hong, Matthew Turk, Jeffrey Carver, Hannah Cohoon, and James Howison;

There’s no such thing as sustainable research software

I was part of a panel in the WORKS workshop at the SC17 conference a couple of months ago, where I gave a short talk called “Expressing and Sharing Workflows” (slides). I’ve written this blog to put the slides into words and add some details I left out. First, what is a workflow? To me, it’s a set of tasks and dependencies between them.

Expressing workflows as code vs data

The idea of a learning curve is fairly well-accepted, that one makes progress in gaining skills at some rate that can be “steep”, where it takes a long time to gain much proficiency, or “gentle”, where a fair amount of proficiency is gained relatively quickly. Here, I want to propose the parallel idea of a software development curve.

Introducing the Software Development Curve

As I’ve been involved in the new FAIR for Research Software (FAIR4RS) activity (jointly run under FORCE11, RDA, and ReSA), I’ve been leading a subgroup looking at the FAIR (data) principles and thinking about how they can be applied to software. One of the things that has become clear to me is how fundamentally different software and data are, and the consequences this has on how software can be made FAIR.

Software and software metadata

After a number of general discussions in the research communication community, mostly focused on software citation, and then a few separate discussions with Anita Bandrowski and Martin Fenner, it’s become clear to me that we need something like a group (perhaps hierarchical) object identifier (GROUPID), which is somewhat different than a DOI, or at least different than how DOIs are commonly used today.

To better understand research communication, we need a GROUPID (group object identifier)

This post is meant to summarize the

 “Our Scholarly Recognition System Doesn’t Still Work” p

anel, held at the Science of Team Science Conference (SciTS), Friday, June 5, 2015, Bethesda, Maryland, USA The panel organizers were Daniel S. Katz (U. of Chicago &amp; Argonne National Laboratory), Amy Brand (Digital Science), Melissa Haendel (Oregon Health &amp;

Summary of “Our Scholarly Recognition System Doesn’t Still Work” panel

The problem I’ve been thinking about software catalogs for a while, mostly since the program I lead at NSF (Software Infrastructure for Sustained Innovation, SI

 2

, see the second half of http://nsf.gov/si2) has funded a fairly large number of projects, around 100.

Catalogs and Indices for Finding (Scientific) Software

Once again, I’m writing in response to Titus Brown, who answered Is software a primary product of science? with “no” in his blog and on Twitter:  Note that Titus cleverly changed the question from asking about research to asking about science. Perhaps we can combine these and think about scientific research. So, what are primary products of scientific research?

Software can be a primary product of scientific research

“Our Scholarly Recognition System Doesn’t Still Work” is the title of a panel at the Science of Team Science Conference that I’m co-organizing with Amy Brand (Digital Science), Melissa Haendel (Oregon Health &amp; Science University), and Holly J. Falk-Krzesinski (Elsevier)  If you are interested in this topic, you may want to consider attending the conference and the panel.

Our Scholarly Recognition System Doesn’t Still Work

I recently attended the hybrid SC21 conference in person, as one of the about 3000 in-person attendees, along with several thousand virtual attendees. (I don’t know the exact numbers, but in the past, an SC conference would have about 12000 attendees in-person.) In December, I’ll be attending the hybrid AGU Fall meeting remotely.

Event formats, including hybrid

While I’ve never been in an improv class, I really like the idea of saying “yes, and” as a way to move the activity forward. As others have also pointed out, this can work in business and research as well.

Collaboration: Working with Yes Anders rather than No Butters

This short blog post is really a pointer to another post that I co-wrote (with Sandra Gesing, Olivier Philippe, and Simon Hettrick), “Results from a US survey about Research Software Engineers,” as the initial post for the US Research Software Sustainability Institute (URSSI) blog.

Results from a US survey about Research Software Engineers

I would like to have a well-functioning, easy-to-use, software citation system in place. In brief, I believe that doing so would increase the recognition of the role of software and its developers and maintainer in research, and would provide them credit in a manner that would be useful in the academic and related settings in which many of them work.

Compact identifiers for software: The last missing link in user-oriented software citation?

This post is a cross-post from the UK Software Sustainability Institute blog on 14 June 2017).

 By Daniel S. Katz, Assistant Director for Scientific Software and Applications at the National Center for Supercomputing Applications, University of Illinois Urbana-Champaign.


 This article is part of our series: A day in the software life, in which we will be asking researchers from all disciplines to discuss the tools that make

Workflow systems for science: programming in the large

This blog is based on part of a talk I gave in January 2017, and the thinking behind it, in turn, is based on my view of a series of recent talks and blogs, and how they might be fit together. The short summary is that general software reproducibly is hard at best, and may not be practical except in special cases.

Is software reproducibility possible and practical?

by Daniel S. Katz and Michelle Barker  Most researchers and policymakers support the idea of making research, and specifically research outputs, findable, accessible, interoperably, and reusable (FAIR). The concept of FAIR has been well-developed for research data, but this is not the case for all research products.

Generalizing FAIR

When thinking about a group adding a new person, we often think that while the group will benefit from adding the new person in the long run, there is a cost to the group in the initial period.

How groups can immediately benefit when new members join

While at WSSSPE4 discussing possible WSSSPE mission statements, the question of what software sustainability is has been raised.

Defining Software Sustainability

(a white paper for the 2019 Collegeville Workshop on Sustainable Scientific Software (CW3S19))

Data-driven software sustainability

Having just returned from a short trip to the RDA Plenary 10, I’ve been thinking about a few things that will be the subject of a couple of hopefully short blogs in the near future.  This first one is about software identification, specifically as related to citation.

Key properties of software: agency and dynamicity

I’m writing this blog post in preparation for a contribution to the upcoming RDA Plenary 19 session “BoF: Sharing code that supports research articles: is it time to align goals and publisher policies?” at 0030 UTC on 23 June. I want to talk about how and why software (code) is shared.

Benefits of sharing code with articles

I’ve been involved in Project CreDIT a bit, since I read an article on it in Nature: Allen, L. et al. 17 April 2014. Publishing: Credit where credit is due.

 Nature

508, 312–313, doi:10.1038/508312a and wrote to the authors, since it seemed to overlap some of the ideas I’d been thinking about called transitive credit: Katz, D.S. 2014.

Project CRediT and Contributorship Roles

This is the second of a set of short blogs based on thoughts at the RDA Plenary 10.  (The first was on agency and dynamicity as key properties of software.)  In the Persistent ID (PID) interest group session at the RDA meeting, there was a talk by Patricia Cruse  about an organization ID project, which aims to create a new set of PIDs for organizations, followed by some discussion.

Group PIDs – a needed abstraction?

This post is written in response to a recent article: “Beyond authorship: attribution, contribution, collaboration, and credit,”

 Learned Publishing

28(2), April 2015 (DOI: 10.1087/20150211). As a member of the CASRAI working group who “provided critical review of the [contributorship] taxonomy,” I am generally supportive of this idea of better and identifying contributions to scientific works.

Measuring Contributions: People vs Products?

While at the Chan-Zuckerberg Initiative’s Open Science 2022 Annual Meeting a couple of weeks ago, I was struck by a comment from Demetris Cheatham about how she hadn’t known about the scientific open-source community until she was introduced to it fairly recently, even though she has a huge amount of experience with the larger open-source community. This was especially confounding when she shared that she realized upon

Inspired by talks at the International Workshop on Science Gateways a few weeks ago, in particular keynotes by David De Roure on Social Machines and Michelle Barker on the evolution of Science Gateways, I started thinking about what would happen if we considered all computational and data software in the form of science gateways. The definition of a science gateway has shifted a bit over time.

What if all research software was a science gateway?

As I’m getting ready for the first meeting of the Chan-Zuckerberg Initiative’s Essential Open Source Software for Science program, with grantees, advisors, funders, researchers, and industry partners, and with the goal “to identify shared needs and how funders might support these needs in the future”, I wonder how much software maintenance should cost, and how we might determine that answer.

How much should software maintenance cost?

In a recent meeting, there was some discussion about a potential workshop recommendation that funding agencies should offer relatively small grants that would just be used for maintaining existing software, but not doing any new development. I had very mixed feelings when I heard this, and decided I would write up my thoughts and see what others felt too.

Daniel S. Katz's blog