Unraveling complex biology with biological knowledge graphs. Content licensed under CC BY 4.0.

Charles Tapley Hoyt

PEP 735 introduced dependency groups in packaging metadata, which are complementary to optional dependencies in that they might not correspond to features in the package, but rather be something like development or release dependencies. I am slowly working towards updating my cookiecutter template cookiecutter-snekpack to use PEP 735. So far, uv and tox have released support - all that’s left is ReadTheDocs.

Dependency Groups and ReadTheDocs

I’m working through making a contribution to pandoc that adds first-class support for author role annotations using the Contribution Role Taxonomy (CRediT) and also outputs compliant Journal Publishing Tag Set (JATS) XML. This has lead me down a (losing) journey with learning the Haskell programming language, so I thought I would post a short note on a function I tried to understand.

Some Haskell I Tried to Write

Graphviz is software for graph visualization written in C. PyGraphviz provides a nice Python wrapper for it. The issue is that getting Python to know about the C headers changes every few months. I’ll try and keep this blog post updated every time there are some changes.

Building Graphviz when installing PyGraphviz

Several controlled vocabularies and ontologies that aren’t themselves OBO Foundry ontologies use unsanctioned OBO PURLs. This post is about how to use the Bioregistry to identify which resources are doing this and to give some insight into how we arrived in this situation. Background on the OBO Foundry   The OBO Foundry is a set of independent, interoperable biomedical ontologies that aspire to using shared development principles.

Resources masquerading as OBO Foundry ontologies

Today’s short post is about three SPARQL queries I wrote to get bibliometric information about journals and publishers out of Wikidata. Each of the following queries can be readily copy-pasted into the Wikidata Query Service and run in the browser.

Querying Journals and Publishers in Wikidata

I’ve just returned from the 17

 th

Annual International Biocuration Conference at the Indian Biological Data Centre (IBDC) in Faridabad, India.

Discussions and Follow-ups from Biocuration 2024

Using Pydantic for encoding data models and FastAPI for implementing APIs on top of them has become a staple for many Python programmers.

Semantic Pydantic

I finally got back into reading! Over winter break 2022, I started the Stormlight Archive then followed up in 2023 by reading the entirety of Brandon Sanderson’s

 Cosmere

, as well as a some other fantasy, science fiction, and literary fiction. Here’s the list.

Books I Read in 2023

Archival Resource Keys (ARKs) are flavor of persistent identifiers like DOIs, URNs, and Handles that have the benefit of being free, flexible with what metadata gets attached, and natively able to resolve to web pages. Name-to-Thing (N2T) implements a resolver for a variety of ARKs, so this blog post is about how that resolver can be re-implemented with the curies Python package. In a lot of ways, ARKs look and act like CURIEs.

Re-implementing the N2T ARK Resolver

This blog is normally about

 very serious science

, but I’m taking a break from that for the evening to advertize my band’s upcoming show on April 8

 th

in the SPH Music Masters Finale (aka, the German Battle of the Bands). We need your support!

The Representatives of Monkey Jack - German Battle of the Bands Finale

Authorships are

The Open Researcher and Contributor Identifier (ORCID) database is an invaluable resource that supports the unambiguous identification of researchers. However, its first party data dump is too complex, verbose, and unstandardized for many use cases. This post describes open source software I wrote that automates downloading, processing, and exporting ORCID into a more usable form. I put the results on Zenodo under the CC0 license.

Easier ORCID

The International Society of Biocuration (ISB) partners with the journal Database to get discounts for its members when they publish there. This means the ISB’s executive committee needs to send a member list to the journal’s editor.

Programmatic Access to a Wordpress User List

The Unified Medical Language System (UMLS) is a widely used biomedical and clinical vocabulary maintained by the United States National Library of Medicine. However, it is notoriously difficult to access and work with due to licensing restrictions and its complex download system. In the same vein as my previous posts about DrugBank and ChEMBL, this post describes open source software I’ve developed for downloading and working with this data.

Unlocking UMLS

I was recently nominated for the International Society for Biocuration’s Excellence in Biocuration Early Career Award (results will be announced on June 14

 th

!). This made me curious about how to model nominations and awards on Wikidata. In this post, I’ll describe how to curate awards, nominations, recipients, and how to make SPARQL queries to get them.

Modeling and Querying Awards in Wikidata

I’ve been working on improving reproducibility in the field of cheminformatics for some time now. For example, I’ve written posts about making data from DrugBank and ChEMBL more actionable. Over the last year, I’ve been preparing a concept with the editors of the Journal of Cheminformatics on how to include an assessment of reproducibility to reviews of manuscripts submitted to the journal.

Reproducibility Pilot in the Journal of Cheminformatics

Biopragmatics

[PEP 735](https://peps.python.org/pep-0735/) introduced dependency
groups in packaging metadata, which are complementary to [optional
dependencies](https://peps.python.org/pep-0631/) in that they might not
correspond to features in the package, but rather be something like
development or release dependencies. I am slowly working towards
[updating](https://github.com/cthoyt/cookiecutter-snekpack/pull/32) my
cookiecutter template
[cookiecutter-snekpack](https://github.com/cthoyt/cookiecutter-snekpack)
to use PEP 735. So far, uv and tox have released support - all that's
left is [ReadTheDocs](https://about.readthedocs.com/). This post
summarizes the issue I added to their issue tracker and the following
discussion.

## Summary of optional dependencies and dependency groups.

Before PEP 735, optional dependencies were often used both for extra
features (e.g., installing `lxml`{.language-plaintext
.highlighter-rouge} for faster XML parsing) as well as development
dependencies (e.g., for testing and documentation build). Here's what
the syntax for this looks like:

:::: {.language-toml .highlighter-rouge}
::: {.highlight}
``` highlight
[project.optional-dependencies]
faster-xml = [
    "lxml",
]
tests = [
    "pytest",
    "coverage",
]
docs = [
    "sphinx>=8",
    "sphinx-rtd-theme>=3.0",
    "sphinx-click",
    "sphinx_automodapi",
]
```
:::
::::

When you're using `pip`{.language-plaintext .highlighter-rouge} (or uv)
to install, you can use square bracket notation to say which ones go in.

:::: {.language-console .highlighter-rouge}
::: {.highlight}
``` highlight
$ pip install .[tests,docs,faster-xml]
```
:::
::::

This is a bit problematic since it conflates what the purpose of
optional dependencies are. Some build tools tried to address with their
on custom configuration for development dependencies in
`pyproject.toml`{.language-plaintext .highlighter-rouge} or elsewhere.
However, dependency groups gives us a more principled approach towards
categorizing dependencies that are not necessarily relevant for the code
itself. Now, we can split up the example from before like this:

:::: {.language-toml .highlighter-rouge}
::: {.highlight}
``` highlight
[project.optional-dependencies]
faster-xml = [
    "lxml",
]

[dependency-groups]
tests = [
    "pytest",
    "coverage",
]
docs = [
    "sphinx>=8",
    "sphinx-rtd-theme>=3.0",
    "sphinx_automodapi",
]
```
:::
::::

When you're using `pip`{.language-plaintext .highlighter-rouge} (or uv)
to install, you can use `--dependency-groups`{.language-plaintext
.highlighter-rouge} to say which ones go in.

:::: {.language-console .highlighter-rouge}
::: {.highlight}
``` highlight
$ pip install --dependency-groups=tests,typing .[faster-xml]
```
:::
::::

## Preparing an environment on ReadTheDocs

ReadTheDocs currently supports specifying optional dependencies (see
https://docs.readthedocs.io/en/stable/config-file/v2.html#packages) with
configuration like the following:

:::: {.language-yaml .highlighter-rouge}
::: {.highlight}
``` highlight
version: 2

python:
  install:
    - method: pip
      path: .
      extra_requirements:
        - docs
```
:::
::::

In my issue
[\#11766](https://github.com/readthedocs/readthedocs.org/issues/11766),
I suggested defining an alternate key
`dependency_groups`{.language-plaintext .highlighter-rouge} that could
work the same way.

:::: {.language-yaml .highlighter-rouge}
::: {.highlight}
``` highlight
version: 2

python:
  install:
    - method: pip
      path: .
      dependency_groups:
        - docs
```
:::
::::

## ReadTheDocs code deep dive and implementation suggestion

I found a few places that would be relevant for a theoretical
implementation. First, there's a data structure that represents the
slots available in the configuration for
`python > install`{.language-plaintext .highlighter-rouge}. Here's the
code
([permalink](https://github.com/readthedocs/readthedocs.org/blob/404d82a448295c81a271c3143d0fc9c10a924555/readthedocs/config/models.py#L77-L82)):

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
class PythonInstall(Base):
    __slots__ = (
        "path",
        "method",
        "extra_requirements",
    )
```
:::
::::

I'd simply add a new slot `dependency_groups`{.language-plaintext
.highlighter-rouge} and update the related processing code
([permalink](https://github.com/readthedocs/readthedocs.org/blob/404d82a448295c81a271c3143d0fc9c10a924555/readthedocs/doc_builder/python_environments.py#L66-L67)):

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
...
if install.extra_requirements:
    extra_req_param = "[{}]".format(",".join(install.extra_requirements))
...
```
:::
::::

Here's what I'd do:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
# Added these next lines
dependency_group_args = []
if install.dependency_groups:
    # not clear if the equals is necessary or if
    # this can be broken into two parts
    dependency_group_args.append("--dependency-groups={}".format(",".join(install.dependency_groups))

extra_req_param = ""
if install.extra_requirements:
    extra_req_param = "[{}]".format(",".join(install.extra_requirements))
self.build_env.run(
    self.venv_bin(filename="python"),
    "-m",
    "pip",
    "install",
    "--upgrade",
    "--upgrade-strategy",
    "only-if-needed",
    "--no-cache-dir",
    "{path}{extra_requirements}".format(
        path=local_path,
        extra_requirements=extra_req_param,
    ),
    *dependency_group_args,
    cwd=self.checkout_path,
    bin_path=self.venv_bin(),
)
```
:::
::::

As a minor note, I would also do a bit of refactoring to store all the
args into the list and then splat all of them into
`run()`{.language-plaintext .highlighter-rouge}

## There Has To Be A Better Way (TM)

As Raymond H. always says, there has to be a better way. Turns out, the
RTD team is already working on a more generic way to override the
installation command via the configuration in
[\#11710](https://github.com/readthedocs/readthedocs.org/pull/11710).

:::: {.language-yaml .highlighter-rouge}
::: {.highlight}
``` highlight
version: 2

build:
  jobs:
    install:
      - pip install --dependency-groups=tests,typing
```
:::
::::

This reduces the need to make potentially lots of update to the rigid
code I worked on above. This makes me very happy!

------------------------------------------------------------------------

I've received a lot of poorly written issues and requests in my open
source work (see the PyKEEN issue tracker) so I thought it was
incredibly important to write this issue very well. It occurred to me
that the thought process here is also maybe generally interesting, so
that's why I copied it to my blog.


[Graphviz](https://graphviz.org) is software for graph visualization
written in C. [PyGraphviz](https://pypi.org/project/pygraphviz) provides
a nice Python wrapper for it. The issue is that getting Python to know
about the C headers changes every few months. I'll try and keep this
blog post updated every time there are some changes.

# November 2024

These days, `pip install graphviz`{.language-plaintext
.highlighter-rouge} seemed to work but
`uv pip install graphviz`{.language-plaintext .highlighter-rouge} gave
the following:

:::: {.language-console .highlighter-rouge}
::: {.highlight}
``` highlight
$ uv pip install pygraphviz
Using Python 3.12.7 environment at /Users/cthoyt/.virtualenvs/orcid
Resolved 1 package in 597ms
error: Failed to prepare distributions
  Caused by: Failed to download and build `pygraphviz==1.14`
  Caused by: Build backend failed to build wheel through `build_wheel` (exit status: 1)

[stdout]
running bdist_wheel
running build
running build_py
creating build/lib.macosx-14.0-arm64-cpython-312/pygraphviz
copying pygraphviz/scraper.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz
copying pygraphviz/graphviz.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz
copying pygraphviz/__init__.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz
copying pygraphviz/agraph.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz
copying pygraphviz/testing.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz
creating build/lib.macosx-14.0-arm64-cpython-312/pygraphviz/tests
copying pygraphviz/tests/test_unicode.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz/tests
copying pygraphviz/tests/test_scraper.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz/tests
copying pygraphviz/tests/test_readwrite.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz/tests
copying pygraphviz/tests/test_string.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz/tests
copying pygraphviz/tests/__init__.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz/tests
copying pygraphviz/tests/test_html.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz/tests
copying pygraphviz/tests/test_node_attributes.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz/tests
copying pygraphviz/tests/test_drawing.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz/tests
copying pygraphviz/tests/test_repr_mimebundle.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz/tests
copying pygraphviz/tests/test_subgraph.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz/tests
copying pygraphviz/tests/test_close.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz/tests
copying pygraphviz/tests/test_edge_attributes.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz/tests
copying pygraphviz/tests/test_clear.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz/tests
copying pygraphviz/tests/test_layout.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz/tests
copying pygraphviz/tests/test_attribute_defaults.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz/tests
copying pygraphviz/tests/test_graph.py -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz/tests
running egg_info
writing pygraphviz.egg-info/PKG-INFO
writing dependency_links to pygraphviz.egg-info/dependency_links.txt
writing top-level names to pygraphviz.egg-info/top_level.txt
reading manifest file 'pygraphviz.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'pygraphviz.egg-info/SOURCES.txt'
copying pygraphviz/graphviz.i -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz
copying pygraphviz/graphviz_wrap.c -> build/lib.macosx-14.0-arm64-cpython-312/pygraphviz
running build_ext
building 'pygraphviz._graphviz' extension
creating build/temp.macosx-14.0-arm64-cpython-312/pygraphviz
clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -DSWIG_PYTHON_STRICT_BYTE_CHAR -I/Users/cthoyt/Library/Caches/uv/builds-v0/.tmpKw7zNP/include -I/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12 -c pygraphviz/graphviz_wrap.c -o build/temp.macosx-14.0-arm64-cpython-312/pygraphviz/graphviz_wrap.o

[stderr]
warning: no files found matching '*.swg'
warning: no files found matching '*.png' under directory 'doc'
warning: no files found matching '*.html' under directory 'doc'
warning: no files found matching '*.txt' under directory 'doc'
warning: no files found matching '*.css' under directory 'doc'
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '.svn' found anywhere in distribution
no previously-included directories found matching 'doc/build'
pygraphviz/graphviz_wrap.c:9:9: warning: 'SWIG_PYTHON_STRICT_BYTE_CHAR' macro redefined [-Wmacro-redefined]
#define SWIG_PYTHON_STRICT_BYTE_CHAR
        ^
:2:9: note: previous definition is here
#define SWIG_PYTHON_STRICT_BYTE_CHAR 1
        ^
pygraphviz/graphviz_wrap.c:3023:10: fatal error: 'graphviz/cgraph.h' file not found
#include "graphviz/cgraph.h"
         ^~~~~~~~~~~~~~~~~~~
1 warning and 1 error generated.
error: command '/usr/bin/clang' failed with exit code 1
  Caused by: This error likely indicates that you need to install a library that provides "graphviz/cgraph.h" for pygraphviz@1.14
```
:::
::::

Here's one solution that work (of the many possible ones):

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
$ export CFLAGS="-I$(brew --prefix graphviz)/include"
$ export LDFLAGS="-L$(brew --prefix graphviz)/lib"
$ uv pip install pygraphviz
Using Python 3.12.7 environment at /Users/cthoyt/.virtualenvs/orcid
Resolved 1 package in 2ms
   Built pygraphviz==1.14
Prepared 1 package in 908ms
Installed 1 package in 1ms
 + pygraphviz==1.14
```
:::
::::

There's also talk of using `--global-option`{.language-plaintext
.highlighter-rouge} to pass
`-I/opt/homebrew/opt/graphviz/include`{.language-plaintext
.highlighter-rouge} and
`-L/opt/homebrew/opt/graphviz/lib`{.language-plaintext
.highlighter-rouge}, but I couldn't figure this out for
`uv pip install`{.language-plaintext .highlighter-rouge}.

# 2023 Era

I was on Python 3.10 on a M2 Mac with macOS 13.
`pygraphviz`{.language-plaintext .highlighter-rouge} didn't have a
pre-built wheel for my systen so
`python -m pip install pygraphviz`{.language-plaintext
.highlighter-rouge} gave the following error:

:::: {.language-console .highlighter-rouge}
::: {.highlight}
``` highlight
$ python -m pip install pygraphviz
...
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -DSWIG_PYTHON_STRICT_BYTE_CHAR -I/Users/cthoyt/.virtualenvs/indra/include -I/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c pygraphviz/graphviz_wrap.c -o build/temp.macosx-13-x86_64-cpython-310/pygraphviz/graphviz_wrap.o
      pygraphviz/graphviz_wrap.c:3020:10: fatal error: 'graphviz/cgraph.h' file not found
      #include "graphviz/cgraph.h"
               ^~~~~~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
```
:::
::::

I figured out based on the Homebrew troubleshooting in the docs
(https://pygraphviz.github.io/documentation/stable/install.html#homebrew)
that the following works:

:::: {.language-shell .highlighter-rouge}
::: {.highlight}
``` highlight
python -m pip install --use-pep517 \
    --config-setting="--global-option=build_ext" \
    --config-setting="--build-option=-I$(brew --prefix graphviz)/include/" \
    --config-setting="--build-option=-L$(brew --prefix graphviz)/lib/" \
    pygraphviz
```
:::
::::


The [International Society of Biocuration
(ISB)](https://www.biocuration.org/) partners with the journal
[Database](https://academic.oup.com/database) to get discounts for its
members when they publish there. This means the ISB's executive
committee needs to send a member list to the journal's editor.
Historically, this has been done manually by exporting the list from the
membership management plugin in the ISB Wordpress blog once per month
and emailing it to th This post is about my journey trying to automate
it

# 1. There must be an API for this

Wordpress has a [programmatic
API](https://developer.wordpress.org/rest-api/), and specifically an
endpoint to list
[users](https://developer.wordpress.org/rest-api/reference/users/).

After logging into the ISB's Wordpress site, I was able to list users by
navigating to the endpoint in browser
https://biocuration.org/wp-json/wp/v2/users. Note: this page won't work
for you unless you're on the EC and have admin powers. I wanted to
replicate accessing this page through a Python script, so I was
suggested by the [official Wordpress
documentation](https://developer.wordpress.org/rest-api/using-the-rest-api/authentication/)
to create an application password and then use simple HTTP
authentication. The docs said try this, after replacing the username
with the Wordpress account (for me, that's `cthoyt`{.language-plaintext
.highlighter-rouge}) and the application password, which is a string
with the form of `XXXX XXXX XXXX XXXX XXXX XXXX`{.language-plaintext
.highlighter-rouge}. The documentation suggested doing the following
curl command, which works:

:::: {.language-shell .highlighter-rouge}
::: {.highlight}
``` highlight
curl --user "USERNAME:PASSWORD" https://biocuration.org/wp-json/wp/v2/users
```
:::
::::

If you want to translate this to Python, it looks like this.

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
import pystow, base64, requests

username = pystow.get_config("isb", "wordpress_username", raise_on_missing=True)
password = pystow.get_config("isb", "wordpress_password", raise_on_missing=True)

token = base64.b64encode(f"{username}:{password}".encode()).decode("utf-8")

res = requests.get(
    'https://www.biocuration.org/wp-json/wp/v2/users',
    headers={"Authorization": f"Basic {token}", "user-agent": "curl"},
)
```
:::
::::

Luckily, encoding the username/password string with basic authorization
is such a common pattern that the `request`{.language-plaintext
.highlighter-rouge} (and any other modern library) has a nice par for
this.

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
import pystow, requests

username = pystow.get_config("isb", "wordpress_username", raise_on_missing=True)
password = pystow.get_config("isb", "wordpress_password", raise_on_missing=True)

res = requests.get(
    'https://www.biocuration.org/wp-json/wp/v2/users',
    auth=(username, password),
    headers={"user-agent": "curl"}
)
```
:::
::::

I burned a ton of time with this because it turns out that Wordpress
blocks both the user agents for the `request`{.language-plaintext
.highlighter-rouge} and `httpx`{.language-plaintext .highlighter-rouge}
libraries. Therefore, you have to explicitly set the
`user-agent`{.language-plaintext .highlighter-rouge} header to something
else, or you get a HTTP 403 forbidden error.

# 2. The API doesn't do what I wanted

It turns out that by default, all the users that never made a post get
filtered out. This is bad since the only users who are making posts on
the ISB are the handful who have or are currently on the executive
committee.

So, obviously, the next step is to start injecting in PHP code to change
how the API works. I found a
[comment](https://github.com/WP-API/WP-API/issues/2300#issuecomment-299202391)
from Tim Jensen, a Wordpress developer, that suggests adding the
following hooks into the Wordpress theme's
`functions.php`{.language-plaintext .highlighter-rouge} file by
navigating to
https://www.biocuration.org/wp-admin/theme-editor.php?file=functions.php&theme=executive:

:::: {.language-php .highlighter-rouge}
::: {.highlight}
``` highlight
function remove_has_published_posts_from_api_user_query($prepared_args, $request)
{
    unset($prepared_args['has_published_posts']);
    return $prepared_args;
}
add_filter('rest_user_query', 'remove_has_published_posts_from_api_user_query', 10, 2);
```
:::
::::

This actually worked! But, it wasn't the end of the story.

# 3. Full names are part of the plugin's metadata

It turns out that the user data model isn't all that comprehensive in
Wordpress. What we really needed was the full name and email address for
each person, and that was stored in an auxillary SQL table created and
managed by the memberships plugin. Note that the plugin lives on top of
Wordpress's first-party users list, and doesn't modify the API access to
the user list.

So obviously, the next step was to add new API endpoints. I learned PHP
at the ripe age of 14 and when I learned some other languages, I never
looked back, so this wasn't so easy. Plus, Wordpress has a huge set of
functions and idioms on top. With a little help from ChatGPT, I was able
to write a new API endpoint to add to
`functions.php`{.language-plaintext .highlighter-rouge} that got users
whose membership expiration date is in the future and joined in the
metadata for their first and last names:

:::: {.language-php .highlighter-rouge}
::: {.highlight}
``` highlight
function export_pmp_members_to_csv() {
    global $wpdb;

    // Set headers to make the response downloadable as a CSV file
    header('Content-Type: text/csv');
    header('Content-Disposition: attachment; filename="members_export.csv";');
    header('Pragma: no-cache');
    header('Expires: 0');

    // Open output stream (in-memory file)
    $output = fopen('php://output', 'w');

    if (!$output) {
        return new WP_Error('file_error', 'Unable to generate CSV file.');
    }

    // Add CSV header row
    fputcsv($output, ['id', 'first', 'last', 'email', 'level', 'start', 'ends']);

    // Query to get PMP members and their membership details
    $results = $wpdb->get_results("
        SELECT 
            u.ID, 
            um_first_name.meta_value AS first_name, 
            um_last_name.meta_value AS last_name, 
            u.user_email, ml.name as membership_name, 
            mu.startdate, 
            mu.enddate
        FROM {$wpdb->users} u
        LEFT JOIN {$wpdb->prefix}pmpro_memberships_users mu ON u.ID = mu.user_id
        LEFT JOIN {$wpdb->prefix}pmpro_membership_levels ml ON mu.membership_id = ml.id
        LEFT JOIN {$wpdb->prefix}usermeta um_first_name ON u.ID = um_first_name.user_id 
            AND um_first_name.meta_key = 'first_name'
        LEFT JOIN {$wpdb->prefix}usermeta um_last_name ON u.ID = um_last_name.user_id 
            AND um_last_name.meta_key = 'last_name'
        WHERE mu.membership_id IS NOT NULL
            AND (mu.enddate > CURDATE() OR mu.enddate IS NULL)
    ");

    // Loop through results and write each row to the CSV
    foreach ($results as $row) {
        fputcsv($output, [
            $row->ID, 
            $row->first_name,
            $row->last_name,
            $row->user_email, 
            $row->membership_name, 
            $row->startdate, 
            $row->enddate
        ]);
    }

    // Close the output stream
    fclose($output);

    // Terminate script to ensure no extra output is sent
    exit;
}

function register_pmp_export_route() {
    // the actual path to the endpoint isn't important here
    register_rest_route('path/to/endpoint/', '/export-members', [
        'methods' => 'GET',
        'callback' => 'export_pmp_members_to_csv'
        'permission_callback' => function() {
            // Optionally, restrict access to logged-in users with specific capabilities
            return current_user_can('manage_options'); // Restrict to admin users
        }
    ]);
}
add_action('rest_api_init', 'register_pmp_export_route');
```
:::
::::

# 4. Automate it

Why stop at just being able to export the sheet? I wanted to go another
mile and make sure that Wordpress sends an email to the right person at
the journal on a monthly basis.

:::: {.language-php .highlighter-rouge}
::: {.highlight}
``` highlight
/ Function to generate and send the CSV via email
function send_pmp_members_csv_via_email() {
    global $wpdb;

    // Create a temporary file to store the CSV
    $tmp_file = tempnam(sys_get_temp_dir(), 'pmp_csv_') . '.csv';
    $output = fopen($tmp_file, 'w');

    if (!$output) {
        return; // Handle error appropriately
    }

    // Add CSV header row
    fputcsv($output, ['ID', 'Username', 'First Name', 'Last Name', 'Email', 'Membership Level Name', 'Joined Date', 'Expires Date']);

    // Query to get PMP members and their membership details
    $results = $wpdb->get_results("
        SELECT u.ID, 
            u.user_login, 
            um_first_name.meta_value AS first_name, 
            um_last_name.meta_value AS last_name,
            u.user_email, 
            ml.name as membership_name
            mu.startdate,
            mu.enddate,
        FROM {$wpdb->users} u
        LEFT JOIN {$wpdb->prefix}pmpro_memberships_users mu ON u.ID = mu.user_id
        LEFT JOIN {$wpdb->prefix}pmpro_membership_levels ml ON mu.membership_id = ml.id
        LEFT JOIN {$wpdb->prefix}usermeta um_first_name ON u.ID = um_first_name.user_id 
            AND um_first_name.meta_key = 'first_name'
        LEFT JOIN {$wpdb->prefix}usermeta um_last_name ON u.ID = um_last_name.user_id 
            AND um_last_name.meta_key = 'last_name'
        WHERE mu.membership_id IS NOT NULL
        AND (mu.enddate > CURDATE() OR mu.enddate IS NULL)
    ");

    // Write data to CSV
    foreach ($results as $row) {
        fputcsv($output, [
            $row->ID, 
            $row->user_login, 
            $row->first_name,
            $row->last_name,
            $row->user_email, 
            $row->membership_name, 
            $row->startdate, 
            $row->enddate
        ]);
    }

    // Close the file
    fclose($output);

    // Prepare email
    $to = ''; // Replace with recipient email
    $subject = 'ISB Monthly Members Report';
    $message = 'Please find the attached CSV file containing the list of active members of the International Society of Biocuration as of today.';
    $headers = array('Content-Type: text/csv; charset=UTF-8');

    // Send email with attachment
    wp_mail($to, $subject, $message, $headers, $tmp_file);

    // Clean up temporary file
    unlink($tmp_file);
}

// Schedule the cron job to run monthly
function schedule_monthly_csv_email() {
    if (!wp_next_scheduled('monthly_pmp_csv_email')) {
        wp_schedule_event(time(), 'monthly', 'monthly_pmp_csv_email');
    }
}
add_action('wp', 'schedule_monthly_csv_email');

// Hook the function to the cron event
add_action('monthly_pmp_csv_email', 'send_pmp_members_csv_via_email');
```
:::
::::

Hopefully I don't have to write anymore PHP for a long time :)


Using [Pydantic](https://docs.pydantic.dev) for encoding data models and
[FastAPI](https://fastapi.tiangolo.com) for implementing APIs on top of
them has become a staple for many Python programmers. When this
intersects with the semantic web, linked open data, and the natural
sciences, we are still lacking a bridge to annotate our data models and
APIs to make them more FAIR (findable, accessible, interoperable, and
reusable). In this post, we build an extension to Pydantic and FastAPI
to annotate data models' fields and API endpoints' query, path, and
other parameters using the [Bioregistry](https://bioregistry.io), a
comprehensive catalog of metadata about semantic spaces from the
semantic web and the natural sciences.

As a demonstration, we will build a data model and API that serves
information about scholars.

## First Steps with Pydantic

We'll use [Open Researcher and Contributor (ORCID)](https://orcid.org/)
identifiers as primary keys, include the researcher's name, and start
with a single cross-reference, e.g., to the author's
[DBLP](https://dblp.org/) identifier. We'll encode this data model using
[Pydantic](https://docs.pydantic.dev/latest/) in the Python programming
language as follows:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from pydantic import BaseModel, Field


class ScholarV1(BaseModel):
    """A model representing a researcher, who might have several IDs on different services."""

    orcid: str = Field(...)
    name: str = Field(...)
    dblp: str | None = Field(None)


print(ScholarV1.schema_json(indent=2))
```
:::
::::

JSON Schema - Version 1

\`\`\`json { \"title\": \"ScholarV1\", \"description\": \"A model
representing a researcher, who might have several IDs on different
services.\", \"type\": \"object\", \"properties\": { \"orcid\": {
\"title\": \"Orcid\", \"type\": \"string\" }, \"name\": { \"title\":
\"Name\", \"type\": \"string\" }, \"dblp\": { \"title\": \"Dblp\",
\"type\": \"string\" } }, \"required\": \[ \"orcid\", \"name\" \] }
\`\`\`

There are several places for improvement here:

1.  Correct capitalization of the titles (`ORCID`{.language-plaintext
    .highlighter-rouge} instead of `Orcid`{.language-plaintext
    .highlighter-rouge} and `DBLP`{.language-plaintext
    .highlighter-rouge} instead of `Dblp`{.language-plaintext
    .highlighter-rouge})
2.  Add useful descriptions of what each field is
3.  Have regular expression patterns to validate input
4.  Give an example

All of these are possible to annotate into Pydantic's
`Field`{.language-plaintext .highlighter-rouge} object, but it requires
lots of effort and takes lots of space. Even worse, this might have to
be partially duplicated if multiple models share the same fields. In the
example below, I annotated ORCID but will skip the others for brevity.

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from pydantic import BaseModel, Field


class ScholarV2(BaseModel):
    """A model representing a researcher, who might have several IDs on different services."""

    orcid: str = Field(
        ...,
        title="ORCID",
        description="A stable, public identifier for a researcher from https://orcid.com",
        pattern="^\d{4}-\d{4}-\d{4}-\d{3}(\d|X)$",
        example="0000-0003-4423-4370",
    )
    name: str = Field(...)
    dblp: str | None = Field(None)


print(ScholarV2.schema_json(indent=2))
```
:::
::::

JSON Schema - Version 2

\`\`\`json { \"title\": \"ScholarV2\", \"description\": \"A model
representing a researcher, who might have several IDs on different
services.\", \"type\": \"object\", \"properties\": { \"orcid\": {
\"title\": \"ORCID\", \"description\": \"A stable, public identifier for
a researcher from https://orcid.com\", \"pattern\":
\"\^\\d{4}-\\d{4}-\\d{4}-\\d{3}(\\d\|X)\$\", \"example\":
\"0000-0003-4423-4370\", \"type\": \"string\" }, \"name\": { \"title\":
\"Name\", \"type\": \"string\" }, \"dblp\": { \"title\": \"Dblp\",
\"type\": \"string\" } }, \"required\": \[ \"orcid\", \"name\" \] }
\`\`\`

However, this was a lot of work. It would be nice if there were some
database of all the semantic spaces in the semantic web and natural
sciences that contained the name, description, regular expression
pattern, and examples. Then, we could draw from this database to
automatically populate our fields.

## Adding Some Bioregistry Magic

The good news is that such a database exists - it's called the
[Bioregistry](https://bioregistry.io). Each semantic space (e.g., ORCID,
DBLP) gets a *prefix* which is usually an acronym for the name of the
resource that serves as the primary key for the semantic space. These
prefixes are also useful in making references to entities in the
semantic space more FAIR (findable, accessible, interoperable, reusable)
using the [compact URI (CURIE)
syntax](https://cthoyt.com/2021/09/14/curies.html), though this isn't
the goal of this demo.

I've mocked some Python code that bridges Pydantic and the Bioregistry
in this repository (https://github.com/cthoyt/semantic-pydantic). I'm
calling it **Semantic Pydantic** because it lets us annotate our data
models with external metadata (and because it rhymes).

Here's the same model as before, but now using a
`SemanticField`{.language-plaintext .highlighter-rouge} that extends
Pydantic's `Field`{.language-plaintext .highlighter-rouge}. It has a
special keyword `prefix`{.language-plaintext .highlighter-rouge} that
lets you give a Bioregistry prefix, then it is smart enough to fill out
all the fields on its own. I also took the liberty of adding several
more semantic spaces that identify scholars like [Web of Science
(`wos`{.language-plaintext
.highlighter-rouge})](https://bioregistry.io/wos.researcher),
[Scopus](https://bioregistry.io/scopus), and even
[GitHub](https://bioregistry.io/github).

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from pydantic import BaseModel, Field

from semantic_pydantic import SemanticField


class ScholarV3(BaseModel):
    """A model representing a researcher, who might have several IDs on different services."""

    orcid: str = SemanticField(..., prefix="orcid")
    name: str = Field(..., example="Charles Tapley Hoyt")

    wos: str | None = SemanticField(default=None, prefix="wos.researcher")
    dblp: str | None = SemanticField(default=None, prefix="dblp.author")
    github: str | None = SemanticField(default=None, prefix="github")
    scopus: str | None = SemanticField(default=None, prefix="scopus")
    semion: str | None = SemanticField(default=None, prefix="semion")
    publons: str | None = SemanticField(default=None, prefix="publons.researcher")
    authorea: str | None = SemanticField(default=None, prefix="authorea.author")


print(ScholarV3.schema_json(indent=2))
```
:::
::::

Finally, we can see a very detailed JSON schema, which includes
everything from before plus additional context from the Bioregistry,
including the prefix itself as well as mappings from the Bioregistry
prefix to external registries like
[BARTOC](https://bioregistry.io/metaregistry/bartoc),
[FAIRsharing](https://bioregistry.io/metaregistry/fairsharing), and
others. Together, these make the data model more FAIR and support
interoperability, since now it is possible to directly match the fields
annotated with Bioregistry prefixes in this model to fields annotated
with the same prefix in other models, even external to the project.

JSON Schema - Version 3

\`\`\`json { \"title\": \"ScholarV3\", \"description\": \"A model
representing a researcher, who might have several IDs on different
services.\", \"type\": \"object\", \"properties\": { \"orcid\": {
\"title\": \"Open Researcher and Contributor\", \"description\": \"

This field corresponds to a local unique identifier from *Open
Researcher and Contributor*.\n

#### Provenance

The semantics of this field are derived from
the\n[`orcid`](\%22https://bioregistry.io/orcid\%22) entry in\nthe
[Bioregistry](\%22https://bioregistry.io\%22): a registry of semantic
web and linked \nopen data compact URI (CURIE) prefixes and URI
prefixes.\n

#### Description of Semantic Space

ORCID (Open Researcher and Contributor ID) is an open, non-profit,
community-based effort to create and maintain a registry of unique
identifiers for individual researchers. ORCID records hold non-sensitive
information such as name, email, organization name, and research
activities.\", \"pattern\": \"\^\\d{4}-\\d{4}-\\d{4}-\\d{3}(\\d\|X)\$\",
\"example\": \"0000-0003-4423-4370\", \"json_schema_extra\": {
\"bioregistry\": { \"prefix\": \"orcid\", \"mappings\": { \"bartoc\":
\"2021\", \"biocontext\": \"ORCID\", \"biolink\": \"ORCID\",
\"fairsharing\": \"FAIRsharing.nx58jg\", \"go\": \"orcid\", \"miriam\":
\"orcid\", \"n2t\": \"orcid\", \"wikidata\": \"P496\" } } }, \"type\":
\"string\" }, \"name\": { \"title\": \"Name\", \"example\": \"Charles
Tapley Hoyt\", \"type\": \"string\" }, \... } } \`\`\` Note that the
extra semantic fields have been truncated for brevity. Each of them also
has lots of detailed information.

## Web Application

Let's take the next step to a web application using FastAPI. The goal of
this web application will be to look up the information for a scholar in
Wikidata based on their ORCID. You don't really have to understand how
the query works other than that it takes in an ORCID string and gives
back an instance of the Scholar model we've been working on above.

The app uses annotations for the query parameters, path parameters, and
other inputs to routes using extensions of Pydantic
`Fields`{.language-plaintext .highlighter-rouge}. So similar to before,
we can extend their custom fields to be semantic in **Semantic
Pydantic**.

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from fastapi import FastAPI
from semantic_pydantic import SemanticPath

app = FastAPI(title="Semantic Pydantic Demo")
Scholar = ...  # defined before


@app.get("/api/orcid/{orcid}", response_model=Scholar)
def get_scholar_from_orcid(orcid: str = SemanticPath(prefix="orcid")):
    """Get xrefs for a researcher in Wikidata, given ORCID identifier."""
    ...  # full implementation in https://github.com/cthoyt/semantic-pydantic
    return Scholar(...)
```
:::
::::

The real power is how this translates to the API, and more importantly,
the automatically generated API documentation. First, the
`SemanticPath`{.language-plaintext .highlighter-rouge} object which we
used in place of a normal `fastapi.Path`{.language-plaintext
.highlighter-rouge} also knows it is for ORCID identifiers. Second, the
response model points to the Scholar class from before which already
knows about its semantics. Below, we see this in a screenshot of the
OpenAPI (formerly known as Swagger) user interface automatically
generated by FastAPI.

![A screenshot of the Swagger UI, enriched by the semantic annotations
described above.](/img/semantic-pydantic-api.png)

There are two big things to note here:

1.  All the detailed information about ORCID makes it into the
    parameter, also giving an example for users to get started
2.  All the detailed information from the response model (Scholar) gets
    shown as a real example.

Now, we have an API that is also annotated with detailed semantics. If
you take a look at the OpenAPI JSON file, it has similar references to
Bioregistry prefixes for the routes themselves, and directly reuses the
JSON schema for the response model.

## Next Steps

So far, this is a proof-of-concept that lives in an *ad hoc* repository.
It's not clear yet if this code is just a neat demo, whether it should
live inside the Bioregistry Python package, I haven't decided yet if
this should go inside the [Bioregistry Python
package](https://github.com/biopragmatics/bioregistry/), or if it should
be in a stand-alone package that might be extensible even further. There
are a few other things to think about in the meantime:

### Infrastructure for FAIR Models and APIs

The first version of this idea just throws the Bioregistry data into the
JSON schema. It would be interesting to develop this infrastructure
further, such as keeping a catalog of all APIs that consume or produce
data models containing semantic fields. A few places this would be
great:

1.  The [Cheminformatics
    Microservice](https://github.com/Steinbeck-Lab/cheminformatics-microservice/issues)
    contains tons of references to [InChi Keys
    (`inchikey`{.language-plaintext
    .highlighter-rouge})](https://bioregistry.io/registry/inchikey)
2.  The Bioregistry could dog-food its own API
3.  The [INDRA Discovery API](https://discovery.indra.bio/apidocs) could
    be refactored to use semantic inputs and outputs

There are also so many more examples, please let me know some services
you think would benefit in the comments on my blog post. Looking
forward, it's also a question on how to automatically discover such
semantic APIs (e.g., by cleverly searching GitHub) or if it would have
to be a manually curated catalog.

### Resolving URLs

A key feature of the Bioregistry is that it provides a way to take a
local unique identifier for an entity in a given semantic space and make
a URL that points to a web page describing the entity. For example, if
you have an ORCID identifier, you can make a URL for the ORCID page
following the format `https://orcid.org/`{.language-plaintext
.highlighter-rouge}. It would be very cool to extend Semantic Pydantic
to add some properties that auto-generate URLs, like in the following:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from pydantic import BaseModel, Field

from semantic_pydantic import SemanticField


class Scholar(BaseModel):
    orcid: str = SemanticField(..., prefix="orcid")
    name: str = Field(...)


charlie = Scholar(orcid="0000-0003-4423-4370", name="Charles Tapley Hoyt")
assert charlie.orcid_url == 'https://orcid.org/0000-0003-4423-4370'
```
:::
::::

### Run the Demo

The demo can be run by cloning the repository, installing its
requirements, and running the self-contained
`app.py`{.language-plaintext .highlighter-rouge}.

:::: {.language-shell .highlighter-rouge}
::: {.highlight}
``` highlight
git clone https://github.com/cthoyt/semantic-pydantic
cd semantic-pydantic
python -m pip install -r requirements.txt
python app.py
```
:::
::::

------------------------------------------------------------------------

This is my first science post of 2024! I'm very happy that the
Bioregistry is currently supported by the Chan Zuckerberg Initiative
(CZI) under award 2023-329850.
