Unraveling complex biology with biological knowledge graphs. Content licensed under CC BY 4.0.

Charles Tapley Hoyt

Graphviz is software for graph visualization written in C. PyGraphviz provides a nice Python wrapper for it. The issue is that getting Python to know about the C headers changes every few months. I’ll try and keep this blog post updated every time there are some changes.

Building Graphviz when installing PyGraphviz

I’m working through making a contribution to pandoc that adds first-class support for author role annotations using the Contribution Role Taxonomy (CRediT) and also outputs compliant Journal Publishing Tag Set (JATS) XML. This has lead me down a (losing) journey with learning the Haskell programming language, so I thought I would post a short note on a function I tried to understand.

Some Haskell I Tried to Write

Several controlled vocabularies and ontologies that aren’t themselves OBO Foundry ontologies use unsanctioned OBO PURLs. This post is about how to use the Bioregistry to identify which resources are doing this and to give some insight into how we arrived in this situation. Background on the OBO Foundry   The OBO Foundry is a set of independent, interoperable biomedical ontologies that aspire to using shared development principles.

Resources masquerading as OBO Foundry ontologies

I’ve just returned from the 17

 th

Annual International Biocuration Conference at the Indian Biological Data Centre (IBDC) in Faridabad, India.

Discussions and Follow-ups from Biocuration 2024

Using Pydantic for encoding data models and FastAPI for implementing APIs on top of them has become a staple for many Python programmers.

Semantic Pydantic

I finally got back into reading! Over winter break 2022, I started the Stormlight Archive then followed up in 2023 by reading the entirety of Brandon Sanderson’s

 Cosmere

, as well as a some other fantasy, science fiction, and literary fiction. Here’s the list.

Books I Read in 2023

Today’s short post is about three SPARQL queries I wrote to get bibliometric information about journals and publishers out of Wikidata. Each of the following queries can be readily copy-pasted into the Wikidata Query Service and run in the browser.

Querying Journals and Publishers in Wikidata

Archival Resource Keys (ARKs) are flavor of persistent identifiers like DOIs, URNs, and Handles that have the benefit of being free, flexible with what metadata gets attached, and natively able to resolve to web pages. Name-to-Thing (N2T) implements a resolver for a variety of ARKs, so this blog post is about how that resolver can be re-implemented with the curies Python package. In a lot of ways, ARKs look and act like CURIEs.

Re-implementing the N2T ARK Resolver

This blog is normally about

 very serious science

, but I’m taking a break from that for the evening to advertize my band’s upcoming show on April 8

 th

in the SPH Music Masters Finale (aka, the German Battle of the Bands). We need your support!

The Representatives of Monkey Jack - German Battle of the Bands Finale

Authorships are

The Open Researcher and Contributor Identifier (ORCID) database is an invaluable resource that supports the unambiguous identification of researchers. However, its first party data dump is too complex, verbose, and unstandardized for many use cases. This post describes open source software I wrote that automates downloading, processing, and exporting ORCID into a more usable form. I put the results on Zenodo under the CC0 license.

Easier ORCID

The International Society of Biocuration (ISB) partners with the journal Database to get discounts for its members when they publish there. This means the ISB’s executive committee needs to send a member list to the journal’s editor.

Programmatic Access to a Wordpress User List

The Unified Medical Language System (UMLS) is a widely used biomedical and clinical vocabulary maintained by the United States National Library of Medicine. However, it is notoriously difficult to access and work with due to licensing restrictions and its complex download system. In the same vein as my previous posts about DrugBank and ChEMBL, this post describes open source software I’ve developed for downloading and working with this data.

Unlocking UMLS

I was recently nominated for the International Society for Biocuration’s Excellence in Biocuration Early Career Award (results will be announced on June 14

 th

!). This made me curious about how to model nominations and awards on Wikidata. In this post, I’ll describe how to curate awards, nominations, recipients, and how to make SPARQL queries to get them.

Modeling and Querying Awards in Wikidata

I’ve been working on improving reproducibility in the field of cheminformatics for some time now. For example, I’ve written posts about making data from DrugBank and ChEMBL more actionable. Over the last year, I’ve been preparing a concept with the editors of the Journal of Cheminformatics on how to include an assessment of reproducibility to reviews of manuscripts submitted to the journal.

Reproducibility Pilot in the Journal of Cheminformatics

Biopragmatics

The [Unified Medical Language System
(UMLS)](https://www.nlm.nih.gov/research/umls) is a widely used
biomedical and clinical vocabulary maintained by the United States
National Library of Medicine. However, it is notoriously difficult to
access and work with due to licensing restrictions and its complex
download system. In the same vein as my previous posts about
[DrugBank](/2020/12/14/taming-drugbank.html) and
[ChEMBL](/2021/08/05/taming-chembl-sql.html), this post describes [open
source software](https://github.com/cthoyt/umls_downloader) I've
developed for downloading and working with this data. It also works for
[RxNorm](https://www.nlm.nih.gov/research/umls/rxnorm/docs/rxnormfiles.html),
[SemMedDB](https://lhncbc.nlm.nih.gov/ii/tools/SemRep_SemMedDB_SKR/SemMedDB_download.html),
[SNOMED-CT](https://www.nlm.nih.gov/healthit/snomedct/international.html),
and any other data accessible through the [UMLS Terminology Services
(UTS) ticket granting
system](https://documentation.uts.nlm.nih.gov/automating-downloads.html).

The first big issue with the UMLS is its licensing. Here's an excerpt
from the [How to License and Access the Unified Medical Language System®
(UMLS®) Data](https://www.nlm.nih.gov/databases/umls.html) page accessed
on August 28^th^, 2023:

> 1. Please sign up for a new UMLS Terminology Services (UTS) account
> with your preferred identity provider at the UTS homepage.
> 2. Complete and submit the license request form. NLM will send the
> license approval e-mail within 5 business days after reviewing
> your authenticated license request.
> 3. You will sign in using identity provider credentials to download
> files or access web interfaces that require UTS authentication
> such as the UTS, VSAC, SNOMED CT, or RxNorm.

These are a few big hurdles:

1. We typically expect scientific data to be available for download
 without login. Specifically, most data can be downloaded by
 following a link that points directly to a file. For example, ChEMBL
 v33 can be downloaded as a gzipped SQLite file from
 <https://ftp.ebi.ac.uk/pub/databases/chembl/ChEMBLdb/latest/chembl_33_sqlite.tar.gz>.
 Rather than providing a data download link, UMLS, has a complicated
 API called the UMLS Terminology Services (UTS) ticket granting
 system that needs to be asked for a specific file, polled for a
 unique access key, then downloaded via an ephemeral (i.e.,
 disappearing) URL that only works once.
2. We typically expect scientific data to be licensed under a standard,
 widely used license such as those from [Creative
 Commons](https://creativecommons.org). Using well-understood
 licenses reduces the cognitive and legal burden of consumers when
 deciding if and how they can reuse, modify, or redistribute data.
 UMLS uses a non-standard [terms of
 service](https://www.nlm.nih.gov/databases/umls.html) that makes it
 more difficult to understand how the data can be stored, modified,
 or redistributed.
3. Further, we hope that data is permissively licensed such that it can
 be re-used, modified, and re-distributed. The Creative Commons [CC
 BY 4.0](https://creativecommons.org/licenses/by/4.0/) and
 [CC0](https://creativecommons.org/publicdomain/zero/1.0/) licenses
 are golden standards of permissibility. UMLS does not have a
 permissive license, meaning (from my best interpretation) that you
 can not redistribute UMLS and you can (probably) not redistribute
 data derived from UMLS. As an aside, Creative Commons also has
 license containing clauses to be explicit about restrictions such as
 the share-alike (SA), non-commercial (NC), and non-distribution
 (ND). While these clauses aren't ideal for scientific data, it would
 at least be nice for UMLS to use a Creative Commons license with the
 appropriate combination of these clauses (I guess all three) to make
 it more explicit about its restrictions.
4. The most bizarre facet of UMLS is that they require you to fill out
 a user survey each year to keep access.

I want to 1) convert UMLS into an [OWL](https://www.w3.org/OWL/)
ontology and 2) extract and encode its semantic mappings to external
vocabularies like the [Medical Subject Headings
(MeSH)](https://bioregistry.iomesh) with [Simple Standard for Sharing
Ontology Mappings (SSSOM)](https://github.com/mapping-commons/sssom).
Given all of these hurdles, it's probably the case that I am not allowed
to redistribute these artifacts.

All together, I consider this a big bummer. The United States National
Library of Medicine (NLM) maintains several highly influential
resources, but I have found in many instances that they lack a community
perspective. Regardless, even as an expat, I pay American taxes, and it
makes me upset that the government funds the development and maintenance
of resources that I can't easily use.

## How To Break Free

![A clip of Queen\'s
](https://media.tenor.com/lfjcIHfOhqoAAAAC/freddie-mercury-queen.gif){i=""
want="" to="" break="" free\"="" music="" video\"=""}

Despite all of this rigamarole, there's a process to subvert these
issues by automating the interaction with the [UMLS Terminology Services
(UTS)](https://uts.nlm.nih.gov/uts) and therefore enabling automated
download of UMLS and the following (non-exhaustive) list of resources:

- [RxNorm](https://www.nlm.nih.gov/research/umls/rxnorm/docs/rxnormfiles.html)
- [SemMedDB](https://lhncbc.nlm.nih.gov/ii/tools/SemRep_SemMedDB_SKR/SemMedDB_download.html)
- [SNOMED-CT](https://www.nlm.nih.gov/healthit/snomedct/international.html)
- potentially more in the future

This has been implemented in the open source
[`umls_downloader`{.language-plaintext
.highlighter-rouge}](https://github.com/cthoyt/umls-downloader) Python
package. It can be installed with the following one-liner in your shell:

:::: {.language-shell .highlighter-rouge}
::: {.highlight}
``` highlight
$ pip install umls_downloader
```
:::
::::

Below, I'll walk you through using it. Throughout, keep in mind that
full documentation for the `umls_downloader`{.language-plaintext
.highlighter-rouge} is available at
[umls-downloader.readthedocs.io](https://umls-downloader.readthedocs.io),
which describes the other functionality and other data that can be
downloaded.

### Usage

UMLS has three different distributions that are described
[here](https://www.nlm.nih.gov/research/umls/licensedcontent/umlsknowledgesources.html).
The following Python code downloads the most simple and straightforward
file, `MRCONSO.RRF`{.language-plaintext .highlighter-rouge} as a zip
archive:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from umls_downloader import download_umls

path = download_umls(version="2023AA", api_key="")
```
:::
::::

This code is smart and does not need to download the file more than
once. It uses [`pystow`{.language-plaintext
.highlighter-rouge}](https://github.com/cthoyt/pystow) to choose a
stable path `~/.data/bio/umls`{.language-plaintext .highlighter-rouge}
relative to the current user's home directory. Inside this directory, it
also uses the version of the data to create a subdirectory. Finally,
this function returns the path to the data, such that no file paths ever
need to be hard-coded.

> **Warning** This still requires an API key, which requires creating an
> account, agreeing to UMLS's terms and conditions, etc. This can be
> done here: <https://uts.nlm.nih.gov/uts/edit-profile>.

### Automating Configuration of UTS Credentials

There are two ways to automatically set the API key, so you don't have
to worry about getting it and passing it around in your python code:

1. Set `UMLS_API_KEY`{.language-plaintext .highlighter-rouge} in the
 environment. This can be done in your interactive session or in the
 configuration for your shell such as in a
 `.bashrc`{.language-plaintext .highlighter-rouge} file for the
 Bourne Again Shell (bash).
2. Create `~/.config/umls.ini`{.language-plaintext .highlighter-rouge}
 and set in the `[umls]`{.language-plaintext .highlighter-rouge}
 section a `api_key`{.language-plaintext .highlighter-rouge} key.
 Mine looks like:

 :::: {.language-ini .highlighter-rouge}
 ::: {.highlight}
 ``` highlight
 [umls]
 api_key=1234567890abcdefghijklmno
 ```
 :::
 ::::

Now you can omit the `api_key`{.language-plaintext .highlighter-rouge}
keyword like in the following:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from umls_downloader import download_umls

# Same path as before
path = download_umls(version="2023AA")
```
:::
::::

### Download the Latest Version

First, you'll have to install [`bioversions`{.language-plaintext
.highlighter-rouge}](https://github.com/cthoyt/bioversions) with
`pip install bioversions`{.language-plaintext .highlighter-rouge}, whose
job it is to look up the latest version of many databases. Then, you can
modify the previous code slightly by omitting the
`version`{.language-plaintext .highlighter-rouge} keyword argument:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from umls_downloader import download_umls

# Same path as before (when run on September 1st, 2023)
path = download_umls()
```
:::
::::

### Download and open the file

The UMLS file is zipped, so it's usually accompanied by the following
boilerplate code:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
import zipfile
from umls_downloader import download_umls

path = download_umls()
with zipfile.ZipFile(path) as zip_file:
 with zip_file.open("MRCONSO.RRF", mode="r") as file:
 for line in file:
 ...
```
:::
::::

This exact code is wrapped with the
`umls_downloader.open_umls()`{.language-plaintext .highlighter-rouge}
using Python's context manager, so it can more simply be written as:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from umls_downloader import open_umls

with open_umls() as file:
 for line in file:
 ...
```
:::
::::

> **Note** The `version`{.language-plaintext .highlighter-rouge} and
> `api_key`{.language-plaintext .highlighter-rouge} arguments work the
> same for `umls_downloader.open_umls()`{.language-plaintext
> .highlighter-rouge} as in
> `umls_downloader.download_umls()`{.language-plaintext
> .highlighter-rouge}

At this point, it's up to you to decide how you want to consume the
`MRCONSO.RRF`{.language-plaintext .highlighter-rouge} file. Below, I
give a demo on how parsed this file in PyOBO in order to convert UMLS to
an OWL ontology.

## Why not an API?

The UMLS provides an
[API](https://documentation.uts.nlm.nih.gov/rest/home.html) for access
to tiny bits of data at a time. There are even two recent (last 5 years)
packages [`umls-api`{.language-plaintext
.highlighter-rouge}](https://pypi.org/project/umls-api)
[`connect-umls`{.language-plaintext
.highlighter-rouge}](https://pypi.org/project/connect-umls) that provide
a wrapper around them. However, API access is generally rate limited,
difficult to use in bulk, and slow. For working with UMLS (or any other
database, for that matter) in bulk, it's necessary to download full
database dumps.

## UMLS Conversions

Building on top of the automated download of UMLS, I implemented a
fit-for-purpose processor with the [PyOBO
framework](https://github.com/pyobo/pyobo) that converts UMLS into an
ontology (encoded either as OWL, OBO, or OBO Graph JSON) which can
therefore be used to generate semantic mappings in the SSSOM format. The
code that implements this can be found
[here](https://github.com/pyobo/pyobo/commit/be67eadf77afa270cf224cad71d5b1905ae78b53).
After installing PyOBO with `pip install pyobo`{.language-plaintext
.highlighter-rouge}, you can automatically download and convert UMLS
first into an ontology encoded in the [OBO flat file
format](https://owlcollab.github.io/oboformat/doc/GO.format.obo-1_4.html),
then convert to OWL with the following code. Note: you'll need
[`robot`{.language-plaintext
.highlighter-rouge}](https://robot.obolibrary.org/) for the second step:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
import pyobo

umls = pyobo.get_ontology("umls")

# Write simple OBO Format
umls.write_obo("umls.obo")

# Convert to OWL
from pyobo.utils.misc import obo_to_owl

obo_to_owl("umls.obo", "umls.owl")
```
:::
::::

In an ideal world, the results of such a conversion could be included as
a part of the [OBO Database
Ingestion](https://github.com/biopragmatics/obo-db-ingest), which
converts database resources available through PyOBO into ontology
artifacts, archives them on GitHub and Zenodo, and gives them PURLs all
on a weekly basis to make sure the most up-to-date version is available
as well as all previous named versions. Instead, we live in a world with
pineapple pizza and restrictive licenses.

One of the nice qualities of UMLS is that it is a semantic mapping hub.
It provides mostly complete mappings between many vocabularies including
[MeSH](https://bioregistry.io/mesh),
[NCIT](https://bioregistry.io/ncit),
[SNOMED-CT](https://bioregistry.io/snomedct),
[HPO](https://bioregistry.io/hpo),
[LOINC](https://bioregistry.io/loinc), and more. However, there are a
few caveats to consider:

1. UMLS mappings aren't all 1-to-1. For example, MeSH mappings
 typically include many UMLS terms (narrower) pointing to the same
 MeSH term (broader). For other vocabularies, such as NCBITaxon, UMLS
 mappings are more reliably 1-to-1. Thanks to Tiago Lubiana for
 pointing this out.
2. Mapping provenance is not available, so the
 `mapping_justification`{.language-plaintext .highlighter-rouge}
 field in SSSOM is uniformly filled with
 `sempav:UnspecifiedMatching`{.language-plaintext
 .highlighter-rouge}.
3. Similarly, UMLS does not apply precise semantic predicates for each
 mapping. This means that they are output in PyOBO and as SSSOM with
 `oboInOwl:hasDbXref`{.language-plaintext .highlighter-rouge} instead
 of more detailed types such as `skos:exactMatch`{.language-plaintext
 .highlighter-rouge}, `skos:narrowMatch`{.language-plaintext
 .highlighter-rouge}, and `skos:broaderMatch`{.language-plaintext
 .highlighter-rouge}. Tools like
 [Boomer](https://github.com/INCATools/boomer) can be used to address
 this (in part). The [Semantic Mapping Reasoning Assembler
 (SeMRA)](https://github.com/biopragmatics/semra) can also be
 configured with prior knowledge about UMLS mapping assumptions when
 aggregating and reasoning over semantic mappings at scale.

With that in mind, anything that can be loaded as an ontology in PyOBO
can also be exported with SSSOM, which I show below. For UMLS, this
looks like:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
import pyobo

df = pyobo.get_sssom_df("umls", names=False)
df.to_csv("umls.sssom.tsv", sep="\t", index=False)
```
:::
::::

> **Note** You can set `names=True`{.language-plaintext
> .highlighter-rouge} to have PyOBO look up the names for all entities,
> but this is a bit of a rabbit hole since it requires getting and
> processing many external resources.

------------------------------------------------------------------------

There's much more to say about UMLS and SSSOM, but this is a good place
to pause and publish this post, since getting UMLS as SSSOM is a task a
lot of people have asked me for help with lately. I might also come back
and explain more about how I use the other resources from UMLS's UTS.