Rants, raves (and occasionally considered opinions) on phyloinformatics, taxonomy, and biodiversity informatics.  For more ranty and less considered opinions, see my Twitter feed.ISSN 2051-8188. Written content on this site is licensed under a Creative Commons Attribution 4.0 International license.

University of Glasgow

Roderic Page

One of the potentially powerful features of TreeBASE II is availability of a RDF version of a study. This means that, in principle, one could take the RDF for a TreeBASE study, combine it with RDF from other sources, and generate a richer view of a particular study.

TreeBASE II RDF

I'm giving a short talk at the Workshop On Open Citations And Open Scholarly Metadata 2020, which will be held online on September 9th.

Workshop On Open Citations And Open Scholarly Metadata 2020 talk

Just a placeholder to mark the ongoing impact of the Internet Archive being attacked (see here, here and here for details).  The impact of this on the Biodiversity Heritage Library (BHL) has been huge, and reveals the extent to which BHL depends on the Archive.

Internet Archive as a single point of failure

Recently I’ve been exploring data downloaded from BOLD. Part of this was motivated by work done with David Schindel for a recent book:  In this blog post I record some struggles I’ve had with the supposedly “Frictionless” data provided by BOLD. I list a serious of issues, and make some recommendations as to how these can be fixed. Previous versions disappear from site   The web page Data Packages lists datasets that can be downloaded.

Exploring BOLD's DNA barcode data releases: there's a fraction too much friction

Following the 2024 BHL meeting, and the departure of Martin Kalfatovic and the uncertainty the departure of such a pivitol person brings, perhaps it’s time to think about the future of BHL. Below I sketch some thoughts, which are hazy at best. I should say at the outset that I think BHL is an extraordinary project. My goal is to think about ways to enhance its utility and impact.

A future for the Biodiversity Heritage Library

TL;DR  These are some brief notes on the latest version (v. 2) of the Data Citation Corpus, relased shortly before the Make Data Count Summit 2024, which also included a discussion on the practical uses of the corpus. I downloaded version 2 from Zenodo doi:10.5281/zenodo.13376773. The data is in JSON format, which I then loaded into CouchDB to play with.

The Data Citation Corpus revisited

How to cite:

 Page, R. (2024). Problems with the DataCite Data Citation Corpus https://doi.org/10.59350/t80g1-xys37

DataCite have released the Data Citation Corpus, together with a dashboard that summarises the corpus. This is billed as: The goal is to build a citation database between scholarly articles and data, such as datasets in repositories, sequences in GenBank, protein structures in PDB, etc.

Problems with the DataCite Data Citation Corpus

How to cite:

 Page, R. (2023). It’s 2023 - why are we still not sharing phylogenies? https://doi.org/10.59350/n681n-syx67

A quick note to support a recent Twitter thread https://twitter.com/rdmpage/status/1729816558866718796?s=61&amp;t=nM4XCRsGtE7RLYW3MyIpMA The article “Diversification of flowering plants in space and time” by Dimitrov et al. describes a genus-level phylogeny for 14,244 flowering plant genera.

It's 2023 - why are we still not sharing phylogenies?

How to cite:

 Page, R. (2023). Where are the plant type specimens? Mapping JSTOR Global Plants to GBIF. https://doi.org/10.59350/m59qn-22v52

This blog post documents my attempts to create links between two major resources for plant taxonomy: JSTOR’s Global Plants and GBIF, specifically between type specimens in JSTOR and the corresponding occurrence in GBIF.

Where are the plant type specimens? Mapping JSTOR Global Plants to GBIF

A System for Converting PDF Documents into Structured XML Format

Unsupervised document structure analysis of digital scientific articles

Header and footer extraction by page association

Ceci n'est pas un hamburger: modelling and representing the scholarly article

Layout-aware text extraction from full-text PDF of scientific articles

VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups

How to cite:

 Page, R. (2023). Document layout analysis. https://doi.org/10.59350/z574z-dcw92

Some notes to self on document layout analysis. I’m revisiting the problem of taking a PDF or a scanned document and determining its structure (for example, where is the title, abstract, bibliography, where are the figures and their captions, etc.). There are lots of papers on this topic, and lots of tools.

Document layout analysis

How to cite:

 Page, R. (2023). The problem with GBIF’s Phylogeny Explorer. https://doi.org/10.59350/v0bt3-zp114

GBIF recently released the Phylogeny Explorer, using legumes as an example dataset. The goal is to enables users to “view occurrence data from the GBIF network aligned to legume phylogeny.” The screenshot below shows the legume phylogeny side-by-side with GBIF data.

The problem with GBIF's Phylogeny Explorer

How to cite:

 Page, R. (2023). Sub-second searching of millions of DNA barcodes using a vector database. https://doi.org/10.59350/qkn8x-mgz20

Recently I’ve been messing about with DNA barcodes.

Sub-second searching of millions of DNA barcodes using a vector database

Pensoft have recently introduced “nanopubs”, small structured publications that can be thought of as containing the minimum possible statement that could be published. Nanopubs are promoted as FAIR, that is findable, accessible, interoperabile, and reusable. I like the idea of nanopubs, but the examples I have seen so far are problematic.

Nanopubs, a way to create even more silos

This blog post has some notes in support of a talk given to the Systematics Association meeting in Reading June 20th, 2024.

 Slides

I will post a link to the slides here once I have given the talk. Page, Roderic (2024). Visualising big trees. figshare. Presentation.

Visualising big trees: a talk at the Systematics Association 2024

How to cite:

 Page, R. (2024). Notes on transforming BHL images https://doi.org/10.59350/2gpbb-98a53

I’ve been down this road before, e.g. BHL, DjVu, and reading the f*cking manual and Demo of full-text indexing of BHL using CouchDB hosted by Cloudant, but I’m revisiting converting BHL page scans to black and white images, partly to clean them up, to make them closer to what a modern reader might expect, and partly to reduce the

Notes on transforming BHL images

How to cite:

 Page, R. (2023). What, if anything, is the Biodiversity Knowledge Hub?

What, if anything, is the Biodiversity Knowledge Hub?

How to cite:

 Page, R. (2023). Adventures in machine learning: iNaturalist, DNA barcodes, and Lepidoptera. https://doi.org/10.59350/5q854-j4s23

Recently I’ve been working with a masters student, Maja Nagler, on a project using machine learning to identify images of Lepidoptera. This has been something of an adventure as I am new to machine learning, and have only minimal experience with the Python programming language.

Adventures in machine learning: iNaturalist, DNA barcodes, and Lepidoptera

How to cite:

 Page, R. (2023). A taxonomic search engine. https://doi.org/10.59350/r3g44-d5s15

Tony Rees commented on my recent post Ten years and a million links. I’ve responded to some of his comments, but I think the bigger question deserves more space, hence this blog post. Tony’s comment My response I think there are several ways to approach this.

A taxonomic search engine

One thing about ChatGPT is it has opened my eyes to some concepts I was dimly aware of but am only now beginning to fully appreciate. ChatGPT enables you ask it questions, but the answers depend on what ChatGPT “knows”. As several people have noted, what would be even better is to be able to run ChatGPT on your own content. Indeed,  ChatGPT itself now supports this using plugins.

ChatGPT, semantic search, and knowledge graphs

I haven’t blogged for a while, work and other reasons have meant I’ve not had much time to think, and mostly I blog to help me think. ChatGPT is obviously a big thing at the moment, and once we get past the moral panic (“students can pass exams using AI!”) there are a lot of interesting possibilities to explore.

ChatGPT, of course

I heard yesterday from Martin Kalfatovic (BHL) that David Remsen has died. Very sad news.

David Remsen

As trailed on a Twitter thread last week I’ve been working on a manuscript describing the efforts to map taxonomic names to their original descriptions in the taxonomic literature. The preprint is on bioRxiv doi:10.1101/2023.05.29.542697  Much of the work has been linking taxa to names, which still has huge gaps.

Ten years and a million links

Some quick notes on interface ideas for digital libraries and/or knowledge graphs. Recently there’s been something of an explosion in bibliographic tools to explore the literature.

Library interfaces, knowledge graphs, and Miller columns

My dad died last weekend. Below is a notice in today's New Zealand Herald. I'm in New Zealand for his funeral. Don't really have the words for this right now.

Dugald Stuart Page 1936-2022

More arm-waving notes on taxonomic databases. I've started to add data to ChecklistBank and this has got me thinking about the issue of data quality.

Can we use the citation graph to measure the quality of a taxonomic database?

Taxonomic treatments have come up in various discussions I'm involved in, and I'm curious as to whether they are actually being used, in particular, whether they are actually being cited. Consider the following quote:    "Traditional" academic citation is from article to article.

Does anyone cite taxonomic treatments?

I keep returning to the problem of viewing large graphs and trees, which means my hard drive has accumulated lots of failed prototypes. Inspired by some recent discussions on comparing taxonomic classifications I decided to package one of these (wildly incomplete) prototypes up so that I can document the idea and put the code somewhere safe.

Large graph viewer experiments

The Plazi project has become one of the major contributors to GBIF with some 36,000 datasets yielding some 500,000 occurrences (see Plazi's GBIF page for details). These occurrences are extracted from taxonomic publication using automated methods.

Problems with Plazi parsing: how reliable are automated methods for extracting specimens from the literature?

Just some thoughts as I work through some datasets linking taxonomic names to the literature. In the diagram above I've tried to capture the different situatios I encounter. Much of the work I've done on this has focussed on case 1 in the diagram: I want to link a taxonomic name to an identifier for the work in which that name was published. In practise this means linking names to DOIs.

Linking taxonomic names to the literature

There are several instances where I have a collection of references that I want to deduplicate and merge.

Deduplicating bibliographic data

I've released a very crude GraphQL endpoint for WikiData. More precisely, the endpoint is for a subset of the entities that are of interest to WikiCite, such as scholarly articles, people, and journals. There is a crude demo at https://wikicite-graphql.herokuapp.com. The endpoint itself is at https://wikicite-graphql.herokuapp.com/gql.php.

GraphQL for WikiData (WikiCite)

uBioRSS: Tracking taxonomic literature using RSS

Aggregating, Tagging and Integrating Biodiversity Research

Over a decade ago RSS (RDF Site Summary or Really Simple Syndication) was attracting a lot of interest as a way to integrate data across various websites. Many science publishers would provide a list of their latest articles in XML in one of three flavours of RSS (RDF, RSS, Atom). This led to tools such as uBioRSS [1] and my own e-Biosphere Challenge: visualising biodiversity digitisation in real time.

Revisiting RSS to monitor the latest taxonomic research

Note to self. The challenge of finding specimen citations in papers keeps coming around. It seems that this is basically the same problem as finding citations to papers, and can be approached in much the same way. If you want to build a database of reference from scratch, one way is to scrape citations from papers (e.g., from the "literature cited" section), convert those strings into structured data, and add those to your database.

Finding citations of specimens

I've created a GitHub repository so that I can keep track of the examples of JSON-LD that I've seen being actively used, for example embedded in web sites, or accessed using an API. The repository is https://github.com/rdmpage/wild-json-ld. The list is by no means exhaustive, I hope to add more examples as I come across them. One reason for doing this is to learn what others are doing.

JSON-LD in the wild: examples of how structured data is represented on the web

If you compare the impact that BHL and Plazi have on GBIF, then it's clear that BHL is almost invisible. Plazi has successfully in carved out a niche where they generate tens of thousands of datasets from text mining the taxonomic literature, whereas BHL is a participant in name only. It's not as if BHL lacks geographic data.

Thoughts on BHL, ALA, GBIF, and Plazi

How to cite:

 Page, R. (2021). Maximum entropy summary trees to display higher classifications https://doi.org/10.59350/af01t-6sw74

A challenge in working with large taxonomic classifications is how you display them to the user, especially if the user probably doesn't want all the gory details.

Maximum entropy summary trees to display higher classifications

D. Ross Robertson has published a paper entitled "Global biogeographical data bases on marine fishes: caveat emptor" (doi:10.1111/j.1472-4642.2008.00519.x - DOI is broken, you can get the article here). The paper concludes:  As I've noted elsewhere on this blog, and as demonstrated by Yesson et al.'s paper on legume records in GBIF (doi:10.1371/journal.pone.0001124) (not cited by Robertson), there are major problems with geographical information

Global biogeographical data bases on marine fishes: caveat emptor

Note to self (basically rewriting last year's Finding citations of specimens). Bibliographic data supports going from identifier to citation string and back again, so we can do a "round trip."

 1.

Given a DOI we can get structured data with a simple HTTP fetch, then use a tool such as citation.js to convert that data into a human-readable string in a variety of formats.

Round trip from identifiers to citations and back again

I've made Species Cite live. This is a web site I've been working on with the GBIF Challenge as a notional deadline so I'll actually get something out the door. "Species Cite" takes as its inspiration the suggestion that citing original taxonomic descriptions (and subsequent revisions) would increase citation metrics for taxonomists, and give them the credit they deserve.

Species Cite: linking scientific names to publications and taxonomists

Quick note on Frankenplace, a cool search tool that displays the geographic distribution of documents that match the user's query as a heatmap. Details of how the tool works are given in:  At the heart of the method is a discrete global grid that divides the world up into small areas of the same size.

Frankenplace, geospatial search, and discrete global grid systems

My paper "Ozymandias: A biodiversity knowledge graph" has been published in PeerJ https://doi.org/10.7717/peerj.6739  The paper describes my entry in GBIF's 2018 Ebbe Nielsen Challenge, which you can explore here. I tweeted about its publication yesterday, and got some interesting responses (and lots of retweets, thanks to everyone for those).  Carl Boettiger (@cboettig) asked where the triples were, as did Kingsley Uyi Idehen (@kidehen). Doh!

Ozymandias: A biodiversity knowledge graph published in PeerJ

I've made a video walkthrough of Ozymandias, which I described in this post. It's a bit, um, long, so I'll need to come up with a shorter version.  Ozymandias - a biodiversity knowledge graph from Roderic Page on Vimeo.

Ozymandias demo

David Shorthouse (@dpsspiders) makes some very cool things, and his latest project World Taxonomists &amp; Systematists is a great example of using automation to assemble a list of the world's taxonomists and systematists. The project uses ORCID.

World Taxonomists and Systematists via ORCID

Following on from previous posts The Semantic Web made fun: d3sparql and The Biodiversity Heritage Library meets Wikidata via Wikispecies: adding author identifiers to BioStor I've put together an example query that can be used to extract a taxonomic classification from Wikidata.

Displaying taxonomic classifications from Wikidata using d3js and SPARQL

It's Friday, so time for either a folly or a rant. BHL have put another user survey into the field http://www.surveymonkey.com/s/BHLsurvey. I loathe user surveys.

Where next for BHL?

In between complaining about the lack of open data in biodiversity (especially taxonomy), and scraping data from various web sites to build stuff I'm interested in, I occasionally end up having interesting conversations with the people whose data I've been scraping, cleaning, cross-linking, and otherwise messing with. Yesterday I had one of those conversations at Kew Gardens.

On asking for access to data

In a recent Twitter conversation including David Shorthous and myself (and other poor souls who got dragged in) we discussed how to demonstrate that adopting JSON-LD as a simple linked-data friendly format might help bootstrap the long awaited "biodiversity knowledge graph" (see below for some suggestions for keeping JSON-LD simple). David suggests partnering with "Three small, early adopting projects". I disagree.

Bootstrapping the biodiversity knowledge graph with JSON-LD

I'm continuing to play with the new version of iSpecies, seeing just how far one can get by simply grabbing JSON from various sources and mashing them up. Since the Open Tree of Life is pretty unresolved ("OMG it's full of stars") I've started to grab trees from TreeBASE and add those.

iSpecies meets TreeBASE

In my (previous post ) I discussed the potential for the Biodiversity Data Journal (BDJ) to be a venue for nano (or near-nano publications). In this post I want to draw attention to what I think is a serious stumbling block, which is the lack of machine readable statements in the journal.

The Biodiversity Data Journal is not machine readable

I've put together a working demo of some code I've been working on to discover GBIF records that correspond to museum specimen codes. The live demo is at http://bionames.org/~rpage/material-examined/ and code is on GitHub. To use the demo, simply paste in a specimen code (e.g., "MCZ 24351") and click

 Find

and it will do it's best to parse the code, then go off to GBIF and see what it can find.

Linking specimen codes to GBIF

I'm going to the TDWG Identifier Workshop this weekend, so I thought I'd jot down a few notes. The biodiversity informatics community has been at this for a while, and we still haven't got identifiers sorted out. From my perspective as both a data aggregator (e.g., BioNames) and a data provider (e.g., BioStor) there are four things I think we need to tackle in order to make significant progress.

On identifiers (again)

Viewing phylogenies on the web: Javascript conversion of Newick tree to SVG

I've just come back from II Iberian Congress of Biological Systematics (CISA2013) in Barcelona, where I had a great time.

Biodiversity informatics in charts

Continuing the theme of the failings of the GBIF classification I've been playing further with cluster maps to visualise the problem (see this earlier post for an introduction).  Browsing through bats in GBIF I keep finding the same species appearing more than once, albeit in different genera.

Cluster maps, papaya plots, and the trouble with GBIF taxonomy

Readers of this blog will know that I'm sceptical about the current value of linked data and RDF in biodiversity informatics. But I came across an interesting paper on RDF and biocuration that suggests a good "use case" for RDF in constructing and curating taxonomic databases. The paper is "Catching inconsistencies with the semantic web: a biocuration case study" (PDF here) by Jerven Bolleman and Sebastien Gehant.

A use case for RDF in taxonomy

My latest tweak to BioNames is to add colour to the phylogenies. Terminal nodes with the same name are labelled with the same background colour. For example, here is a tree for fiddler and ghost crabs: The colours make it easier to see that this tree has a mixture of a few sequences from divergent taxa, and a lot of sequences from the same taxa. Note that you can now also download the SVG drawing of the tree.

BioNames - colourful phylogenies and downloadable SVG

Thinking about next steps for my BioStor project, one thing I keep coming back to is the problem of how to dramatically scale up the task of finding taxonomic literature online. While I personal find it oddly therapeutic to spend a little time copying and pasting citations into BioStor's OpenURL resolver and trying to find these references in BHL, we need something a little more powerful.

Next steps for BioStor: citation matching

Quick note to self about possible way to using fuzzy matching when searching for taxonomic names. Now that I'm using Cloudant to host CouchDB databases (e.g., see BioStor in the the cloud) I'd like to have a way to support fuzzy matching so that if I type in a name and misspelt it, there's a reasonable chance I will still find that name. This is the "did you mean?" feature beloved by Google users.

Fuzzy matching taxonomic names using ngrams

Following on from the last post, I've now set up a trivial NCBI RDF service at bioguid.info/taxonomy/ (based on the ISSN resolver I released yesterday and announced on the Bibliographic Ontology Specification Group).  If you visit it in a web browser it's nothing special. However, if you choose to display XML you'll see some simple RDF.

NCBI RDF

Prompted by a conversation with Vince Smith at the recent Online Taxonomy meeting at the Linnean Society in London I've been revisiting touch-based displays of large trees. There are a couple of really impressive examples of what can be done. Perceptive Pixel   I've blogged about this before, but came across another video that better captures the excitement of touch-based navigation of a taxonomy.

Touching the tree of life

If we are ever going to link biodiversity data together we need to have some way of ensuring persistent links between digital records. This isn't going to happen unless people take persistent identifiers seriously. I've been trying to link specimen codes in publications to GBIF, with some success, so imagine my horror when it started to fall apart.

Dear GBIF, please stop changing occurrenceIDs!

In any discussion of data gathering or data cleaning the term "crowdsourcing" inevitably comes up. A example where this approach has been successful is the Encyclopedia of Life's Flickr pool, where Flickr users upload images that are harvested by EOL.

Where is the "crowd" in crowdsourcing? Mapping EOL Flickr photos

Here is my presentation from today's Anchoring Biodiversity Information: From Sherborn to the 21st century and beyond meeting.

Sherborn presentation on Open Taxonomy

This is a follow up to my previous post TDWG Challenge - what is RDF good for? where I'm being, frankly, a pain in the arse, and asking why we bother with RDF? In many ways I'm not particularly anti-RDF, but it bothers me that there's a big disconnect between the reasons we are going down this route and how we are actually using RDF.

Reflections on the TDWG RDF "Challenge"

One side effect of playing with ways to visualise and integrate biology databases is that you stumble across the weird and wonderful stuff that living organisms get up to. My earliest papers were on crustacean taxonomy, so I thought I'd try my latest toy on them.

Fun things about crustaceans

More zoom viewer experiments (see previous post), this time with a linked map that updates as you browse the tree (SVG-capable browser required). As you browse the frog classification the map updates to show the location of georeferenced sequences in GenBank from the taxa in the part of the tree you are looking at. The map is limited to not more than 200 localities, and many frog sequences aren't georeferenced, but it's a fun way to combine

Geography and genes: zoomable view of frog NCBI classification with linked map

Continuing on from my previous post Viewing scientific articles on the iPad: towards a universal article reader, here are some brief notes on the PLoS iPad app that I've previously been critical of.  There are two key things to note about this app. The first is that it uses the page turning metaphor. The article is displayed as a PDF, a page at a time, and the user swipes the page to turn it over.

Viewing scientific articles on the iPad: the PLoS Reader

To much fanfare (e.g.,

 Nature News

, "Linnaeus meets the Internet" doi:10.1038/news.2010.221), on May 5th

 PLoS ONE

published Sandy Knapp's "Four New Vining Species of

 Solanum

(Dulcamaroid Clade) from Montane Habitats in Tropical America" doi:10.1371/journal.pone.0010502.

Linnaeus meets the Internet: PLoS + Botany =  #fail

My short note on "Wikipedia as an Encyclopaedia of Life" has appeared in

 Organisms Diversity &amp;

Wikipedia paper out

This week seems to be API week. The Encyclopedia of Life API Beta Test has been out since August 12th. By comparison with the Mendeley API that I've spent rather too much time trying to get to grips with, the EOL API release seems rather understated.

Navigating the Encyclopedia of Life tree on the desktop and the iPhone

Being in an unusually constructive mood, I've spent the last couple of days playing with the TreeBASE II API, in an effort to find out how hard it would be to replace TreeBASE's frankly ghastly interface. After some hair pulling and bad language I've got something to work. It's very crude, but gives a glimpse at what can be done.

Show me the trees! Playing with the TreeBASE API

Every so often I revisit the idea of browsing a collection of documents (or specimens, or phylogenies) geographically. It's one thing to display a map of localities for single document (as I did most recently for

 Zootaxa

), it's quite another to browse a large collection.

Browsing a digital library using a map

I've been playing a little with TreeBASE II, and the more I do the more I want to pull my hair out.

 Broken URLs

The old TreeBASE had a URL API, which databases such as NCBI made use of. For example, the NCBI page for

 Amphibolurus nobbi

has a link to this taxon in TreeBASE.

TreeBASE II makes me pull my hair out

Time for a Friday folly. I've made a clunky screencast showing an example of linking biodiversity data together, using bioGUID as the universal wrapper around various data sources. I started with GenBank sequence EF013683, added another, EF013555, then explored some links (specimen, publication, taxon, journal), using the OpenLink RDF Browser:  Linking biodiversity data from Roderic Page on Vimeo.

Linking biodiversity data

Just noticed that BioStor now has just over 70,000 articles extracted from the Biodiversity Heritage Library. This number is a little "soft" as there are some duplicates in the database that I need to clean out, but it's a nice sounding number. Each article has full text available, and in most cases reasonably complete metadata.

70,000 articles extracted from the Biodiversity Heritage Library

Yesterday I fired off a stream of tweets, starting with:  Various people commented on this, either on twitter or in emails, e.g.:  So, to clarify, I'm not abandoning wikis. I'm just frustrated with the limitations of Semantic Mediawiki (SMW). Now, SMW is a great piece of software with some cool features.

Wiki frustration

This post is simply a quick note on some experiments with DjVu that I haven't finished. Much of BHL's content is available as DjVu files, which contain both the scanned images and OCR text, complete with co-ordinates of each piece of text. This means that it would, in principle, be trivial to lay out the bounding boxes of each text element on a web page.

DjVu XML to HTML

As part of the discussion on whether legacy biodiversity literature matters a graph from the following paper came up:  So, why is the Sarkar et al. graph bogus? Here is their graph (Fig. 3) for animals:    This is the number of new animal species described each year, estimated by parsing taxonomic names and extracting the date in the taxonomic authority. There are two prominent "spikes" which are worrying.

Rate of description of new animal species and *that* Taxatoy graph

My BioStor project has reached over 13,000 articles, making it a sizeable respository of open access articles on biodiversity. It's still a tiny fraction of what could be extracted from the Biodiversity Heritage Library (BHL), but perhaps it's worth taking stock of what's there.

 Coverage

One pleasing discovery is that, despite the 1923 cut-off due to U.S. copyright, BHL contains a lot of post-1923 articles.

BioStor so far

One of the more glaring limitations of my BHL viewer described in the previous post is that it can take a while to load all the page thumbnails (there can be hundreds). Given that one of the original motivations for this project was a faster viewer, this kinda sucks. What I'd like to do is load the thumbnails only when I need them, rather than all at once at the start -- in other words I'd like to implement lazy loading.

BHL Viewer now with go faster stripes

The Science Commons has released a short video by Jesse Dylan, who made the Yes We Can video.

Yes We Can - "scientists are the ultimate remixers"

I've been playing recently with the Biodiversity Heritage Library (BHL), and am starting to get a sense for the complexities (and limitations) of the metadata BHL stores about publications. The more I look at BHL the more I think the resource is (a) wonderfully useful and (b) hampered by some dodgy metadata.

Biodiversity Heritage Library, Google books, and  metadata quality

One thing I find myself doing a lot is creating Excel spreadsheets and filling them will lists of taxonomic names and bibliographic references, for which I then try to extract identifiers (such as DOIs). This is a tedious business, but the hope is that by doing it once I can create a useful resource. However, often I get bored and the spreadsheets lie forgotten in some deep recess of my computer's hard drive.

Using Google Spreadsheets and RSS feeds to edit IPNI

TreeView X, the open source version of TreeView, has been slowly suffering bit rot as C++ compilers and operating systems change. Every so often I'd tweak the code to build on some Linux version or other, but this isn't something I've a lot of time for. Moreover, because of the hassle of rebuilding binaries and source tar balls the updated versions weren't uploaded to the TreeView X web site.

TreeView X now on Google Code

For the last two days I've been participating in a NESCent meeting on Dryad, a "repository of data underlying scientific publications, with an initial focus on evolutionary biology and related fields". The aim of Dryad is to provide a durable home for the kinds of data that don't get captured by existing databases such as GenBank and TreeBASE (for example, the Excel spreadsheets, Word files, and tarballs of data that, if they are lucky, make it

Dryad, DOIs, and why data matters more than journal articles

For a short, but reasonably technical sumary of what I think the issues are, please read this "Technical Report", which I presented at the Workshop on Database Issues in Biological Databases (DBiBD) in Edinburgh in January 2005. This document is itself based on a BBSCR grant proposal which was funded. Here is the abstract.

Towards a Taxonomically Intelligent Phylogenetic Database

It's Friday, so time for some random, half-baked ideas. Imagine that we have a database of evolutionary trees, and these overlap for a set of taxa that we are interested in. How do we summarise these trees? One approach is to make a supertree. It would be useful to display the subtrees that went into making this supertree, if only to give an idea of how much they agree with the supertree. How to do this?

Photosynth for trees - supertrees revisited

Vince Smith has produced a nice flyer for my forthcoming talk at The Natural History Museum on March 17th (11-12).    It will be a busy day as I'm also talking at the British Library in the evening (6pm - 8:30pm), for which Sarah Kemmitt has produced a flyer, and set up a discussion forum on Nature Network. With all this effort going into the artwork, I'd better actually come up with something useful to say.

Talks at The Natural History Museum and British Library

Last night BBC One aired David Attenborough's Charles Darwin and the Tree of Life, which featured a lovely "fly through" the tree of life:  In conjunction with the TV show, the Wellcome Trust has launched the Interactive Tree of Life, a Flash-based view of the tree of life. There's also a blog about the project. Here's a demo of the tree:  The tree looks very nice, and a lot of work has gone into it, but I am somewhat underwhelmed.

Thoughts on the Wellcome Interactive Tree of Life

I'm in the US on UK time, so this is probably a bad idea to write this, but the paper by Malte Ebach et al. ("O Cladistics, Where Art Thou?", doi:10.1111/j.1096-0031.2008.00225.x) in the latest

 Cladistics

just annoys me too much.

Death throes of Cladistics

Given that the Twitter stream tagged #vizbi will fade away soon, I've grabbed most of the links I tweeted during VIZBI 2011 and have put them here. This isn't intended as a comprehensive list, merely the things which caught my eye, and didn't flash by faster than I could tweet.

Some VIZBI 2011 links

Next few weeks will be busy with term starting, kids visiting, and other commitments, so time to jot down some ideas. The first is to have a Wiki for taxonomic names. Bit like Wikispecies, but actually useful, by which I mean useful for working biologists. This would mean links to digital literature (DOIs, Handles, etc.), use of identifiers for names and taxa (such as NCBI taxids, LSIDs, etc.), and having it pre-populated with data.

Half-baked ideas. I. Wiki for taxonomy

Starting to get serious about the Grand Challenge. First step is to parse the XML data Elsevier made available. Sadly this is only for Molecular Phylogenetics and Evolution for 2007, I would have liked the whole journal in XML to avoid hassles with parsing PDF. However, XML is not without it's own problems.

Hell is other people's data

The good news is that the merger of Blackwell's digital content with that of Wiley's has not affected the DOIs, which is exactly as you'd expect, and is a nice demonstration of the power of identifiers that use indirection (although there was a time when Wiley was offline).  For example, the article identified by doi:10.1111/j.1095-8312.2003.00274.x had the URL http://www.blackwell-synergy.com/doi/abs/10.1111/j.1095-8312.2003.00274.x and now has

DOIs, the good news and the bad news

I've added a service to bioGUID that takes a PDF and attempts to extract latitude and longitude data from the PDF, returning those co-ordinates in either a Google Earth KML file, or in JSON format. This is one of a bunch of services that I'm adding to bioGUID to support some of the data mining that I'm doing.

From PDFs to Google Earth

From Nature's blog on web technology and science comes this post on Open Text Mining Interface (OTMI): and further Currently playing in iTunes:

 By the Time I Get to Phoenix

by Glen Campbell

iPhylo

Adding Solr to BioStor: searching for real