I have been thinking, this week, about the observability of AWS Lambda functions in API Gateway contexts. The major challenge is that Prometheus metrics pose a problem as they are pull-only (via a scraping endpoint). Prometheus metrics are stored in a temporary disk cache and then pulled off-site by Grafana etc.
LocalStack is a great cloud emulation layer. It lets you simulate interaction with AWS, which is great for writing integration tests. However, I wanted a system that, when run locally, would spin up the LocalStack server and then destroy it when done. But when running the test on GitLab CI, it will use the “service” provision of their continuous integration system and connect to that.
In my new role at Crossref I work on a series of data pipelines for research and development projects. These are resource-intensive data processing tasks that need to be executed periodically on a schedule, with good observability, but also with parallel processing capacity. Amazon’s Managed Workflows for Apache Airflow (MWAA) seems like an ideal solution for this.
I am currently conducting a research project at Crossref that requires me to build a database using large backend files (e.g. building a relational database from a 3GB XML file). We need to rebuild this monthly, so Apache Airflow seemed a good tool to run these periodic tasks. There are, however, lots of “gotchas” in this framework that can trip up a newcomer and I thought it might be helpful to document some of these.
So, after three years of shielding, I got Covid. I contracted it at hospital (or on my way there). How do I know? Because I don’t go anywhere else. I thought, though, that it might be useful to document my experience of getting antiviral treatments for Covid as an extremely clinically vulnerable individual. I developed a sore throat on Saturday evening (17th December 2022) but tested negative.
As I posted a while ago, from January 2023 I will be working at Crossref while retaining my university Professorship. I wanted, here, to outline a few of the projects that I hope to work on once I get started there. I should say upfront: I am afraid there is no time estimate on these and we can’t guarantee to prioritise any particular project. But if there is one that stands out to you, do let me know, as this serves as a useful community gauge.
Like many years, 2022 was a year of health problems for me. The entire year has been overshadowed by the episode of kidney failure that I suffered as a result of BK virus associated nephropathy. It is fair to say that I have been quite seriously unwell. I also spent a long period of this year, in my spare time, campaigning for protection of the immunocompromised with the monoclonal antibody therapy, Evusheld.
As many of you know, I have been involved for the past few months in a campaign to get Evusheld – a protective/prophylactic drug for immunocompromised people who do not respond well to Covid vaccines – available on the NHS in the UK. We have not succeeded on that front yet and I am not giving up. However, for my own protection, since AstraZeneca made the drug privately available last week, I have been able to procure it privately.
The other day I wrote about the response that we had from the UK’s Department of Health and Social Care. In that reply the DHSC said that it had a problem with some of the evidence we had put forward because “it should also be noted that most of these studies are pre-prints (therefore have not been scrutinised through peer review)”. A few further notes on this.
This week has been one in which my personal and professional lives have aligned in interesting ways. As you may know, one strand of my work focuses on the study of academic publishing, including peer review. This has featured discussions of the idea of “excellence” but also how preprints are viewed in the world.
