Scott Chamberlain

I recently had a use case at work where I wanted to check that file paths given in a Python script actually existed. These paths were in various GitHub repositories, so all I had to do was pull out the paths and check if they exist on GitHub. There were a few catches though.

Python, ast, and redbaron

Soooo, my last job at Deck was amazing. I loved it. I was doing data engineer stuff there, mostly maintaining infrastructure for data pipelines. Everyone was great and the mission was amazing: helping Democrats win. Yet the company was shut down about a month ago, sending me on another job search, the 3rd since early/mid 2021.

Working at Fred Hutchinson Cancer Center

This blog is now using Hugo. Important - if you subscribe to the RSS for this blog you likely have to delete/remove the old one and add the new RSS link.

Moved to Hugo

taxizedb arose from pain in using taxize when dealing with large amounts of data in a single request or doing a lot of requests of any data size. taxize works with remote data sources on the web, so there’s a number of issues that can slow the response down: internet speed, server response speed (was a response already cached or not; or do they even use caching), etc.

taxizedb: an update

TL;DR  In 6 months (end of November 2022) the CRAN Checks API https://cranchecks.info/ will be gone You can still get badges at https://badges.cranchecks.info You can use the new badges like: [![cran checks](https://badges.cranchecks.info/worst/dplyr.svg)](https://cran.r-project.org/web/checks/check_results_dplyr.html)  Find more details at https://github.com/sckott/cchecksbadges Sunsetting the CRAN Checks API   If you contribute an R package to

CRAN Checks API and Badges

In February this year I wroute about how many parameters functions should have, looking at some other languages, with a detailed look at R. On a related topic …  As I work on many R packages that are API clients for various web services, I began wondering: What is the best way to deal with API routes that have a lot of parameters?

API client design: how to deal with lots of parameters?

notes to self for next job hunt (some of which may be generally useful): don’t apply to general tech companies anymore for many reasons. heard back from very very few but that may be b/c I don’t know many people at general tech companies never been able to get through interviews; they’re presumably looking for computer science grads (not me) most of their missions are probably not stuff I’d be happy about at the end of the day.

Job searching notes

I was working on a work task last week, and needed to filter out one instance of a class from a list of class instances. No matter how you do this speed doesn’t matter too much if you’re doing this operation once or a few times. However, I this operation needs to be done about 100K times each time the script runs - so speed definitely does matter in this case.

List comprehension vs. filter vs. key lookup

At work I’ve been using Quarto quite a bit for website and books for work projects.

Software rules and Quarto

I recently opened an issue in a repository for a package I’m working on to think about potential footguns and how to avoid them. That word “footguns” got me thinking;

Avoiding the word footgun(s)

I worked on a refactor of an R package at work the other day. Here’s some notes about that after doing the work. This IS NOT a best practices post - it’s just a collection of thoughts. For context, the package is an API client.

Refactoring notes

I wrote the other day about overcoming an issue with Shiny. Another issue I ran into concurrently was about file inputs. The issue was that file inputs (i.e., shiny::fileInput) was difficult to clear.

Shiny file inputs

You’ve experienced an HTTP redirect (or URL redirect, or URL forwarding) even if you haven’t noticed. We all use browsers (I assume, since you are reading this), either on a phone or laptop/desktop computer. Browsers don’t show all the HTTP requests going on in the background, some of which are redirects.

Mocking HTTP redirects

I’ve been working on an inherited Shiny app at work for the past few months. One of the many frustrating things about Shiny lately has been around buttons. Well, it wasn’t really about buttons, but that’s where it started. I noticed that a number of buttons - some having file inputs, some having text inputs - did not trigger every time, and I expected them to trigger every time no matter what.

Shiny button weirdness

My parents just found this email they had printed out from me from May 19, 2006, when I was 26. I chatted about some family stuff, then had this rambling string of weird thoughts below. I thought others might appreciate a good cringe - or cringy laugh - at my expense. It’s especially funny because I’m most def an atheist. I don’t know, those Tucson sunsets really are transformative.

Weird thoughts

It’s been interesting switching jobs with respect to programming languages. I used to write 95% R - now I write 95% Python.  I have been using Python for many years, but not seriously or getting paid either. I’ve learned alot in the first 6 months.  Some Python things learned: Functions and methods   I used to think functions and methods were the same thing. But during the last 6 months I learned that functions and methods are not the same.

Notes on Python

\  Code is often arranged in packages for any given language. Packages are often cataloged in a package registry of some kind: NPM for node, crates.io for Rust, etc. For R, that registry is either CRAN or Bioconductor (for the most part).  CRAN has the concept of an archived package.

stories behind archived packages

Recology

I recently had a use case at work where I wanted to check that file paths given in a Python script actually existed. These paths were in various GitHub repositories, so all I had to do was pull out the paths and check if they exist on GitHub.

There were a few catches though.

First, I couldn't simply get any string out of each Python script - they needed to be strings specficied by a specific function parameter, and match a regex (e.g., start with 'abc').

Second, the script paths lack the GitHub repository root name. This name was part of the function name - so I needed to get access to the function that the path was specified within, and then parse the function name to get the repository name.

The obvious solution I thought was the [ast](https://docs.python.org/3/library/ast.html) library.

## ast library

I started by using `ast`{.language-plaintext .highlighter-rouge}. The `ast.NodeVisitor`{.language-plaintext .highlighter-rouge} class seemed like it would do the trick.

An example script ("my_script.py"):

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
def hello(path, stuff=None):
    return path


if __name__ == "__main__":
    print(hello(path="hello/world.py", stuff="hello mars"))
```
:::
:::

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
import ast

class CollectStrings(ast.NodeVisitor):
    def visit_Module(self, node):
        self.out = set()
        self.generic_visit(node)
        return list(filter(lambda w: w.startswith("hello") and w.endswith(".py"), self.out))

    def visit_Str(self, node):
        self.out.add(node.s)

file = "my_script.py"
with open(file, "r") as f:
    body = ast.parse(f.read())

coll = CollectStrings()
coll.visit(body)
## ['hello/world.py']
```
:::
:::

That worked great at fetching paths - only because all the paths I was looking for started with the same text and all have the same file extension.

HOWEVER - I also needed the function name that the `path`{.language-plaintext .highlighter-rouge} argument was called from. I tried to make this work with `ast.NodeVisitor`{.language-plaintext .highlighter-rouge} but couldn't get it to work.

I eventually got frustrated enough and figured there must be some libraries that build on top of `ast`{.language-plaintext .highlighter-rouge} that make it easier to work with ast's in Python.

## redbaron

Enter [redbaron](https://github.com/PyCQA/redbaron). I found this library pretty quickly upon searching for a library building on top of `ast`{.language-plaintext .highlighter-rouge}.

Another example script ("their_script.py"):

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
def hello(path, stuff=None):
    return path


def goodbye(path, stuff=None):
    return path


def world():
    path_str = hello(path="src/world.py", stuff="hello mars")
    other_path_str = goodbye(path="src/world.py", stuff="hello saturn")

    return path_str, other_path_str


if __name__ == "__main__":
    print(world())
```
:::
:::

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
import re
from redbaron import RedBaron

file = "their_script.py"
with open(file, "r") as src:
  red = RedBaron(src.read())

red
## 0   def hello(path, stuff=None):
##         return path
##     
##     
##     
## 1   def goodbye(path, stuff=None):
##         return path
##     
##     
##     
## 2   def world():
##         path_str = hello(path="src/world.py", stuff="hello mars")
##         other_path_str = goodbye(path="src/world.py", stuff="hello saturn")
##     
##         return path_str, other_path_str
##     
##     
##     
## 3   if __name__ == "__main__":
##         print(world())
## 
```
:::
:::

Even just the resulting object you get from parsing something is useful:

And with `.help()`{.language-plaintext .highlighter-rouge} you get a very detailed map of the structure of the thing you're trying to navigate (only printing first 20 lines):

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
red.help()
## 0 -----------------------------------------------------
## DefNode()
##   # identifiers: def, def_, defnode, funcdef, funcdef_
##   # default test value: name
##   async=False
##   name='hello'
##   return_annotation ->
##     None
##   decorators ->
##   arguments ->
##     * DefArgumentNode()
##         # identifiers: def_argument, def_argument_, defargument, defargumentnode
##         target ->
##           NameNode() ...
##         annotation ->
##           None
##         value ->
##           None
##     * DefArgumentNode()
##         # identifiers: def_argument, def_argument_, defargument, defargumentnode
...
```
:::
:::

Looking at the result from `red.help()`{.language-plaintext .highlighter-rouge} I can then use `.find_all()`{.language-plaintext .highlighter-rouge} to find certain nodes in the ast.

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
nodes = red.find_all("AtomtrailersNode")
nodes = list(filter(lambda w: "hello" in w.dumps(), nodes))
nodes
## [hello(path="src/world.py", stuff="hello mars"), goodbye(path="src/world.py", stuff="hello saturn")]
```
:::
:::

Then I can write some okay code to extract out the function name, and ugly code to get the string supplied to the `path`{.language-plaintext .highlighter-rouge} parameter. Then f-string those together to get the path I'm after.

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
paths = []
for node in nodes:
    fxn_name = node.name.value
    command = re.search("src/.*\\.py", node.dumps()).group()
    paths.append(f"{fxn_name}/{command}")

for path in paths:
    print(path)
## hello/src/world.py
## goodbye/src/world.py
```
:::
:::

Not super proud of this but gets the job done for my use case - and when you're not making open source for others, you don't need to worry about other use cases :)

I'll definitely try to learn how to properly extract stuff using `redbaron`{.language-plaintext .highlighter-rouge} - but it got me to answer much faster than the `ast`{.language-plaintext .highlighter-rouge} library.


In February this year I wroute about [how many parameters functions should have](https://recology.info/2020/02/how-many-parameters/), looking at some other languages, with a detailed look at R. On a related topic ...

As I work on many R packages that are API clients for various web services, I began wondering: What is the best way to deal with API routes that have a lot of parameters?

The general programming wisdom I've seen is that a function should have no more than 3-4 parameters (e.g., [this long SO thread](https://stackoverflow.com/questions/174968/how-many-parameters-are-too-many), or [this one](https://softwareengineering.stackexchange.com/questions/331803/techniques-for-minimising-number-of-function-arguments)). So should one do anything different from a normal function when that function is connecting to a web API route with a lot of parameters? I've not found very much spilled ink on this exact topic, but I'll discuss what I have found.

## Use cases?

A Software Engineering StackExchange thread [How to handle many arguments in an API wrapper?](https://softwareengineering.stackexchange.com/questions/196895/how-to-handle-many-arguments-in-an-api-wrapper) had a couple ideas. One idea is to consider use cases, and then make separate functions covering those use cases. This might work, but I haven't explored it thoroughly for a real API route yet. Pondering on it though I doubt this would work since you'd have to pre-emptively think about all the different scenarios users might dream up, which seems like a fools errand.

## Builder pattern

Another concept brought up in the thread mentioned above was the [Builder pattern](https://en.wikipedia.org/wiki/Builder_pattern). It's hard for me to understand the idea in abstract - [here's a nicer discussion of this in Ruby](https://medium.com/kkempin/builder-design-pattern-in-ruby-dfa2d557ff1b).

This is a good concept to know about, but I don't think is appropriate for the issue at hand, of how to handle many API parameters.

## Named parameter map

Another idea in that thread was to use a named parameter map. In R this would look something like this (imagine a lot more parameters though):

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
foo <- function(args) {
  get("/some-api-route", args)
}
api_args <- list(query = "*:*", limit = 10)
my_args <- modifyList(api_args, list(query = "bears", limit = 300))
foo(my_args)
```
:::
:::

That is, the above would replace this:

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
foo <- function(query = "*:*", limit = 10) {
  get("/some-api-route", list(query = query, limit = limit))
}
foo(query = "*:*", limit = 10)
```
:::
:::

So in the first code block the function no longer has a lot of parameters in it. The drawback of this in R (and I'm sure is similar in other languages) is that users lose the autocomplete helpers that most modern IDE's and text editors have - helping users type less and quickly get a tip on what each parameter is intended to do and importantly (if the developer has documented the function well) what types the parameters expects and what (if any) options there are to pass to the parameter.

Of course a user can "just" read the docs to figure out what each parameter expects, but it sure can save a lot of time if the help is right there in the tooltips of the IDE/text editor. In addition, in R there's automated checking that parameters in functions are also documented, which is nice for making sure parameters and docs don't get out of sync. You'd lose this by using a parameter map - though you could document the parameter map - and perhaps wire together some custom code to make sure the parameters in the parameter map are all handled by the function. This does seem like a lot of fuss though compared to simply having the parameters in the function itself.

This approach probably becomes more attractive if a client has many functions that take the same parameters - in which a named parameter map could handle the parameters and any logic behind checking those parameters.

## Include no parameters in the function

In other words: Pass all parameters on to the API w/o including any of them in the function - i.e., let the API handle any problems in parameters.

Another approach I've not seen written about but that I've seen in code is having a rather lite wrapper around an API and letting the API itself sort out any problems due to user inputs.

An example is the [gh R package](https://gh.r-lib.org/), a client for the GitHub API. For query parameters you can pass in named parameters through the ellipsis `...`{.language-plaintext .highlighter-rouge}, all of which are passed as query parameters. The gh package does no checking of these parameters (that I know of); simply passes them to the GitHub API. The GitHub API happens to apparently ignore invalid (silently drop) parameters and invalid valuses of parameters (here, "stuff" is an invalid value for the `page`{.language-plaintext .highlighter-rouge} parameter).

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
x <- gh("GET /users/{username}/repos", username = "gaborcsardi", page = "stuff")
length(x)
#> 30
```
:::
:::

I don't hate this solution, but I don't love it either. This approach is highly dependent on a well designed API that fails gracefully, with informative error messages and with correct status codes, etc. I would say most APIs are not as nice as GitHub's, at least in the scientific API space in which I work.

One plus side of this approach is the R package gh only has one parameter (`...`{.language-plaintext .highlighter-rouge}) to handle all query parameters, so you do solve the too many query parameters problem.

Another upside to this approach is you do not have to keep up with any changes in parameters on each API route - for example, an API route could drop one parameter, and add another, and the R client wouldn't have to change anything (assuming the change in parameters wasn't associated with a change that breaks code in the client).

A major downside of this approach is that the user often has to mount a time-consuming expedition to figure out what parameters are accepted. Some API clients may document them, and some will simply direct users to the web APIs docs. I think this part alone makes this solution (include no parameters in the function) not a good one since the user experience can be so bad if the documentation is not good. And all developers know its much easier for their docs to get out of date than their code.

## Include some parameters in the function

Another approach is to define some query parameters in the function, and handle all others via R's ellipsis (`...`{.language-plaintext .highlighter-rouge}) - or similar in other languages. I've seen this relatively often and have used it myself. It's often used when there's a clear smaller set of important parameters - those can be put in the function as named parameters. And then there's a long tail of other parameters that the maintainer thinks are not likely to be used very often. Those can be looked up by the user in the API docs for whatever API the client interacts with.

An example of this is the rOpenSci package [rtweet](https://docs.ropensci.org/rtweet/) - a client for the Twitter API. In the [search_tweets() function](https://docs.ropensci.org/rtweet/reference/search_tweets.html#arguments) there are a half dozen or so named parameters in the function, but then the ellipsis handles all other parameters.

The drawback to this approach is that no two APIs behave the same way. In the case of Twitter they silently ignore/drop parameters they do not support (same as the GitHub API, see above). For example:

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
library(rtweet)
search_tweets("hillaryclinton", n = 3, foo = "bar")
```
:::
:::

Works just fine even though `foo`{.language-plaintext .highlighter-rouge} is absolutely not a parameter supported by the Twitter API. They must ignore parameters they don't support. This is same behavior as the GitHub API we saw above.

In the case of Twitter and GitHub one might want to raise errors on unsupported parameters client side in rtweet to avoid any use confusion of parameters being silently dropped.

## Grouping similar parameters together

Many threads (e.g., [this one](https://softwareengineering.stackexchange.com/a/352676/329940)) suggest that similar parameters could be grouped together to reduce the number of parameters passed to a function. For example, if a function has the parameters `latitude`{.language-plaintext .highlighter-rouge} and `longitude`{.language-plaintext .highlighter-rouge} you could group those into a single parameter called e.g., `coordinates`{.language-plaintext .highlighter-rouge}.

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
# Original function, each parameter separate
foo <- function(latitude, longitude) {
    # do something with latitude/longitude
    latitude
    longitude
}

# Modified function, grouping the two parameters into one
foo <- function(coordinates) {
    # do something with latitude/longitude
    coordinates$latitude
    coordinates$longitude
}
```
:::
:::

Though I've not tried this approach myself, it might be a good compromise between a function not handling any parameters (i.e., just passing all to the web API unhandled) and handling every parameter individually.

## Closing thoughts

The benefit of documenting API query parameters in a client package is that you can tell the user what each parameters expects in language they can understand. That is, if you simply direct users to the docs for the web API with which the client interacts, the API docs could be not very good and/or specify types expected that the user may not understand. In addition, there may be edge cases or similar with some parameters that are not documented in the API docs but that you can document in the client docs for each parameter.

I would say the vast majority of web API clients I use that do succeed in having very few parameters also have docs in which it's a nightmare trying to figure out what parameters each method accepts. That is, the pursuit of very few parameters at least is correlated with a very poor user experience - in my experience.
