Chris von Csefalvay is a computational epidemiologist/data scientist working at the intersection of AI, epidemiology and public health.

Chris von Csefalvay

The year is 1959. Eisenhower is on his second term, Castro just kicked Batista out of the country and Ray Charles’s Let the Good Times Roll is topping the charts.

The 95% myth

Over at the work blog, I’m discussing what knowledge means for large language models (LLMs), and the ways in which we can leverage this knowledge dividend for better inference.  Read the full post here.

The knowledge dividend of large language models

The awesome thing about language is that, well, we all mostly speak it, to some extent or another. This gives us an immensely powerful tool to manipulate transformational tasks. For the purposes of this post, I consider a transformational task to be essentially anything that takes an input and is largel intended to return some version of the same thing. This is not a very precise definition, but it will have to do for now.

LAIR - Language As Intermediate Representation

As the new Omicron variant of concern of COVID-19 emerged over Thanksgiving week (of all times!), attention has been turning to Paxlovid (PF-07321332). If, like me, you torment yourself by perusing conspiracy theories, you might have heard it referred to as Pfizermectin.

Peace, love, Paxlovid and ‘Pfizermectin’

Tip  If you’re here for the DOI matching script, the link is here. Quarto is a great tool for reproducible research. It is also a great tool for building websites. After about a decade of Wordpress-based websites, I’ve moved to Quarto primarily for two reasons. First, most of my content is largely static.

Quarto project scripts are awesomeness

There’s a notion in artificial intelligence known as Moravec’s paradox: it’s relatively easy to teach a computer to play chess or checkers at a pretty decent level, but near impossible to teach it something as trivial as bipedal motion. (Hassabis 2017) The sensorimotor tasks that our truly wonderful brains have mastered by our second birthday are much harder to teach a computer than something arguably as ‘complex’ as beating a chess grandmaster.

The hardest AI problem you’ve never heard of.

It appears that in what is clearly a wonderful little PR stunt, a Polish rum company managed to do a Sophia and appoint an ‘AI-driven’ ‘robot’ as its ‘CEO’.  The other guilty party to this pile of steaming bovine excrement is Hanson Robotics, famous for giving us Sophia, the “world’s first robot citizen”. Most of what I’m saying here goes just as well for Sophia.

The skeuomorphic fallacy

Yesterday, I completed a marathon on the SkiErg, with a pretty respectable time (see results here). I know I can do better, and I know I

 will

do better. On the other hand, this was a marathon where I pretty much did everything wrong. And that was sort of an instructive experience in its own right. The title is somewhat misleading. Not much went wrong – the marathon itself was smooth sailing.

How to do a SkiErg marathon entirely the wrong way (but still finish)

There’s something special about language. It is ‘our own’, it is ‘us’, in a profound way, and quite surprisingly, more so than art. I was deeply struck by this when I first saw reactions to large generative language models that created realistic, human-ish prose.

Beyond Broca

There might be valid concerns about the mRNA vaccines. Then there are the clearly insane ones, such as the claims that it involves the injection of a microchip. The argument that mRNA vaccines are a form of ‘gene therapy’ are somewhere in the middle, and that makes responding to it somewhat difficult.

Why the COVID-19 vaccines are not gene therapy

The temple of Asclepios, the Greek god of healing arts and medicine, at Epidaurus was pretty much the ancient Greek world’s equivalent of the Mayo Clinic. It then tells us a lot about the Greek worldview of healing that one of the things the temple complex prominently featured was a theatre. The Greeks believed in drama therapy, but in perhaps a slightly different way of how we think of it today.

The Moral Pulse of the Machine

Every year, the Commandant of the Marine Corps publishes a reading list of books that often only bear on warfighting tangentially at best. The idea behind this is that those entrusted with the lives of servicemembers should have an understanding of the world that goes beyond the profession of arms. In much the same way, I have been advising data scientists to go beyond professional literature.

Five non-data-science books for data scientists

I used to be on Quora. I’m not anymore. But I used to be. And I wrote some answers. Here are some of my favourites, categorised by field… sort of.

 AI

If I wrote an AI and trained it to score 200 on a standardized IQ test, what could a human of 100 IQ still do better? Read the full answer here. Will true AI come before the ability to completely simulate a human brain? Is there a difference? Read the full answer here.

My favourite Quora answers

In my latest post for the Starschema blog, I discuss the end of dashboards, and what comes next:  Read the full post here.

As we may see: the world after dashboards

In the middle of the green and pleasant countryside of England lies a small piece of commons –– land, that is, over which a number of people share certain rights. In this hypothetical scenario, it’s six farmers,  to . Each of them has a hypothetical cow, which differs from ordinary cows in that it grazes a fixed unit, and produces a fixed amount of milk (unlike real cows, which vary wildly in their input and output parameters, or so I’m told).

Ransomware attacks as n-person prisoner’s dilemmas

Say you’re busing tables and you’re trying to pass someone in a wheelchair. What do you do? Do you say “excuse me” and wait for them to move? Do you say “excuse me” and then try to pass them? Do you just try to pass them? Do you say nothing and just try to pass them? All of these are, actually, pretty legitimate answers. Now, say you’re a robot.

What I learned from getting bodied by a robot.

Asemantic Induction of Hallucinations in LLMs

There’s a style of visual design I’m inordinately fond of called Raygun Gothic. It’s hard to describe what the hell exactly one needs to be on to enjoy it, but think of it like the aesthetic from the latter Fallout games with a more optimistic outlook on the future. Gibson described it as “the future that never was”, and I think that’s a pretty apt description. We get these competing futures in technology all the time.

Prompt Engineering: The Art of Yesterday

It’s not every day that you find out you have climbed the exalted heights of another discipline. My work is pretty interdisciplinary, but it shocked me, too, that I’m apparently holding forth on neoliberalism and the epistemic question in African universities (archive link):  This, of course, came at some surprise to me, as I have never written anything on the topic.

Stochastic parrots, cap and gown edition

In the first four entries (1 2 3 4) of this sequence, I have focused primarily on what LLMs aren’t, can’t, won’t, wouldn’t and shouldn’t. It’s probably time to conclude this series by that much awaited moment in all stories, where the darkest night finally turns into a glorious dawn, where we finally arrive at the promised land, where we finally get to talk about what LLMs

 could

be. What I see as the most successful potential model of

Teams of Rivals

My recent post on the Starschema blog discusses the need for better data products to tackle future pandemic challenges:  Read the full post here.

Data for the next pandemic

I have probably spent more time looking at Poussin’s

 Dance to the Music of Time

than any other work of art. Sneaking off to the Wallace Collection in London and just looking at the

 Dance

was my comfort activity while living in London – a time that was not exactly devoid of its trials. It’s not, by any measure,

 great

art, insofar as such judgments can be made with any objectivity.

The Lyre of Hephaestus

In a recent paper that has attracted the interest of popular media as well, Fabio Urbina and colleagues examined the use (or rather, the abuse) of computational chemistry models of toxicity for generating toxic compounds and potential chemical agent candidates.(Urbina et al. 2022) Urbina and colleagues conclude that Urbina, Fabio, Filippa Lentzos, Cédric Invernizzi, and Sean Ekins. 2022.

A different shade of grey

I love posts that allow me to merge some of my addictions.

Auto-DOI for Quarto posts via Rogue Scholar

::: {.page-columns .page-full}
There's something special about language. It is 'our own', it is 'us',
in a profound way, and quite surprisingly, more so than art. I was
deeply struck by this when I first saw reactions to large generative
language models that created realistic, human-ish prose. Notably, those
mature enough to reach a non-professional audience -- ChatGPT based on
GPT-3 and later GPT-4 -- came quite some time after models that could
create fairly acceptable visual 'art'.^1^ The appearance of synthetic
language-like products (SLLPs), as I like to call the output of such
generative models, came well after the appearance of synthetic simulacra
of visual art,^2^ yet elicited much less fervent responses.

::: {.no-row-height .column-margin .column-container}
^1^ I don't mean to insinuate that what Stable Diffusion and DALL·E
produce are 'art' in the sense we understand that concept. However,
neither is what GPT produces 'language'. They are both simulators of
outcomes based on stochastic approximations over a sufficiently large
training set to be able to approximate the outcome of human activities
we know as 'art' and 'language', respectively.

^2^ For a given value of "well after". Time, in this discipline, moves
with an unsettling alacrity.
:::
:::

This was quite striking, for three reasons.

- For one, computationally, the probability space that a model seeking
 to create a realistic image has to navigate is exponentially larger
 than what's required to produce human-like prose.
- Secondly, we consider making art to be a very deeply human endeavour.
 Animals may to some minimal extent be taught to create poor simulacra
 of human artistic endeavours like painting, but nobody would confuse a
 trained elephant's 'paintings' to art [(Ross 2019)]{.citation
 cites="ross2019consciousness"}. Art is not just a product, it's also
 an activity, one that proceeds with a subjective element in the
 artist, and no machine can replicate the process, no matter how well
 it may approximate the outcome.
- Most importantly, however, despite the previous point, lay audiences
 saw a connection between a simulacrum of language and human-like
 intelligence that was absent from a simulacrum of art.

::: {.no-row-height .column-margin .column-container}
::: {#ref-ross2019consciousness .csl-entry}
Ross, Don. 2019. 'Consciousness, Language, and the Possibility of
Non-Human Personhood: Reflections on Elephants'. *Journal of
Consciousness Studies* 26 (3-4): 227--51.
:::
:::

Which, of course, leads us to the key question: what if we got one of
the most deeply enshrined beliefs about language, intelligence and the
relationship between the two *utterly, dreadfully wrong*?

::: {#our-precious-words .section .level1}
# Our precious words

A large language model (LLM) is, essentially, a very simple machine that
knows a large number of conditional probabilities. Given a sequence of
tokens
![](https://latex.codecogs.com/png.latex?k_0,%20k_1,%20%5Ccdots,%20k_%7Bn%7D),
it associates every possible token
![](https://latex.codecogs.com/png.latex?k%5E%7B%5Cprime%7D) with a
probability
![](https://latex.codecogs.com/png.latex?p(k_%7Bn%20+%201%7D%20=%20k%5E%7B%5Cprime%7D%20%7C%20k_0,%20k_1,%20%5Ccdots,%20k_%7Bn%7D))
-- or in other words, given a token sequence
![](https://latex.codecogs.com/png.latex?k_0,%20k%7B1%7D,%20%5Ccdots,%20k_%7Bn%7D),
it assigns to every point in a probability space a conditional
likelihood that that point's corresponding token will be the
![](https://latex.codecogs.com/png.latex?k_%7Bn+1%7D)-th token. Or,
using my preferred formulation, which looks at the inverse probability:
given the token sequence, it creates a probability distribution of the
next token and draws stochastically, weighted by token likelihood, so
that a draw from the region of highest probability is most likely.

It turns out that if the model's understanding of these conditional
probabilities is sufficiently good, it can simulate knowledge quite
well, [a point I belaboured
elsewhere](https://chrisvoncsefalvay.com/posts/knowledge-dividend-of-llms/).
This is not overly surprising. If a model knew the conditional
probability of rain on day ![](https://latex.codecogs.com/png.latex?d)
-- let's call this ![](https://latex.codecogs.com/png.latex?p_r(d)),
given a vector ![](https://latex.codecogs.com/png.latex?%5Ctheta_r) of
length ![](https://latex.codecogs.com/png.latex?n) that tells us whether
it rained on days ![](https://latex.codecogs.com/png.latex?d-1),
![](https://latex.codecogs.com/png.latex?d-2), ...,
![](https://latex.codecogs.com/png.latex?d-n), we'd trust it to tell us
whether we'd need our raincoat on that given day. All it would have to
do for that is to learn the conditional probability of
![](https://latex.codecogs.com/png.latex?p_r(d)%20%7C%20%5Ctheta_n),
which of course it could easily do by representing
![](https://latex.codecogs.com/png.latex?p_r(d)%20%7C%20%5Ctheta_n) as
![](https://latex.codecogs.com/png.latex?f(d,%20%5Ctheta_n)), then
learning the parameters of that function so as to minimise a loss
function
![](https://latex.codecogs.com/png.latex?J(f(d,%20%5Ctheta_n),%20r(d),%20%5Ctheta_n(d))),
where ![](https://latex.codecogs.com/png.latex?r(d)) is of course
whether it rained on day ![](https://latex.codecogs.com/png.latex?d) and
![](https://latex.codecogs.com/png.latex?%5Ctheta_n(d)) is the
![](https://latex.codecogs.com/png.latex?%5Ctheta_n) history vector for
the day ![](https://latex.codecogs.com/png.latex?d). Iterate this often
enough (over not single values of
![](https://latex.codecogs.com/png.latex?r(d)) and
![](https://latex.codecogs.com/png.latex?%5Ctheta_n(d)) but vectors
thereof), and you can learn a pretty decent conditional probability
function. The model would know no more about rain or shine than LLMs
know about language or the subject matters of language, but simulating
tokens gets you quite a long way towards being useful as a simulacrum of
knowledge.

Indeed, this is to the point that what comes out of such a model might
well appear human-like: modern GPT implementations can produce prose
that is a little stilted at times, but certainly often only
distinguishable from human prose by the conspicuous absence of
grammatical and spelling errors. This is interesting because of how it
was perceived: quite immediately, this was connected to a kind of
intelligence that was almost human, or indeed at times better than
human. People suddenly started to worry about a dumb token simulator
taking over their jobs.

Clearly, language hit a nerve.
:::

::: {#the-medium-is-the-message .section .level1 .page-columns .page-full}
# The medium is the message

JARVIS. Siri. Alexa. WOPR. The AIs of fiction and our every-days have
one thing in common: they use language as the presentation layer. This
is deceptive, because neither of these systems are, well, particularly
smart. Compared to models that can, say, quantitatively infer the
activity of a small molecule drug from its structure [(QSAR models, on
which see Karpov, Godin, and Tetko 2020; and also see Zhong and Guan
2023; Guntuboina et al. 2023)]{.citation
cites="karpov2020transformer zhong2023developing guntuboina2023peptidebert"},
Siri is pretty pathetic. However, it has something QSAR models and other
very impressive applications of machine learning don't: the human
presentation layer, i.e. language.

::: {.no-row-height .column-margin .column-container}
::: {#ref-karpov2020transformer .csl-entry}
Karpov, Pavel, Guillaume Godin, and Igor V Tetko. 2020.
'Transformer-CNN: Swiss Knife for QSAR Modeling and Interpretation'.
*Journal of Cheminformatics* 12 (1): 1--12.
:::

::: {#ref-zhong2023developing .csl-entry}
Zhong, Shifa, and Xiaohong Guan. 2023. 'Developing Quantitative
Structure--Activity Relationship (QSAR) Models for Water Contaminants'
Activities/Properties by Fine-Tuning GPT-3 Models'. *Environmental
Science & Technology Letters*.
:::

::: {#ref-guntuboina2023peptidebert .csl-entry}
Guntuboina, Chakradhar, Adrita Das, Parisa Mollaei, Seongwon Kim, and
Amir Barati Farimani. 2023. 'PeptideBERT: A Language Model Based on
Transformers for Peptide Property Prediction'. *arXiv Preprint
arXiv:2309.03099*.
:::

::: {#ref-endicott2006properties .csl-entry}
Endicott, Timothy, Joshua Getzler, and Edwin Peel. 2006. 'Properties of
Law: Essays in Honour of Jim Harris'.
:::
:::

How we treat a system seems to be conditional on how it talks to us. In
that sense, the medium is profoundly how we treat the message. To use
the terminology of J.W. Harris's writings on human rights, we associate
the 'right' to be considered to be intrinsically connected to being
capable of engaging in human 'discourse [(see Endicott, Getzler, and
Peel 2006)]{.citation cites="endicott2006properties"}. And that, of
course, means language.

This is not overly surprising, either. Our understanding of language has
been that of a watershed moment in evolution. Humans became what they
are when they learned to use language. Tool use is great, but tool use
only makes *a human* at best. What makes *humans*, plural, is language.
This is intrinsically connected, of course, to society. Language is not
an arbitrarily selected activity, nor is it really necessarily the kind
of evolutionary game changer that tool use is. Rather, it is the tool,
the *sine qua non*, the cornerstone and the absolutely fundamental
instrument of social interaction. Language creates society. Society
recognises human individuals and gives that recognition a meaning. The
fact that I am a human being, and recognised as such (I hope), has a
meaning that is different from me recognising that my dog is an
individual of the species *Canis lupus familiaris*, because it does not
merely acknowledge me as being of a certain species, but also of being
of a certain kind of agent capable not only of having rights but also of
speaking for them. Language is how all that happens [(e.g. Budwig 2000;
but see Browning 2023)]{.citation
cites="budwig2000language browning2023personhood"}.

::: {.no-row-height .column-margin .column-container}
::: {#ref-budwig2000language .csl-entry}
Budwig, Nancy. 2000. 'Language, Practices and the Construction of
Personhood'. *Theory & Psychology* 10 (6): 769--86.
:::

::: {#ref-browning2023personhood .csl-entry}
Browning, Jacob. 2023. 'Personhood and AI: Why Large Language Models
Don't Understand Us'. *AI & SOCIETY*, 1--8.
:::
:::
:::

::: {#the-language-of-intelligence-or-vice-versa .section .level1 .page-columns .page-full}
# The language of intelligence (or vice versa)

What, then, if we got one of the most important things about humanity,
and human intelligence, dreadfully wrong altogether? What if language is
not a *product* of intelligence (as we understand it in the human
context) but rather a necessary instrument thereof?

::: {.page-columns .page-full}
The evolution of something as crucial as language remains shrouded in a
perplexing mystery to this date. What we know is that at some point,
about 50-100,000 years ago, *something* happened that gave rise to
language. We don't quite know what it was, or how it specifically
transpired. Indeed, despite advances in our understanding of cognitive
neuroscience, we haven't found evidence of the 'language faculty'
proposed by [Hauser, Chomsky, and Fitch (2002; but see the criticisms by
Jackendoff and Pinker 2005)]{.citation
cites="hauser2002faculty jackendoff2005nature"}.^3^ The genetics of
language production -- which centres around FOXP2 these days [(see Enard
et al. 2002; Enard 2011)]{.citation
cites="enard2002molecular enard2011foxp2"} -- hasn't gotten us a lot
further, and there are way too many edge cases (*dissociations*, as the
term in evolutionary neuroscience goes), where either there is a
significant intellectual deficit despite preserved language ability
(Williams syndrome being the textbook example of this, *viz.* [Bellugi
et al. (2013)]{.citation cites="bellugi2013dissociation"}) or the
inverse (e.g. Developmental Verbal Dyspraxia, where there is impairment
to language production but not to overall intellect, *viz.*
[Vargha-Khadem et al. (2005)]{.citation cites="vargha2005foxp2"}) to be
able to confidently make this connection on an individual level.

::: {.no-row-height .column-margin .column-container}
^3^ not to be confused with the brain areas responsible for speech,
which perplexingly are part, but not the whole, of the language faculty

::: {#ref-enard2002molecular .csl-entry}
Enard, Wolfgang, Molly Przeworski, Simon E Fisher, Cecilia SL Lai,
Victor Wiebe, Takashi Kitano, Anthony P Monaco, and Svante Pääbo. 2002.
'Molecular Evolution of FOXP2, a Gene Involved in Speech and Language'.
*Nature* 418 (6900): 869--72.
:::

::: {#ref-enard2011foxp2 .csl-entry}
Enard, Wolfgang. 2011. 'FOXP2 and the Role of Cortico-Basal Ganglia
Circuits in Speech and Language Evolution'. *Current Opinion in
Neurobiology* 21 (3): 415--24.
:::

::: {#ref-bellugi2013dissociation .csl-entry}
Bellugi, Ursula, Shelly Marks, Amy Bihrle, and Helene Sabo. 2013.
'Dissociation Between Language and Cognitive Functions in Williams
Syndrome'. In *Language Development in Exceptional Circumstances*,
177--89. Psychology Press.
:::

::: {#ref-vargha2005foxp2 .csl-entry}
Vargha-Khadem, Faraneh, David G Gadian, Andrew Copp, and Mortimer
Mishkin. 2005. 'FOXP2 and the Neuroanatomy of Speech and Language'.
*Nature Reviews Neuroscience* 6 (2): 131--38.
:::
:::
:::

::: {.no-row-height .column-margin .column-container}
::: {#ref-hauser2002faculty .csl-entry}
Hauser, Marc D, Noam Chomsky, and W Tecumseh Fitch. 2002. 'The Faculty
of Language: What Is It, Who Has It, and How Did It Evolve?' *Science*
298 (5598): 1569--79.
:::

::: {#ref-jackendoff2005nature .csl-entry}
Jackendoff, Ray, and Steven Pinker. 2005. 'The Nature of the Language
Faculty and Its Implications for Evolution of Language (Reply to Fitch,
Hauser, and Chomsky)'. *Cognition* 97 (2): 211--25.
:::
:::

On the other hand, on a broader level, it is hard to discount the
relationship. What is more complex is the direction of this
relationship. There are, really, three possible scenarios:

1. Language is a consequence of human intelligence. The kind of
 intelligence we associate with modern human cognitive capabilities
 necessarily presupposes, absent some marginal exceptions, language.
2. Language is an epiphenomenon of human intelligence. It evolved in
 parallel, but neither requires human intelligence (see Williams
 syndrome) nor does human intelligence require it (see Developmental
 Verbal Dyspraxia).
3. Human intelligence is largely a consequence of language, which is
 its necessary but not sufficient condition. It is the evolutionarily
 most stable representation layer for information, and allows
 reasoning through complexity.

While the second of these is a convenient way to hand-wave away the
entire question and account for the edge cases I discussed above, I find
the third of these much more compelling. It is not defeated by the
argument from either of the edge cases: it is not defeated by arguments
from intact language despite intellectual deficits, because it does not
assume that language is sufficient, merely that it is necessary. It is
not defeated by the inverse, either, because it permits a small number
of deviations. Language is not the only possible representational layer
that could underpin intelligence. It is, however, vastly more
evolutionarily advantageous through its efficiency. It is so much
stronger and so much more efficient that it can be considered almost
absolutely dominant -- which indeed accounts for the fact that disorders
of language with preserved intellectual functioning are vanishingly
rare. If the efficiency of language as the 'operating system' of
intelligence weren't so strongly dominant, such disorders would not be
disorders, indeed, but alternate ways of cognitive existence that are
equally evolutionarily stable.
:::

::: {#the-golden-link .section .level1 .page-columns .page-full}
# The golden link

Which leads us to what I shall call the "golden link" of intelligence --
and perhaps the most frightening finding that derives from LLMs. We
intuit, correctly, that a realistic simulacrum of language is an
indication of intelligence. We once more intuit, correctly, that even if
we're aware of the limits of LLMs' 'language', it displays more than a
scintilla of whatever makes up intelligence. Just as Stable Diffusion is
not art but a simulacrum of the end result of the process we know as
'art', ChatGPT isn't really 'language' but a simulacrum, by way of
extending token sequences, of the end result of the process we know as
'language' -- but no matter how deeply we understand this, it is hard to
deny that ChatGPT does speak to us, to quote Kipling, "as a man would
talk to a man". Or, to put it this way: all the amazing things genuinely
complicated artificial intelligence can do, such as predict protein
structures or binding affinities or interpret histology specimens or
optimise mathematical problems, is a *praxis* -- something the system
*does*. Producing language is, or at least is some way towards, a
*hexis* -- something the system *is*. Ande that makes all the
difference.

And so, our trepidation and the 'uncanny valley' sensation of LLMs'
'intelligence' is quite instructive [(Floridi 2023)]{.citation
cites="floridi2023ai"}. It shows, clear as day, the intrinsic link
between language and intelligence, but more importantly, that language
is not a consequence of intelligence but a fundamental pre-requisite and
indeed the communication protocol on which efficient human intelligence
rests. Language is neither sufficient nor necessary for human
intelligence (and perhaps other forms of intelligence do exist that do
not require language at all), but it is the evolutionarily dominant
stable strategy for representing information in a manner that can
support intelligence.

::: {.no-row-height .column-margin .column-container}
::: {#ref-floridi2023ai .csl-entry}
Floridi, Luciano. 2023. 'AI as Agency Without Intelligence: On ChatGPT,
Large Language Models, and Other Generative Models'. *Philosophy &
Technology* 36 (1): 15.
:::
:::

There lies the scariest revelation of LLMs. It's not that LLMs will
supplant us (they won't), or that we'll be condemned to a lifetime of
reading books written by LLMs (have you tried to get ChatGPT to write a
story on a novel premise?), nor that LLMs will steal our jobs and take
over the planet. Rather, the great revelation is that LLMs cast light on
what might have been one of the longest standing fallacies of humans
reasoning about reasoning -- that language is the product of
intelligence, and not its operating system.
:::

::: {#quarto-appendix .default}
::: {.section .quarto-appendix-contents}
## Citation {#citation .anchored .quarto-appendix-heading}

<div>

::: {.quarto-appendix-secondary-label}
BibTeX citation:
:::

``` {.sourceCode .code-with-copy .quarto-appendix-bibtex}
@misc{csefalvay2023,
 author = {Chris von Csefalvay},
 title = {Beyond {Broca}},
 date = {2023-10-15},
 url = {https://chrisvoncsefalvay.com/posts/llms-language},
 doi = {10.59350/vynvf-0k137},
 langid = {en-GB}
}
```

::: {.quarto-appendix-secondary-label}
For attribution, please cite this work as:
:::

::: {#ref-csefalvay2023 .csl-entry .quarto-appendix-citeas}
Chris von Csefalvay. 2023. "Beyond Broca."
<https://doi.org/10.59350/vynvf-0k137>.
:::

</div>
:::
:::