Meta, Salvage Accumulation, and Babel

"Now all the earth used the same language and the same words.
And the people said to one another, 'Come, let us build ourselves a city, and a tower with its top in the heavens, and let us make a name for ourselves...'
The Lord came down to see the city and the tower, which mortals had built. And the Lord said, 'Look, they are one people, and they have all one language, and this is only the beginning of what they will do; nothing that they propose to do will now be impossible for them. Come, let us go down and confuse their language....'
So the Lord scattered them abroad from there over the face of all the earth, and they left off building the city. Therefore it was called Babel..."

— Genesis 11:1, 4, 5–6, 8–9, New Revised Standard Version

"Most Christians would not regard the New Testament, and translations thereof, as too sacred to be used in machine learning," Meta's AI team wrote in a summary of their latest AI audio processing capabilities. Released in May 2023, Meta's Massively Multilingual Speech (MMS) is a version of it's 2020 model, Wav2vec 2, but which "supports speech-to-text and text-to-speech for 1,107 languages and language identification for over 4,000 languages." The next most capable model, OpenAI's Whisper, only supports 100 languages. Meta's goal for its model is impossibly simple: "We envision a future where a single model can solve several speech tasks for all languages."

With such a model, Meta's AI team predicts that language barriers to technological access would be virtually eliminated. As it stands, computing and communication technologies within the framework of global capitalism operate as a form of neocolonial soft power: these technologies only exist within a relatively small number of languages, causing rapid erosion in language diversity. The stated goal of the single universal model is increased language access to technologies of communication and computing—"they have all one language, and this is only the beginning of what they will do; nothing that they propose to do will now be impossible for them."

Such a vast improvement in the language support came less from an innovation in Wav2vec 2 than from a newly implemented dataset: thousands and thousands of hours of audio recordings of the New Testament.

This new dataset, first produced through the conservative–evangelical movement of 1980s–2010s, serves as the foundation for Meta's universal speech model. The production, re-production, and re-configuration of this data shows the ways that data moves as a commodity outside the realm of profit. In this sense, this data demonstrates the ways that AI and its requisite data operates as a form of salvage accumulation—production that is violently absorbed and irreversibly translated into capitalist value forms. It is accumulation for its own sake; labor crystalized into data crystalized into capital; a single shared economic language.

Pericapitalist Production

In The Mushroom at the End of the World, Anna Tsing describes capitalism as a system of salvage. For Tsing, capitalism is unable to produce many of its most profitable resources for extraction: coal, oil, labor, matsutake mushrooms—all represent forms vital to capitalism that it is unable to produce. As Neil Smith elaborates in Uneven Development, this reliance on nature as a "universal means of production" positions capitalism as the ultimate beneficiary of all forms of production, capitalist or otherwise. Capitalism expands globally to form a generalized extractive relation with nature. Tsing describes this production that is subsumed into capitalism as "pericapitalist"—sites of production that exist both inside and outside capital, but within a system of salvage accumulation.¹ Tsing's argument, however, de-stabilizes "nature" within this realm of production to focus on the ways that all forms of production become subject to extractive/salvage accumulation. The grammar of extraction becomes more consequential than the specific categories. As Gilles Deleuze describes in "Postscript on the Societies of Control", capitalism as a system of enclosure and concentration has given way to a capitalism that is no longer involved in production or products, but movements and data.²

Data and its production, particularly in "Postscript on the Societies of Control," operates as shorthand for capitalist forms of value. Deleuze describes the transformation from the individual to the dividual—a form composed of individual data points, captured movements, patterns of behavior, or habits. Such data operates as the foundation for high-order, a-material production: marketing, speculation, financial trading, etc. The financial crisis, crypto-currency, and tech-unicorn bubble (Spotify, Uber, Netflix, or any other fundamentally low-profit/high-market-cap "disruptor" company) of 2008 to 2021 represents the apotheosis of this de-materialized capital. But the economic falter of 2021, the collapse of the crypto-currency market, and the rapid (and extremely expensive) developments in Generative AI and Natural Language Processing has produced new spaces and grammars for salvage accumulation.

AI Grammars

AI, at its core, is pattern re-production. Despite the predictions of an AI-fueled utopia and apocalyptic prophesies, the real technical potential of AI relies on a foundation of data.

An image-generation AI like OpenAI's DALL-E operates by translating across two sets of data: image and text. This raw data must be processed. The role of the AI model in the pipeline is to parse the grammars of the data. Minute patterns, repetitions, predictable series. Images and sentences alike blown apart in search of the underlying patterns of pixels and words to be stored as tokens or values. These grammars of action are not built around a complete image or sentence, but of its divided and sub-divided parts.³ A bear isn't a bear because it exists in any way we understand it to; it is a bear because a grammar of pixels appears across a vast dataset of images labeled bear. DALL-E is the alignment of language with pixel-order, of sub-patterns and non-linguistic grammars. For every pattern DALL-E understands, it can now produce infinite iterations through the re-combination of grammars.

The aesthetic production of AI is salvage par excellence. It is the pixel-level copy and pasting of thousands of patterns, merging and re-merging in various organizations to produce the vaguely recognizable forms that follow familiar logic. There is nothing "new", insofar there is nothing that doesn't pre-exist in the pixel order of its dataset. The innovation of AI is the obliteration of the object into its dividuated pixels.

Through this salvage emerges a collage—the re-purposing of materials to create something which both references its original and its newness. It is a transformative act, but it cannot avoid reference to the original. Collage is permanently locked in dialogue with the pieces that comprise it. Like the system of capital that produces it, it is only able to salvage. It can only recombine previous patterns. The pixel orders that compose any object are fused with pixels orders of another, and while the combinations may be novel, they are bound—like any production—to an episteme, to knowledge that defines the conditions of possibility.⁴

When DALL-E creates "digital image noise," it cannot do so without its database of tokens and pixels. It moves within the epistemology of the dataset. The "wholeness" of the image obscures the mechanisms of salvage that produce it. The operating principle of the model only becomes visible when the database of pixel-patterns begins to collapse into one another. When the pixel-patterns that form the AI generated content become indistinguishable in the database, those patterns begin to emerge and disrupt the illusion of "wholeness" in the collage.

As researchers at Rice University have shown in the paper "Self-Consuming Generative Models Go MAD", feeding AI generated content into AI models rapidly erodes the capabilities as pixel-patterns become indistinguishable within the dataset. This condition, which the researchers call "Model Autophagy Disorder" or MAD, results in generations that either begin to look identical, or degrade into unrecognizable syntheses. The research team shows that AI models cannot synthesize their own data—the models are bound to the possibilities of their own dataset. The ways that the models descend into MADness reveal their underlying operation. When an AI model produces the same image over and over, or fails to produce any image at all, it indicates, as the researchers suggest, an entropic metamorphosis of the underlying data. The patterns in the data erode and melt into a solid mass. Pixels and tokens become locked in a loop of self-reference.⁵ The pixel-patterns that comprise the dataset become indistinguishable from each other, and when tasked with assembling a collage, the AI model reaches for a single, undifferentiated object.⁶

There is, as Beth Coleman describes in "Technology of the Surround," an "ontological entropy of AI system design, which is constrained in its reproduction of a biopolitics of hierarchy and valuation".⁷ As Coleman points out, AI models are black boxes, only legible by controlling a series of inputs and outputs—model A produces this result with this data; model B produces that result. The inability to directly intervene in the algorithmic logic of the model renders data as the only axis of control.

The problem of epistemology, in fact, moves beyond the technical details of the model. It is impossible to imagine DALL-E using anything except data as the basis for its creation precisely because the epistemology of the dataset is not limited to the dataset itself. The logic of salvage, of the dividual, of disconnected references (AI production | finance) to a base reality (data | materiality) unites both capital its underlying technologies as part of the same assemblage. The current conjuncture of capital is co-constituted with the epistemology of AI.

Companies like Meta, Google, and OpenAI have no recourse for producing the data that drives their models. Data must be harvested from non-productive spaces, translated from spaces of pericapitalist production to machine-legible information banks. Wikipedia is a common source of data harvest, as are the 600,000 emails recoved from Enron's corporate email servers. OpenAI could be said to harvest the image-recognition knowledge of the workers in Kenya and elsewhere in the global south who it severely underpays, in addition to recycling the user chat data from ChatGPT into GPT-4. Google's AI (and many others) reaped its data from the massive Google Books Corpus, which itself embodied a form of salvage accumulation since its inception in 2002. And Meta, for its universal audio model, turned to a long-standing evangelical project for its data harvest.

Evangelical Products

New Testament audio-translation has been a project deeply entwined with the evangelical movement in the United States. Meta's dataset specifically comes from the non-profit "Faith Comes By Hearing" (FCBH), a multi-million dollar non-profit in the field of bible distribution. The non-profit, originally named "Hosanna," began as a Christian tape library in 1972 before pivoting in the mid-1980s to produce dramatized recordings of the Bible in multiple languages. FCBH was one of many Christian media companies propelled by the rise of the evangelical movement from the 1970s to the 2010s. In 1976, a Newsweek cover story declared it "the year of the evangelical," noting a shift in the religious overtones of the 1976 presidential election.⁸ Evangelicalism is, in many ways, a political ideology rather than a theological or spiritual one, particularly from 1976 forwards. Though broadly Protestant, the conservative-evangelical approach to social issues operated above issues of theology by couching the broadly conservative of the hetero-patriarchal nuclear family into the long-standing American fundamentalist approach to scripture.⁹ As Robert Self argues in All in the Family, the family served as a conceptual bedrock for a range of conservative ideologies—chief among them the evangelical movement. One of the critical theological innovations of the conservative-evangelical movement was its ability to cast engagement in politics as a vital expression of faith—and thus that the state could enact the political will of the church (broadly defined).

While denominational and theological differences were over-coded through political engagement around issues like abortion, queerness, drugs, and other domestic issues, as Gene Zubovich argues in "The U.S. Culture Wars Abroad," a crucial aspect of this political engagement revolved around issues of de-colonization, imperialism, and global structures of race and class.¹⁰ Conservative-evangelicalism was concerned not only with domestic politics, but the continuation of those conservative domestic politics abroad. As part of this project, conservative-evangelical nonprofits like Faith Comes by Hearing articulated a critique of secular monoculture (couched as authoritarianism—a blend of concerns about communism and imperialism deeply embedded in Cold War ideology) in favor of spiritual monoculture (virtues and civilized morality implicit in Christianity). Faith Comes by Hearing, along with other Bible-distribution organizations, served as a catalyst for spreading Christianity, and along with it "Christian values" globally.

In 2004, Faith Comes By Hearing made its first move towards the techno-evangelism with the release of "The Proclaimer"—a MP3 player with a solar panel and hand crank that exclusively plays the recorded versions of the New Testament that would later be fed into Meta's AI model.

Even compared to Faith Comes By Hearing's other dedicated bible MP3 player, the "Bible Stick", "The Proclaimer" looks like camping gear. Developed after a three-day fasting session, "The Proclaimer" marked a shift in the marketing methodology of the organization. Attempting to latch onto the growing legitimacy of the tech-world after the 2008 financial crisis, Faith Comes By Hearing would position itself as an outlet for techno-evangelism. Yet, at its core was its audio library. "The Proclaimer" and the "Bible Stick" including its "Kidz" and "Military" variants operated as means of distribution for its growing library of bible audio recordings.

Crucially for both Faith Comes By Hearing and Meta, its growing library of audio translations are not created directly by the non-profit. Rather, Faith Comes By Hearing holds exclusive license on the audio recordings, which are produced as part of a grant system. Rather than award the grant to support ongoing work, the grant is made available to cover the costs of the completed work.

The following table shows the yearly expenditure to produce the language data that now powers Meta's audio AI model:

Each new recording is an arduous process. The New Testament in Catalan (recorded in 2020 and released in 2021) has a final run-time of 38 hours and 56 minutes. The 2022 recording of the New Testament in Toba-Maskoy has a similar run-time of 38 hours and 36 minutes. Spread across 8-hour days, this project would take at least 5 days to complete a single, perfect take. In addition to the reading itself, the audio mastering, post-production, and equipment also contribute to the sum labor required to produce these recordings.

The content of these recordings and language data is not incidental to the conditions of production. The underlying religious mission of the recordings blurs the division between labor and spiritual devotion. Faith Comes By Hearing's grant system makes it difficult to determine the precise distribution of funds for the recording contractors grant recipients and the multiple people who produce the recordings. What is clear is that those who actually created the recordings are not credited anywhere except the IRS tax forms as organizations that received grants—not on Faith Comes By Hearing's website, not in the audio recording metadata, and not in the recordings themselves.

Licensing the Word of the Lord

Faith Comes By Hearing holds exclusive license for every recording in their database. The following is an excerpt of Faith Comes By Hearing's License for all the recordings available through its web portal (https://live.bible.is/terms):

"FCBH [Faith Comes By Hearing] allows you to use Bible.is in accordance with these Terms for your personal, non-commercial use. Except as provided in these Terms, you may not copy, download, stream, capture, reproduce, duplicate, archive, upload, modify, translate, publish, broadcast, transmit, retransmit, distribute, perform, display, sell or otherwise use any Content appearing on the Site.

• Use the Bible.is Services to download, read, listen, or view the Content for personal, non-commercial use, in accordance with its pre-defined functionality only"

The relationship between copyright and the Bible is a longer, more complicated history, but most recent (and popular) translations of the fall under a copyright for publishing reasons—New International Version, English Standard Version, King James Version (in the UK only, under a perpetual copyright owned by the Crown), New Living Translation, etc. Faith Comes By Hearing seems to be no exception.

It is unclear in either Meta's marketing materials for MMS or the accompanying research paper whether the data foundation of MMS was used with the permission of Faith Comes By Hearing. The recordings which power the AI model are easy to scrape (or manually download) from the Faith Comes By Hearing site. Moreover, unlike some of Meta's other AI licenses, many of Meta's non-Large Language Models are released broadly under the MIT license. Oddly, Massively Multilingual Speech is released under Creative Commons Attribution-NonCommercial 4.0—something Creative Commons itself explicitly recommends against. This odd licensing decision is related to Faith Comes By Hearing's copyrights. Faith Comes By Hearing has not acknowledged Meta's use of their recordings on their social media or in their last three organization reports since Meta announced the model. It is possible Faith Comes By Hearing is not aware of their role in the AI world at all.

AI companies and investment firms like a16z (Andreessen Horowitz) have been explicit in their use of copyrighted materials as the data foundation of their models. In November 2023, a16z wrote in a Notice of Inquiry to the U.S. Copyright Office that "imposing the cost of actual or potential copyright liability on the creators of AI models will either kill or significantly hamper their development".¹¹ a16z argued that "the use of copyrighted works en masse to train an AI model ... does not infringe copyright," attempting to couch the data foundation as "non-exploitive" use.¹² This fast and loose approach to copyright has come to a head with a high-profile lawsuit between the New York Times and Microsoft/OpenAI.¹³ a16z's "non-exploitive" data harvesting refers to the act of salvage in the AI model, not the production of the data itself. Crucially, a16z's defense of copyright infringement hinges less on the argument that the AI output is a transformative work, but rather that the outputs produced by the AI model do not resemble the inputs at all—consistent with the belief in the divine productive possibilities held by disciples of Artificial General Intelligence.¹⁴ Rather than an epistemology of data, a16z positions AI as a machine which transforms the nature of data itself through the process of salvage.

Spiritual Information → Linguistic Information

a16z's defense of the salvage accumulation of data hinges on the idea of the data operating as a structure. Rather than content, AI operates through the relation between content. Words become tokens. The signifying power of words are reduced to nothing more than the relationship between signifiers. A word has a meaning, referring to something; a token refers to a position within a formula. The trick of AI is producing strings of tokens which resemble (sometimes verbatim, according to the lawsuit filed by the New York Times) sentences of words. Data moves through the AI assemblage not as semantic content, but as linguistic content.

This movement of data is epitomized in Meta's hijacking of the recordings produced by Faith Comes By Hearing. Faith Comes By Hearing produces the recordings of the bible in a wide range of languages explicitly because of the spiritual content embedded in those recordings. The following recording of Revelations 7:1 exists as a form of spiritual information, designed to communicate a divine significance.

When harvested by Meta, the recording is processed in an entirely different manner. The spiritual information is transformed into its raw sonic and linguistic components, stripped of its significance and reprocessed. The following recording is AI generated using Massively Multilingual Speech, reading the same passage: Revelations 7:1.

The similarity between the voices in these two recordings is unsurprising: the AI model can only produce based on its data epistemology. Yet, for Meta or a16z or OpenAI, these recordings represent sound which is fundamentally transformed. When a16z defends the copyright infringement AI is predicated on, it attempts to cleanly parse the voice in the first recording and the second recording as entirely separate from one another. The labor, cost, information, and ideology which produced the first recording are meant to disappear, separated completely from the second recording. But the second recording is premised upon the first—they are inextricably bound together through reliance on the same underlying data.

The relationship between Meta and the conservative-evangelical movement is not one of continuity. Meta and the evangelical movement represent two entirely different modes of political power, of materiality, or temporality. The heart of the issue is not whether Meta is aligned with evangelicalism, but how evangelicalism serves as a foundational movement upon which technology and, by extension, reactionary techno-optimism is able to develop and expand. Meta is, interestingly, agnostic. It has no opinion on the source or history of its dataset. It has no interest in theology. It values only the data itself: the pattern of syllables aligned to text, the linguistic content, the possible implementations.

Massively Multilingual Speech speaks in the voices of the readers for Faith Comes By Hearing. It doesn't matter. They can say anything now.