The Logical Structure of Human Civilization (John Searle)
https://www.youtube.com/watch?v=wO6WcPX7BR0


The Chinese Room Argument
First published Fri Mar 19, 2004; substantive revision Thu Feb 20, 2020
The argument and thought-experiment now generally known as the Chinese Room Argument was first published in a 1980 article by American philosopher John Searle (1932– ). It has become one of the best-known arguments in recent philosophy. Searle imagines himself alone in a room following a computer program for responding to Chinese characters slipped under the door. Searle understands nothing of Chinese, and yet, by following the program for manipulating symbols and numerals just as a computer does, he sends appropriate strings of Chinese characters back out under the door, and this leads those outside to mistakenly suppose there is a Chinese speaker in the room.

The narrow conclusion of the argument is that programming a digital computer may make it appear to understand language but could not produce real understanding. Hence the “Turing Test” is inadequate. Searle argues that the thought experiment underscores the fact that computers merely use syntactic rules to manipulate symbol strings, but have no understanding of meaning or semantics. The broader conclusion of the argument is that the theory that human minds are computer-like computational or information processing systems is refuted. Instead minds must result from biological processes; computers can at best simulate these biological processes. Thus the argument has large implications for semantics, philosophy of language and mind, theories of consciousness, computer science and cognitive science generally. As a result, there have been many critical replies to the argument.
1. Overview
Work in Artificial Intelligence (AI) has produced computer programs that can beat the world chess champion, control autonomous vehicles, complete our email sentences, and defeat the best human players on the television quiz show Jeopardy. AI has also produced programs with which one can converse in natural language, including customer service “virtual agents”, and Amazon’s Alexa and Apple’s Siri. Our experience shows that playing chess or Jeopardy, and carrying on a conversation, are activities that require understanding and intelligence. Does computer prowess at conversation and challenging games then show that computers can understand language and be intelligent? Will further development result in digital computers that fully match or even exceed human intelligence? Alan Turing (1950), one of the pioneer theoreticians of computing, believed the answer to these questions was “yes”. Turing proposed what is now known as ‘The Turing Test’: if a computer can pass for human in online chat, we should grant that it is intelligent. By the late 1970s some AI researchers claimed that computers already understood at least some natural language. In 1980 U.C. Berkeley philosopher John Searle introduced a short and widely-discussed argument intended to show conclusively that it is impossible for digital computers to understand language or think.

Searle argues that a good way to test a theory of mind, say a theory that holds that understanding can be created by doing such and such, is to imagine what it would be like to actually do what the theory says will create understanding. Searle (1999) summarized his Chinese Room Argument (herinafter, CRA) concisely:

Imagine a native English speaker who knows no Chinese locked in a room full of boxes of Chinese symbols (a data base) together with a book of instructions for manipulating the symbols (the program). Imagine that people outside the room send in other Chinese symbols which, unknown to the person in the room, are questions in Chinese (the input). And imagine that by following the instructions in the program the man in the room is able to pass out Chinese symbols which are correct answers to the questions (the output). The program enables the person in the room to pass the Turing Test for understanding Chinese but he does not understand a word of Chinese.
Searle goes on to say, “The point of the argument is this: if the man in the room does not understand Chinese on the basis of implementing the appropriate program for understanding Chinese then neither does any other digital computer solely on that basis because no computer, qua computer, has anything the man does not have.”

Thirty years after introducing the CRA Searle 2010 describes the conclusion in terms of consciousness and intentionality:

I demonstrated years ago with the so-called Chinese Room Argument that the implementation of the computer program is not by itself sufficient for consciousness or intentionality (Searle 1980). Computation is defined purely formally or syntactically, whereas minds have actual mental or semantic contents, and we cannot get from syntactical to the semantic just by having the syntactical operations and nothing else. To put this point slightly more technically, the notion “same implemented program” defines an equivalence class that is specified independently of any specific physical realization. But such a specification necessarily leaves out the biologically specific powers of the brain to cause cognitive processes. A system, me, for example, would not acquire an understanding of Chinese just by going through the steps of a computer program that simulated the behavior of a Chinese speaker (p.17).
“Intentionality” is a technical term for a feature of mental and certain other things, namely being about something. Thus a desire for a piece of chocolate and thoughts about real Manhattan or fictional Harry Potter all display intentionality, as will be discussed in more detail in section 5.2 below.

Searle’s shift from machine understanding to consciousness and intentionality is not directly supported by the original 1980 argument. However the re-description of the conclusion indicates the close connection between understanding and consciousness in Searle’s later accounts of meaning and intentionality. Those who don’t accept Searle’s linking account might hold that running a program can create understanding without necessarily creating consciousness, and conversely a fancy robot might have dog level consciousness, desires, and beliefs, without necessarily understanding natural language.

In moving to discussion of intentionality Searle seeks to develop the broader implications of his argument. It aims to refute the functionalist approach to understanding minds, that is, the approach that holds that mental states are defined by their causal roles, not by the stuff (neurons, transistors) that plays those roles. The argument counts especially against that form of functionalism known as the Computational Theory of Mind that treats minds as information processing systems. As a result of its scope, as well as Searle’s clear and forceful writing style, the Chinese Room argument has probably been the most widely discussed philosophical argument in cognitive science to appear since the Turing Test. By 1991 computer scientist Pat Hayes had defined Cognitive Science as the ongoing research project of refuting Searle’s argument. Cognitive psychologist Steven Pinker (1997) pointed out that by the mid-1990s well over 100 articles had been published on Searle’s thought experiment – and that discussion of it was so pervasive on the Internet that Pinker found it a compelling reason to remove his name from all Internet discussion lists.

This interest has not subsided, and the range of connections with the argument has broadened. A search on Google Scholar for “Searle Chinese Room” limited to the period from 2010 through 2019 produced over 2000 results, including papers making connections between the argument and topics ranging from embodied cognition to theater to talk psychotherapy to postmodern views of truth and “our post-human future” – as well as discussions of group or collective minds and discussions of the role of intuitions in philosophy. In 2007 a game company took the name “The Chinese Room” in joking honor of “...Searle’s critique of AI – that you could create a system that gave the impression of intelligence without any actual internal smarts.” This wide-range of discussion and implications is a tribute to the argument’s simple clarity and centrality.

2. Historical Background
2.1 Leibniz’ Mill
Searle’s argument has four important antecedents. The first of these is an argument set out by the philosopher and mathematician Gottfried Leibniz (1646–1716). This argument, often known as “Leibniz’ Mill”, appears as section 17 of Leibniz’ Monadology. Like Searle’s argument, Leibniz’ argument takes the form of a thought experiment. Leibniz asks us to imagine a physical system, a machine, that behaves in such a way that it supposedly thinks and has experiences (“perception”).

17. Moreover, it must be confessed that perception and that which depends upon it are inexplicable on mechanical grounds, that is to say, by means of figures and motions. And supposing there were a machine, so constructed as to think, feel, and have perception, it might be conceived as increased in size, while keeping the same proportions, so that one might go into it as into a mill. That being so, we should, on examining its interior, find only parts which work one upon another, and never anything by which to explain a perception. Thus it is in a simple substance, and not in a compound or in a machine, that perception must be sought for. [Robert Latta translation]
Notice that Leibniz’s strategy here is to contrast the overt behavior of the machine, which might appear to be the product of conscious thought, with the way the machine operates internally. He points out that these internal mechanical operations are just parts moving from point to point, hence there is nothing that is conscious or that can explain thinking, feeling or perceiving. For Leibniz physical states are not sufficient for, nor constitutive of, mental states.

2.2 Turing’s Paper Machine
A second antecedent to the Chinese Room argument is the idea of a paper machine, a computer implemented by a human. This idea is found in the work of Alan Turing, for example in “Intelligent Machinery” (1948). Turing writes there that he wrote a program for a “paper machine” to play chess. A paper machine is a kind of program, a series of simple steps like a computer program, but written in natural language (e.g., English), and implemented by a human. The human operator of the paper chess-playing machine need not (otherwise) know how to play chess. All the operator does is follow the instructions for generating moves on the chess board. In fact, the operator need not even know that he or she is involved in playing chess – the input and output strings, such as “N–QB7” need mean nothing to the operator of the paper machine.

As part of the WWII project to decipher German military encryption, Turing had written English-language programs for human “computers”, as these specialized workers were then known, and these human computers did not need to know what the programs that they implemented were doing.

One reason the idea of a human-plus-paper machine is important is that it already raises questions about agency and understanding similar to those in the CRA. Suppose I am alone in a closed room and follow an instruction book for manipulating strings of symbols. I thereby implement a paper machine that generates symbol strings such as “N-KB3” that I write on pieces of paper and slip under the door to someone ouside the room. Suppose further that prior to going into the room I don’t know how to play chess, or even that there is such a game. However, unbeknownst to me, in the room I am running Turing’s chess program and the symbol strings I generate are chess notation and are taken as chess moves by those outside the room. They reply by sliding the symbols for their own moves back under the door into the room. If all you see is the resulting sequence of moves displayed on a chess board outside the room, you might think that someone in the room knows how to play chess very well. Do I now know how to play chess? Or is it the system (consisting of me, the manuals, and the paper on which I manipulate strings of symbols) that is playing chess? If I memorize the program and do the symbol manipulations inside my head, do I then know how to play chess, albeit with an odd phenomenology? Does someone’s conscious states matter for whether or not they know how to play chess? If a digital computer implements the same program, does the computer then play chess, or merely simulate this?

By mid-century Turing was optimistic that the newly developed electronic computers themselves would soon be able to exhibit apparently intelligent behavior, answering questions posed in English and carrying on conversations. Turing (1950) proposed what is now known as the Turing Test: if a computer could pass for human in on-line chat, it should be counted as intelligent.

A third antecedent of Searle’s argument was the work of Searle’s colleague at Berkeley, Hubert Dreyfus. Dreyfus was an early critic of the optimistic claims made by AI researchers. In 1965, when Dreyfus was at MIT, he published a circa hundred page report titled “Alchemy and Artificial Intelligence”. Dreyfus argued that key features of human mental life could not be captured by formal rules for manipulating symbols. Dreyfus moved to Berkeley in 1968 and in 1972 published his extended critique, “What Computers Can’t Do”. Dreyfus’ primary research interests were in Continental philosophy, with its focus on consciousness, intentionality, and the role of intuition and the inarticulated background in shaping our understandings. Dreyfus identified several problematic assumptions in AI, including the view that brains are like digital computers, and, again, the assumption that understanding can be codified as explicit rules.

However by the late 1970s, as computers became faster and less expensive, some in the burgeoning AI community started to claim that their programs could understand English sentences, using a database of background information. The work of one of these, Yale researcher Roger Schank (Schank & Abelson 1977) came to Searle’s attention.Schank developed a technique called “conceptual representation” that used “scripts” to represent conceptual relations (related to Conceptual Role Semantics). Searle’s argument was originally presented as a response to the claim that AI programs such as Schank’s literally understand the sentences that they respond to.

2.3 The Chinese Nation
A fourth antecedent to the Chinese Room argument are thought experiments involving myriad humans acting as a computer. In 1961 Anatoly Mickevich (pseudonym A. Dneprov) published “The Game”, a story in which a stadium full of 1400 math students are arranged to function as a digital computer (see Dneprov 1961 and the English translation listed at Mickevich 1961, Other Internet Resources). For 4 hours each repeatedly does a bit of calculation on binary numbers received from someone near them, then passes the binary result onto someone nearby. They learn the next day that they collectively translated a sentence from Portuguese into their native Russian. Mickevich’s protagonist concludes “We’ve proven that even the most perfect simulation of machine thinking is not the thinking process itself, which is a higher form of motion of living matter.” Apparently independently, a similar consideration emerged in early discussion of functionalist theories of minds and cognition (see further discussion in section 5.3 below), Functionalists hold that mental states are defined by the causal role they play in a system (just as a door stop is defined by what it does, not by what it is made out of). Critics of functionalism were quick to turn its proclaimed virtue of multiple realizability against it. While functionalism was consistent with a materialist or biological understanding of mental states (arguably a virtue), it did not identify types of mental states (such as experiencing pain, or wondering about OZ) with particular types of neurophysiological states, as “type-type identity theory” did. In contrast with type-type identity theory, functionalism allowed sentient beings with different physiology to have the same types of mental states as humans – pains, for example. But it was pointed out that if extraterrestrial aliens, with some other complex system in place of brains, could realize the functional properties that constituted mental states, then, presumably so could systems even less like human brains. The computational form of functionalism, which holds that the defining role of each mental state is its role in information processing or computation, is particularly vulnerable to this maneuver, since a wide variety of systems with simple components are computationally equivalent (see e.g., Maudlin 1989 for discussion of a computer built from buckets of water). Critics asked if it was really plausible that these inorganic systems could have mental states or feel pain.

Daniel Dennett (1978) reports that in 1974 Lawrence Davis gave a colloquium at MIT in which he presented one such unorthodox implementation. Dennett summarizes Davis’ thought experiment as follows:

Let a functionalist theory of pain (whatever its details) be instantiated by a system the subassemblies of which are not such things as C-fibers and reticular systems but telephone lines and offices staffed by people. Perhaps it is a giant robot controlled by an army of human beings that inhabit it. When the theory’s functionally characterized conditions for pain are now met we must say, if the theory is true, that the robot is in pain. That is, real pain, as real as our own, would exist in virtue of the perhaps disinterested and businesslike activities of these bureaucratic teams, executing their proper functions.
In “Troubles with Functionalism”, also published in 1978, Ned Block envisions the entire population of China implementing the functions of neurons in the brain. This scenario has subsequently been called “The Chinese Nation” or “The Chinese Gym”. We can suppose that every Chinese citizen would be given a call-list of phone numbers, and at a preset time on implementation day, designated “input” citizens would initiate the process by calling those on their call-list. When any citizen’s phone rang, he or she would then phone those on his or her list, who would in turn contact yet others. No phone message need be exchanged; all that is required is the pattern of calling. The call-lists would be constructed in such a way that the patterns of calls implemented the same patterns of activation that occur between neurons in someone’s brain when that person is in a mental state – pain, for example. The phone calls play the same functional role as neurons causing one another to fire. Block was primarily interested in qualia, and in particular, whether it is plausible to hold that the population of China might collectively be in pain, while no individual member of the population experienced any pain, but the thought experiment applies to any mental states and operations, including understanding language.

Thus Block’s precursor thought experiment, as with those of Davis and Dennett, is a system of many humans rather than one. The focus is on consciousness, but to the extent that Searle’s argument also involves consciousness, the thought experiment is closely related to Searle’s. Cole (1984) tries to pump intuitions in the reverse direction by setting out a thought experiment in which each of his neurons is itself conscious, and fully aware of its actions including being doused with neurotransmitters, undergoing action potentials, and squirting neurotransmitters at its neighbors. Cole argues that his conscious neurons would find it implausible that their collective activity produced a consciousness and other cognitive competences, including understanding English, that the neurons lack. Cole suggests the intuitions of implementing systems are not to be trusted.

3. The Chinese Room Argument
In 1980 John Searle published “Minds, Brains and Programs” in the journal The Behavioral and Brain Sciences. In this article, Searle sets out the argument, and then replies to the half-dozen main objections that had been raised during his earlier presentations at various university campuses (see next section). In addition, Searle’s article in BBS was published along with comments and criticisms by 27 cognitive science researchers. These 27 comments were followed by Searle’s replies to his critics.

In the decades following its publication, the Chinese Room argument was the subject of very many discussions. By 1984, Searle presented the Chinese Room argument in a book, Minds, Brains and Science. In January 1990, the popular periodical Scientific American took the debate to a general scientific audience. Searle included the Chinese Room Argument in his contribution, “Is the Brain’s Mind a Computer Program?”, and Searle’s piece was followed by a responding article, “Could a Machine Think?”, written by philosophers Paul and Patricia Churchland. Soon thereafter Searle had a published exchange about the Chinese Room with another leading philosopher, Jerry Fodor (in Rosenthal (ed.) 1991).

The heart of the argument is Searle imagining himself following an symbol processing program written in English (which is what Turing called “a paper machine”). The English speaker (Searle) sitting in the room follows English instructions for manipulating Chinese symbols, whereas a computer “follows” (in some sense) a program written in a computing language. The human produces the appearance of understanding Chinese by following the symbol manipulating instructions, but does not thereby come to understand Chinese. Since a computer just does what the human does – manipulate symbols on the basis of their syntax alone – no computer, merely by following a program, comes to genuinely understand Chinese.

This narrow argument, based closely on the Chinese Room scenario, is specifically directed at a position Searle calls “Strong AI”. Strong AI is the view that suitably programmed computers (or the programs themselves) can understand natural language and actually have other mental capabilities similar to the humans whose behavior they mimic. According to Strong AI, these computers really play chess intelligently, make clever moves, or understand language. By contrast, “weak AI” is the much more modest claim that computers are merely useful in psychology, linguistics, and other areas, in part because they can simulate mental abilities. But weak AI makes no claim that computers actually understand or are intelligent. The Chinese Room argument is not directed at weak AI, nor does it purport to show that no machine can think – Searle says that brains are machines, and brains think. The argument is directed at the view that formal computations on symbols can produce thought.

We might summarize the narrow argument as a reductio ad absurdum against Strong AI as follows. Let L be a natural language, and let us say that a “program for L” is a program for conversing fluently in L. A computing system is any system, human or otherwise, that can run a program.

If Strong AI is true, then there is a program for Chinese such that if any computing system runs that program, that system thereby comes to understand Chinese.I could run a program for Chinese without thereby coming to understand Chinese.Therefore Strong AI is false.
The first premise elucidates the claim of Strong AI. The second premise is supported by the Chinese Room thought experiment. The conclusion of this narrow argument is that running a program cannot endow the system with language understanding. (There are other ways of understanding the structure of the argument. It may be relevant to understand some of the claims as counterfactual: e.g. “there is a program” in premise 1 as meaning there could be a program, etc. On this construal the argument involves modal logic, the logic of possibility and necessity (see Damper 2006 and Shaffer 2009)).

It is also worth noting that the first premise above attributes understanding to “the system”. Exactly what Strong-AI supposes will acquire understanding when the program runs is crucial to the success or failure of the CRA. Schank 1978 has a title that claims their group’s computer, a physical device, understands, but in the body of the paper he claims that the program [“SAM”] is doing the understanding: SAM, Schank says “...understands stories about domains about which it has knowledge” (p. 133). As we will see in the next section (4), these issues about the identity of the understander (the cpu? the program? the system? something else?) quickly came to the fore for critics of the CRA. Searle’s wider argument includes the claim that the thought experiment shows more generally that one cannot get semantics (meaning) from syntax (formal symbol manipulation). That and related issues are discussed in section 5: The Larger Philosophical Issues.

4. Replies to the Chinese Room Argument
Criticisms of the narrow Chinese Room argument against Strong AI have often followed three main lines, which can be distinguished by how much they concede:

(1) Some critics concede that the man in the room doesn’t understand Chinese, but hold that nevertheless running the program may create comprehension of Chinese by something other than the room operator. These critics object to the inference from the claim that the man in the room does not understand Chinese to the conclusion that no understanding has been created. There might be understanding by a larger, smaller, or different, entity. This is the strategy of The Systems Reply and the Virtual Mind Reply. These replies hold that the output of the room might reflect real understanding of Chinese, but the understanding would not be that of the room operator. Thus Searle’s claim that he doesn’t understand Chinese while running the room is conceded, but his claim that there is no understanding of the questions in Chinese, and that computationalism is false, is denied.

(2) Other critics concede Searle’s claim that just running a natural language processing program as described in the CR scenario does not create any understanding, whether by a human or a computer system. But these critics hold that a variation on the computer system could understand. The variant might be a computer embedded in a robotic body, having interaction with the physical world via sensors and motors (“The Robot Reply”), or it might be a system that simulated the detailed operation of an entire human brain, neuron by neuron (“the Brain Simulator Reply”).

(3) Finally, some critics do not concede even the narrow point against AI. These critics hold that the man in the original Chinese Room scenario might understand Chinese, despite Searle’s denials, or that the scenario is impossible. For example, critics have argued that our intuitions in such cases are unreliable. Other critics have held that it all depends on what one means by “understand” – points discussed in the section on The Intuition Reply. Others (e.g. Sprevak 2007) object to the assumption that any system (e.g. Searle in the room) can run any computer program. And finally some have argued that if it is not reasonable to attribute understanding on the basis of the behavior exhibited by the Chinese Room, then it would not be reasonable to attribute understanding to humans on the basis of similar behavioral evidence (Searle calls this last the “Other Minds Reply”). The objection is that we should be willing to attribute understanding in the Chinese Room on the basis of the overt behavior, just as we do with other humans (and some animals), and as we would do with extra-terrestrial Aliens (or burning bushes or angels) that spoke our language. This position is close to Turing’s own, when he proposed his behavioral test for machine intelligence.

In addition to these responses specifically to the Chinese Room scenario and the narrow argument to be discussed here, some critics also independently argue against Searle’s larger claim, and hold that one can get semantics (that is, meaning) from syntactic symbol manipulation, including the sort that takes place inside a digital computer, a question discussed in the section below on Syntax and Semantics.

4.1 The Systems Reply
In the original BBS article, Searle identified and discussed several responses to the argument that he had come across in giving the argument in talks at various places. As a result, these early responses have received the most attention in subsequent discussion. What Searle 1980 calls “perhaps the most common reply” is the Systems Reply.

The Systems Reply (which Searle says was originally associated with Yale, the home of Schank’s AI work) concedes that the man in the room does not understand Chinese. But, the reply continues, the man is but a part, a central processing unit (CPU), in a larger system. The larger system includes the huge database, the memory (scratchpads) containing intermediate states, and the instructions – the complete system that is required for answering the Chinese questions. So the Sytems Reply is that while the man running the program does not understand Chinese, the system as a whole does.

Ned Block was one of the first to press the Systems Reply, along with many others including Jack Copeland, Daniel Dennett, Douglas Hofstadter, Jerry Fodor, John Haugeland, Ray Kurzweil and Georges Rey. Rey (1986) says the person in the room is just the CPU of the system. Kurzweil (2002) says that the human being is just an implementer and of no significance (presumably meaning that the properties of the implementer are not necessarily those of the system). Kurzweil hews to the spirit of the Turing Test and holds that if the system displays the apparent capacity to understand Chinese “it would have to, indeed, understand Chinese” – Searle is contradicting himself in saying in effect, “the machine speaks Chinese but doesn’t understand Chinese”.

Margaret Boden (1988) raises levels considerations. “Computational psychology does not credit the brain with seeing bean-sprouts or understanding English: intentional states such as these are properties of people, not of brains” (244). “In short, Searle’s description of the robot’s pseudo-brain (that is, of Searle-in-the-robot) as understanding English involves a category-mistake comparable to treating the brain as the bearer, as opposed to the causal basis, of intelligence”. Boden (1988) points out that the room operator is a conscious agent, while the CPU in a computer is not – the Chinese Room scenario asks us to take the perspective of the implementer, and not surprisingly fails to see the larger picture.

Searle’s response to the Systems Reply is simple: in principle, he could internalize the entire system, memorizing all the instructions and the database, and doing all the calculations in his head. He could then leave the room and wander outdoors, perhaps even conversing in Chinese. But he still would have no way to attach “any meaning to the formal symbols”. The man would now be the entire system, yet he still would not understand Chinese. For example, he would not know the meaning of the Chinese word for hamburger. He still cannot get semantics from syntax.

In some ways Searle’s response here anticipates later extended mind views (e.g. Clark and Chalmers 1998): if Otto, who suffers loss of memory, can regain those recall abilities by externalizing some of the information to his notebooks, then Searle arguably can do the reverse: by internalizing the instructions and notebooks he should acquire any abilities had by the extended system. And so Searle in effect concludes that since he doesn’t acquire understanding of Chinese by internalizing the external components of the entire system (e.g. he still doesn’t know what the Chinese word for hamburger means), understanding was never there in the partially externalized system of the original Chinese Room.
⋯以下暫省略，时字數超标了