Roger’s Takeaway

本文與上一篇一樣，再次驗證資料庫AI與真正AI的差別在於他是不是能驗證自己的行為，並且不需要人類大量的去維護知識庫。

摘要

Rich Sutton 探討了 AI 自主驗證其知識的必要性，而非依賴人類設計者。文章對比了需要「基礎（grounding）」與「驗證（verification）」的 AI 系統（如機器人）與傳統的「資料庫 AI」（如 CYC）方法，後者依賴人類維護知識本體。

Highlight

1.

賦予 AI 驗證其知識的能力並非小事。事實上，這是一件非常重大的事，不容易做到。通常，一點知識可以寫得很精簡，但其驗證卻非常複雜。說「桌上有一本書」很容易，但要表達其驗證過程（哪怕是很小一部分），例如拿起書所涉及的視覺和觸覺感受，卻非常複雜。

2.

雖然驗證需要所有的執行細節，但執行細節本身並不是驗證。前往午餐室有一個程序，但與此分開的，應該還有一個驗證器（verifier）來確定它是否成功。一個程序完全有可能建立在（grounded in）行動和感知上，卻完全排除了驗證器，從而排除了自主知識維護的可能性。雖然可能有點以偏概全，但這正是現代 AI 機器人系統中典型的情況。它們擁有廣泛的「基礎（grounded）知識」，但仍然無法驗證其中幾乎任何一部分。

3.

我所提議的 AI 與所謂的經典「資料庫 AI」（database AI）之間存在巨大對比。有許多大型 AI 計畫致力於將大量知識編纂（codify）到資料庫或「知識本體」（ontologies）中，其中 Doug Lenat 的 CYC 只是最廣為人知的一個。在這些努力中，「由人類維護知識」的想法是被接受的。他們強調特殊的知識表達方法和工具，以便讓人們更容易理解和存取知識，並試圖保持其正確性。

4.

無論這種符號「資料庫 AI」方法的最終成就如何，很明顯，它與我所倡導的截然相反。資料庫方法需要付出巨大努力來組織和輸入一個客觀、公開且「脫離實體」（disembodied）的知識庫。

5.

我所倡導的 AI，是能夠在與動態環境即時互動的同時，維護其自身的（可能與他人不同的）表徵（representations）。最重要的是，資料庫方法擁抱由人類來維護和組織 AI 的知識。而我所倡導的，是將這些功能自動化，讓 AI 能夠充分理解其知識，以便自行驗證。

逐字稿

人工智慧的自主知識驗證

如果我們不希望 AI 的人類設計者承擔確保 AI 知識正確性的重擔，那麼 AI 就必須能自己確保。它必須能夠驗證它所獲得或被賦予的知識。

賦予 AI 驗證其知識的能力並非小事。事實上，這是一件非常重大的事，不容易做到。通常，一點知識可以寫得很精簡，但其驗證卻非常複雜。說「桌上有一本書」很容易，但要表達其驗證過程（哪怕是很小一部分），例如拿起書所涉及的視覺和觸覺感受，卻非常複雜。定義一個操作（例如「我下樓一層就能到午餐室」）很容易，但要驗證這一點，就必須參考可執行的程序，如尋找和走下樓梯、辨識午餐室等。這些程序涉及的細節和閉迴路應變措施（contingencies），遠比知識本身要多得多，例如開門、樓梯可能被封閉，或在路上遇到人。在使用這些知識時（例如在規劃時），人們通常可以忽略所有這些細節，但要驗證知識，就必須在低層次（low level）上對其進行規範。增加未經驗證的知識的簡易性，與加入自主驗證機制的複雜性，兩者完全無法相比。

請注意，雖然驗證需要所有的執行細節，但執行細節本身並不是驗證。前往午餐室有一個程序，但與此分開的，應該還有一個驗證器（verifier）來確定它是否成功。一個程序完全有可能建立在（grounded in）行動和感知上，卻完全排除了驗證器，從而排除了自主知識維護的可能性。雖然可能有點以偏概全，但這正是現代 AI 機器人系統中典型的情況。它們擁有廣泛的「基礎（grounded）知識」，但仍然無法驗證其中幾乎任何一部分。它們使用視覺程序來辨識門和走廊，並根據這些結論做出決定，但它們自己無法糾正錯誤。如果某物被辨識為「門口」卻無法通過，這個失敗將不會被辨識，也不會被用來糾正未來的門口辨識，除非由人類來完成。

另一方面，一旦擁有了「基礎」（grounding），加入驗證的下一步就不那麼令人望而生畏了。我們只需在執行程序中附加適當的測試和終止條件，這些條件在某種程度上衡量原始陳述的真實性，同時詳細說明其真正含義。什麼是椅子？不僅僅是會觸發你視覺椅子偵測器的東西！那會是「基礎知識」（grounded knowledge），但無法驗證；它將依賴人類來判斷哪些是椅子、哪些不是。但假設你有嘗試「坐下」的程序。那麼你需要的驗證器，就只是能夠衡量你「坐下」的成功程度。這樣你就可以自己驗證、改進和維護你的「可坐物」辨識器。

我所提議的 AI 與所謂的經典「資料庫 AI」（database AI）之間存在巨大對比。有許多大型 AI 計畫致力於將大量知識編纂（codify）到資料庫或「知識本體」（ontologies）中，其中 Doug Lenat 的 CYC 只是最廣為人知的一個。在這些努力中，「由人類維護知識」的想法是被接受的。他們強調特殊的知識表達方法和工具，以便讓人們更容易理解和存取知識，並試圖保持其正確性。這些系統傾向於強調靜態的世界知識，例如「Springfield 是 Illinois 的首府」、「金絲雀是一種鳥」，甚至「你下午 3:30 與 John 有個會議」，而不是機器人與環境即時互動所需的動態知識。一個主要問題是讓人們在輸入知識時使用相同的類別和術語，更重要的是，讓他們所指的意義相同。人們在尋找一種終極的「知識本體」，即對所有物體及其可能關係的編纂，以便能對它們做出清晰的陳述。但到目前為止，這還未能實現；似乎總有更多不適用的案例遠多於適用的。人類擅長靈活運用他們的概念，並且知道何時概念不適用。

無論這種符號「資料庫 AI」方法的最終成就如何，很明顯，它與我所倡導的截然相反。資料庫方法需要付出巨大努力來組織和輸入一個客觀、公開且「脫離實體」（disembodied）的知識庫。我所倡導的 AI，是能夠在與動態環境即時互動的同時，維護其自身的（可能與他人不同的）表徵（representations）。最重要的是，資料庫方法擁抱由人類來維護和組織 AI 的知識。而我所倡導的，是將這些功能自動化，讓 AI 能夠充分理解其知識，以便自行驗證。

Verification

Rich Sutton

11/14/2001

If the human designers of an AI are not to be burdened with ensuring that what their AI knows is correct, then the AI will have to ensure it itself. It will have to be able to verify the knowledge that it has gained or been given.

Giving an AI the ability to verify its knowledge is no small thing. It is in fact a very big thing, not easy to do. Often a bit of knowledge can be written very compactly, whereas its verification is very complex. It is easy to say "there is a book on the table", but very complex to express even a small part of its verification, such as the visual and tactile sensations involved in picking up the book. It is easy to define an operator such as "I can get to the lunchroom by going down one floor", but to verify this one must refer to executable routines for finding and descending the stairs, recognizing the lunchroom, etc. These routines involve enormously greater detail and closed-loop contingencies, such as opening doors, the possibility of a stairway being closed, or meeting someone on the way, than does the knowledge itself. One can often suppress all this detail when using the knowledge, e.g., in planning, but to verify the knowledge requires its specification at the low level. There is no comparison between the ease of adding unverified knowledge and the complexity of including a means for its autonomous verification.

Note that although all the details of execution are needed for verification, the execution details are not themselves the verification. There is a procedure for getting to the lunchroom, but separate from this would be the verifier for determining if it has succeeded. It is perfectly possible for the procedure to be fully grounded in action and sensation, while completely leaving out the verifier and thus the possibility of autonomous knowledge maintenance. At the risk of being too broad-brush about it, this is what typically happens in modern AI robotics systems. They have extensive grounded knowledge, but still no way of verifying almost any of it. They use visual routines to recognize doors and hallways, and they make decisions based on these conclusions, but they cannot themselves correct their errors. If something is recognized as a "doorway" yet cannot be passed through, this failure will not be recognized and not used to correct future doorway recognitions, unless it is done by people.

On the other hand, once one has grounding, the further step to include verification is less daunting. One need only attach to the execution procedures appropriate tests and termination conditions that measure in some sense the veracity of the original statement, while at the same time specifying what it really means in detail. What is a chair? Not just something that lights up your visual chair detector! That would be grounded knowledge, but not verifiable; it would rely on people to say which were and were not chairs. But suppose you have routines for trying to sit. Then all you need for a verifier is to be able to measure your success at sitting. You can then verify, improve, and maintain your "sittable thing" recognizer on your own.

There is a great contrast between the AI that I am proposing and what might be considered classical "database AI". There are large AI efforts to codify vast amounts of knowledge in databases or "ontologies", of which Doug Lenat's CYC is only the most widely known. In these efforts, the idea of people maintaining the knowledge is embraced. Special knowledge representation methods and tools are emphasized to make it easier for people to understand and access the knowledge, and to try to keep it right. These systems tend to emphasize static, world knowledge like "Springfield is the capital of Illinois", "a canary is a kind of bird", or even "you have a meeting scheduled with John at 3:30", rather than the dynamic knowledge needed say by a robot to interact in real time with its environment. A major problem is getting people to use the same categories and terms when they enter knowledge and, more importantly, to mean the same things by them. There is a search for an ultimate "ontology", or codification of all objects and their possible relationships, so that clear statements can be made about them. But so far this has not proven possible; there always seem to be far more cases that don't fit than do. People are good about being fluid with there concepts, and knowing when they don't apply.

Whatever the ultimate success of the symbolic "database AI" approach, it should be clear that it is the anti-thesis of what I am calling for. The database approach calls for heroic efforts organizing and entering an objective, public, and disembodied knowledge base. I am calling for an AI that maintains its own representations, perhaps different from those of others, while interacting in real time with a dynamic environment. Most important of all, the database approach embraces human maintenance and human organization of the AI's knowledge. I am calling for automating these functions, for the AI being able to understand its knowledge well enough to verify it itself.