Roger’s Takeaway

這是一篇很短的文章，但點出AI最大的重點，就是當AI必須交由外部來驗證知識的真偽，以及藉由外部提供知識的話，我們就很難真正建立起龐大的知識系統。

Highlight

1.一個 AI 系統能夠創造與維護知識的程度，取決於它能自行驗證這些知識的程度。

2.這個 AI 系統能否驗證自己的知識，還是需要人們介入來偵測錯誤和未預見的交互作用，並進行修正。只要後者成立，我們就永遠無法建立真正龐大的知識系統。

逐字稿

對於一位 AI 研究者來說，聲稱自己對於領域的發展方向有獨到的見解或計畫，似乎有點不合時宜。如果他真有這樣的想法，何不直接去實踐，如果他是對的，就展示其獨特的成果？儘管如此，對整個領域進行評估與分析，診斷那些反覆困擾領域的弊病，並提出通用的解決方案，仍然有其價值。

我所要提出的見解是，一個成功的人工智慧的關鍵在於它能夠自行判斷自己是否運作正確。在某個層面上，這是一個務實的問題。如果 AI 無法自行判斷其運作是否得當，那麼就必須由某個人來進行評估並做出任何必要的修改。一個能夠自我評估的 AI，或許也能夠自行修改。

驗證原則：

一個 AI 系統能夠創造與維護知識的程度，取決於它能自行驗證這些知識的程度。

成功的驗證存在於所有基於搜尋的 AI 系統中，例如規劃器、遊戲玩家，甚至是遺傳演算法。舉例來說，Deep Blue 透過廣泛的搜尋為其每一步可能的棋步產生評分。它之所以相信某一步棋是好棋，是透過搜尋樹所驗證的，該搜尋樹顯示了這步棋將不可避免地帶來一個好的局面。這些系統不需要被告知該做什麼選擇；它們可以自己判斷。試想一下，如果我們透過告訴西洋棋程式在每種棋局下該走哪種棋來編寫它。許多早期的西洋棋程式就是這樣建構的。問題當然在於，西洋棋的棋局種類繁多。而且，程式設計師給予的走棋建議和規則越多，系統就變得越複雜，規則之間意想不到的交互作用也越多。這些程式變得脆弱且不可靠，需要不斷維護，不久之後，這種方法就完全輸給了「暴力」搜尋法。

雖然基於搜尋的規劃器在選擇棋步的層面進行驗證，但它們通常無法在其他層面進行驗證。例如，它們通常將其狀態評估的評分函數視為給定的。即使是 Deep Blue 也無法搜尋到遊戲的終局，而是依賴一個由人類調整的位置評分函數，它無法自行評估這個函數。西洋雙陸棋冠軍程式 TD-Gammon 的一個主要優勢在於，它確實會評估並改進自己的評分函數。

另一個基於搜尋的規劃器幾乎從未進行驗證的重要層面，是那些指定棋步、行動或操作結果的層面。在像西洋棋這樣合法棋步數量有限的遊戲中，我們可以輕易地想像準確地編寫所有棋步的後果。但如果我們在更廣泛的 AI 情境下想像規劃，那麼許多被允許的行動，其結果將不會是完全已知的。如果我把貝果帶到 Leslie 的辦公室，她會在那裡嗎？開車去上班需要多長時間？我今天能完成這份報告嗎？我們每天做的許多決定都帶有不確定且變動的影響。然而，現代的 AI 系統幾乎從不考慮這一點。它們假設所有的行動模型都將由人工準確輸入，即使這些模型可能構成了系統中大部分的知識，或是系統曾經產生的所有知識。

最後，讓我們對一般性的知識提出同樣的觀點。思考任何一個 AI 系統及其擁有的知識。它可能是一個專家系統，或是一個像 CYC 這樣的大型資料庫。或者它可能是一個擁有建築物佈局知識，或是在各種情況下如何反應的知識的機器人。在所有這些案例中，我們都可以問，這個 AI 系統能否驗證自己的知識，還是需要人們介入來偵測錯誤和未預見的交互作用，並進行修正。只要後者成立，我們就永遠無法建立真正龐大的知識系統。它們將永遠是脆弱且不可靠的，並且其規模將受限於人們自身所能監控和理解的範圍。

「永遠不要編寫任何比你的腦袋還大的程式」

然而，今日的 AI 系統絕大多數都無法驗證自身的知識。大型的本體論和知識庫被建立起來，卻完全依賴於人類的建構與維護。它們說「鳥有翅膀」，但它們當然無法自行驗證這一點。

November 15, 2001

It is a bit unseemly for an AI researcher to claim to have a special insight or plan for how his field should proceed. If he has such, why doesn't he just pursue it and, if he is right, exhibit its special fruits? Without denying that, there is still a role for assessing and analyzing the field as a whole, for diagnosing the ills that repeatedly plague it, and to suggest general solutions.

The insight that I would claim to have is that the key to a successful AI is that it can tell for itself whether or not it is working correctly. At one level this is a pragmatic issue. If the AI can't tell for itself whether it is working properly, then some person has to make that assessment and make any necessary modifications. An AI that can assess itself may be able to make the modifications itself.

The Verification Principle:

An AI system can create and maintain knowledge only to the extent that it can verify that knowledge itself.

Successful verification occurs in all search-based AI systems, such as planners, game-players, even genetic algorithms. Deep Blue, for example, produces a score for each of its possible moves through an extensive search. Its belief that a particular move is a good one is verified by the search tree that shows its inevitable production of a good position. These systems don't have to be told what choices to make; they can tell for themselves. Image trying to program a chess machine by telling it what kinds of moves to make in each kind of position. Many early chess programs were constructed in this way. The problem, of course, was that there were many different kinds of chess positions. And the more advice and rules for move selection given by programmers, the more complex the system became and the more unexpected interactions there were between rules. The programs became brittle and unreliable, requiring constant maintainence, and before long this whole approach lost out to the "brute force" searchers.

Although search-based planners verify at the move selection level, they typically cannot verify at other levels. For example, they often take their state-evaluation scoring function as given. Even Deep Blue cannot search to the end of the game and relies on a human-tuned position-scoring function that it does not assess on its own. A major strength of the champion backgammon program, TD-Gammon, is that it does assess and improve its own scoring function.

Another important level at which search-based planners are almost never subject to verification is that which specifies the outcomes of the moves, actions, or operators. In games such as chess with a limited number of legal moves we can easily imagine programming in the consequences of all of them accurately. But if we imagine planning in a broader AI context, then many of the allowed actions will not have their outcomes completely known. If I take the bagel to Leslie's office, will she be there? How long will it take to drive to work? Will I finish this report today? So many of the decisions we take every day have uncertain and changing effects. Nevertheless, modern AI systems almost never take this into account. They assume that all the action models will be entered accurately by hand, even though these may be most of the knowledge in or ever produced by the system.

Finally, let us make the same point about knowledge in general. Consider any AI system and the knowledge that it has. It may be an expert system or a large database like CYC. Or it may be a robot with knowledge of a building's layout, or knowledge about how to react in various situations. In all these cases we can ask if the AI system can verify its own knowledge, or whether it requires people to intervene to detect errors and unforeseen interactions, and make corrections. As long as the latter is the case we will never be able to build really large knowledge systems. They will always be brittle and unreliable, and limited in size to what people can monitor and understand themselves.

"Never program anything bigger than your head"

And yet it is overwhelmingly the case that today's AI systems are not able to verify their own knowledge. Large ontologies and knowledge bases are built that are totally reliant on human construction and maintenance. "Birds have wings" they say, but of course they have no way of verifying this.