Welcome to the online introduction to artificial intelligence.My name is Sebastian Thrun. >>I'm Peter Norvig.We are teaching this class at Stanford, and now we are teaching it online for the entire world.We are really excited about this.It's great to have you all here.It's exciting to have such a record-breaking number of people.We think we can deliver a good introduction to artificial intelligence.We hope you'll stick with it.It's going to be a lot of work,but we think it's going to be very rewarding.The way that it is going to be organized is that every week there is going to be new videos and with these videos, quizes.With these quizzes, you can test your knowledge about AI.We also post for the advanced version of this class, homework assignments and exams on which you'll be quizzed.We're going to grade those to give you a final score to seeif you can actually master artificial intelligence the same way any good student at Stanford would do it.If you do that, then at the end of the class, we'll sign a letter of accomplishment, and let you know that you've achieved this and what your rank in the class was.So I hope you have fun. Watch us on videotape.We will teach you AI.Participate in the discussion forum. Ask your questions, and help others answer questions.I hope we have a fantastic time ahead of us in the next 10 weeks.Welcome to the class. We'll see you online.Welcome to the first unit of Online Introduction to Artificial Intelligence.I will be teaching you the very, very basics today.This is Unit 1 of Artificial Intelligence.Welcome.The purpose of this class is twofold:Number 1, to teach you the very basics of artificial intelligenceso you'll be able to talk to people in the fieldand understand the basic tools of the trade;and also, very importantly, to excite you about the field.I have been in the field of artificial intelligence for about 20 years,and it's been truly rewarding.So I want you to participate in the beauty and the excitement of AIso you can become a professional who gets the same reward and excitement out of this field as I do.The basic structure of this class involves videos in which Peter or I will teach you something new,then also quizzes, which we will ask you about your ability to answer AI questions,and finally, answer videos in which we tell you what the right answer would have beenfor the quiz that you might have falsely or incorrectly answered before.This will all be reiterated, and every so often you get a homework assignment,also in the form of quizzes but without the answers.And then we also have video exams.If you check our website, there's requirementson how you have to do assignments and exams.Please go to ai-class.org in this class.An AI program is called wetware, a formula, or an intelligent agent.Pick the one that fits best.[Thrun] The correct answer is intelligent agent.Let's talk about intelligent agents.Here is my intelligent agent,and it gets to interact with an environment.The agent can perceive the state of the environmentthrough its sensors,and it can affect its state through its actuators.The big question of artificial intelligence is the function that maps sensors to actuators.That is called the control policy for the agent.So all of this class will deal with how does an agent make decisionsthat it can carry out with its actuators based on past sensor data.Those decisions take place many, many times,and the loop of environment feedback to sensors, agent decision,actuator interaction with the environment and so on is called perception action cycle.So here is my very first quiz for you.Artificial intelligence, AI, has successfully been used in finance,robotics, games, medicine, and the Web.Check any or all of those that apply.And if none of them applies, check the box down here that says none of them.So the correct answer is all of those-- finance, robotics, games, medicine, the Web, and many more applications.So let me talk about them in some detail. There is a huge number of applications of artificial intelligence in finance,very often in the shape of making trading decisions--in which case, the agent is called a trading agent. And the environment might be things like the stock market or the bond market or the commodities market.And our trading agent can sense the course of certain things, like the stock or bonds or commodities.It can also read the news online and follow certain events.And its decisions are usually things like buy or sell decisions--trades. There's a huge history of artificial intelligence finding methods to look at data over timeand make predictions as to how courses develop over time--and then put in trades behind those. And very frequently, people using artificial intelligence trading agentshave made a good amount of money with superior trading decisions. There's also a long history of AI in Robotics. Here is my depiction of a robot. Of course, there are many different types of robotsand they all interact with their environments through their sensors,which include things like cameras, microphones, tactile sensor or touch.And the way they impact their environments is to move motors around, in particular, their wheels, their legs, their arms, their grippers.They can also say things to people using voice. Now there's a huge history of using artificial intelligence in robotics.Pretty much, every robot that does something interesting today uses AI. In fact, often AI has been studied together with robotics, as one discipline.But because robots are somewhat special in that they use physical actuatorsand deal with physical environments, they are a little bit different from just artificial intelligence, as a whole. When the Web came out, the early Web crawlers were called robots and to block a robot from accessing your website, to the present day, there's a file called robot.txt, that allows you to deny any Web crawler to access and retrieve that information from your website. So historically, robotics played a huge role in artificial intelligenceand a good chunk of this class will be focusing on robotics. AI has a huge history in games--to make games smarter or feel more natural.There are 2 ways in which AI has been used in games, as a game agent. One is to play against you, as a human user.So for example, if you play the game of Chess,then you are the environment to the game agent.The game agent gets to observe your moves, and it generates its own moveswith the purpose of defeating you in Chess. So most adversarial games, where you play against an opponentand the opponent is a computer program,the game agent is built to play against you--against your own interests--and make you lose. And of course, your objective is to win. That's an AI games-type situation. The second thing is that games agents in AI also are used to make games feel more natural.So very often games have characters inside, and these characters act in some way.And it's important for you, as the player, to feel that these characters are believable. There's an entire sub-field of artificial intelligence to use AIto make characters in a game more believable--look smarter, so to speak--so that you, as a player, think you're playing a better game. Artificial intelligence has a long history in medicine as well. The classic example is that of a diagnostic agent. So here you are--and you might be sick, and you go to your doctor. And your doctor wishes to understand what the reason for your symptoms and your sickness is.The diagnostic agent will observe you through various measurements-- for example, blood pressure and heart signals, and so on--and it'll come up with the hypothesis as to what you might be suffering from.But rather than intervene directly, in most cases the diagnostic of your disease is communicated to the doctor, who then takes on the intervention.This is called a diagnostic agent. There are many other versions of AI in medicine.AI is used in intensive care to understand whether there are situationsthat need immediate attention.It's been used for life-long medicine to monitor signs over long periods of time.And as medicine becomes more personal, the role of artificial intelligencewill definitely increase. We already mentioned AI on the Web.The most generic version of AI is to crawl the Web and understand the Web,and assist you in answering questions.So when you have this search box over hereand it says "Search" on the left,and "I'm Feeling Lucky" on the right,and you type in the words,what AI does for you is it understands what words you typed inand finds the most relevant pages. That is really co-artificial intelligence. It's used by a number of companies, such as Microsoft and Googleand Amazon, Yahoo, and many others.And the way this works is that there's a crawling agent that can goto the World Wide Web and retrieve pages, through just a computer program.It then sorts these pages into a big database inside the crawlerand also analyzes developments of each page to any possible query. When you then come and issue a query, the AI system is able to give you a response-- for example, a collection of 10 best Web links. In short, every time you try to write a piece of software,that makes your computer software smartlikely you will need artificial intelligence.And in this class, Peter and I will teach you many of the basic tricks of the tradeto make your software really smart.It will be good to introduce some basic terminologythat is commonly used in artificial intelligence to distinguish different types of problems.The very first word I will teach you is fully versus partially observable.An environment is called fully observable if what your agent can senseat any point in time is completely sufficient to make the optimal decision. So, for example, in many card games, when all the cards are on the table, the momentary site of all those cardsis really sufficient to make the optimal choice.That is in contrast to some other environments where you need memoryon the side of the agent to make the best possible decision.For example, in the game of poker, the cards aren't openly on the table,and memorizing past moves will help you make a better decision.To fully understand the difference, consider the interaction of an agent with the environment to its sensors and its actuators, and this interaction takes place over many cycles, often called the perception-action cycle.For many environments, it's convenient to assume that the environment has some sort of internal state.For example, in a card game where the cards are not openly on the table, the state might pertain to the cards in your hand.An environment is fully observable if the sensors can always see the entire state of the environment. It's partially observable if the sensors can only see a fraction of the state,yet memorizing past measurements gives us additional information of the statethat is not readily observable right now.So any game, for example, where past moves have information about what might be in a person's hand, those games are partially observable,and they require different treatment. Very often agents that deal with partially observable environmentsneed to acquire internal memory to understand what the state of the environment is, and we'll talk extensively when we talk about hidden Markov models about how this structurehas such internal memory. A second terminology for environments pertains to whether the environmentis deterministic or stochastic. Deterministic environment is one where your agent's actionsuniquely determine the outcome. So, for example, in chess, there's really no randomness when you move a piece.The effect of moving a piece is completely predetermined,and no matter where I'm going to move the same piece, the outcome is the same. That we call deterministic. Games with dice, for example, like backgammon, are stochastic. While you can still deterministically move your pieces, the outcome of an action also involves throwing of the dice, and you can't predict those. There's a certain amount of randomness involved for the outcome of dice,and therefore, we call this stochastic. Let me talk about discrete versus continuous.A discrete environment is one where you have finitely many action choices, and finitely many things you can sense. So, for example, in chess, again, there's finitely many board positions,and finitely many things you can do.That is different from a continuous environment where the space of possible actions or things you could sense may be infinite. So, for example, if you throw darts, there's infinitely many ways to angle the dartsand to accelerate them. Finally, we distinguish benign versus adversarial environments. In benign environments, the environment might be random.It might be stochastic, but it has no objective on its own that would contradict the own objective. So, for example, weather is benign.It might be random. It might affect the outcome of your actions.But it isn't really out there to get you. Contrast this with adversarial environments, such as many games, like chess,where your opponent is really out there to get you.It turns out it's much harder to find good actions in adversarial environmentswhere the opponent actively observes you and counteracts what you're trying to achieverelative to benign environment, where the environment might merely be stochasticbut isn't really interested in making your life worse.So, let's see to what extent these expressions make sense to youby going to our next quiz. So here are the 4 concepts again: partially observable versus fully,stochastic versus deterministic, continuous versus discrete, adversarial versus benign.And let me ask you about the game of checkers.Check one or all of those attributes that apply.So, if you think checkers is partially observable, check this one.Otherwise, just don't check it.If you think it's stochastic, check this one,continuous, check this one, adversarial, check this one.If you don't know about checkers, you can check the Web and Google itto find a little more information about checkers.So, checkers is an interesting game.Here's the typical board of the game of checkers.Your pieces might look like this, and your opponent's pieces might look like this.And apart from some very cryptic rules in checkers,which I won't really discuss here, the board basically tells youeverything there is to know about checkers, so it's clearly fully observable.It is deterministic because your move and your opponent's movevery clearly affect the state of the board in ways that have absolutely no stochasticity. It is also discrete because there's finitely many action choicesand finitely many board positions,and obviously, it is adversarial, since your opponent is out to get you.[Male narrator] The game of poker--is this partially observable, stochastic, continuous, or adversarial?Please check any or all of those that apply.[Male narrator] I would argue poker is partially observablebecause it can't be seen what is in your opponent's hands.It is stochastic because you're being dealt cards that are kind of coming at random.It is not continuous; it's just finding many cardsand finding many actions you can do, even though you might arguethat there's a huge number of different monies you can bet. It's still finite, and it is clearly adversarial.If you've ever played poker before, you know how brutal it can be.[Male narrator] --a favorite, a robotic car.I wish to know whether it is partially observable, stochastic, continuous, or adversarial. That is, is the problem of driving robotically--say, in a city--subject to any of those 4 categories?Please check any or all that might apply.Well, the robotic car clearly deals with a partially observable environmentif you just look at momentary sensing input, you can't even tell how fast other cars are going.So, you need to memorize something.It is stochastic because it's inherently unpredictable what's going to happen next with other cars. It is continuous. There's the infinitely many ways to set your steeringor push your gas pedal or your brake,and, well, you can argue with adversarial or not.Depending on where you live, it might be highly adversarial. Where I live, it isn't.I'm going to briefly talk of AI as something else, which is AI is the technique of uncertainty management in computer software.Put differently, AI is the discipline that you apply when you want to know what to dowhen you don't know what to do.Now, there's many reasons why there might be uncertainty in a computer program.There could be a sensor limit.That is, your sensors are unable to tell me what exactly is the case outside the AI system.There could be adversaries who act in a way that makes it hard for youto understand what is the case.There could be stochastic environments.Every time you roll the dice in a dice game,the stochasticity of the dice will make it impossible for youto be absolutely certain of what's the situation. There could be laziness.So perhaps you can actually compute what the situation is,but your computer program is just too lazy to do it.And here's my favorite: ignorance, plain ignorance. Many people are just ignorant of what's going on.They could know it, but they just don't care.All of these things are cause for uncertainty.AI is the discipline that deals with uncertainty and manages it in decision making. Now we've had an introduction to AI.We've heard about some of the properties of environments, and we've seen some possible architecture for agents. I'd like next to show you some examples of AI in practice.And Sebastian and I have some experience personally in things we have doneat Google, at NASA, and at Stanford.And I want to tell you a little bit about some of those.One of the best successes of AI technology at Google has been the machine translation system. Here we see an example of an article in Italian automatically translated into English.Now, these systems are built for 50 different languages,and we can translate from any of the languages into any of the other languages.So, that's over 2,500 different systems, and we've done this all using machine learning techniques, using AI techniques,rather than trying to build them by hand.And the way it works is that we go out and collect examples of textthat's a line between the 2 languages.So we find, say, a newspaper that publishes 2 editions, an Italian edition and an English edition, and now we have examples of translations. And if anybody ever asked us for exactly the translation of this one particular article,then we could just look it up and say "We already know that."But of course, we aren't often going to be asked that.Rather, we're going to be asked parts of this.Here are some words that we've seen before, and we have to figure out which words in this article correspond to which words in the translation article.And when we do that by examining many, many millions of words of text in the 2 languages and making the correspondence, and then we can put that all together.And then when we see a new example of text that we haven't seen before,we can just look up what we've seen in the past for that correspondence.So, the task is really two parts.Off-line, before we see an example of text we want to translate,we first build our translation model.We do that by examining all of the different examplesand figuring out which part aligns to which.Now, when we're given a text to translate, we use that model, and we go through and find the most probable translation.So, what does it look like?Well, let's look at it in some example text.And rather than look at news articles, I'm going to look at something simpler. I'm going to switch from Italian to Chinese.Here's a bilingual text.Now, for a large-scale machine translation, examples are found on the Web.This example was found in a Chinese restaurant by Adam Lopez.Now, it's given, for a text of this form, that a line in Chinese corresponds to a line in English,and that's true for each of the individual lines.But to learn from this text, what we really want to discoveris what individual words in Chinese correspond to individual wordsor small phrases in English.I've started that process by highlighting the word "wonton" in English.It appears 3 times throughout the text.Now, in each of those lines, there's a character that appears,and that's the only place in the Chinese text where that character appears.So, that seems like it's a high probability that this character in Chinesecorresponds to the word "wonton" in English.Let's see if we can go farther.My question for you is what word or what character or characters in Chinesecorrespond to the word "chicken" in English?And here we see "chicken" appears in these locations.Click on the character or characters in Chinese that corresponds to "chicken."The answer is that chicken appears here,here, here, and here.Now, I don't know for sure, 100%, that that is the character for chicken in Chinese,but I do know that there is a good correspondence.Every place the word chicken appears in English,this character appears in Chinese and no other place.Let's go 1 step farther.Let's see if we can work out a phrase in Chineseand see if it corresponds to a phrase in English.Here's the phrase corn cream.Click on the characters in Chinese that correspond to corn cream.The answer is: these 2 characters hereappear only in these 2 locationscorresponding to the words corn cream which appear only in these locations in the English text.Again, we're not 100% sure that's the right answer,but it looks like a strong correlation.Now, 1 more question.Tell me what character or characters in Chinesecorrespond to the English word soup.The answer is that soup occurs in most of these phrasesbut not 100% of them.It's missing in this phrase.Equivalently, on the Chinese side we see this character occursin most of the phrases,but it's missing here.So we see that the correspondence doesn't have to be 100%to tell us that there is still a good chance of a correlation.When we're learning to do machine translationwe use these kinds of alignments to learn probability tablesof what is the probability of one phrase in one language corresponding to the phrase in another language.So congratulations, you just finished unit 1.You just finished unit 1 of this class, where I told you about key applicationsof artificial intelligence,I told you about the definition of an intelligent agent,I gave you 4 key attributes of intelligent agents(partial observability, stochasticity, continuous spaces, and adversarial natures),I discussed sources and management of uncertainty,and I briefly mentioned the mathematical concept of rationality.Obviously, I only touched any of these issues superficially,but as this class goes on you're going to dive into any of thoseand learn much more about what it takes to make a truly intelligent AI system.Thank you.[PROBLEM SOLVING]In this unit we're going to talk about problem solving.The theory and technology of building agentsthat can plan ahead to solve problems.In particular, we're talking about problem solving where the complexity of the problem comes from the idea that there are many states.As in this problem here.A navigation problem where there are many choices to start with.And the complexity comes from picking the right choice now and picking the right choice at thenext intersection and the intersection after that.Streaming together a sequence of actions.This is in contrast to the type of complexity shown in this picture,where the complexity comes from the partial observabilitythat we can't see through the fog where the possible paths are.We can't see the results of our actionsand even the actions themselves are not known.This type of complexity will be covered in a later unit. Here's an example of a problem.This is a route-finding problem where we're given a start city,in this case, Arad, and a destination, Bucharest, the capital of Romania,from which this is a corner of the map.And the problem then is to find a route from Arad to Bucharest.The actions that the agent can execute when driving from one city to the next along one of the roads shown on the map.The question is, is there a solution that the agent can come up with given the knowledge shown here to the problem of driving from Arad to Bucharest?And the answer is no.There is no solution that the agent can come up with because Bucharest doesn't appear on the map,and so the agent doesn't know any actions that can arrive there. So let's give the agent a better chance. Now we've given the agent the full map of Romania.To start, he's in Arad, and the destination--or goal--is in Bucharest.And the agent is given the problem of coming up with a sequence of actionsthat will arrive at the destination.Now, is it possible for the agent to solve this problem?And the answer is yes. There are many routes or steps or sequences of actions that will arrive at the destination. Here is one of them:Starting out in Arad, taking this step first, then this one, then this one, then this one, and then this one to arrive at the destination. So that would count as a solution to the problem.So sequence of actions, chained together, that are guaranteed to get us to the goal. [DEFINITION OF A PROBLEM]Now let's formally define what a problem looks like. A problem can be broken down into a number of components. First, the initial state that the agent starts out with. In our route finding problem, the initial state was the agent being in the city of Arad. Next, a function--Actions--that takes a state as input and returns a set of possible actions that the agent can execute when the agent is in this state. [ACTIONS (s) {a,a2,a3...}]In some problems, the agent will have the same actions available in all statesand in other problems, he'll have different actions dependent on the state. In the route finding problem, the actions are dependent on the state.When we're in one city, we can take the routes to the neighboring cities--but we can't go to any other cities. Next we have a function called Result, which takes, as input, a state and an actionand delivers, as its output, a new state. So, for example, if the agent is in the city of Arad, and takes--that would be the state--and takes the action of driving along Route E-671 towards Timisoara,then the result of applying that action in that state would be the new state--where the agent is in the city of Timisoara.Next, we need a function called Goal Test,which takes a state and returns a Boolean value-- true or false--telling us if this state is a goal or not. In a route-finding problem, the only goal would be being in the destination city--the city of Bucharest--and all the other states would return false for the Goal Test. And finally, we need one more thing which is a Path Cost function--which takes a path, a sequence of state/action transitions,and returns a number, which is the cost of that path.Now, for most of the problems we'll deal with, we'll make the Path Cost function be additiveso that the cost of the path is just the sum of the costs of the individual steps. And so we'll implement this Path Cost function, in terms of a Step Cost function. The Step Cost function takes a state, an action, and the resulting state from that action and returns a number--n--which is the cost of that action. In the route finding example, the cost might be the number of miles traveled or maybe the number of minutes it takes to get to that destination. Now let’s see how the definition of a problemmaps onto the route finding, the domain.First, the initial state was given. Let’s say we start off in Arad,and the goal test,let’s say that the state of being in Bucharestis the only state that counts as a goal,and all the other states are not goals.Now the set of all of the states hereis known as the state space,and we navigate the state space by applying actions.The actions are specific to each city,so when we are in Arad, there are three possible actions,to follow this road, this one, or this one.And as we follow them, we build pathsor sequences of actions.So just being in Arad is the path of length zero,and now we could start exploring the spaceand add in this path of length one,this path of length one,and this path of length one.We could add in another path here of length two and another path here of length two.Here is another path of length two.Here is a path of length three.Another path of length two, and so on.Now at ever point, we want to separate the state out into three parts.First, the ends of the paths—The farthest paths that have been explored,we call the frontier.And so the frontier in this caseconsists of these statesthat are the farthest out we have explored.And then to the left of that in this diagram,we have the explored part of the state.And then off to the rigtht,we have the unexplored.So let’s write down those three components.We have the frontier.We have the unexplored region,and we have the explored region.One more thing,in this diagram we have labeled the step cost of each action along the route.So the step cost of going between Neamt to Iasiwould be 87 corresponding to a distance of 87 kilometers,and the path cost is just the sum of the step costs.So the cost of the pathof going from Arad to Oradeawould be 71 plus 75.[Narrator] Now let's define a function for solving problems.It's called Tree Search because it superimposes a search tree over the state space.Here's how it works: It starts off byinitializing the frontier to be the pathconsisting of only the initial states,and then it goes into a loop in which it first checks to see do we still have anything left in the frontier?If not we fail, there can be no solution.If we do have something, then we make a choice.Tree Search is really a family of functionsnot a single algorithm which depends on how we make that choice,and we'll see some of the options later.If we go ahead and make a choice of one of the paths on the frontier and remove that path from the frontier, we find the statewhich is at the end of the path, and if thatstate's a go then we're done.We found a path to the goal; otherwise, we do what's called expanding that path.We look at all the actions from that state,and we add to the path the actions and the result of that state; so we get a new path that has the old path, the action and the result of that action, and westick all of those paths back onto the frontier.Now Tree Search represents a whole family of algorithms, and where you get the familyresemblance is that they're all looking at the frontier, copying items off and and looking to see if their goal tests,but where you get the difference is right here,in the choice of how you're going to expand the next item on the frontier, which path do we look at first, and we'll go throughdifferent sets of algorithms that make different choices for which path to look at first.The first algorithm I want to consider is called Breadth-First Search.Now it could be called shortest-first searchbecause what it does is always chooseof the frontier one of the paths that hadn't beenconsidered yet that's the shortest possible. So how does it work?Well we start off with the path of length 0, starting in the start state, and that's the only path in the frontier soit's the shortest one so we pick it,and then we expand it, and we add inall the paths that result from applying all the possible actions.So now we've removed this path from the frontier,but we've added in 3 new paths. This one, this one, and this one.Now we're in a position wherewe have 3 paths on the frontier, andwe have to pick the shortest one.Now in this case all 3 paths have the same length, length 1, so we break the tie at random or using someother technique, and let's suppose that in this case we choose this pathfrom Arad to Sibiu.Now the question I want you to answer is once we remove that from the frontier,what paths are we going to add next?So show me by checking off the citiesthat ends the paths, which paths are going to be added to the frontier?[Male narrator] The answer is that in Sibiu, the action function gives us 4 actionscorresponding to traveling along these 4 roads, so we have to add in paths for each of those actions. One of those paths goes here, the other path continues from Arad and goes out here. The third path continues out hereand then the fourth path goes from here--from Arad to Sibiuand then backtracks back to Arad.Now, it may seem silly and redundant to have a path that starts in Arad,goes to Sibiu and returns to Arad.How can that help us get to our destination in Bucharest?But we can see if we're dealing with a tree search, why it's natural to have this type of formulation and why the tree search doesn't even notice that it's backtracked. What the tree search does is superimpose on top of the state spacea tree of searches, and the tree looks like this. We start off in state A, and in state A, there were 3 actions,so we gave those paths going to Z, S, and T.And from S, there were 4 actions, so that gave us paths going from O, F, R, and A,and then the tree would continue on from here. We'd take one of the next itemsand we'd move it and continue on, but notice that we returned to the A statein the state space, but in the tree, it's just another item in the tree. Now, here's another representation of the search spaceand what's happening is as we start to explore the state, we keep track of the frontier, which is the set of states that are at the end of the pathsthat we haven't explored yet, and behind that frontieris the set of explored states, and ahead of the frontier is the unexplored states. Now the reason we keep track of the explored statesis that when we want to expand and we find a duplicate--so say when we expand from here, if we pointed back to state T,if we hadn't kept track of that, we would have to add in a new state for T down here.But because we've already seen it and we know that this is actually a regressive stepinto the already explored state, now, because we kept track of that,we don't need it anymore. Now we see how to modify the Tree Search Functionto make it be a Graph Search Functionto avoid those repeated paths.What we do, is we start off and initialize a setcalled the explored set of states that we have already explored.Then, when we consider a new path,we add the new state to the set of already explored states,and then when we are expanding the path and adding in new states to the end of it,we don’t add that in if we have already seen that new statein either the frontier or the explored.Now back to Breadth First Search.Let’s assume we are using the Graph Searchso that we have eliminated the duplicate paths.Arad is crossed off the list.The path that goes from Arad to Sibiuand back to Arad is removed,and we are left with these one, two, three,four, five possible paths.Given these 5 paths,show me which ones are candidates to be expanded nextby the Breadth First Search Algorithm.[Male narrator] And the answer is that Breadth - First Search always considers the shortest paths first, and in this case, there's 2 paths of length 1,and 1, the paths from Arad to Zerind and Arad to Timisoara,so those would be the 2 paths that would be considered. Now, let's suppose that the tie is broken in some way and we chose this path from Arad to Zerind.Now, we want to expand that node. We remove it from the frontier and put it in the explored listand now we say, "What paths are we going to add?"So check off the ends of the paths the cities that we're going to add.[Male narrator] In this case, there's nothing to addbecause of the 2 neighbors, 1 is in the explored list and 1 is in the frontier,and if we're using graph search, then we won't add either of those. [Male narrator] So we move on, we look for another shortest path.There's one path left of length 1, so we look at that path, we expand it,add in this path, put that one on the explored list, and now we've got 3 paths of length 2. We choose 1 of them, and let's say we choose this one. Now, my question is show me which states we add to the pathand tell me whether we're going to terminate the algorithm at this point because we've reached the goal or whether we're going to continue.[Male narrator] The answer is that we add 1 more path, the path to Bucharest. We don't add the path going back because it's in the explored list,but we don't terminate it yet.True, we have added a path that ends in Bucharest,but the goal test isn't applied when we add a path to the frontier. Rather, it's applied when we remove that path from the frontier,and we haven't done that yet. [Male narrator] Now, why doesn't the general tree search or graph search algorithm stopwhen it adds a goal node to the frontier? The reason is because it might not be the best path to the goal.Now, here we found a path of length 2and we added a path of length 3 that reached the goal.The general graph search or tree search doesn't know that there might be some other path that we could expandthat would have a distance of say, 2-1/2,but there's an optimization that could be made.If we know we're doing Breadth - First Searchand we know there's no possibility of a path of length 2-1/2.Then we can change algorithm so that it checks statesas soon as they're added to the frontierrather than waiting until they're expandedand in that case, we can write a specific Breadth - First Search routine that terminates early and gives us a result as soon as we add a goal state to the frontier.Breadth - First Search will find this path that ends up in Bucharest, and if we're looking for the shortest path in terms of number of steps, Breadth - First Search is guaranteed to find it, But if we're looking for the shortest path in terms of total costby adding up the step costs, then it turns out that this path is shorter than the path found by Breadth - First Search. So let's look at how we could find that path. An algorithm that has traditionally been called uniform-cost searchbut could be called cheapest-first search,is guaranteed to find the path with the cheapest total cost. Let's see how it works.We start out as before in the start state.And we pop that empty path off.Move it from the frontier to explored,and then add in the paths out of that state.As before, there will be 3 of those paths.And now, which path are we going to pick nextin order to expand according to the rules of cheapest first?Cheapest first says that we pick the path withthe lowest total cost.And that would be this path.It has a cost of 75 compared to the cost of 118 and 140for the other paths.So we get here. We take that path off the frontier,put it on the explored list, add in its neighbors.Not going back to Arad,but adding in this new path.Summing up the total cost of that path, 71 + 75 is 146 for this path.And now the question is, which path gets expanded next?Of the 3 paths on the frontier, we have ones with a cost of 146, 140, and 118.And that's the cheapest, so this one gets expanded.We take it off the frontier, move it to explored,add in its successors. In this case it's only 1.And that has a path total of 229.Which path do we expand next?Well, we've got 146, 140, and 229So 140 is the lowest.Take it off the frontier. Put it on explored.Add in this pathfor a total cost of 220.And this path for a total cost of 239.And now the question is, which path do we expand next?The answer is this one, 146.Put it on explored.But there's nothing to add becauseboth of its neighbors have already been explored.Which path do we look at next?The answer is this one. Two-twenty is less than 229 or 239.Take it off the frontier. Put it on explored.Add in 2 more paths and sum them up.So, 220 plus 146 is 366.And 220 plus 97 is 317.Okay, and now, notice that we're closing in on Bucharest.We've got 2 neighbors almost there, but neither of them is their turn yet.Instead, the cheapest path is this one over here,so move it to the explored list.Add 70 to the path cost so far,and we get 299.Now the cheapest node is 239 here,so we expand, finally, into Bucharest at a cost of 460.And now the question is are we done? Can we terminate the algorithm?[Male] And the answer is no, we're not done yet.We've put Bucharest, the gold state, onto the frontier,but we haven't popped it off the frontier yet.And the reason is because we've got to look around and see if there's a better paththat can reach it, Bucharest.And so, let's continue.Look at everything on the frontier.Here's the cheapest one over here.Expand that.Now, what's the cheapest next one?Well, over here.Oops, forgot to take this one off the list.So now, we're at 317 plus 101 gives us another path into Bucharest,and this is a better path. This is 418, gives us another route in.But we have to keep going.The best path on the frontier is 366,so pop that off, and that would give us 2 more routes into here,and eventually we pop off all of these.And then we get to the point where 418 was the best path on the frontier.We pop that off, and then we recognize that we'd reach the goal,and the reason that uniform cost finds the optimal path, the cheapest cost, is because it's guaranteed that it will first pop off this cheapest path, the 418, before it gets to the more expensive path, like the 460.So, we've looked at 2 search algorithms.One, breadth-first search, in which we always expand first the shallowest paths, the shortest paths.Second, cheapest-first search, in which we always expand first the pathwith the lowest total cost.And I'm going to take this opportunity to introduce a third algorithm, depth-first search,which is in a way the opposite of breadth-first search.In depth-first search, we always expand first the longest path, the path with the most lengths in it.Now, what I want to ask you to do is for each of these nodes in each of the trees,tell us in what order they're expanded,first, second, third, fourth, fifth and so on by putting a number into the box.And if there are ties, put that number in and resolve the ties in left to right order.Then I want you to ask one more question or answer one more questionwhich is are these searches optimal?That is, are they guaranteed to find the best solution?And for breadth-first search, optimal would mean finding the shortest path.If you think it's guaranteed to find the shortest path, check here.For cheapest first, it would mean finding the path with the lowest total path cost.Check here if you think it's guaranteed to do that.And we'll allow the assumption that all costs have to be positive.And in depth first, cheapest or optimal would mean, again,as in breadth first, finding the shortest possible path in terms of number of lengths.Check here if you think depth first will always find that.Here are the answers.Breadth-first search, as the name implies, expands nodes in this order.One, 2, 3, 4, 5, 6, 7. So, it's going across a stripe at a time, breadth first.Is it optimal?Well, it's always expanding in the shortest paths first,and so wherever the goal is hiding, it's going to find it by examiningno longer paths, so in fact, it is optimal.Cheapest first, first we expand the path of length zero, then the path of length 2.Now there's a path of length 4, path of length 5, path of length 6, a path of length 7, and finally, a path of length 8.And as we've seen, it's guaranteed to find the cheapest path of all,assuming that all the individual step costs are not negative.Depth-first search tries to go as deep as it can first,so it goes 1, 2, 3, then backs up, 4, then backs up, 5, 6, 7.And you can see that it doesn't necessarily find the shortest path of all.Let's say that there were goals in position 5 and in position 3.It would find the longer path to position 3 and find the goal thereand would not find the goal in position 5.So, it is not optimal.Given the non-optimality of depth-first search,why would anybody choose to use it?Well, the answer has to do with the storage requirements.Here I've illustrated a state space consisting of a very large or even infinite binary tree.As we go to levels 1, 2, 3, down to level n, the tree gets larger and larger.Now, let's consider the frontier for each of these search algorithms. For breadth-first search, we know a frontier looks like that, and so when we get down to level n, we'll require a storage space of 2 to the n of pass in a breadth-first search.For cheapest first, the frontier is going to be more complicated.It's going to sort of work out this contour of cost,but it's going to have a similar total number of nodes.But for depth-first search, as we go down the tree, we start going down this branch,and then we back up, but at any point, our frontier is only going to have n nodesrather than 2 to the n nodes, so that's a substantial savings for depth-first search.Now, of course, if we're also keeping track of the explored set, then we don't get that much savings.But without the explored set, depth-first search has a huge advantage in terms of space saved.One more property of the algorithms to consideris the property of completeness, meaning if there is a goal somewhere,will the algorithm find it?So, let's move from very large trees to infinite trees, and let's say that there's some goal hidden somewhere deep down in that tree.And the question is, are each of these algorithms complete?That is, are they guaranteed to find a path to the goal?Mark off the check boxes for the algorithms that you believe are complete in this sense.The answer is that breadth-first search is complete,so even if the tree is infinite, if the goal is placed at any finite level,eventually, we're going to march down and find that goal.Same with cheapest first. No matter where the goal is, if it has a finite cost,eventually, we're going to go down and find it.But not so for depth-first search.If there's an infinite path, depth-first search will keep following that,so it will keep going down and down and down along this pathand never get to the path that the goal consists ofand never get to the path on which the goal sits. So, depth-first search is not complete.Let's try to understand a little better how uniform cost search works.We start at a start state, and then we start expanding out from there looking at different paths, and what we end of doing is expanding in terms of contours like on a topological map,where first we span out to a certain distance, then to a farther distance,and then to a farther distance.Now at some point we meet up with a goal. Let's say the goal is here.Now we found a path from the start to the goal.But notice that the search really wasn't directed at any way towards the goal.It was expanding out everywhere in the space and depending on where the goal is, we should expect to have to explore half the space, on average, before we find the goal.If the space is small, that can be fine,but when spaces are large, that won't get us to the goal fast enough.Unfortunately, there is really nothing we can do, with what we know, to do better than that,and so if we want to improve, if we want to be able to find the goal faster, we're going to have to add more knowledge.The type of knowledge that is proven most useful in search is an estimate of the distance from the start state to the goal.So let's say we're dealing with a route-finding problem, and we can move in any direction--up or down, right or left--and we'll take as our estimate, the straight line distance between a state and a goal,and we'll try to use that estimate to find our way to the goal fastest.Now an algorithm called greedy best-first search does exactly that.It expands first the path that's closest to the goal according to the estimate.So what do the contours look like in this approach?Well, we start here, and then we look at all the neighboring states,and the ones that appear to be closest to the goal we would expand first.So we'd start expanding like this and like this and like this and like thisand that would lead us directly to the goal.So now instead of exploring whole circles that go out everywhere with a certain space,our search is directed towards the goal.In this case it gets us immediately towards the goal, but that won't always be the case if there are obstacles along the way.Consider this search space. We have a start state and a goal,and there's an impassable barrier.Now greedy best-first search will start expanding out as before,trying to get towards the goal,and when it reaches the barrier, what will it do next?Well, it will try to increase along a path that's getting closer and closer to the goal.So it won't consider going back this way which is farther from the goal.Rather it will continue expanding out along these lineswhich always get closer and closer to the goal,and eventually it will find its way towards the goal.So it does find a path, and it does it by expanding a small number of nodes,but it's willing to accept a path which is longer than other paths.Now if we explored in the other direction, we could have found a much simpler path,a much shorter path, by just popping over the barrier, and then going directly to the goal.but greedy best-first search wouldn't have done that because that would have involved getting to this point, which is this distance to the goal, and then considering states which were farther from the goal.What we would really like is an algorithm that combines the best parts of greedy search which explores a small number of nodes in many cases and uniform cost search which is guaranteed to find a shortest path.We'll show how to do that next using an algorithm called the A-star algorithm.[Male narrator] A* Search works by always expanding the path that has a minimum value of the function fwhich is defined as a sum of the g + h components.Now, the function g of a pathis just the path cost,and the function h of a path is equal to the h value of the state, which is the final state of the path,which is equal to the estimated distance to the goal.Here's an example of how A* works.Suppose we found this path through the state's base to a state xand we're trying to give a measure to the value of this path.The measure f is a sum of g, the path cost so far, and h, which is the estimated distance that the path will take to complete its path to the goal. Now, minimizing g helps us keep the path shortand minimizing h helps us keep focused on finding the goaland the result is a search strategy that is the best possiblein the sense that it finds the shortest length pathwhile expanding the minimum number of paths possible.It could be called "best estimated total path cost first,"but the name A* is traditional. Now let's go back to Romania and apply the A* algorithmand we're going to use a heuristic, which is a straight line distancebetween a state and the goal. The goal, again, is Bucharest,and so the distance from Bucharest to Bucharest is, of course, 0.And for all the other states, I've written in redthe straight line distance.For example, straight across like that. Now, I should say that all the roads here I've drawn as straight lines,but actually, roads are going to be curved to some degree, so the actual distance along the roads is going to be longerthan the straight line distance. Now, we start out as usual--we'll start in Arad as a start state--and we'll expand out Arad and so we'll add 3 pathsand the evaluation function, f, will be the sum of the path length,which is given in black, and the estimated distance, which is given in red. And so the path length from this path will be 140+253 or 393;for this path, 75+374, or 449;and for this path, 118+329, or 447.And now, the question is out of all the paths that are on the frontier,which path would we expand next under the A* algorithm? The answer is that we select this path first--the one from Arad to Sibiu--because it has the smallest value--393--of the sum f=g+h.Let's go ahead and expand this node now. So we're going to add 3 paths.This one has a path cost of 291and an estimated distance to the goal of 380, for a total of 671.This one has a path cost of 239and an estimated distance of 176, for a total of 415. And the final one is 220+193=413.And now the question is which state to we expand next? The answer is we expand this path nextbecause its total, 413, is less than all the other ones on the front tier--although only slightly less than the 415 for this path. So we expand this node,giving us 2 more paths--this one with an f-value of 417,and this one with an f-value of 526.The question again--which path are we going to expand next?And the answer is that we expand this path, Fagaras, next,because its f-total, 415, is less than all the other paths in the front tier. Now we expand Fagaras and we get a path that reaches the goaland it has a path length of 450 and an estimated distance of 0for a total f value of 450,and now the question is: What do we do next?Click here if you think we're at the end of the algorithm and we don't need to expand nextor click on the node that you think we will expand next.The answer is that we're not done yet,because the algorithm works by doing the goal test, when we take a path off the front tier, not when we put a path on the front tier. Instead, we just continue in the normal way and choose the nodeon the front tier which has the lowest value.That would be this one--the path through Pitesti, with a total of 417.So let's expand the node at Pitesti. We have to go down this direction, up,then we reach a path we've seen before,and we go in this direction.Now we reach Bucharest, which is the goal,and the h value is going to be 0 because we're at the goal, and the g value works out to 418.Again, we don't stop here just because we put a path onto the front tier,we put it there, we don't apply the goal test next,but, now we go back to the front tier,and it turns out that this 418 is the lowest-cost path on the front tier.So now we pull it off, do the goal test,and now we found our path to the goal,and it is, in fact, the shortest possible path.In this case, A-star was able to find the lowest-cost path.Now the question that you'll have to think about,because we haven't explained it yet,is whether A-star will always do this.Answer yes if you think A-star will always find the shortest cost path,or answer no if you think it depends on the particular problem given,or answer no if you think it depends on the particular heuristic estimate function, h.The answer is that it depends on the h function.A-star will find the lowest-cost path if the h function for a state is less than the true costof the path to the goal through that state.In other words, we want the h to never overestimate the distance to the goal.We also say that h is optimistic.Another way of stating thatis that h is admissible, meaning is it admissible to use it to find the lowest-cost path.Think of all of these of being the same wayof stating the conditions under which A-star finds the lowest-cost path.Here we give you an intuition as to whyan optimistic heuristic function, h, finds the lowest-cost path.When A-star ends, it returns a path, p, with estimated cost, c.It turns out that c is also the actual cost,because at the goal the h component is 0,and so the path cost is the total cost as estimated by the function.Now, all the paths on the front tierhave an estimated cost that's greater than c,and we know that because the front tier is explored in cheapest-first order.If h is optimistic, then the estimated cost is less than the true cost,so the path p must have a cost that's less than the true costof any of the paths on the front tier.Any paths that go beyond the front tiermust have a cost that's greater than thatbecause we agree that the step cost is always 0 or more.So that means that this path, p, must be the minimal cost path.Now, this argument, I should say, only goes throughas is for tree search.For graph search the argument is slightly more complicated, but the general intuitions hold the same.So far we've looked at the state space of cities in Romania--a 2-dimensional, physical space.But the technology for problem solving through searchcan deal with many types of state spaces,dealing with abstract properties, not just x-y position in a plane.Here I introduce another state space--the vacuum world.It's a very simple world in which there are only 2 positionsas opposed to the many positions in the Romania state space.But there are additional properties to deal with as well.The robot vacuum cleaner can be in either of the 2 conditions,but as well as that each of the positions can either have dirt in it or not have dirt in it.Now the question is to represent this as a state spacehow many states do we need?The number of states can fill in this box here.And the answer is there are 8 states.There are 2 physical states that the robot vacuum cleaner can be in--either in state A or in state B.But in addition to that, there are states about how the world isas well as where the robot is in the world.So state A can be dirty or not.That's 2 possibilities.And B can be dirty or not.That's 2 more possibilities.We multiply those together. We get 8 possible states.Here is a diagram of the state space for the vacuum world. Note that there are 8 states, and we have the actions connecting the statesjust as we did in the Romania problem.Now let's look at a path through this state.Let's say we start out in this position,and then we apply the action of moving right.Then we end up in a position where the state of the world looks the same,except the robot has moved from position 'A' to position 'B'.Now if we turn on the sucking action,then we end up in a state where the robot is in the same positionbut that position is no longer dirty.Let's take this very simple vacuum worldand make a slightly more complicated one.First, we'll say that the robot has a power switch,which can be in one of three conditions: on, off, or sleep.Next, we'll say that the robot has a dirt-sensing camera,and that camera can either be on or off.Third, this is the deluxe model of robotin which the brushes that clean up the dustcan be set at 1 of 5 different heights to be appropriate for whatever level of carpeting you have.Finally, rather that just having the 2 positions,we'll extend that out and have 10 positions.Now the question is how many states are in this state space?The answer is that the number of states is the cross product of the numbers of all the variables, since they're each independent,and any combination can occur.For the power we have 3 possible positions.The camera has 2.The brush height has 5.The dirt has 2 for each of the 10 positions.That's 2^10 or 1024.Then the robot's position can be any of those 10 positions as well. That works out to 307,200 states in the state space.Notice how a fairly trivial problem--we're only modeling a few variables and only 10 positions--works out to a large number of state spaces.That's why we need efficient algorithms for searching through states spaces.I want to introduce one more problem that can be solved with search techniques.This is a sliding blocks puzzle, called a 15 puzzle.You may have seen something like this.So there are a bunch of little squares or blocks or tilesand you can slide them around.and the goal is to get into a certain configuration. So we'll say that this is the goal state, where the numbers 1-15 are in orderleft to right, top to bottom.The starting state would be some state where all the positions are messed up.Now the question is: Can we come up with a good heuristic for this?Let's examine that as a way of thinking about where heuristics come from.The first heuristic we're going to considerwe'll call h1, and that is equal to the number of misplaced blocks.So here 10 and 11 are misplaced because they should be there and there, respectively,12 is in the right place, 13 is in the right place,and 14 and 15 are misplaced. That's a total of 4 misplaced blocks.The 2nd heuristic, h2, is equal tothe sum of the distances that each block would have to move to get to the right position.For this position, 10 would have to move 1 space to get to the right position,11 would have to move 1, so that's a total of 2 so far, 13 is in the right place,14 is 1 displaced, and 15 is 1 displaced,so that would also be a total of 4.Now, the question is: Which, if any, of these heuristics are admissible?Check the boxes next to the heuristics that you think are admissible.H1 is admissible, because every tile that's in the wrong position must be moved at least once to get into the right position.So h1 never overestimates.How about h2?H2 is also admissible, because every tile in the wrong positioncan be moved closer to the correct position no faster than 1 space per move.Therefore, both are admissible.But notice that h2 is always greater than or equal to h1.That means that, with the exception of breaking ties,an A* search using h2 will always expandfewer paths than one using h1Now, we're trying to build an artificial intelligencethat can solve problems like this all on its own.You can see that the search algorithms do a great jobof finding solutions to problems like this.But, you might complain that in order for the search algorithms to work,we had to provide it with a heurstic function.A heurstic function came from the outside.You might think that coming up with a good heurstic function is really where all the intelligence is.So, a problem solver that uses an heurstic function given to itreally isn't intelligent at all.So let's think about where the intelligence could come fromand can we automatically come up with good heurstic functions.I'm going to sketch a description ofa program that can automatically come up with good heursticsgiven a description of a problem.Suppose this program is given a description of the sliding blocks puzzlewhere we say that a block can move from square A to square Bif A is adjacent to B and B is blank.Now, imagine that we try to loosen this restriction.We cross out "B is blank,"and then we get the rule"a block can move from A to B if A is adjacent to B,"and that's equal to our heurstic h2because a block can move anywhere to an adjacent state.Now, we could also cross out the other part of the rule,and we now get "a block can move from any square Ato any square B regardless of any condition.That gives us heurstic h1.So we see that both of our heurstics can be derived from a simple mechanical manipulationof the formal description of the problem.Once we've generated automatically these candidate heuristics,another way to come up with a good heurstic is to say that a new heurstic, h, is equal to the maximum of h1 and h2,and that's guaranteed to be admissible as long as h1 and h2 are admissiblebecause it still never overestimates, and it's guaranteed to be better because its getting closer to the true value.The only problem with combining multiple heuristics like thisis that there is some cause to compute the heuristicand it could take longer to compute even if we end up expanding pure paths.Crossing out parts of the rules like thisis called "generating a relaxed problem."What we've done is we've taken the original problem,where it's hard to move squares around,and made it easier by relaxing one of the constraints.You can see that as adding new links in the state space,so if we have a state space in which there are only particular links,by relaxing the problem it's as if we are adding new operatorsthat traverse the state in new ways.So adding new operators only makes the problem easier,and thus never overestimates, and thus is admissible.We've seen what search can do for problem solving.It can find the lowest-cost path to a goal,and it can do that in a way in which we never generate more paths than we have to.We can find the optimal number of paths to generate,and we can do that with a heuristic function that we generate on our ownby relaxing the existing problem definition.But let's be clear on what search can't do.All the solutions that we have found consist of a fixed sequence of actions.In other words, the agent Hirin Arad, thinks, comes up with a plan that it wants to executeand then essentially closes his eyes and starts driving,never considering along the way if something has gone wrong.That works fine for this type of problem,but it only works when we satisfy the following conditions.[Problem solving works when:]Problem-solving technology works when the following set of conditions is true:First, the domain must be fully observable.In other words, we must be able to see what initial state we start out with.Second, the domain must be known.That is, we have to know the set of available actions to us.Third, the domain must be discrete.There must be a finite number of actions to chose from.Fourth, the domain must be deterministic.We have to know the result of taking an action.Finally, the domain must be static.There must be nothing else in the world that can change the world except our own actions.If all these conditions are true, then we can search for a planwhich solves the problem and is guaranteed to work.In later units, we will see what to do if any of these conditions fail to hold.Our description of the algorithm has talked about paths in the state space.I want to say a little bit now about how to implement that in terms of a computer algorithm.We talk about paths, but we want to implement that in some ways.In the implementation we talk about nodes.A node is a data structure, and it has four fields.The state field indicates the state at the end of the path.The action was the action it took to get there.The cost is the total cost,and the parent is a pointer to another node.In this case, the node that has state "S", and it will have a parent which points to the node that has state "A",and that will have a parent pointer that's null.So we have a linked list of nodes representing the path.We'll use the word "path" for the abstract idea, and the word "node" for the representation in the computer memory.But otherwise, you can think of those two terms as being synonyms,because they're in a one-to-one correspondence.Now there are two main data structures that deal with nodes.We have the "frontier" and we have the "explored" list.Let's talk about how to implement them.In the frontier the operations we have to deal withare removing the best item from the frontier and adding in new ones.And that suggests we should implement it as a priority queue,which knows how to keep track of the best items in proper order.But we also need to have an additional operation of a membership test as a new item in the frontier.And that suggests representing it as a set,which can be built from a hash table or a tree.So the most efficient implementations of search actually have both representations.The explored set, on the other hand, is easier.All we have to do there is be able to add new members and check for membership.So we represent that as a single set, which again can be done with either a hash table or tree.Congratulations.You just made assignment 1.This is homework assignment #1.This is a question about peg solitaire.In peg solitaire, a single player facesthe following kind of board.Initially, all pieces are occupied except for the center piece.You can find more information on peg solitare at the following URL.[http://en.wikipedia.org/wiki/peg_solitaire]I wish to know whether this game is partially observable,Please say yes or no.I wish to know whether it is stochastic.Please say yes if it is and no if it's deterministic.Let me know if it's continuous, yes or no,and let me know if it's adversarial, yes or no.>>Peg Solitaire is not partially observable because you can see the board at all times.It is not stochastic because you just make all the moves, and they have very different mystic effects.It is not continuous. It's just finding many choices of actions and finding many board positions, so therefore, it is not continuous.and it's not adversarial because there is no adversaries--just you playing.I am going to ask you about the problem to learn about a loaded coin.A loaded coin is a coin,that if you flip it, might have a non 0.5 chanceof coming up heads or tails.Fair coins always come up 50% heads or tails.Loaded coins might come up, for example, 0.9 chance heads and 0.1 chance tails.Your task will be to understand,from coin flips,whether a coin is loaded,and if so, at what probability.I don't want you to solve the problem,but I want you to answer the following questions:Is it partially observable?Yes or no.Is it stochastic?Yes or no.Is it continuous? [Yes or no.]And finally, is it adversarial?Yes or no.[Thrun] So the loaded coin example is clearly partially observable,and the reason is it is actually used for the memoryif you flip it more than 1 time so you can learn more about what the actual probability is.Therefore, looking at the most recent coin flip is insufficient to make your choice.It is stochastic because you flip a coin.It is not continuous because there's only 1 action--a flip--and 2 outcomes.And it isn't really adversarial because while you do your learning taskno adversary interferes.Let's talk about the problem of finding a path through a maze.Let me draw you a maze. Suppose you wish to find the path from the start to your goal.I don't want to you to solve this problem.Rather I want you to tell me whether it's partially observable.Yes or no.It is stochastic?Yes or no.Is it continuous?Yes or no.[Thrun] The path through the maze is clearly not partially observablebecause you can see the maze entirely at all times.It is not stochastic. There is no randomness involved.It isn't really continuous. There's typically just finitely many choices--go left or right.And it isn't adversarial because there's no real adversary involved.This is a search question. Suppose we are given the following search tree. We are searching from the top, the start node, to the goal, which is over here. Assume we expand from left to right. Tell me how many nodes are expanded if we expand from left to right, counting the start node and the goal node in your answer. And give me the same answer for Depth First Search. Now, let's assume you're going to search from right to left. How many nodes would we now expand in Breadth First Search,and how many do we expand in Depth First Search?[Thrun] Breadth first from left to right is 6--1, 2, 3, 4, 5, 6.Depth first from left to right is 4--1, 2, 3, 4.Breadth first searched from right to left is 9--1, 2, 3, 4, 5, 6, 7, 8, 9.And depth first from right to left is 9--1, 2, 3, 4, 5, 6, 7, 8, 9.Another search problem--Consider the following search tree,where this is the start node.Now, assume we search from left to right.I would like you to tell me the number of nodes expanded from Breadth-First Searchand Depth-First Search.Please do count the start and the goal node,and please give me the same numbers for Right-to-Left Search,for Breadth-First, and Depth-First. [Thrun] The correct answer for breadth first left to right is 13--1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13.And for depth first it is 10--1, 2, 3, 4, 5, 6, 7, 8, 9, and 10.For right to left search, the right answer for breadth first is 11--1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11.And for depth first the right answer is 7--1, 2, 3, 4, 5, 6, 7.This is another search problem.Let's assume we have a search graph.It isn't quite a tree but looks like this.Obviously in the structure we can reach nodes through multiple paths.So let's assume that our search never expands the same node twice.Let's also assume this start node is on top. We search down.And this over here is our goal node.So left-to-right search, tell me how many nodesbreadth first would expand--do count the start and goal node in the final answer.Give me the same result for a depth-first search.Again counting the start and the goal node in your answer.And again give me your answer for breadth-firstand for depth-first in the right-to-left search paradigm.[Thrun] The right answer over here is 10 for breadth first from left to right--1, 2, 3, 4, 5, 6, 7, 8, 9, 10.Depth first is 16, or all nodes--1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16.And notice how I never expanded a node twice.Correct answer for breadth first right to left is 7--1, 2, 3, 4, 5, 6, 7.And the correct answer for depth first from right to left is 4--1, 2, 3, and 4.Let's talk about a star search.Let's assume we have the following grid.The start state is right here.And the goal state is right here.And just for convenience, I will give each here a little number.A. B. C. D.Let me draw a heuristic function.Please take a look for a momentand tell me whether this heuristic function is admissable.Check here if yes and here if no.Which one is the first node a star would expand?B1 or A2?What's the second node to expand?B1, C1, A2, A3, or B2?And finally, what is the third node to expand?D1, C2, B3, or A4? [Thrun] Clearly this is an admissable heuristic because the distance to the goalis strictly underestimated.From here it would take 1 step,from here it will take 1, 2 steps, so the answer is yes.Now, to understand A*, let me also draw the g function for development part of this table.Clearly g is 0 over here.To understand which node to expand, this one or this one,let's project the g function, which is 1,and we will see that 3 plus 1 is smaller than 4 plus 1;therefore, this is the second node to expand, which is b1.Now let me for the next step explain the g function from this guy here, 2 and 2.So 2 plus 2 is 4 versus 3 plus 2 is 5, so we expand this node next, which is c1.And finally, the g function from here would go 3 and 3.3 plus 1 is better than 3 plus 2, so we would expand d1 next.And notice how in the sum of g and h,this node over here, which has a total of 4, is better than any other node that is unexpanded.So in particular, 4 plus 1 is 5, and 3 plus 2 is 5 as well, and 2 plus 3 is 5 as well, so this is the next one to expand.So the next units will be concerned with probabilitiesand particularly with structured probabilities using Bayes networks.This is some of the most involved material in this class.And since this is a Stanford level class, you will find out that some of the quizzes are actually really hard.So as you go through the material, I hope the hardness of the quizzes won't discourage you;it'll really entice you to take a piece of paper and a pen and work them out.Let me give you a flavor of a Bayes network using an example.Suppose you find in the morning that your car won't start.Well, there's many causes why your car might not start.One is that your battery is flat.Even for a flat battery there is multiple causes.One, it's just plain dead,and one is that the battery is okay but it's not charging.The reason why a battery might not charge is that the alternator might be brokenor the fan belt might be broken.If you look at this influence diagram, also called a Bayes network,you'll find there's many different ways to explain that the car won't start.And a natural question you might have is, "Can we diagnose the problem?"One diagnostic tool is a battery meter,which may increase or decrease your belief that the battery may cause your car failure.You might also know your battery age.Older batteries tend to go dead more often.And there's many other ways to look at reasons why the car might not start.You might inspect the lights, the oil light, the gas gauge.You might even dip into the engine to see what the oil level is with a dipstick.All of those relate to alternative reasons why the car might not be starting, like no oil, no gas, the fuel line might be blocked, or the starter may be broken.And all of these can influence your measurements,like the oil light or the gas gauge, in different ways.For example, the battery flat would have an effect on the lights.It might have an effect on the oil light and on the gas gauge,but it won't really affect the oil you measure with the dipstick.That is affected by the actual oil level, which also affects the oil light.Gas will affect the gas gauge, and of course without gas the car doesn't start.So this is a complicated structure that really describes one way to understandhow a car doesn't start.A car is a complex system.It has lots of variables you can't really measure immediately,and it has sensors which allow you to understand a little bit about the state of the car.What the Bayes network does,it really assists you in reasoning from observable variables, like the car won't startand the value of the dipstick, to hidden causes, like is the fan belt brokenor is the battery dead.What you have here is a Bayes network.A Bayes network is composed of nodes.These nodes correspond to events that you might or might not knowthat are typically called random variables.These nodes are linked by arcs, and the arcs suggest that a child of an arcis influenced by its parent but not in a deterministic way.It might be influenced in a probabilistic way, which means an older battery, for example,has a higher chance of causing the battery to be dead,but it's not clear that every old battery is dead.There is a total of 16 variables in this Bayes network.What the graph structure and associated probabilities specifyis a huge probability distribution in the space of all of these 16 variables.If they are all binary, which we'll assume throughout this unit,they can take 2 to the 16th different values, which is a lot.The Bayes network, as we find out, is a complex representationof a distribution over this very, very large joint probability distribution of all of these variables.Further, once we specify the Bayes network,we can observe, for example, the car won't start.We can observe things like the oil light and the lights and the battery meterand then compute probabilities of the hypothesis, like the alternator is brokenor the fan belt is broken or the battery is dead.So in this class we're going to talk about how to construct this Bayes network,what the semantics are, and how to reason in this Bayes network to find out about variables we can't observe, like whether the fan belt is broken or not.That's an overview.Throughout this unit I am going to assume that every event is discrete--in fact, it's binary.We'll start with some consideration of basic probability,we'll work our way into some simple Bayes networks,we'll talk about concepts like conditional independenceand then define Bayes networks more generally,move into concepts like D-separation and start doing parameter counts.Later on, Peter will tell you about inference in Bayes networks.So we won't do this in this class.I can't overemphasize how important this class is.Bayes networks are used extensively in almost all fields of smart computer system,in diagnostics, for prediction, for machine learning, and fields like finance,inside Google, in robotics.Bayes networks are also the building blocks of more advanced AI techniquessuch as particle filters, hidden Markov models, MDPs and POMDPs, Kalman filters, and many others.These are words that don't sound familiar quite yet,but as you go through the class, I can promise you you will get to know what they mean.So let's start now at the very, very basics.[Thrun] So let's talk about probabilities.Probabilities are the cornerstone of artificial intelligence.They are used to express uncertainty,and the management of uncertainty is really key to many, many things in AIsuch as machine learning and Bayes network inference and filtering and robotics and computer vision and so on.So I'm going to start with some very basic questions,and we're going to work our way up from there.Here is a coin.The coin can come up heads or tails, and my question is the following:Suppose the probability for heads is 0.5.What's the probability for it coming up tails?[Thrun] So the right answer is a half, or 0.5,and the reason is the coin can only come up heads or tails.We know that it has to be either one.Therefore, the total probability of both coming up is 1.So if half of the probability is assigned to heads, then the other half is assigned to tail.[Thrun] Let me ask my next quiz.Suppose the probability of heads is a quarter, 0.25.What's the probability of tail?[Thrun] And the answer is 3/4. It's a loaded coin, and the reason is, well, each of them come up with a certain probability.The total of those is 1. The quarter is claimed by heads.Therefore, 3/4 remain for tail, which is the answer over here.[Thrun] Here's another quiz.What's the probability that the coin comes up heads, heads, heads, three times in a row,assuming that each one of those has a probability of a halfand that these coin flips are independent?[Thrun] And the answer is 0.125.Each head has a probability of a half.We can multiply those probabilities because they are independent events,and that gives us 1 over 8 or 0.125.[Thrun] Now let's flip the coin 4 times, and let's call Xi the result of the i-th coin flip.So each Xi is going to be drawn from heads or tail.What's the probability that all 4 of those flips give us the same result,no matter what it is, assuming that each one of those has identicallyan equally distributed probability of coming up heads of the half?[Thrun] And the answer is, well, there's 2 ways that we can achieve this.One is the all heads and one is all tails.You already know that 4 times heads is 1/16,and we know that 4 times tail is also 1/16.These are completely independent events.The probability of either one occurring is 1/16 plus 1/16, which is 1/8, which is 0.125.[Thrun] So here's another one.What's the probability that within the set of X1, X2, X3, and X4there are at least three heads?[Thrun] And the solution is let's look at different sequences in which head occurs at least 3 times.It could be head, head, head, head, in which it comes 4 times.It could be head, head, head, tail and so on, all the way to tail, head, head, head.There's 1, 2, 3, 4, 5 of those outcomes.Each of them has a 16th for probability, so it's 5 times a 16th, which is 0.3125.[Thrun] So we just learned a number of things.One is about complementary probability.If an event has a certain probability, p,the complementary event has the probability 1-p.We also learned about independence.If 2 random variables, X and Y, are independent, which you're going to write like this,that means the probability of the joint that any 2 variables can assume is the product of the marginals.So rather than asking the question, "What is the probability"for any combination that these 2 coins or maybe 5 coins could have taken?"we can now look at the probability of each coin individually,look at its probability and just multiply them up.[Thrun] So let me ask you about dependence. Suppose we flip 2 coins.Our first coin is a fair coin, and we're going to denote the outcome by X1.So the chance of X1 coming up heads is half.But now we branch into picking a coin based on the first outcome.So if the first outcome was heads, you pick a coin whose probability of coming up heads is going to be 0.9.The way I word this is by conditional probability,probability of the second coin flip coming up headsprovided that or given that X1, the first coin flip, was heads, is 0.9.The first coin flip might also come up tails,in which case I pick a very different coin.In this case I pick a coin which with 0.8 probability will once again give me tails,conditioned on the first coin flip coming up tails.So my question for you is,what's the probability of the second coin flip coming up heads?[Thrun] The answer is 0.55.The way to compute this is by the theorem of total probability.Probability of X2 equals heads.There's 2 ways I can get to this outcome.One is via this path over here, and one is via this path over here.Let me just write both of them down.So first of all, it could be the probability of X2 equals headsgiven that and I will assume X1 was head already.Now I have to add the complementary event.Suppose X1 came up tails.Then I can ask the question, what is the probability that X2 comes up heads regardless,even though X1 was tails?Plugging in the numbers gives us the following.This one over here is 0.9 times a half.The probability of tails is 0.8,thereby my head probability becomes 1 minus 0.8, which is 0.2.Adding all of this together gives me 0.45 plus 0.1,which is exactly 0.55.So, we actually just learned some interesting lessons.The probability of any random variable Y can be written asprobability of Y given that some other random variable X assumes value itimes probability of X equals i,sums over all possible outcomes i for the (inaudible) variable X.This is called total probability.The second thing we learned has to do with negation of probabilities.We found that probability of not X given Y is 1 minus probability of X given Y.Now, you might be tempted to say "What about the probability of X given not Y?""Is this the same as 1 minus probability of X given Y?"And the answer is absolutely no.That's not the case.If you condition on something that has a certain probability value,you can take the event you're looking at and negate this,but you can never negate your conditional variable and assume these values add up to 1.We assume there is sometimes sunny days and sometimes rainy days,and on day 1, which we're going to call D1, the probability of sunny is 0.9.And then let's assume that a sunny day follows a sunny day with 0.8 chance,and a rainy day follows a sunny day with--well--Well, the correct answer is 0.2, which is a negation of this event over here.A sunny day follows a rainy day with 0.6 chance, and a rainy day follows a rainy day--please give me your number.0.4So, what are the chances that D2 is sunny?Suppose the same dynamics apply from D2 to D3, so just replace D3 over here with D2s over there.That means the transition probabilities from one day to the next remain the same.Tell me, what's the probability that D3 is sunny?So, the correct answer over here is 0.78,and over here it's 0.756.To get there, let's complete this one first.The probability of D2 = sunny.Well, we know there's a 0.9 chance it's sunny on D1,and then if it is sunny, we know it stays sunny with a 0.8 chance. So, we multiply these 2 things together, and we get 0.72.We know there's a 0.1 chance of it being rainy on day 1, which is the complement,but if it's rainy, we know it switches to sunny with 0.6 chance, so you multiply these 2 things, and you get 0.06.Adding those two up equals 0.78.Now, for the next day, we know our prior for sunny is 0.78.If it is sunny, it stays sunny with 0.8 probability.Multiplying these 2 things gives us 0.624.We know it's rainy with 0.2 chance, which is the complement of 0.78,but a 0.6 chance if it was (inaudible) sunny.But if you multiply those, 0.132.Adding those 2 things up gives us 0.756.So, to some extents, it's tedious to compute these values,but they can be perfectly computed, as shown here.Next example is a cancer example.Suppose there's a specific type of cancer which exists for 1% of the population.I'm going to write this as follows.You can probably tell me now what the probability of not having this cancer is.And yes, the answer is 0.99.Let's assume there's a test for this cancer, which gives us probabilistically an answer whether we have this cancer or not.So, let's say the probability of a test being positive, as indicated by this + sign,given that we have cancer, is 0.9.The probability of the test coming out negative if we have the cancer is--you name it.0.1, which is the difference between 1 and 0.9.Let's assume the probability of the test coming out positivegiven that we don't have this cancer is 0.2.In other words, the probability of the test correctly sayingwe don't have the cancer if we're cancer free is 0.8.Now, ultimately, I'd like to know what's the probability they have this cancer given they just received a single, positive test?Before I do this, please help me filling out some other probabilitiesthat are actually important.Specifically, the joint probabilities.The probability of a positive test and having cancer.The probability of a negative test and having cancer, and this is not conditional anymore.It's now a joint probability.So, please give me those 4 values over here.And here the correct answer is 0.009,which is the product of your prior, 0.01, times the conditional, 0.9.Over here we get 0.001, the probability of our prior cancer times 0.1.Over here we get 0.198,the probability of not having cancer is 0.99times still getting a positive reading, which is 0.2.And finally, we get 0.792, which is the probability of this guy over here, and this guy over here.Now, our next quiz, I want you to fill in the probability of the cancer given that we just received a positive test.And the correct answer is 0.043.So, even though I received a positive test,my probability of having cancer is just 4.3%,which is not very much given that the test itself is quite sensitive.It really gives me a 0.8 chance of getting a negative result if I don't have cancer.It gives me a 0.9 chance of detecting cancer given that I have cancer. Now, what comes (inaudible) small?Well, let's just put all the cases together.You already know that we received a positive test. Therefore, this entry over here, and this entry over here are relevant.Now, the chance of having a positive test and having cancer is 0.009.Well, I might--when I receive a positive test--have cancer or not cancer,so we will just normalize by these 2 possible causes for the positive test,which is 0.009 + 0.198.We know both these 2 things together gets 0.009 over 0.207, which is approximately 0.043.Now, the interesting thing in this equation is that the chances of having seen a positive test result in the absence of cancersare still much, much higher than the chance of seeing a positive result in the presence of cancer, and that's because our prior for canceris so small in the population that it's just very unlikely to have cancer.So, the additional information of a positive test only erased my posterior probability to 0.043.So, we've just learned about what's probably the most importantpiece of math for this class in statistics called Bayes Rule.It was invented by Reverend Thomas Bayes, who was a British mathematicianand a Presbyterian minister in the 18th century.Bayes Rule is usually stated as follows: P of A given B where B is the evidenceand A is the variable we care about is P of B given A times P of A over P of B.This expression is called the likelihood.This is called the prior, and this is called marginal likelihood.The expression over here is called the posterior.The interesting thing here is the way the probabilities are reworded.Say we have evidence B.We know about B, but we really care about the variable A.So, for example, B is a test result.We don't care about the test result as much as we care about the factwhether we have cancer or not.This diagnostic reasoning--which is from evidence to its causes--is turned upside down by Bayes Rule into a causal reasoning, which is given--hypothetically, if we knew the cause,what would be the probability of the evidence we just observed.But to correct for this inversion, we have to multiplyby the prior of the cause to be the case in the first place,in this case, having cancer or not,and divide it by the probability of the evidence, P(B),which often is expanded using the theorem of total probability as follows.The probability of B is a sum over all probabilities of B conditional on A, lower caps a, times the probability of A equals lower caps a.This is total probability as we already encountered it.So, let's apply this to the cancer case and say we really care about whether you have cancer,which is our cause, conditioned on the evidence that is the result of this hidden cause, in this case, a positive test result.Let's just plug in the numbers.Our likelihood is the probability of seeing a positive test resultgiven that you have cancer multiplied by the prior probabilityof having cancer over the probability of the positive test result,and that is--according to the tables we looked at before--0.9 times a prior of 0.01 over--now we're going to expand this right over here according to total probability which gives us 0.9 times 0.01.That's the probability of + given that we do have cancer.So, the probability of + given that we don't have cancer is 0.2,but the prior here is 0.99.So, if we plug in the numbers we know about, we get 0.009 over 0.009 + 0.198.That is approximately 0.0434, which is the number we saw before.So, if you want to draw Bayes rule graphically, we have a situation where we have an internal variable A, like whether I'm going to die of cancer, but we can't sense A.Instead, we have a second variable, called B, which is our test, and B is observable, but A isn't.This is a classical example of a Bayes network. The Bayes network is composed of 2 variables, A and B.We know the prior probability for A, and we know the conditional. A causes B--whether or not we have cancer, causes the test result to be positive or not, although there was some randomness involved.So, we know what the probability of B given the different values for A,and what we care about in this specific instance is called diagnostic reasoning, which is the inverse of the causal reasoning,the probability of A given B or similarly, probability of A given not B.This is our very first Bayes network, and the graphical representationof drawing 2 variables, A and B, connected with an arcthat goes from A to B is the graphical representation of a distributionof 2 variables that are specified in the structure over here,which has a prior probability and has a conditional probability as shown over here.Now, I do have a quick quiz for you.How many parameters does it take to specifythe entire joint probability within A and B, or differently, the entire Bayes network?I'm not looking for structural parameters that relate to the graph over here.I'm just looking for the numerical parameters of the underlying probabilities.And the answer is 3.It takes 1 parameter to specify P of A from which we can derive P of not A.It takes 2 parameters to specify P of B given A and P given not A, from which we can derive P not B given A and P of not B given not A.So, it's a total of 3 parameters for this Bayes network.So, we just encountered our very first Bayes networkand did a number of interesting calculations.Let's now talk about Bayes Rule and look into more complex Bayes networks.I will look at Bayes Rule again and make an observationthat is really non-trivial.Here is Bayes Rule, and in practice, what we find is this term here is relatively easy to compute.It's just a product, whereas this term is really hard to compute.However, this term over here does not depend on what we assume for variable A.It's just the function of B.So, suppose for a moment we also care about the complementary event of not Agiven B, for which Bayes Rule unfolds as follows.Then we find that the normalizer, P(B), is identical, whether we assume A on the left side or not A on the left side.We also know from prior work that P(A) given B plusP of not A given B must be one because these are 2 complementary events.That allows us to compute Bayes Rule very differentlyby basically ignoring the normalizer, so here's how it goes.We compute P(A) given B--and I want to call this prime, because it's not a real probability--to be just P(B) given A times P(A),which is the normalizer, so the denominator of the expression over here.We do the same thing with not A.So, in both cases, we compute the posterior probability non-normalizedby omitting the normalizer B.And then we can recover the original probabilities by normalizingbased on those values over here, so the probability of A given B, the actual probability, is a normalizer, eta, times this non-normalized form over here.The same is true for the negation of A over here.And eta is just the normalizer that results by adding these 2 values over here togetheras shown over here, and dividing them for one.So, take a look at this for a moment.What we've done is we deferred the calculation of the normalizer over hereby computing pseudo probabilities that are non-normalized.This made the calculation much easier, and when we were done with everything,we just folded it back into the normalizer based on the resulting pseudo probabilities and got the correct answer.The reason why I gave you all this is because I want you to apply it nowto a slightly more complicated problem, which is the 2-test cancer example.In this example, we again might have our unobservable cancer C, but now we're running 2 tests, test 1 and test 2.As before, the prior probability of cancer is 0.01.The probability of receiving a positive test result for either test is 0.9.The probability of getting a negative result given they're cancer free is 0.8.And from those, we were able to compute all the other probabilities,and we're just going to write them down over here.So, take a moment to just verify those.Now, let's assume both of my tests come back positive,so T1 = + and T2 = +.What's the probability of cancer now written in short form probability of C given ++?I want you to tell me what that is, and this is a non-trivial question.So, the correct answer is 0.1698 approximately,and to compute this, I used the trick I've shown you before.Let me write down the running count for cancer and for not canceras I integrate the various multiplications in Bayes Rule.My prior for cancer was 0.01 and for non-cancer was 0.99.Then I get my first +, and the probability of a + given they have cancer is 0.9,and the same for non-cancer is 0.2.So, according to the non-normalized Bayes Rule, I now multiply these 2 things together to get my non-normalized probabilityof having cancer given the plus.Since multiplication is commutative, I can do the same thing again with my 2nd test result, 0.9 and 0.2, and I multiply all of these 3 things together to get my non-normalized probability P prime to be the following: 0.0081, if you multiply those things together,and 0.0396 if you multiply these facts together.And these are not a probability.If we add those for the 2 complementary of cancer/non-cancer,I get 0.0477.However, if I now divide, that is, I normalizethose non-normalized probabilities over here by this factor over here,I actually get the correct posterior probability P of cancer given ++.And they look as follows:approximately 0.1698 and approximately 0.8301.Calculate for me the probability of cancer given that I received one positive and one negative test result.Please write your number into this box.We apply the same trick as beforewhere we use the exact same prior of 0.01.Our first + gives us the following factors: 0.9 and 0.2.And our minus gives us the probability 0.1 for a negative first test result given that we have cancer,and a 0.8 for the inverse of a negative result of not having cancer. We multiply those together.We get our non-normalized probability.And if we now normalize by the sum of those two thingsto turn this back into a probability, we get 0.009over the sum of those two things over here, and this is 0.0056for the chance of having cancer and 0.9943 for the chance of being cancer free.And this adds up approximately to 1, and therefore, is a probability distribution.I want to use a few words of terminology.This, again, is a Bayes network, of which the hidden variable Ccauses the still stochastic test outcomes T1 and T2.And what is really important is that we assume not just that T1 and T2 are identically distributed.We use the same 0.9 for test 1 as we use for test 2,but we also assume that they are conditionally independent.We assumed that if God told us whether we actually had cancer or not,if we knew with absolute certainty the value of the variable C,that knowing anything about T1 would not help us make a statement about T2.Put differently, we assumed that the probability of T2 given C and T1is the same as the probability of T2 given C.This is called conditional independence, which is given the value of the cancer variable C.If you knew this for a fact, then T2 would be independent of T1.It's conditionally independent because the independence only holds trueif we actually know C, and it comes out of this diagram over here.If we look at this diagram, if you knew the variable C over here,then C separately causes T1 and T2.So, as a result, if you know C, whatever counted over here is kind of cut off causally from what happens over here.That causes these 2 variables to be conditionally independent.So, conditional independence is a really big thing in Bayes networks.Here's a Bayes network where A causes B and C,and for a Bayes network of this structure, we know that given A, B and C are independent.It's written as B conditionally independent of C given A.So, here's a question.Suppose we have conditional independence between B and C given A.Would that imply--and there's my question--that B and C are independent?So, suppose we don't know A.We don't know whether we have cancer, for example.What that means is that the test results individually are still independent of each othereven if we don't know about the cancer situation.Please answer yes or no.And the correct answer is NoIntuitively, getting a positive test result about cancergives us information about whether you have cancer or not.So if you get a positive test resultyou're going to raise the probability of having cancerrelative to the prior probability.With that increased probability we will predictthat another test will with a higher likelihoodgive us a positive response than if we hadn't taken the previous test.That's really important to understandSo that we understand it let me make you calculate those probabilitiesLet me draw the cancer example again with two tests.Here's my cancer variableand then there's two conditionally independent tests T1 and T2.And as before let me assume that the prior probability of cancer is 0.01What I want you to compute for me is the probability of the second testto be positive if we know that the first test was positive.So write this into the following box.So, for this one, we want to apply total probability.This thing over here is the same as probability of test 2 to be positive, which I'm going to abbreviate with a +2 over here, conditioned on test 1 being positive and me having cancertimes the probability of me having cancer given test 1 was positive plusthe probability of test 2 being positive conditioned on test 1 being positiveand me not having cancer times the probability of me not having cancergiven that test 1 is positive.That's the same as the theorem of total probability,but now everything is conditioned on +1. Take a moment to verify this.Now, here I can plug in the numbers.You already calculated this one before, which is approximately 0.043,and this one over here is 1 minus that, which is 0.957 approximately.And this term over here now exploits conditional independence, which is given that I know C, knowledge of the first test gives me no more information about the second test.It only gives me information if C was unknown, as was the case over here.So, I can rewrite this thing over here as follows:P of +2 given that I have cancer.I can drop the +1, and the same is true over here.This is exploiting my conditional independence. I knew that P of +1 or +2 conditioned on Cis the same as P of +2 conditioned on C and +1.I can now read those off my table over here, which is 0.9 times 0.043 plus 0.2,which is 1 minus 0.8 over here times 0.957, which gives me approximately 0.2301.So, that says if my first test comes in positive,I expect my second test to be positive with probably 0.2301.That's an increased probability to the default probability, which we calculated before, which is the probability of any test, test 2 come in as positive before was the normalizer of Bayes rule which was 0.207.So, my first test has a 20% chance of coming in positive.My second test, after seeing a positive test,has now an increased probability of about 23% of coming in positive.So, now we've learned about independence,and the corresponding Bayes network has 2 nodes. They're just not connected at all.And we learned about conditional independence, in which case we have a Bayes network that looks like this.Now I would like to know whether absolute independence implies conditional independence.True or false?And I'd also like to know whether conditional independence implies absolute independence.Again, true or false?And the answer is both of them are false.We already saw that conditional independence, as shown over here,doesn't give us absolute independence.So, for example, this is test #1 and test #2.You might or might not have cancer. Our first test gives us information about whether you have cancer or not.As a result, we've changed our prior probability for the second test to come in positive.That means that conditional independence does not imply absolute independence,which means this assumption here falls, and it also turns out that if you have absolute independence, things might not be conditionally independent for reasons that I can't quite explain so far,but that we will learn about next.[Thrun] For my next example, I will study a different type of a Bayes network.Before, we've seen networks of the following type,where a single hidden cause caused 2 different measurements.I now want to study a network that looks just like the opposite.We have 2 independent hidden causes,but they get confounded within a single observational variable.I would like to use the example of happiness.Suppose I can be happy or unhappy.What makes me happy is when the weather is sunny or if I get a raise in my job,which means I make more money.So let's call this sunny, let's call this a raise, and call this happiness.Perhaps the probability of it being sunny is 0.7,probability of a raise is 0.01.And I will tell you that the probability of being happy is governed as follows.The probability of being happy given that both of these things occur--I got a raise and it is sunny--is 1.The probability of being happy given that it is not sunny and I still got a raise is 0.9.The probability of being happy given that it's sunny but I didn't get a raise is 0.7.And the probability of being happy given that it is neither sunny nor did I get a raise is 0.1.This is a perfectly fine specification of a probability distributionwhere 2 causes affect the variable down here, the happiness.So I'd like you to calculate for me the following questions.Probability of a raise given that it is sunny, according to this model.Please enter your answer over here.[Thrun] The answer is surprisingly simple.It is 0.01.How do I know this so fast?Well, if you look at this Bayes network,both the sunniness and the question whether I got a raise impact my happiness.But since I don't know anything about the happiness,there is no way that just the weather might implicate or impact whether I get a raise or not.In fact, it might be independently sunny, and I might independently get a raise at work.There is no mechanism of which these 2 things would co-occur.Therefore, the probability of a raise given that it's sunny is just the same as the probability of a raise given any weather, which is 0.01.[Thrun] Let me talk about a really interesting special instance of Bayes net reasoningwhich is called explaining away.And I'll first give you the intuitive answer,then I'll wish you to compute probabilities for me that manifest the explain away effectin a Bayes network of this type.Explaining away means that if we know that we are happy,then sunny weather can explain away the cause of happiness.If I then also know that it's sunny, it becomes less likely that I received a raise.Let me put this differently.Suppose I'm a happy guy on a specific dayand my wife asks me, "Sebastian, why are you so happy?""Is it sunny, or did you get a raise?"If she then looks outside and sees it is sunny,then she might explain to herself,"Well, Sebastian is happy because it is sunny.""That makes it effectively less likely that he got a raise"because I could already explain his happiness by it being sunny."If she looks outside and it is rainy,that makes it more likely I got a raise,because the weather can't really explain my happiness.In other words, if we see a certain effect that could be caused by multiple causes,seeing one of those causes can explain away any other potential causeof this effect over here.So let me put this in numbers and ask you the challenging question ofwhat's the probability of a raise given that I'm happy and it's sunny?[Thrun] The answer is approximately 0.0142,and it is an exercise in expanding this term using Bayes' rule,using total probability, which I'll just do for you.Using Bayes' rule, you can transform this into P of H given R comma S times P of R given S over P of H given S.We observe the conditional independence of R and Sto simplify this to just P of R,and the denominator is expanded by folding in R and not R,P of H given R comma Stimes P of R plus P of H given not R and Stimes P of not R, which is total probability.We can now read off the numbers from the tables over here,which gives us 1 times 0.01 divided by this expressionthat is the same as the expression over here, so 0.01 plus this thing over here,which you can find over here to be 0.7, times this guy over here, which is 1 minus the value over here, 0.99,which gives us approximately 0.0142.[Thrun] Now, to understand the explain away effect,you have to compare this to the probability of a raise given that we're just happyand we don't know anything about the weather.So let's do that exercise next.So my next quiz is, what's the probability of a raise given that all I know is that I'm happyand I don't know about the weather?This happens to be once again a pretty complicated question, so take your time.[Thrun] So this is a difficult question.Let me compute an auxiliary variable, which is P of happiness.That one is expanded by looking at the different conditions that can make us happy.P of happiness given S and R times P of S and R, which is of course the product of those 2because they are independent,plus P of happiness given not S R, probability of not as Rplus P of H given S and not R times the probability of P of S and not R plus the last case,P of H given not S and not R.So this just looks at the happiness under all 4 combinations of the variablesthat can lead to happiness.And you can plug those straight in.This one over here is 1, and this one over here is the product of S and R, which is 0.7 times 0.01.And as you plug all of those in, you get as a result 0.5245.That's P of H.Just take some time and do the math by going through these different casesusing total probability, and you get this result.Armed with this number, the rest now becomes easy,which is we can use Bayes' rule to turn this around.P of H given R times P of R over P of H.P of R we know from over here, the probability of a raise is 0.01.So the only thing we need to compute now is P of H given R.And again, we apply total probability.Let me just do this over here.We can factor P of H given R as P of H given R and S, sunny,times probability of sunny plus P of H given R and not sunnytimes the probability of not sunny.And if you plug in the numbers with this, you get 1 times 0.7plus 0.9 times 0.3.That happens to be 0.97.So if we now plug this all back into this equation over here,we get 0.97 times 0.01 over 0.5245.This gives us approximately as the correct answer 0.0185.[Thrun] And if you got this right, I will be deeply impressedabout the fact you got this right.But the interesting thing now to observe is if we happen to know it's sunnyand I'm happy, then the probability of a raise is 14%, 0.014.If I don't know about the weather and I'm happy,then the probability of a raise goes up to about 18.5%.Why is that? Well, it's the explaining away effect.My happiness is well explained by the fact that it's sunny.So if someone observes me to be happy and asks the question,"Is this because Sebastian got a raise at work?"well, if you know it's sunny and this is a fairly good explanation for me being happy,you don't have to assume I got a raise.If you don't know about the weather, then obviously the chances are higherthat the raise caused my happiness,and therefore this number goes up from 0.014 to 0.018.Let me ask you one final question in this next quiz,which is the probability of the raise given that I look happy and it's not sunny.This is the most extreme case for making a raise likelybecause I am a happy guy, and it's definitely not caused by the weather.So it could be just random, or it could be caused by the raise.So please calculate this number for me and enter it into this box.[Thrun] Well, the answer follows the exact same scheme as before,with S being replaced by not S.So this should be an easier question for you to answer.P of R given H and not S can be inverted by Bayes' rule to be as follows.Once we apply Bayes' rule, as indicated over here where we swapped H to the left sideand R to the right side, you can observe that this value over herecan be readily found in the table.It's actually the 0.9 over there.This value over here, the raise is independent of the weatherby virtue of our Bayes network, so it's just 0.01.And as before, we apply total probability to the expression over here,and we obtain off this quotient over here that these 2 expressions are the same.P of H given not S, not R is the value over here,and the 0.99 is the complement of probability of R taken from over here,and that ends up to be 0.0833.This would have been the right answer.[Thrun] It's really interesting to compare this to the situation over here.In both cases I'm happy, as shown over here,and I ask the same question, which is whether I got a raise at work, as R over here.But in one case I observe that the weather is sunny; in the other one it isn't.And look what it does to my probability of having received a raise.The sunniness perfectly well explains my happiness,and my probability of having received a raise ends up to be a mere 1.4%, or 0.014.However, if my wife observes it to be non-sunny, then it is much more likelythat the cause of my happiness is related to a raise at work,and now the probability is 8.3%, which is significantly higher than the 1.4% before.This is a Bayes network of which S and R are independentbut H adds a dependence between S and R.Let me talk about this in a little bit more detail on the next paper.So here is our Bayes network again.In our previous exercises, we computed for this networkthat the probability of a raise of R given any of these variables shown here was as follows.The really interesting thing is that in the absence of information about H,which is the middle case over here,the probability of R is unaffected by knowledge of S--that is, R and S are independent.This is the same as probability of R,and R and S are independent.However, if I know something about the variable H,then S and R become dependent--that is, knowing about my happiness over here renders S and R dependent.This is not the same as probability of just R given H.Obviously, it isn't because if I now vary S from S to not S,it affects my probability for the variable R.That is a really unusual situationwhere we have R and S are independentbut given the variable H, R and S are not independent anymore.So knowledge of H makes 2 variables that previously were independent non-independent.Offered differently, 2 variables that are independent may not be in certain casesconditionally independent.Independence does not imply conditional independence.[Thrun] So we're now ready to define Bayes networks in a more general way.Bayes networks define probability distributions over graphs or random variables.Here is an example graph of 5 variables,and this Bayes network defines the distribution over those 5 random variables.Instead of enumerating all possibilities of combinations of these 5 random variables,the Bayes network is defined by probability distributions that are inherent to each individual node.For node A and B, we just have a distribution P of A and P of Bbecause A and B have no incoming arcs.C is a conditional distribution conditioned on A and B.D and E are conditioned on C.The joint probability represented by a Bayes networkis the product of various Bayes network probabilitiesthat are defined over individual nodeswhere each node's probability is only conditioned on the incoming arcs.So A has no incoming arc; therefore, we just want it P of A.C has 2 incoming arcs, so we define the probability of C conditioned on A and B.And D and E have 1 incoming arc that's shown over here.The definition of this joint distribution by using the following factorshas one really big advantage.Whereas the joint distribution over any 5 variables requires 2 to the 5 minus 1,which is 31 probability values,the Bayes network over here only requires 10 such values.P of A is one value, for which we can derive P of not A.Same for P of B.P of C given A B is derived by a distribution over Cconditioned on any combination of A and B, of which there are 4 of A and B as binary.P of D given C is 2 parameters for P of D given C and P of D given not C.And the same is true for P of E given C.So if you add those up, you get 10 parameters in total.So the compactness of the Bayes networkleads to a representation that scales significantly better to large networksthan the common natorial approach which goes through all combinations of variable values.That is a key advantage of Bayes networks, and that is the reason why Bayes networks are being used so extensivelyfor all kinds of problems.So here is a quiz.How many probability values are required to specify this Bayes network?Please put your answer in the following box.[Thrun] And the answer is 13.One over here, 2 over here, and 4 over here.Simplifiably speaking, any variable that has K inputs requires 2 to the K such variables.So in total we have 1, 9, 13.[Thrun] Here's another quiz.How many parameters do we need to specify the joint distribution for this Bayes network over herewhere A, B, and C point into D, D points into E, F, and Gand C also points into G?Please write your answer into this box.[Thrun] And the answer is 19.So 1 here, 1 here, 1 here, 2 here, 2 here, 2 arcs point into G, which makes for 4,and 3 arcs point into D. Two to the 3 is 8.So we get 1, 2, 3, 8, 2, 2, 4. If you add those up, it's 19.[Thrun] And here is our car network which we discussed at the very beginning of this unit.How many parameters do we need to specify this network?Remember, there are 16 total variables,and the naive joint over the 16 will be 2 to the 16th minus 1, which is 65,535.Please write your answer into this box over here.[Thrun] To answer this question, let us add up these numbers.Battery age is 1, 1, 1.This has 1 incoming arc, so it's 2.Two incoming arcs makes 4.One incoming arc is 2, 2 equals 4.Four incoming arcs makes 16.If we add all the right numbers, we get 47.[Thrun] So it takes 47 numerical probabilities to specify the jointcompared to 65,000 if you didn't have the graph-like structure.I think this example really illustrates the advantage of compact Bayes network representations over unstructured joint representations.[Thrun] The next concept I'd like to teach you is called D-separation.And let me start the discussion of this concept by a quiz.We have here a Bayes network,and I'm going to ask you a conditional independence question.Is C independent of A?Please tell me yes or no.Is C independent of A given B?Is C independent of D?Is C independent of D given A?And is E independent of C given D?[Thrun] So C is not independent of A.In fact, A influences C by virtue of B.But if you know B, then A becomes independent of C,which means the only determinate into C is B.If you know B for sure, then knowledge of A won't really tell you anything about C.C is also not independent of D, just the same way C is not independent of A.If I learn something about D, I can infer more about C.But if I do know A, then it's hard to imagine how knowledge of D would help me with Cbecause I can't learn anything more about A than knowing A already.Therefore, given A, C and D are independent.The same is true for E and C.If we know D, then E and C become independent.[Thrun] In this specific example, the rule that we could apply is very, very simple.Any 2 variables are independent if they're not linked by just unknown variables.So for example, if we know B, then everything downstream of Bbecomes independent of anything upstream of B.E is now independent of C, conditioned on B.However, knowledge of B does not render A and E independent.In this graph over here, A and B connect to C and C connects to D and to E.So let me ask you, is A independent of E,A independent of E given B,A independent of E given C,A independent of B,and A independent of B given C?[Thrun] And the answer for this one is really interesting.A is clearly not independent of E because through C we can see an influence of A to E.Given B, that doesn't change.A still influences C, despite the fact we know B.However, if we know C, the influence is cut off.There is no way A can influence E if we know C.A is clearly independent of B.They are different entry variables. They have no incoming arcs.But here is the caveat.Given C, A and B become dependent.So whereas initially A and B were independent,if you give C, they become dependent.And the reason why they become dependent we've studied before.This is the explain away effect.If you know, for example, C to be true, then knowledge of A will substantially affect what we believe about B.If there's 2 joint causes for C and we happen to know A is true,we will discredit cause B.If we happen to know A is false, we will increase our belief for the cause B.That was an effect we studied extensively in the happiness example I gave you before.The interesting thing here is we are facing a situationwhere knowledge of variable C renders previously independent variables dependent.[Thrun] This leads me to the general study of conditional independence in Bayes networks,often called D-separation or reachability.D-separation is best studied by so-called active triplets and inactive tripletswhere active triplets render variables dependentand inactive triplets render them independent.Any chain of 3 variables like this makes the initial and final variable dependentif all variables are unknown.However, if the center variable is known--that is, it's behind the conditioning bar--then this variable and this variable become independent.So if we have a structure like this and it's quote-unquote cut off by a known variable in the middle, that separates or deseparates the left variable from the right variable, and they become independent.Similarly, any structure like this renders the left variable and the right variable dependentunless the center variable is known,in which case the left and right variable become independent.Another active triplet now requires knowledge of a variable.This is the explain away case.If this variable is known for a Bayes network that converges into a single variable,then this variable and this variable over here become dependent.Contrast this with a case where all variables are unknown.A situation like this means that this variable on the left or on the right are actually independent.In a single final example, we also get dependence if we have the following situation:a direct successor of a conversion variable is known.So it is sufficient if a successor of this variable is known.The variable itself does not have to be known,and the reason is if you know this guy over here,we get knowledge about this guy over here.And by virtue of that, the case over here essentially applies.If you look at those rules, those rules allow you to determine for any Bayes networkwhether variables are dependent or not dependent given the evidence you have.If you color the nodes dark for which you do have evidence,then you can use these rules to understand whether any 2 variables are conditionally independent or not.So let me ask you for this relatively complicated Bayes network the following questions.Is F independent of A?Is F independent of A given D?Is F independent of A given G?And is F independent of A given H?Please mark your answers as you see fit.[Thrun] And the answer is yes, F is independent of A.What we find for our rules of D-separation is that F is dependent on Dand A is dependent on D.But if you don't know D, you can't govern any dependence between A and F at all.If you do know D, then F and A become dependent.And the reason is B and E are dependent given D,and we can transform this back into dependence of A and Fbecause B and A are dependent and E and F are dependent.There is an active path between A and F which goes across here and herebecause D is known.If we know G, the same thing is true because G gives us knowledge about D,and D can be applied back to this path over here.However, if you know H, that's not the case.So H might tell us something about G,but it doesn't tell us anything about D,and therefore, we have no reason to close the path between A and F.The path between A and F is still passive, even though we have knowledge of H.[Thrun] So congratulations. You learned a lot about Bayes networks.You learned about the graph structure of Bayes networks,you understood how this is a compact representation,you learned about conditional independence,and we talked a little bit about application of Bayes networkto interesting reasoning problems.But by all means this was a mostly theoretical unit of this class,and in future classes we will talk more about applications.The instrument of Bayes networks is really essential to a number of problems.It really characterizes the sparse dependence that exists in many readable problemslike in robotics and computer vision and filtering and diagnostics and so on.I really hope you enjoyed this class,and I really hope you understood in depth how Bayes networks work.[Probabilistic Interference][Male] Welcome back. In the previous unit, we went over the basicsof probability theory and saw howa Bayes network could concisely represent a joint probability distribution,including the representation of independence between the variables.In this unit, we will see how to do probabilistic inference.That is, how to answer probability questions using Bayes nets.Let's put up a simple Bayes net.We'll use the familiar example of the earthquakewhere we can have a burglary or an earthquake setting off an alarm, and if the alarm goes off, either John or Mary might call.Now, what kinds of questions can we ask to do inference about?The simplest type of question is the same question we askwith an ordinary subroutine or function in a programming language.Namely, given some inputs, what are the outputs?So, in this case, we could say given the inputs of B and E, what are the outputs, J and M?Rather than call them input and output variables,in probabilistic inference, we'll call them evidence and query variables.That is, the variables that we know the values of are the evidence,and the ones that we want to find out the values of are the query variables.Anything that is neither evidence nor query is known as a hidden variable.That is, we won't tell you what its value is.We won't figure out what its value is and report it,but we'll have to compute with it internally.And now furthermore, in probabilistic inference, the output is not a single number for each of the query variables,but rather, it's a probability distribution.So, the answer is going to be a complete, joint probability distribution over the query variables.We call this the posterior distribution, given the evidence, and we can write it like this.It's the probability distribution of one or more query variablesgiven the values of the evidence variables.And there can be zero or more evidence variables, and each of them are given an exact value.And that's the computation we want to come up with.There's another question we can ask.Which is the most likely explanation?That is, out of all the possible values for all the query variables,which combination of values has the highest probability?We write the formula like this, asking which Q values are maxable given the evidence values.Now, in an ordinary programming language, each function goes only one way.It has input variables, does some computation,and comes up with a result variable or result variables.One great thing about Bayes nets is that we're not restricted to going only in one direction.We could go in the causal direction, giving as evidencethe route nodes of the tree and asking as query values the nodes at the bottom.Or, we could reverse that causal flow. For example, we could have J and M be the evidence variablesand B and E be the query variables,or we could have any other combination.For example, we could have M be the evidence variableand J and B be the query variables.Here's a question for you.Imagine the situation where Mary has called to report that the alarm is going off,and we want to know whether or not there has been a burglary.For each of the nodes, click on the circle to tell usif the node is an evidence node, a hidden node, or a query node. The answer is that Mary calling is the evidence node.The burglary is the query node,and all the others are hidden variables in this case.Now we're going to talk about how to do inference on Bayes net.We'll start with our familiar network, and we'll talk about a methodcalled enumeration,which goes through all the possibilities, adds them up,and comes up with an answer.So, what we do is start by stating the problem.We're going to ask the question of what is the probability that the burglar alarm occurred given that John called and Mary called?We'll use the definition of conditional probability to answer this.So, this query is equal to the joint probability distributionof all 3 variables divided by the conditionalized variables. Now, note I'm using a notation here where instead of writing out the probabilityof some variable equals true, I'm just using the notation plus and then the variable name in lower case, and if I wanted the negation, I would use negation sign.Notice there's a different notation where instead of writing out the plus and negation signs, we just use the variable name itself, P(e),to indicate E is true.That notation works well, but it can get confusing between does P(e) mean E is true, or does it mean E is a variable?And so we're going to stick to the notation where we explicitly have the pluses and negation signs.To do inference by enumeration, we first take a conditional probabilityand rewrite it as unconditional probabilities.Now we enumerate all the atomic probabilities and calculate the sum of products.Let's look at just the complex term on the numerator first.The procedure for figuring out the denominator would be similar, and we'll skip that.So, the probability of these 3 terms togethercan be determined by enumerating all possible values of the hidden variables.In this case, there are 2, E and A, so we'll sum over those variables for all values of E and for all values of A.In this case, they're boolean, so there's only 2 values of each.We ask what's the probability of this unconditional term?And that we get by summing out over all possibilities, E and A being true or false.Now, to get the values of these atomic events, we'll have to rewrite this equation in a form that correspondsto the conditional probability tables that we have associated with the Bayes net.So, we'll take this whole expression and rewrite it.It's still a sum over the hidden variables E and A, but now I'll rewrite this expression in terms of the parentsof each of the nodes in the network.So, that gives us the product of these 5 terms,which we then have to sum over all values of E and A.If we call this product f(e,a), then the whole answer is the sum of F for all values of E and A,so as the sum of 4 terms where each of the terms is a product of 5 numbers.Where do we get the numbers to fill in this equation?From the conditional probability tables from our model, so let's put the equation back up, and we'll ask you for the casewhere both E and A are positive to look up in the conditional probability tables and fill in the numbersfor each of these 5 terms, and then multiply them together and fill in the product.We get the answer by reading numbers off the conditional probability tables,so probability of B being positive is 0.001.Of E being positive, because we're dealing with the positive case nowfor the variable E, is 0.002.The probability of A being positive, because we're dealing with that case, given that B is positive and the case for an E is positive, that we can read off here as 0.95.The probability that J is positive given that A is positive is 0.9.And finally, the probability that M is positive given that A is positivewe read off here as 0.7.We multiple all those together, it's going to be a small numberbecause we've got the .001 and the .002 here.Can't quite fit it in the box, but it works out to .000001197.That seems like a really small number, but remember, we have to normalize by the P(+j,+m) term, and this is only 1 of the 4 possibilities.We have to enumerate over all 4 possibilities for E and A,and in the end, it works out that the probability of the burglar alarm being truegiven that John and Mary calls, is 0.284.And we get that number because intuitively, it seems that the alarm is fairly reliable.John and Mary calling are very reliable,but the prior probability of burglary is low.And those 2 terms combine together to give us the 0.284 valuewhen we sum up each of the 4 terms of these products.[Norvig] We've seen how to do enumeration to solve the inference problemon belief networks.For a simple network like the alarm network, that's all we need to know.There's only 5 variables, so even if all 5 of them were hidden,there would only be 32 rows in the table to sum up.From a theoretical point of view, we're done.But from a practical point of view, other networks could give us trouble.Consider this network, which is one for determining insurance for car owners.There are 27 different variables.If each of the variables were boolean, that would give us over 100 million rows to sum out.But in fact, some of the variables are non-boolean,they have multiple values, and it turns out that representing this entire networkand doing enumeration we'd have to sum over a quadrillion rows.That's just not practical, so we're going to have to come up with methodsthat are faster than enumerating everything.The first technique we can use to get a speed-up in doing inference on Bayes netsis to pull out terms from the enumeration.For example, here the probability of b is going to be the same for all values of E and a.So we can take that term and move it out of the summation,and now we have a little bit less work to do.We can multiply by that term once rather than having it in each row of the table.We can also move this term, the P of e, to the left of the summation over a,because it doesn't depend on a.By doing this, we're doing less work.The inner loop of the summation now has only 3 terms rather than 5 terms.So we've reduced the cost of doing each row of the table.But we still have the same number of rows in the table,so we're going to have to do better than that.The next technique for efficient inference is to maximize independence of variables.The structure of a Bayes net determines how efficient it is to do inference on it.For example, a network that's a linear string of variables,X1 through Xn, can have inference done in time proportional to the number n,whereas a network that's a complete networkwhere every node points to every other node and so on could take time 2 to the nif all n variables are boolean variables.In the alarm network we saw previously, we took careto make sure that we had all the independence relations represented in the structure of the network.But if we put the nodes together in a different order,we would end up with a different structure.Let's start by ordering the node John calls firstand then adding in the node Mary calls.The question is, given just these 2 nodes and looking at the node for Mary calls,is that node dependent or independent of the node for John calls?[Norvig] The answer is that the node for Mary calls in this networkis dependent on John calls.In the previous network, they were independent given that we knew that the alarm had occurred.But here we don't know that the alarm had occurred,and so the nodes are dependentbecause having information about one will affect the information about the other.[Norvig] Now we'll continue and we'll add the node A for alarm to the network.And what I want you to do is click on all the other variablesthat A is dependent on in this network.[Norvig] The answer is that alarm is dependent on both John and Mary.And so we can draw both nodes in, both arrows in.Intuitively that makes sense because if John calls, then it's more likely that the alarm has occurred,likely as if Mary calls, and if both called, it's really likely.So you can figure out the answer by intuitive reasoning,or you can figure it out by going to the conditional probability tablesand seeing according to the definition of conditional probabilitywhether the numbers work out.[Norvig] Now we'll continue and we'll add the node B for burglaryand ask again, click on all the variables that B is dependent on.[Norvig] The answer is that B is dependent only on A.In other words, B is independent of J and M given A.[Norvig] And finally, we'll add the last node, E,and ask you to click on all the nodes that E is dependent on.[Norvig] And the answer is that E is dependent on A.That much is fairly obvious.But it's also dependent on B. Now, why is that?E is dependent on A because if the earthquake did occur,then it's more likely that the alarm would go off.On the other hand, E is also dependent on Bbecause if a burglary occurred, then that would explain why the alarm is going off,and it would mean that the earthquake is less likely.[Norvig] The moral is that Bayes nets tend to be the most compactand thus the easier to do inference on when they're written in the causal direction--that is, when the networks flow from causes to effects.Let's return to this equation, which we use to show how to do inference by enumeration.In this equation, we join up the whole joint distributionbefore we sum out over the hidden variables.That's slow, because we end up repeating a lot of work.Now we're going to show a new technique called variable elimination,which in many networks operates much faster.It's still a difficult computation, an NP-hard computation, to do inference over Bayes nets in general.Variable elimination works faster than inference by enumeration in most practical cases.It requires an algebra for manipulating factors,which are just names for multidimensional arrays that come out of these probabilistic terms.We'll use another example to show how variable elimination works.We'll start off with a network that has 3 boolean variables.R indicates whether or not it's raining.T indicates whether or not there's traffic, and T is dependent on whether it's raining.And finally, L indicates whether or not I'll be late for my next appointment,and that depends on whether or not there's traffic.Now we'll put up the conditional probability tables for each of these 3 variables.And then we can use inference to figure out the answer to questions likeam I going to be late?And we know by definition that we could do that through enumeration by going through all the possible values for R and T and summing up the product of these 3 nodes. Now, in a simple network like this, straight enumeration would work fine,but in a more complex network, what variable elimination does is give us a wayto combine together parts of the network into smaller partsand then enumerate over those smaller parts and then continue combining.So, we start with a big network.We eliminate some of the variables.We compute by marginalizing out, and then we have a smaller network to deal with,and we'll show you how those 2 steps work.The first operation in variable elimination is called joining factors.A factor, again, is one of these tables.It's a multidimensional matrix, and what we do is choose 2 of the factors, 2 or more of the factors.In this case, we'll choose these 2, and we'll combine them togetherto form a new factor which represents the joint probability of all the variables in that factor.In this case, R and T.Now we'll draw out that table.In each case, we just look up in the corresponding table, figure out the numbers, and multiply them together. For example, in this row we have a +r and a +t,so the +r is 0.1, and the entry for +r and +t is 0.8,so multiply them together and you get 0.08.Go all the way down. For example, in the last row we have a -r and a -t.-r is 0.9. The entry for -r and -t is also 0.9.Multiply those together and you get 0.81.So, what have we done?We used the operation of joining factors on these 2 factors, getting us a new factor which is part of the existing network.Now we want to apply a second operation called elimination,also called summing out or marginalization, to take this table and reduce it. Right now, the tables we have look like this.We could sum out or marginalize over the variable Rto give us a table that just operates on T.So, the question is to fill in this table for P(T)--there will be 2 entries in this table, the +t entry, formed by summing out all the entries here for all values of r for which t is positive, and the -t entry, formed the same way, by looking in this tableand summing up all the rows over all values of r where t is negative.Put your answers in these boxes.The answer is that for +t we look up the 2 possible values for r,and we get 0.08 or 0.09.Sum those up, get 0.17,and then we look at the 2 possible values of R for -t,and we get 0.02 and 0.81.Add those up, and we get 0.83.So, we took our network with RT and L. We summed out over R.That gives us a new network with T and Lwith these conditional probability tables.And now we want to do a join over T and Land give us a new table with the joint probability of P(T, L).And that table is going to look like this.The answer, again, for joining variables is determined by pointwise multiplication,so we have 0.17 times 0.3 is 0.051,+t and +l, 0.17 times 0.7 is 0.119.Then we go to the minuses.Minus 0.83 times 0.1 is 0.083.And finally, 0.83 times 0.9 is 0.747.Now we're down to a network with a single node, T, L, with this joint probability table, and the only operation we have left to dois to sum out to give us a node with just L in it.So, the question is to compute P(L) for both values of L,+l and -l.The answer is that the +l values, 0.051 plus 0.083 equals 0.134.And the negative values, 0.119 plus 0.747 equals 0.886.No subtitles...So, that's how variable elimination works.It's a continued process of joining together factors to form a larger factor and then eliminating variables by summing out.If we make a good choice of the order in which we apply these operations,then variable elimination can be much more efficient than just doing the whole enumeration.Now I want to talk about approximate inferenceby means of sampling.What do I mean by that?Say we want to deal with a joint probability distribution,say the distribution of heads and tails over these 2 coins.We can build a table and then start counting by sampling.Here we have our first sample. We flip the coins and the one-cent piece came up heads,and the five-cent piece came up tails,so we would mark down one count.Then we'd toss them again.This time the five cents is heads, and the one cent is tails,so we put down a count there, and we'd repeat that processand keep repeating it until we got enough counts that we could estimatethe joint probability distribution by looking at the counts.Now, if we do a small number of samples, the counts might not be very accurate.There may be some random variation that causes them not to converge to their true values, but as we add more counts,the counts--as we add more samples, the counts we get will come closer to the true distribution.Thus, sampling has an advantage over inference in that we know a procedurefor coming up with at least an approximate value for the joint probability distribution,as opposed to exact inference, where the computation may be very complex.There's another advantage to sampling, which is if we don't know what the conditional probability tables are, as we did in our other models,if we don't know these numeric values, but we can simulate the process,we can still proceed with sampling, whereas we couldn't with exact inference.Here's a new network that we'll use to investigatehow sampling can be used to do inference.In this network, we have 4 variables. They're all boolean.Cloudy tells us if it's cloudy or not outside, and that can have an effect on whether the sprinklers are turned on, and whether it's raining.And those 2 variables in turn have an effect on whether the grass gets wet. Now, to do inference over this network using sampling, we start off with a variable where all the parents are defined.In this case, there's only one such variable, Cloudy.And it's conditional probability table tells us that the probability is 50% for Cloudy,50% for not Cloudy, and so we sample from that.We generate a random number, and let's say it comes up with positive for Cloudy.Now that variable is defined, we can choose another variable.In this case, let's choose Sprinkler, and we look at the rows in the tablefor which Cloudy, the parent, is positive, and we see we should samplewith probability 10% to +s and 90% a -s.And so let's say we do that sampling with a random number generator,and it comes up negative for Sprinkler.Now let's jump over here. Look at the Rain variable.Again, the parent, Cloudy, is positive,so we're looking at this part of the table.We get a 0.8 probability for Rain being positive,and a 0.2 probability for Rain being negative.Let's say we sample that randomly, and it comes up Rain is positive.And now we're ready to sample the final variable,and what I want you to do is tell me which of the rowsof this table should we be considering and tell me what's more likely.Is it more likely that we have a +w or a -w?The answer to the question is that we look at the parents.We find that the Sprinkler variable is negative,so we're looking at this part of the table.And the Rain variable is positive, so we're looking at this part.So, it would be these 2 rows that we would consider,and thus, we'd find there's a 0.9 probability for w, the grass being wet, and only 0.1 for it being negative,so the positive is more likely.And once we've done that, then we generated a complete sample,and we can write down the sample here.We had +c, -s, +r.And assuming we got a probability of 0.9 came out in favor of the +w, that would be the end of the sample.Then we could throw all this information out and start over againby having another 50/50 choice for cloudy and then working our way through the network.Now, the probability of sampling a particular variable, choosing a +w or a -w, depends on the values of the parents.But those are chosen according to the conditional probability tables,so in the limit, the count of each sampled variablewill approach the true probability.That is, with an infinite number of samples, this procedure computes the truejoint probability distribution.We say that the sampling method is consistent.We can use this kind of sampling to compute the complete joint probability distribution,or we can use it to compute a value for an individual variable.But what if we wanted to compute a conditional probability?Say we wanted to compute the probability of wet grassgiven that it's not cloudy.To do that, the sample that we generated here wouldn't be helpful at allbecause it has to do with being cloudy, not with being not cloudy.So, we would cross this sample off the list.We would say that we reject the sample, and this technique is called rejection sampling.We go through ignoring any samples that don't match the conditional probabilities that we're interested inand keeping samples that do, say the sample -c, +s, +r, -w.We would just continue going through generating samples, crossing off the ones that don't match, keeping the ones that do.And this procedure would also be consistent.We call this procedure rejection sampling.But there's a problem with rejection sampling.If the evidence is unlikely, you end up rejecting a lot of the samples.Let's go back to the alarm network where we had variables for burglary and for an alarmand say when arrested, in computing the probability of a burglary,given that the alarm goes off.The problem is that burglaries are very infrequent,so most of the samples we would get would end up being--we start with generating a B, and we get a -b and then a -a.We go back and say does this match?No, we have to reject this sample,so we generate another sample, and we get another -b, -a.We reject that. We get another -b, -a.And we keep rejecting, and eventually we get a +b,but we'd end up spending a lot of time rejecting samples.So, we're going to introduce a new method called likelihood weightingthat generates samples so that we can keep every one. With likelihood weighting, we fix the evidence variables.That is, we say that A will always be positive,and then we sample the rest of the variables,so then we get samples that we want.We would get a list like -b, +a, -b, +a,+b, +a.We get to keep every sample, but we have a problem.The resulting set of samples is inconsistent.We can fix that, however, by assigning a probability to each sample and weighing them correctly.In likelihood weighting, we're going to be collecting samples just like before,but we're going to add a probabilistic weight to each sample.Now, let's say we want to compute the probability of raingiven that the sprinklers are on, and the grass is wet.We start as before.We make a choice for Cloudy, and let's say that, again, we choose Cloudy being positive.Now we want to make a choice for Sprinkler, but we're constrained to always choose Sprinkler being positive, so we'll make that choice.And we know we were dealing with Cloudy being positive,so we're in this row, and we're forced to make the choice of Sprinkler being positive,and that has a probability of only 0.1, so we'll put that 0.1 into the weight.Next, we'll look at the Rain variable,and here we're not constrained in any way, so we make a choiceaccording to the probability tables with Cloudy being positive.And let's say that we choose the more popular choice, and Rain gets the positive value.Now, we look at Wet Grass.We're constrained to choose positive, and we know that the parentsare also positive, so we're dealing with this row here.Since it's a constrained choice, we're going to add in or multiply in an additional weight,and I want you to tell me what that weight should be.The answer is we're looking for the probability of having a +w given a +s and a +r, so that's in this row, so it's 0.99.So, we take our old weight and multiply it by 0.99, gives us a final weight of 0.099 for a sample of +c, +s, +r and +w.When we include the weights, counting this sample that was forced to have a +s and a +wwith a weight of 0.099, instead of counting it as a full one sample,we find that likelihood weighting is also consistent.Likelihood weighting is a great technique,but it doesn't solve all our problems. Suppose we wanted to compute the probability of C given +s and +r.In other words, we're constraining Sprinkler and Rain to always be positive.Since we use the evidence when we generate a node that has that evidence as parents,the Wet Grass node will always get good values based on that evidence.But the Cloudy node won't, and so it will be generating values at randomwithout looking at these values, and most of the time, or some of the time,it will be generating values that don't go well with the evidence.Now, we won't have to reject them like we do in rejection sampling, but they'll have a low probability associated with them.A technique called Gibbs sampling,named after the physicist Josiah Gibbs, takes all the evidence into account and not just the upstream evidence.It uses a method called Markov Chain Monte Carlo, or MCMC.The idea is that we resample just one variable at a timeconditioned on all the others.That is, we have a set of variables, and we initialize them to random variables, keeping the evidence values fixed.Maybe we have values like this, and that constitutes one sample, and now, at each iteration through the loop,we select just one non-evidence variable and resample itbased on all the other variables. And that will give us another sample, and repeat that again.Choose another variable.Resample that variable and repeat.We end up walking around in this space of assignments of variables randomly.Now, in rejection and likelihood sampling, each sample was independent of the other samples.In MCMC, that's not true.The samples are dependent on each other, and in fact, adjacent samples are very similar.They only vary or differ in one place.However, the technique is still consistent. We won't show the proof for that.Now, just one more thing.I can't help but describe what is probably the most famous probability problem of all.It's called the Monty Hall Problem after the game show host.And the idea is that you're on a game show, and there's 3 doors:door #1, door #2, and door #3.And behind each door is a prize, and you know that one of the doorscontains an expensive sports car, which you would find desirable,and the other 2 doors contain a goat, which you would find less desirable.Now, say you're given a choice, and let's say you choose door #1.But according to the conventions of the game, the host, Monty Hall,will now open one of the doors, knowing that the door that he openscontains a goat, and he shows you door #3.And he now gives you the opportunity to stick with your choiceor to switch to the other door.What I want you to tell me is, what is your probability of winningif you stick to door #1, and what is the probability of winningif you switched to door #2?The answer is that you have a 1/3 chance of winning if you stick with door #1and a 2/3 chance if you switch to door #2.How do we explain that, and why isn't it 50/50?Well, it's true that there's 2 possibilities, but we've learned from probability that just because there are 2 optionsdoesn't mean that both options are equally likely.It's easier to explain why the first door has a 1/3 probabilitybecause when you started, the car could be in any one of 3 places. You chose one of them. That probability was 1/3.And that probability hasn't been changed by the revealing of one of the other doors.Why is door #2 two-thirds?Well, one way to explain it is that the probability has to sum to 1, and if 1/3 is here, the 2/3 has to be here.But why doesn't the same argument that you use for 1 hold for 2?Why can't we say the probability of 2 holding the car was 1/3 before this door was revealed?Why has that changed 2 and has not changed 1?And the reason is because we've learned something about door #2.We've learned that it wasn't the door that was flipped over by the host,and so that additional information has updated the probability,whereas we haven't learned anything additional about door #1because it was never an option that the host might switch door #1.And in fact, in this case, if we reveal the door, we find that's where the car actually is.So you see, learning probability may end up winning you something.Now, as a final epilogue, I have here a copy of a letter written by Monty Hall himselfin 1990 to Professor Lawrence Denenberg of Harvardwho, with Harry Lewis, wrote a statistics book in which they used the Monty Hall Problem as an example,and they wrote to Monty asking him for permission to use his name.Monty kindly granted the permission, but in his letter,he writes, "As I see it, it wouldn't make any difference after the playerhas selected Door A, and having been shown Door C--why should he then attempt to switch to Door B?So, we see Monty Hall himself did not understand the Monty Hall Problem.[Thrun] Given the following Bayes network with P of A equal to 0.5,P of B given the A equals 0.2,and P of B given not A 0.8,calculate the following probability. [Thrun] Consider a network of the following type:a variable, A, that is binary connects to three variables, X1, X2, and X3,that are also binary.The probability of A is 0.5, and for all variable XI we have the probability of XI given A is 0.2,and the probability of XI given not A equals 0.6.I would like to know from you the probability of A given that we observed X1, X2, and not X3.Notice that these variables over here are conditionally independent given A.[Thrun] Let us consider the same network again.I would like to know the probability of X3 given that I observed X1.[Thrun] In this next homework assignment I will be drawing you a Bayes networkand will ask you some conditional independence questions.Is B conditionally independent of C? And say yes or no.Is B conditionally independent of C given D? And say yes or no.Is B conditionally independent of C given A? And say yes or no.And is B conditionally independent given A and D? And say yes or no.[Thrun] Consider the following network.I would like to know whether the following statements are true or false.C is conditionally independent of E given A.B is conditionally independent of D given C and E.A is conditionally independent of C given E.And A is conditionally independent of C given B.Please check yes or no for each of these questions.[Thrun] In my final question I'll look at the exact same network as before,but I would like to know the minimum number of numerical parameterssuch as the values to define probabilities and conditional probabilitiesthat are necessary to specify the joint distribution of all 5 variables.[Thrun] The answer is 0.2,and this follows directly from Bayes' rule.In this formula, we can read off the first 2 values straight from the table over here,and we expand the denominator by total probability.Observing that this is exactly the same expression as up here,we get 0.1 divided by 0.1 plus this expression over here can be copied from over here,and P of not A is directly obtained up here.Hence we get 0.5 over here, and as a result we get 0.2.[Thrun] For this question we will be exploring a little trickabout non-normalized probability.We will observe that P of A given X1, X2 and not X3,the expression on the left can be resolved by Bayes' rule into this expression over here.We will take X3 to the left and replace it by A,both conditioned on the variables X1 and X2.Then we have PA given X1, X2 divided by P not X3, X1, X2.Next we employ 2 things.One is the denominator does not depend on A,so whether I put an A or not A has no bearing on any calculation here,which means I can defer its calculation until later, and it will turn out to be important.So I'm going to be proportional to just the stuff over here.And second, I export my conditional independencewhereby I can omit X1 and X2 from the probability of not X3 conditioned on A.These variables are conditionally independent.This gives me the following recursionwhere I now removed the third variable from the estimation problemand just retained the first 2 relative to my initial expression.If I keep expanding this, I get the following solution.P of not X3 given A, P X2 given A, P X1 given A times P of A.You might take a minute to just verify this,but this is exploiting the conditional independence very much as in the first step I showed you over here.This step lacks the normalizer,so let me work on the normalizer by expressing the opposite probability, P of not A given the same events, X1, X2, and not X3,which resolves to P of not X3 given not A,P of X2 given not A, P of X1 given not A,and P of not A.I can now plug in the values from above.So the first term gives me 0.8 times 0.2 times 0.2 times 0.5.In the second term I get 0.4 times 0.6 times 0.6 times 0.5,which resolves to 0.016 and 0.072.This is clearly not a probability because we left out the normalizer.But as we know, the normalizer does not depend on whether I put A or not A in here.As a result, it will be the same for both of these expressions,and I can obtain it by just adding these non-normalized probabilitiesand then subsequently divide these non-normalized probabilities accordingly.So let me just do this.We get for the desired probability over here 0.1818and for the inverse probability over here 0.8182.Our desired answer therefore is 0.1818.This was not an easy question.[Thrun] The answer is a little bit involved.We use total probability to re-express this by bringing in A.P of X3 given X1 is the sum of P of X3 given X1 and Atimes P of A given X1 plus the A complement, which is X3, conditional X1 and not Atimes P of not A given X1.That is just total probability.Next we utilized conditional independence by which we can simplify this expressionto drop X1 in the conditional variablesand we transform this expression by Bayes' rule again.The same applies to the right side with not A replacing A.All of those expressions over here can be foundeither in the table up there or just by their complements,with the exception of P of X1.But P of X1 can again be just obtained by total probability,which resolves to 0.2 times 0.5 plus 0.6 times 0.5, which gives me 0.4.We are now in a position to calculate the last term over here, which goes as follows.This expression is 0.2 times 0.2 times 0.5 over 0.4 plus 0.6 times 0.6 times 0.5 over 0.4,which gives us as a final result 0.5.[Thrun] And the answer is as follows.No, no, yes, and no.B and C in the absence of any other information are dependent through A,which is if you learn something about B, you can infer something about A,and then we'll know more about C.If you know D, that doesn't change a thing.You can just take D out of the pool.If you know A, B and C become conditionally independent.This dependence goes away, and ignorance of D doesn't render B and C dependent.However, if we add D back to the mix,then knowledge of D will render B and C dependent by way of the explaining away effect.[Thrun] So the correct answer is tricky in this case.It is no, no, no, and yes.The first one is straightforward.C and E are conditionally independent based on D,and knowledge of A doesn't change anything.B and D are conditionally independent through A,and knowledge of C or E doesn't change that.A and C is interesting.A and C is independent. But if you know D, they become dependent.It turns out if you know E, you can know something about D,and as a result, A and C become dependent through the explain away effect.That doesn't apply if you know B.Even though B tells you something about E, it tells you nothing about D because B and D are independent.Therefore, knowing B tells you nothing about D,and the explain away effect does not occur between A and C.The answer here is yes.[Thrun] The correct answer is 16.The probability of A and C require 1 parameter each.The complement of not A and not C follows by 1 minus that parameter.This guy over here requires 2 parameters.You need to know the probability of B given A and B given not A.The complements can be obtained easily.The probability of D is conditioned on 2 variables which can take 4 possible values.Hence the number is 4.And E is conditioned on 3 variables, so it can take a total of 8 different values,2 to the 3rd, which is 8.If you add 8 plus 4 plus 2 plus 1 plus 1, you get 16.Welcome to the machine learning unit.Machine learning is a fascinating area.The world has become immeasurably data-rich.The world wide web has come up over the last decade.The human genome is being sequenced.Vast chemical databases, pharmaceutical databases,and financial databases are now availableon a scale unthinkable even 5 years ago.To make sense out of the data,to extract information from the data, machine learning is the discipline to go.Machine learning is an important subfeed of artificial intelligence,it's my personal favorite next to roboticsbecause I believe it has a huge impact on societyand is absolutely necessary as we move forward.So in this class, I teach you some of the very basics ofmachine learning, and in our next unitPeter will tell you some more about machine learning.We'll talk about supervised learning, which is one side of machine learning,and Peter will tell you about unsupervised learning,which is a different style.Later in this class we will also encounter reinforcement learning,which is yet another set of machine learning.Anyhow, let's just dive in.Welcome to the first class on machine learning.So far we talked a lot about Bayes Networks.And the way we talked about themis all about reasoning within Bayes Networksthat are known.Machine learning addresses the problemof how to find those networksor other modelsbased on data.Learning models from datais a major, major area of artificial intelligenceand it's perhaps the onethat had the most commercial success.In many commercial applications the models themselves are fittedbased on data.For example, Googleuses data to understandhow to respond to each search query.Amazon uses datato understand how to place products on their website.And these machine learning techniquesare the enabling techniques that make that possible.So this classwhich is about supervised learningwill go through some very basic methodsfor learning models from datain particular, specific types of Bayes Networks.We will complement thiswith a class on unsupervised learningthat will be taught next after this class.Let me start off with a quiz.The quiz is: What companies are famousfor machine learning using data?Google for mining the web.Netflix for mining what peoplewould like to rent on DVDs.Which is DVD recommendations.Amazon.com for product placement.Check any or alland if none of those applycheck down here.And, not surprisingly, the answer isall of those companies and many, many, many moreuse massive machine learning for making decisionsthat are really essential to the businesses.Google mines the web and uses machine learning for translation,as we've seen in the introductory level. Netflix has usedmachine learning extensively for understanding what type of DVD to recommend to you next.Amazon composes its entire product pages using machine learning by understanding how customers respond to different compositions and placements of their products,and many, many other examples exist.I would argue that in Silicon Valley,at least half the companies dealing with customers and online productsdo extensively use machine learning,so it makes machine learning a really exciting discipline.In my own research, I've extensively used machine learning for robotics.What you see here is a robot my students and I built at Stanfordcalled Stanley, and it won the DARPA Grand Challenge.It's a self-driving car that drives without any human assistance whatsoever,and this vehicle extensively uses machine learning.The robot is equipped with a laser systemI will talk more about lasers in my robotics class,but here you can see how the robot is able to build3-D models of the terrain ahead.These are almost like video game models that allow it to makeassessments where to drive and where not to drive.Essentially, it's trying to drive on flat ground.The problem with these lasers is that they don't see very far.They see about 25 meters out, so to drive really fastthe robot has to see further.This is where machine learning comes into play.What you see here is camera images delivered by the robotsuperimposed with laser data that doesn't see very far,but the laser is good enough to extract samplesof driveable road surface that can then be machine learnedand extrapolated into the entire camera image.That enables the robot to use the camerato see driveable terrain all the way to the horizonup to like 200 meters out, enough to drive really, really fast.This ability to adapt its vision by driving its own training examples using lasersbut seeing out 200 meters or morewas a key factor in winning the race.Machine learning is a very large fieldwith many different methodsand many different applications.I will now define some of the very basic terminologythat is being used to distinguish different machine learning methods.Let's start with the what. What is being learned?You can learn parameterslike the probabilities of a Bayes Network.You can learn structurelike the arc structure of a Bayes Network.And you might even discover hidden concepts.For exampleyou might find that certain training exampleform a hidden group.For example Netflixyou might find that there's different types of customerssome that care about classic moviessome of them care about modern moviesand those might form hidden conceptswhose discovery can really help youmake better sense of the data.Next is what from?Every machine learning methodis driven by some sort of target informationthat you care about.In supervised learningwhich is the subject of today's classwe're given specific target labelsand I give you examples just in a second. We also talk about unsupervised learningwhere target labels are missingand we use replacement principlesto find, for examplehidden concepts. Later there will be a class in reinforcement learningwhen an agent learns from feedback with the physical environmentby interacting and trying actionsand receiving some sort of evaluation from the environmentlike "Well done" or "That works."Again, we will talk about those in detail later.There's different things you could try to dowith machine learning technique.You might care about prediction.For example you might want to care about what's going to happen with the futurein the stockmarket for example.You might care to diagnose somethingwhich is you get data and you wish to explain itand you use machine learning for that.Sometimes your objective is to summarize something.For example if you read a long articleyour machine learning method might aim toproduce a short article that summarizes the long article.And there's many, many, many more different things.You can talk about the how to learn.We use the word passiveif your learning agent is just an observerand has no impact on the data itself.Otherwise, you call it active.Sometimes learning occurs onlinewhich means while the data is being generatedand some of it is offlinewhich means learning occurs after the data has been generated.There's different types of outputs of a machine learning algorithm.Today we'll talk about classificationversus regression.In classification the output is binaryor a fixed number of classesfor example something is either a chair or not.Regression is continuous.The temperature might be 66.5 degreesin our prediction.And there's tons of internal detailswe will talk about.Just to name one.We will distinguish generativefrom discriminative.Generative seeks to model the data as generally as possibleversus discriminative methodsseek to distinguish dataand this might sound like a superficial distinctionbut it has enormous ramificationon the learning algorithm.Now to tell you the truthit took me many years to fully learn all these words hereand I don't expect you to pick them all upin one classbut you should as well know that they exist.And as they come upI'll emphasize themso you can resort any learning methodI tell you back into the specific taxonomy over here.The vast amount of work in the fieldfalls into the area of supervised learning.In supervised learningyou're given for each training examplea feature vector and a target label named Y.For example, for a credit rating agencyX1, X2, X3 might be a featuresuch as is the person employed?What is the salary of the person?Has the person previously defaulted on a credit card?And so on.And Y is a predictorwhether the person is to default on the credit or not.Now machine learningis to be carried out on past datawhere the credit rating agency might have collected features just like theseand actual occurances of default or not. What it wishes to produceis a function that allows usto predict future customers.So the new person comes inwith a different feature vector.Can we predict as good as possiblethe functional relationshipbetween these features X1 to Xn all the way to Y?You can apply the exact same examplein image recognitionwhere X might be pixels of images or it might be features of things found in imagesand Y might be a label that sayswhether a certain object is containedin an image or not. Now in supervised learningyou're given many such examples.X21 to X2nleads to Y2all way the index m.This is called your data.If we call each input vector Xmand we wish to find out the functiongiven any Xm or any future vector X produces as close as possible my target signal Y.Now this isn't always possibleand sometimes it's acceptablein fact preferableto tolerate a certain amount of errorin your training data.But the subject of machine learningis to identify this function over here.And once you identify ityou can use it for future Xsthat weren't part of the training setto produce a predictionthat hopefully is really, really good.So let me ask you a question.And this is a question for which I haven't given you the answerbut I'd like to appeal to your intuition.Here's one data setwhere the X is one dimensionally plotted horizontally and the Y is verticallyand suppose there looks like this.Suppose my machine learning algorithmgives me 2 hypotheses.One is this function over herewhich is a linear functionand one is this function over here.I'd like to know which of the functionsyou find preferableas an explanation for the data.Is it function A?Or function B?Check here for Ahere for Band here for neither.And I hope you guessed function A.Even though both perfectly describe the dataB is much more complex than A.In fact, outside the dataB seems to go to a minus infinity much fasterthan these data pointsand to plus infinity much fasterwith these data points over here.And in betweenwe have wide oscillations that don't correspond to any data.So I would argueA is preferable. The reason why I asked this questionis because of something called Occam's Razor.Occam can be spelled in many different ways. And what Occam says is that everything else being equalchose the less complex hypothesis.Now in practice there's actually a trade-offbetween a really good data fitand low complexity.Let me illustrate this to youby a hypothetical example.Consider the following graphwhere the horizontal axis graphs complexity of the solution.For example, if you use polynomialsthis might be a high-degree polynomial over here and maybe a linear function over herewhich is a low-degree polynomialyour training data errortends to go like this.The more complex the hypothesis you allowthe more you can just fit your data.However, in realityyour generalization error on unknown datatends to go like this.It is the sum of the training data errorand another functionwhich is called the overfitting error.Not surprisinglythe best complexity is obtainedwhere the generalization error is minimum.There are methodsto calculate the overfitting error.They go into a statistical fieldunder the name Bayes variance methods.However, in practiceyou're often just given the training data error.You find if you don't find the modelthat minimizes the training data errorbut instead pushes back the complexityyour algorithm tends to perform betterand that is something we will study a little bitin this class.However, this slide is really importantfor anybody doing machine learning in practice.If you deal with dataand you have ways to fit your databe aware that overfittingis a major source of poor performanceof a machine learning algorithm.And I give you examples in just one second.So a really important exampleof machine learning is SPAM detection.We all get way too much emailand a good number of those are SPAM.Here are 3 examples of email.Dear Sir: First I must solicit your confidencein this transaction, this is by virtue of its naturebeing utterly confidential and top secret...This is likely SPAM.Here's another one.In upper caps.99 MILLION EMAIL ADDRESSES FOR ONLY $99This is very likely SPAM.And here's another one.Oh, I know it's blatantly OTbut I'm beginning to go insane.Had an old Dell Dimension XPS sitting in the cornerand decided to put it to use.And so on and so on.Now this is likely not SPAM.How can a computer programdistinguish between SPAM and not SPAM?Let's use this as an exampleto talk about machine learning for discriminationusing Bayes Networks.In SPAM detectionwe get an emailand we wish to categorize iteither as SPAMin which case we don't even show as to the whereor what we call HAMwhich is the technical word foran email worth passing on to the person being emailed.So the function over here is the function we're trying to learn.Most SPAM filters use human input.When you go through emailyou have a button called IS SPAMwhich allows you as a user to flag SPAMand occasionally you will say an email is SPAM.If you look at thisyou have a typical supervised machine learning situationwhere the input is an emailand the output is whether you flag it as SPAMor if we don't flag itwe just think it's HAM.Now to make this amenable toa machine learning algorithmwe have to talk about how to represent emails.They're all using different words and different charactersand they might have different graphics included.Let's pick a representation that's easy to process.And this representation is often calledBag of Words.Bag of Words is a representationof a documentthat just counts the frequencyof words.If an email were to say HelloI will say Hello.The Bag of Words representationis the following.2-1-1-1for the dictionarythat contains the 4 wordsHello I will say.Now look at the subtlety here.Rather than representing each individual wordwe have a count of each wordand the count is obliviousto the order in which the words were stated.A Bag of Words representationrelative to a fixed dictionaryrepresents the counts of each wordrelative to the words in the dictionary.If you were to use a different dictionarylike hello and good-byeour counts would be 2 and 0.However, in most casesyou make sure that all the words foundin messagesare actually included in the dictionary.So the dictionary might be very, very large.Let me make up an unofficial exampleof a few SPAM and a few HAM messages.Offer is secret.Click secret link.Secret sports link.Obviously those are contrivedand I tried to retain the recoveryto a small number of wordsto make this example workable.In practice we need thousandsof such messagesto get good information.Play sports today.Went play sports.Secret sports event.Sport is today.Sport costs money.My first quiz isWhat is the size of the vocabularythat contains all words in these messages?Please enter the value in this box over here.Well let's count. Offer is secret click. Secret occurs over here alreadyso we don't have to count it twice.Link, sports, play, today, went, eventcosts money.So the answer is 12.There's 12 different words contained in these 8 messages. [Narrator] Another quiz. What is the probability that a random message that arrives to fall into the spam bucket? Assuming that those messages are all drawn at random. [writing on page][Narrator] And the answer is:there's 8 different messages of which 3 are spam.So the maximum likelihood estimateis 3/8.[writing on paper]So, let's look at this a little bit more formally and talk about maximum likelihood.Obviously, we're observing 8 messages: spam, spam, spam, and 5 times ham.And what we care about is what's our prior probability of spamthat maximizes the likelihood of this data?So, let's assume we're going to assign a value of pi to this, and we wish to find the pi that maximizes the likelihood of this data over here,assuming that each email is drawn independently according to an identical distribution.The probability of the p(yi) data item is then pi if yi = spam, and 1 - pi if yi = ham.If we rewrite the data as 1, 1, 1, 0, 0, 0, 0, 0,we can write p(yi) as follows: pi to the yi times (1 - pi) to the 1 - yi.It's not that easy to see that this is equivalent, but say yi = 1.Then this term will fall out. It's not proficient by 1 because the exponent is zero, and we get pi as over here.If yi = 0, then this term falls out, and this one here becomes 1 - pi as over here.Now assuming independence, we get for the entire data set that the joint probability of all data items is the product of the individual data items over here, which can now be written as follows:pi to the count of instances where yi = 1 times 1 - pi to the count of the instances where yi = 0.And we know in our example, this count over here is 3,and this count over here is 5, so we get pi to the 3rd times 1 - pi to the 5th.We now wish to find the pi that maximizes this expression over here.We can also maximize the logarithm of this expression,which is 3 times log pi + 5 times log (1 - pi)Optimizing the log is the same as optimizing p because the log is monotonic to p.The maximum of this function is attained with a derivative of 0,so let's compute with a derivative and set it to 0.This is the derivative, 3 over pi - 5 over 1 - pi.We now bring this expression to the right side,multiply the denominators up, and sort all the expressions containing pi to the left,which gives us pi = 3/8, exactly the number we were at before.We just derived mathematically that the data likelihood maximizing numberfor the probability is indeed the empirical count, which means when we looked at this quiz beforeand we said a maximum likelihood for the prior probability of spam is 3/8,by simply counting 3 over 8 emails were spam, we actually followed proper mathematical principles to do maximum likelihood estimation.Now, you might not fully have gotten the derivation of this, and I recommend you to watch it again, but it's not that importantfor the progress in this class. So, here's another quiz.I'd like the maximum likelihood, or ML solutions, for the following probabilities.The probability that the word "secret" comes up,assuming that we already know a message is spam,and the probability that the same word "secret" comes up if we happen to know the message is not spam, it's ham.And just as beforewe count the word secretin SPAM and in HAMas I've underlined here.Three out of 9 words in SPAM are the word secretso we have a third over hereor 0.333and only 1 out of all the 15 words in HAMare secretso you get a fifteenthor 0.0667.By now, you might have recognized what we're really building up is a Bayes networkwhere the parameters of the Bayes networks are estimated using supervised learningby a maximum likelihood estimator based on training data.The Bayes network has at its root an unobservable variable called spam, which is binary, and it has as many children as there are words in a message,where each word has an identical conditional distributionof the word occurrence given the class spam or not spam.If you write on our dictionary over here, you might remember the dictionary had 12 different words, so here is 5 of the 12, offer, is, secret, click and sports.Then for the spam class, we found the probability of secret given spam is 1/3,and we also found that the probability of secret given ham is 1/15, so here's a quiz.Assuming a vocabulary size of 12, or put differently,the dictionary has 12 words, how many parameters do we need to specify this Bayes network?And the correct answer is 23.We need 1 parameter for the prior p (spam),and then we have 2 dictionary distributions of any word,i given spam, and the same for ham.Now, there's 12 words in a dictionary, but this distribution only needs 11 parameters, so 12 can be figured out because they have to add up to 1.And the same is true over here, so if you add all these together,we get 23.So, here's a quiz.Let's assume we fit all the 23 parameters of the Bayes networkas explained using maximum likelihood.Let's now do classification and see what class and message it ends up with.Let me start with a very simple message, and it contains a single wordjust to make it a little bit simpler.What's the probability that we classify this one word message as spam?And the answer is 0.1667 or 3/18.How do I get there? Well, let's apply Bayes rule.This form is easily transformed into this expression over here,the probability of the message given spam times the prior probability of spamover the normalizer over here.Now, we know that the word "sports" occurs 1 in our 9 words of spam,and our prior probability for spam is 3/8,which gives us this expression over here.We now have to add the same probabilities for the class ham."Sports" occurs 5 times out of 15 in the ham class,and the prior probability for ham is 5/8,which gives us 3/72 divided by 18/72, which is 3/18 or 1/6.This gets to a more complicated quiz.Say the message now contains 3 words."Secret is secret," not a particularly meaningful email,but the frequent occurrence of "secret" seems to suggest it might be spam.What's the probability you're going to judge this to be spam?And the answer is surprisingly high. It's 25/26, or 0.9615.To see if we apply Bayes rule, which multiples the prior for spam-nesswith the conditional probability of each word given spam."Secret" carries 1/3, "is" 1/9, and "secret" 1/3 again.We normalize this by the same expression plus the probability for the non-spam case.5/8 is a prior."Secret" is 1/15."Is" is 1/15, and "secret" again.This resolves to 1/216 over this expression plus 1/5400,and when you work it all out is 25/26.The final quiz, let's assume our message is "Today is secret."And again, it might look like spam because the word "secret" occurs.I'd like you to compute for me the probability of spam given this message.And surprisingly, the probability for this message to be spam is 0.It's not 0.001. It's flat 0.In other words, it's impossible, according to our model,that this text could be a spam message.Why is this?When we apply the same rule as before, we get the prior for spam which is 3/8.And we multiple the conditional for each word into this.For "secret," we know it to be 1/3.For "is," to be 1/9, but for today, it's 0.It's 0 because the maximum of the estimate for the probability of "today" in spam is 0."Today" just never occurred in a spam message so far.Now, this 0 is troublesome because as we compute the outcome--and I'm plugging in all the numbers as before--none of the words matter anymore, just the 0 matters.So, we get 0 over something which is plain 0.Are we overfitting? You bet.We are clearly overfitting.It can't be that a single word determines the entire outcome of our analysis.The reason is that our model, to assign a probability of 0 for the word "today" to be in the class of spam is just too aggressive.Let's change this.One technique to deal with the overfitting problem is called Laplace smoothing.In maximum likelihood estimation, we assign towards our probabilitythe quotient of the count of this specific event over all events in our data set.For example, for the prior probability, we found that 3/8 messages are spam.Therefore, our maximum likelihood estimate for the prior probability of spam was 3/8.In Laplace Smoothing, we use a different estimate.We add the value k to the countand normalize as if we added k to every single class that we've tried to estimate something over.This is equivalent to assuming we have a couple of fake training exampleswhere we add k to each observation count.Now, if k equals 0, we get our maximum likelihood estimator.But if k is larger than 0 and n is finite, we get different answers.Let's say k equals 1,and let's assume we get one message, and that message was spam, so we're going to write it one message, one spam.What is p (spam) for the Laplace smoothing of k + 1?Let's do the same with 10 messages, and we get 6 spam.And 100 messages, of which 60 are spam.Please enter your numbers into the boxes over here.The answer here is 2/3 or 0.667 and is computed as follows.We have 1 message with 1 as spam, but we're going to add k =1.We're going to add k = 2 over here because there's 2 different classes.K = 1 times 2 = 2, which gives us 2/3.The answer over here is 7/12.Again, we have 6/10 but we add 2 down here and 1 over here, so you get 7/12.And correspondingly, we get 61/102 is 60 + 1 over 100 +2.If we look at the numbers over here, we get 0.5833 and 0.5986.Interestingly, the maximum likelihood on the last 2 cases over herewill give us .6, but we only get a value that's closer to .5,which is the effect of our smoothing prior for the Laplacian smoothing.Let's use the Laplacian smoother with K=1to calculate the few interesting probabilities-- P of SPAM, P of HAM, and then the probability of the words "today", given that it's in the SPAM class or the HAM class.And you might assume that our recovery sizeis about 12 different words here.This one is easy to calculate for SPAM and HAM.For SPAM, it's 2/5,and the reason is, we had previously3 out of 8 messages assigned to SPAM.But thanks to the Laplacian smoother, we add 1 over here.And there are 2 classes, so we add 2 times 1 over here,which gives us 4/10, which is 2/5.Similarly to get 3/5 over here.Now the tricky part comes up over here. Before, we had 0 occurances of the word "today" in the SPAM class,and we had 9 data points.But now we are going to add 1 for Laplacian smoother,and down here, we are going to add 12.And the reason that we add 12 is becausethere's 12 different words in our dictionaryHence, for each word in the dictonary, we are going to add 1.So we have a total of 12, which gives us the 12 over here.That makes 1/21.In the HAM class, we had 2 occurrencesof the word "today"--over here and over here. We add 1, normalize by 15,plus 12 for the dictionary size,which is 3/27 or 1/9.This was not an easy question.We come now to the final quiz here,which is--I would like to compute the probabilitythat the message "today is secret"falls into the SPAM box with Laplacian smoother using K=1.Please just enter your number over here.This is a non-trivia question.It might take you a while to calculate this.In the approximate probabilities--0.4858.How did we get this?Well, the prior probability for SPAMunder the Laplacian smoothing is 2/5."Today" doesn't occur, but we have already calculated this to be 1/21."Is" occurs once, so we get 2 over here over 21."Secret" occurs 3 times, so we get a 4 over here over 21,and we normalize this by the same expression over here.Plus the prior for HAM, which is 3/5,we have 2 occurrences of "today", plus 1, equals 3/27."Is" occurs once--2/27.And "secret" occurs once--again 2/27.When you work this all out, you get this number over here.So we learned quite a bit.We learned about Naive Bayesas our first supervised learning methods.The setup was that we had features of documents or trading examples and labels.In this case, SPAM or not SPAM.And from those pieces,we made a generative model for the SPAM classand the non-SPAM classthat described the condition of probabilityof each individual feature.We then used first maximum likelihoodand then a Laplacian smootherto fit those primers over here.And then using Bayes rule,we could take any training examples over hereand figure out what the class probability was over here.This is called a generative modelin that the condition of probabilities all aim to maximizethe probability of individual features as if thosedescribe the physical world.We also used what is called a bag of words model,in which our representation of each emailwas such that we just counted the occurrences of words,irrespective of their order. Now this is a very powerful method for fighting SPAM.Unfortunately, it is not powerful enough.It turns out spammers know about Naive Bayes,and they've long learned to come up with messagesthat are fooling your SPAM filter if it uses Naive Bayes.So companies like Google and othershave become much more involvedin methods for SPAM filtering.Now I can give you some more examples how to filter SPAM,but all of those quite easily fit with the same Naive Bayes model.[Narrator] So here features that you might consider when you writein an advance spam filter.For example, does the email come from a known spamming IP or computer?Have you emailed this person before?In which case it is less likely to be spam.Here's a powerful one:have 1000 other peoplerecently received the same message?Is the email header consistent? So example if the from field says your bankis the IP address really your bank? Surprisingly is the email all caps?Strangely many spammers believe if you write things in all caps you'll pay more attention to it. Do the inline URLs point to those pages where they say they're pointing to?Are you addressed by your correct name?Now these are some features,I'm sure you can think of more.You can toss them easily into the naive base model and get better classification.In fact model spam filters keep learningas people flag emails as spam, and of course spammers keep learning as welland trying to fool modern spam filters.Who's going to win?Well so far the spam filters are clearly winning.Most of my spam I never see, but who knowswhat's going to happen with the future?It's a really fascinating machine learning problem. [Narrator] Naive Bayes can also be applied to the problem of hand written digits recognition.This is a sample of hand-written digits takenfrom a U.S. postal data set where hand written zip codes on letters are being scanned and automatically classified.The machine-learning problem here istaken a symbol just like this.What is the corresponding number?Here it's obviously 0.Here it's obviously 1. Here it's obviously 2, 1.For the one down here, it's a little bit harder to tell.Now when you apply Naive Bayes,the input vector could be the pixel valuesof each individual pixel so we havea 16 x 16 input resolution.You would get 256 different values corresponding to the brightness of each pixel.Now obviously given sufficiently made training example, you might hope to recognize digits,but one of the deficiencies of this approach isit is not particularly shifted range. So for example a pattern like thiswill look fundamentally different from a pattern like this.Even though the pattern on the right is obtainedby shifting the pattern on the left by 1 to the right. There's many different solutions, but a common one could be to use smoothing in a different way from the way we discussed it before.Instead of just counting 1 pixel value's count, you could mix it with counts of the neighboring pixel values so if all pixels are slightly shifted,we get about the same statistics as the pixel itself.Such a method is called input smoothing. You can what's technically called convolve the input vector equals pixel value variable, and you might get better results than if you do Naive Bayes on the raw pixels.Now to tell you the truth for digit recognition of this type,Naive Bayes is not a good choice.The conditional independence assumption of each pixel, given the class,is too strong an assumption in this case,but it's fun to talk about image recognition in the context of Naive Bayes regardless.So, let me step back a step and talk a bit aboutoverfitting prevention in machine learningbecause it's such an important topic.We talked about Occam's Razor,which in a generalized way suggests there is a tradeoff between how well we can fit the dataand how smooth our learning algorithm is.In our class in smoothing, we already found 1 wayto let Occam's Razor play, which is by selecting the value K to make our statistical counts smoother.I alluded to a similar way in the image recognition domainwhere we smoothed the image so the neighboring pixels count similar.This all raises the question of how to choose the smoothing parameter.So, in particular, in Laplacian smoothing, how to choose the K.There is a method called cross-validationwhich can help you find an answer.This method assumes there is plenty of training examples, butto tell you the truth, in spam filtering there is more than you'd ever want.Take your training dataand divide it into 3 buckets.Train, cross-validate, and test.Typical ratios will be 80% goes into train, 10% into cross-validate,and 10% into test.You use the train to find all your parameters.For example, the probabilities of a base network.You use your cross-validation setto find the optimal K, and the way you do this isyou train for different values of K,you observe how well the training model performs on the CV data,not touching the test data,and then you maximize over all the Ks to get the best performanceon the cross-validation set.You iterate this many times until you find the best K.When you're done with the best K, you train again, and then finallyonly one you touch the test datato verify the performance,and this is the performance you report.It's really important in cross-validationsplit apart a cross-validation set that's different from the test set.If you were to use the test set to find the optimal K,then your test set becomes an effective part of your training routine,and you might overfit your test data,and you wouldn't even know.By keeping the test data separate from the beginning,and train on the training data, you usethe cross-validation data to find how good your train data is doing,and the unknown parameters of K to fine-tune the K.Finally, only once you use the test datado you get a fair answer to the question,"How well will your model perform on future data?"So, pretty much everybody in machine learninguses this model. You can redo the split between training and the cross-validation part,people often use the word 10-fold cross-validationwhere they do 10 different forwardingsand run the model 10 times to find the optimal Kor smoothing parameter.No matter which way you do it, find the optimal smoothing parameterand then use a test set exactly once to verify in a report.Let me back up a step further, and let's look at supervised learning more generally.Our example so far was one of classification.The characteristic of classifcation is that the target labels or the target class is discrete.In our case it was actually binary.In many problems, we try to predict a continuous quantity.For example, in the interval 0 to 1 or perhaps a real number.Those machine learning problems are called regression problems.Regression problems are fundamentally different from classification problems.For example, our base network doesn't afford us an answerto a problem where the target value could be at 0,1.A regression problem, for example, would be one topredict the weather tomorrow.Temperature is a continuous value. Our base number would not be ableto predict the temperature, it only can predict discrete classes.A regression algorithm is able to give us a continuous predictionabout the temperature tomorrow.So let's look at the regression next.So here's my first quiz for you on regression.This scatter plot shows for Berkeley California for a period of timethe data for each house that was sold.Each dot is a sold house.It graphs the size of the house in square feetto the sales price in thousands of dollars.As you can see, roughly speaking,as the size of the house goes up, so does the sales price.I wonder, for a house of about 2500 square feet,what is the approximate sales price you would assumebased just on the scatter plot data?Is it 400k, 600k, 800k, or 1000k?My answer is, there seems to be a roughly linear relationship,maybe not quite linear, between the house size and the price.So we look at a linear graph that best describes the data--you get this dashed line over here.And for the dashed line, if you walk up the 2500 square feet, you end up with roughly 800K.So this would have been the best answer.Now obviously you can answer this question without understanding anything about regression.But what you find is this is different from classification as before. This is not a binary concept anymore of like expensive and cheap.It really is a relationship between two variables.One you care about--the house price, and one that you can observe,which is the house size in square feet.And your goal is to fit a curve that best explains the data.Once again, we have a case where we can play Occam's razor.There clearly is a data fit that is not linear that might be better,like this one over here.And when you go to hide the linear curves, you might even be inclined to draw a curve like this.Now of course the curve I'm drawing right now is likely an overfit.And you don't want to postulate that this is the general relationship between the size of a house and the sales price.So even though my black curve might describe the data better,the blue curve or the dashed linear curve over here might be a better explanation overture of Occam's razor.So let's look a little bit deeper into what we call regression.As in all regression problems, our data will be comprised of input vectors of length in that map to another continuous value.And we might be given a total of M data points.This is from the classification case, except this time the Ys are continuous.Once again, we're looking for function f that maps our vector x into y.In linear regression, the function has a particular form which is W1 times X plus W0.In this case X is one dimensional which is N = 1.Or in the high-dimensional space, we might just write W times X plus W0,where W is a vector and X is a vector.And this is the inner product of these 2 vectors over here. Let's for now just consider the one-dimensional case. In this quiz, I've given you a linear regression form with 2 unknown parameters, W1 and W0.I've given you a data set.And this data set happens to be fittable by a linear regression model without any residual error.Without any math, can you look at this and find out to me what the 2 parameters, W0 and W1 are? This is a suprisingly challenging question.If you look at these numbers from 3 to 6.When we increase X by 3, Y decreases by 3,which suggests W1 is -1.Now let's see if this holds. If we increase X by 3, it decreases Y by 3.If we increase X by 1, we decrease Y by 1.If we increase X by 2, we decrease Y by 2.So this number seems to be an exact fit.Next we have to get the constant W0 right. For X = 3, we get -3 as an expression over here,because we know W1 = -1.So if this has to equal zero in the end, then W0 has to be 3. Let's do a quick check.-3 plus 3 is 0.-6 plus 3 is -3.And if we plug in any of the numbers, you find those are correct. Now this is the case of an exact data set.It gets much more challenging if the data set cannot be fit with a linear function. To define linear regression,we need to understand what we are trying to minimize.The word is called here, are loss functionand the loss function is the amount of residual error we obtainafter fitting the linear function as good as possible.The residual error is the sum of all training examples, J of YJ, which is the target label, minus our prediction, which is W1 XJ minus W0 to the square.This is the quadratic error between our target tablesand what our best hypothesis can produce.The minimizing of lossis used for linear regression of a new regression problem,and you can write it as follows:Our solution to the regression problem W* is the arg min of the loss over all possible vectors W.The problem of minimizing quadratic loss for linear functions can be solved in closed form.When I reduce, I will do this for the one-dimensional case on paper.I will also give you the solution for the case where your input space is multidimensional,which is often called "multivariant regression."We seek to minimize a sum of a quadratic expressionwhere the target labels are subtracted with the output of our linear regression modelparameterized by w1 and w2.The summation here is overall training examples,and I leave the index of the summation out if not necessary.The minimum of this is obtained where the derivative of this function equals zero.Let's call this function "L."For the partial derivative with respect to w0, we get this expression over here,which we have to set to zero.We can easily get rid of the -2 and transform this as follows:Here M is the number of training examples.This expression over here gives us w0 as a function of w1,but we don't know w1. Let's do the same trick for w1and set this to zero as well,which gets us the expression over here.We can now plug in the w0 over here into this expression over hereand obtain this expression over here, which looks really involved but is relatively straightforward.With a few steps of further calculation, which I'll spare you for now,we get for w1 the following important formula:This is the final quotient for w1,where we take the number of training examples times of the sum of all xy minus the sum of x times the sum of y divided by this expression over here.Once we've computed w1, we can go back to our original articulation of w0 over hereand plug w1 into w0 and obtain w0.These are the two important formulas we can also find in the textbook.I'd like to go back and use those formulas to calculate these two coefficients over here.You get 4 times the sum of x and the sum of y, which is -32minus the product of the sum of x, which is 18, and the sum of y, which is -6,divided by the sum of x squared, which is 86, times 4, minus the sum of x squared, which is 18 times 18, which is 324.If you work this all out, it becomes -1, which is w1.W0 is now obtained by completing the quarter times sum of all y, which is -6, minus -1/4 times sum of all x.If you plug this all in, you get 3, as over here. Our formula is actually correct.Here is another quiz for linear regression. We have the follow data:Here is the data plotted graphically.I wonder what the best regression is.Give me w0 and w1. Apply the formulas I just gave you.And the answer is W0 = 0.5, and W1 = 0.9.If I were to draw a line, it would go about like this.It doesn't really hit the two points at the end.If you were thinking of something like this, you were wrong.If you draw a curve like this, your quadratic error becomes 2.One over here, and one over here.The quadratic error is smaller for the line that goes in between those points.This is easily seen by computing as shown in the previous slide.W1 equals (4 x 118 - 20 x 20) / (4 x 120 - 400) which is 0.9.This is merely plugging in those numbers into the formulas I gave you.W0 then becomes ¼ x 20. Now we plug in W1-- 0.9 / 4 x 20 equals 0.5.This is an example of linear regression,in which case there is a residual error,and the best-fitting curve is the one that minimizesthe total of the residual vertical error in this graph over here.So linear regression works wellif the data is approximately linear,but there are many examples when linear regression performs poorly.Here's one where we have acurve that is really nonlinear.This is an interesting one where we seem to have a linear relationshipthat is flatter than the linear regression indicates,but there is one outlier.Because if you are minimizing quadratic error,outliers penalize you over-proportionately.So outliers are particularly bad for linear regression.And here is a case,where the data clearly suggestsa very different phenomena for linear.We have only two ?? variables even being used,and this one has a strong frequencyand a strong vertical spread.Clearly a linear regression modelis a very poor one to explainthis data over here.Another problem with linear regressionis that as you go to infinity in the X space,your Ys also become infinite.In some problems that isn't a plausible model.For example, if you wish to predict the weatheranytime into the future,it's implausible to assume the further the prediction goes out,the hotter or the cooler it becomes.For such situations there is a model called logistic regression, which uses a slightly more complicatedmodel than linear regression, which goes as follows:.Let F of XP, or linear function,and the output of logistic regressionis obtained by the following function:One over one plus exponential of minus F of X.So here's a quick quiz for you.What is the range in which Z might fallgiven this function over here,and ??? the linear function of F or X over here.Is it zero, one?Is it minus one, one?Is it minus one, zero?Minus two, two?Or none of the above?The answer is zero, one.If this expression over here,F of X,grows to positive infinity,then Z becomes one.And the reason isas this term over here becomes very large,E to the minus of that term approaches zero;one over one equals one.If F of X goes to minus infinity,then Z goes to zero.And the reason is,if this expression over here goes to minus infinity,E to the infinity becomes very large;one over something very large becomes zero.When we plot the logistic function it looks like this:So it's approximately lineararound F of X equals zero,but it levels off to zero and oneas we go to the extremes.Another problem with linear regression has to do with the regularization or complexity control.Just like before, we sometimes wish to havea less complex model.So in regularization, the loss function is either the sumof the loss of a data function and a complexity control term,which is often called the loss of the parameters.The loss of the data is simply curvatic loss, as we discussed before.The loss of parameters might just be a function that penalizes the parameters to become largeup to some known P, where P is usually either 1 or 2.If you draw this graphically,in a parameter space comprised of 2 parameters,your curvatic term for minimizing the data errormight look like this, where the minimum sits over here.Your term for regularization might pull these parameters toward 0.It pulls it toward 0, along the circle if you use curvatic error,and it does it in a diamond-shaped way.For L1 regularization--either one works well.L1 has the advantage in that parameters tend to get really sparse.If you look at this diagram, there is a tradeoff between W-0 and W-1.In the L1 case, that allows one of them to be driven to 0.In the L2 case, parameters tend not to be as sparse.So L1 is often preferred.This all raises the question,how to minimize more complicated loss functionsthan the one we discussed so far.Are there closed-form solutions of the type we found for linear regression?Or do we have to resort to iterative methods?The general answer is, unfortunantly, we have to resort to iterative methods.Even though there are special cases in which corresponding solutions may exist,in general, our loss functions now become complicated enough that all we can do is iterate.Here is a prototypical loss functionand the method for interation will be called gradient descent.In gradient descent, you start with an initial guess,W-0, where 0 is your iteration number,and then you up with it iteratively.Your i plus 1st parameter guess will be obtained by taking your i-th guessand subtracting from it the gradient of your loss function,and that guess multiplied by a small learning weight alpha,where alpha is often as small as 0.01.I have a couple of questions for you.Consider the following 3 points.We call them A, B, C.I wish to know, for points A, B, and C,Is the gradient at this point positive, about zero, or negative?For each of those, check exactly one of those cases.In case A, the gradient is negative.If you move to the right in the X space,then your loss decreases.In B, it's about zero.In C, it's pointing up; it's positive.So if you apply the rule over here,if you were to start at A as your W-zero,then your gradient is negative.Therefore, you would add something to the value of W.You move to the right, and your loss has decreased.You do this until you find yourselfwith what's called a local minimum, where B resides.In this instance over here, gradient descent starting at Awould not get you to the global minimum,which sits over here because there's a bump in between.Gradient methods are known to be subject to local minimum.I have another gradient quiz.Consider the following quadratic arrow function.We are considering the gradient in 3 different places.a. b. and c.And they ask you which gradient is the largest.a, b, or c or are they all equal?In which case, you would want to check the last box over hereAnd the answer is C.The derivative of a quadratic function is a linear function.Which would look about like this.And as we go outside, our gradient becomes larger and larger.This over here is much steeper than this curve over here.[Thrun] Here is a final gradient descent quiz.Suppose we have a loss function like thisand our gradient descent starts over here.Will it likely reach the global minimum?Yes or no.Please check one of those boxes.[Thrun] And the answer is yes,although, technically speaking, to reach the absolute global minimumwe need the learning rates to become smaller and smaller over time.If they stay constant, there is a chance this thing might bounce around between 2 points in the end and never reach the global minimum.But assuming that we implement gradient descent correctly,we will finally reach the global minimum.That's not the case if you start over here, where we can get stuck over hereand settle for the minimum over here, which is a local minimumand not the best solution to our optimization problem.So one of the important points to take away from this is gradient descent is universally applicable to more complicated problems--problems that don't have a plausible solution.But you have to check whether there is many local minima,and if so, you have to worry about this.Any optimization book can tell you tricks how to overcome this.I won't go into any more depth here in this class.[Thrun] It's interesting to see how to minimize a loss function using gradient descent.In our linear case, we have L equals sum over the correct labelsminus our linear function to the square,which we seek to minimize.We already know that this has a closed form solution,but just for the fun of it, let's look at gradient descent.The gradient of L with respect to W1 is minus 2, sum of all Jof the difference as before but without the square times Xj.The gradient with respect to W0 is very similar.So in gradient descent we start with W1 0 and W0 0 where the upper cap 0 corresponds to the iteration index of gradient descent.And then we iterate.In the M iteration we get our new estimate by using the old estimateminus a learning rate of this gradient over heretaking the position of the old estimate W1, M minus 1.Similarly, for W0 we get this expression over here.And these expressions look nasty,but what it really means is we subtract an expression like thisevery time we do gradient descent from W1and an expression like this every time we do gradient descent from W0,which is easy to implement, and that implements gradient descent.Now, there are many different ways to apply linear functions in machine learning.We so far have studied linear functions for regression, but linear functions are also used for classification,and specifically for an algorithm called the perceptron algorithm.This algorithm happens to be a very early model of a neuron,as in the neurons we have in our brains, and was invented in the 1940s.Suppose we give a data set of positive samples and negative samples.A linear separator is a linear equation that separates positive from negative examples.Obviously, not all sets possess a linear separator, but some do.For those we can define the algorithm of the perceptron and it actually converges.To define a linear separator, let's start with our linear equation as before--w1x + w0 in cases where x is higher dimensional this might actually be a vector--never mind.If this is larger or equal to zero, then we call our classification 1.Otherwise, we call it zero.Here's our linear separation classification functionwhere this is our common linear function.Now, as I said, perceptron only converges if the data is linearly separable,and then it converges to a linear separation of the data,which is quite amazing.Perceptron is an iterative algorithm that is not dissimilar from grade descent.In fact, the update rule echoes that of grade descent, and here's how it goes.We start with a random guess for w1 and w0,which may correspond to a random separation line,but usually is inaccurate.Then the mth weight-i is obtained by using the old weight plus some learning rate alphatimes the difference between the desired target labeland the target label produced by our function at the point m-1.Now, this is an online learning rule, which is we don't process all the data in batch.We process one data at a time, and we might go through the data many, many times--hence the j over here--but every time we do this, we apply this rule over here.What this rule gives us is a method to adapt our weights in proportion to the error.If the prediction of our function f equals our target label,and the error is zero, then no update occurs.If there is a difference, however, we update in a way so as to minimize the error.Alpha is a small learning weight.Once again, perceptron converges to a correct linear separatorif such linear separator exists.Now, the case of linear separation has recently received a lot of attention in machine learning.If you look at the picture over here, you'll find there are many different linear separators.There is one over here. There is one over here. There is one over here.One of the questions that has recently been researched extensively is which one to prefer.Is it a, b, or c?Even though you probably have never seen this literature,I will just ask your intuition in this following quiz.Which linear separator would you prefer if you look at these three different linear separators--a, b, c, or none of them?[Narrator] And intuitively I would argue it's B,and the reason why isC comes really close to examples.So if these examples are noisy,it's quite likely that by being so close to these examples that future examples cross the line.Similarly A comes close to examples.B is the one that stays really far away from any example.So there's this entire region over herewhere there's no example anywhere near B. This region is often called the margin.The margin of the linear separator is the distance of the separatorto the closest training example.The margin is a really important concept in machine learning.There is an entire class of maximum margin learning algorithms,and the 2 most popular are support vector machines and boosting.If you are familiar with machine learning, you've come across these terms.These are very frequently used these days in actual discrimination learning tasks.I will not go into any details because it would go way beyond the scope of this introduction to artificial intelligence class, but let's see a few abstract words specifically aboutsupport vector machines or SVMs.As I said before a support vector machinederives a linear separator, and it takes the one that actually maximizes the marginas shown over here.By doing so it attains additional robost-ness over perceptron which only picks a linear separator without consideration of the margin.Now the problem of finding the margin maximizing linear separator can be solved by a quadratic programwhich is an integer method for finding the bestlinear separator that maximizes the margin.One of the nice things that supportvector machines do in practice isthey use linear techniques to solve nonlinear separation problems,and I'm just going to give you a glimpse ofwhat's happening without going into any detail.Suppose the data looks as follows:we have a positive class which is near the origin of a coordinate system and a negative class that surrounds the positive class.Clearly these 2 classes are not linearly separable because there's no line I can draw that separates the negative examples from the positive examples.An idea that underlies SVMs,that will ultimately be known as the kernel trick, is to augment the feature set by new features.Suppose this is X1, and this is X2, and normally X1 and X2 will be the input features.In this example, you might derivea 3rd one.Let me pick a 3rd one Suppose X3 equals the square root of X1 square + X2 square.In other words X3 is the distanceof any data point from the center of the coordinate system.Then things do become linearly separableso that just along the 3rd dimensionall the positive examples end up to be close to the origin,and all the negative examples are further away, and the line is orthogonal to the 3rd input feature solves the separation problem.Map back into the space over hereis actually a circle which is a set of all values of X3 that are equidistantto the center of the origin.Now this trick could be done in any linear learning algorithm, and it's really an amazing trick.You can take any nonlinear problem, add features of this type or any other type,and use linear techniques and get better solutions.This is a very deep machine learning insight that you can extend your feature space in this way, and there's numerous papers written about this.In SVMs, the extension of the feature space is mathematically done by what's called a kernel.I can't really tell you about this in this class,but it makes it possible to write very large new feature spaces includinginfinitely dimensional new feature spaces.These messages are very powerful. It turns out you never really compute all those features.They are implicitly represented by so called kernels, and if you care about this,I recommend you to dive deeper into the literature of support vector machines.This is meant to just give you an overview of the essence of what support vector machines are all about.So in summary,linear methods we learned about using them for regression and also classification.We learned about exact solutionsversus iterative solutions.We talked about smoothing,and we even talked about using linear methods for nonlinear problems.So we covered quite a bit of ground.This is a really significant cross sectionof machine learning.As the final method in this unit, I'd like now to talk about k-nearest neighbors.And the distinguishing factor of k-nearest neighbors is that it is a nonparametric machine learning method.So far we've talked about parametric methods. Parametric methods have parameters, like probabilities or weights,and the number of parameters is constant.Or to put it differently, the number of parameters is independent of the training set size.So for example in the Naive Bayes, if we bring up more data, the number of condition probabilities will stay the same.Well, that wasn't technically always the case.Our vocabulary might increase and as such the number of parameters.But for any fixed dictionary, the number of parameters are truly independent of the training set size.The same was true, for example, in our regression caseswhere the number of regression weights is independent of the number of data points. Now this is very different from non-parametricwhere the number of parameters can grow. In fact, it can grow a lot over time. Those techniques are called non-parametric. Nearest neighbor is so straightforward.I'd really like to introduce you using a quiz. So here's my quiz.Suppose we have a number of data points.I want you for 1-nearest neighbor to check those squared areasthat you believe will carry a positive label.And I will give you the label of the existing data points.So please check any of those boxes that you believe are now 1-nearest neighbor that carry a positive label.And the algorithm, of course, searches for the nearest point in this Euclidean space and just copies its label. And the answer was: This is a positive point,and this is a positive point.These 2 points over here are negative. So let's define k-nearest neighbors. The algorithm is really blatantly simple. In the learning step, you simply memorize all data.If a new example comes along with the input value you knowbut which you wish to classify, you do the following.You first find the k-nearest neighbors.And then you return the majority class label as your final class label for the new example.Simple, isn't it?So here's a somewhat contrived situation of the data point we wish to classify where the label data lies on the spiral of increasing diameter as it goes outwards. Please answer for me in this quiz what class label you'd assignfor k = 1, k = 3, 5, 7, and all the way to 9.And this is an easy answer. The nearest neighbor is this point over here, so for 1 we say plus.For 3 neighbors, we get 2 positive, 1 negative. It's still plus.For 5 neighbors--1, 2, 3, 4, 5--we get 3 negative, 2 positive. It's a minus. For 7, we get 3 positive but still 4 negative, so it's negative.And for 9, the positives outweigh the negative, so you get a plus. Obviously, as you can see, as K increases, more and more data points are being consulted.So when K finally becomes 9, all those data points are in and make a much smoother result. Just as in the Laplacian smoothing example before, the value of k is a smoothing parameter. It makes the function less scattered. Here is an example of k=1for a 2-class nearest neighbor problem.You can see the separation boundary is what's called a Voronoi diagrambetween the positive and negative class, and in cases where there is noise between these class boundaries, you'll find really funny, complex boundaries as indicated over here. Particularly interesting is this guy over here where the class of this circle over hereprotrudes way into the otherwise solid class.Now, as you go to k=3, you get this graph over here,which is smoother.So if you are over here, your two nearest neighbors are of this type over there,and you get a uniform class over here. In this region over here, you get uniform classes as solid classesas shown over here. The more you drive up k, the more clean this decision boundary becomes, but the more outliers are actually misclassified as well. So if I go back to my k-nearest neighbor method,we just learned that k is a regularizer.It controls the complexity of the k-nearest neighbor algorithm. and the larger k is, the smoother the output. We can, once again, use cross-validation to find the optimal kbecause there is an inherent trade off--between the complexity of what we want to fitand the goodness of the fit. What are the problems of kNN?Well, I would argue that there're two.One is very large data sets,and one is very large feature spaces.Now the first one results in lengthy searcheswhen you try to find K's nearest neighbors.Now, fortunately there aremethods to search efficiently.Often you represent your datanot by a linear list, in which case the searchwould be linear in the number of data points,but by a tree, where the search becomes logarithmic.The method of choice is called kDD treeswhere there are many other ways to represent data points as trees.Now very large feature spaces cause more of a problem.It turns out computing nearest neighbors,as the feature space for the input vector increases,becomes increasingly difficult,and the tree methods become increasingly brittle.And the reason is shown in the following graph:If your graph input dimension to the average edge length of your neighborhoodyou'll find that for randomly chosen pointsvery quickly all points are really far away.The edge length of one is obtainedif your query pointis unit one away from the nearest neighbor.If you have one hundred dimensions,that is almost certain.Why is that?Well, in one hundred dimensions,they are to be one where just by chanceyour're far away.The number of points you needto get something closegrows exponentially with the number of dimensions.So, for any fixed data set sizeyou will find yourself in a situationwhere all your neighbors are far away.Nearest neighbor works really wellfor small input spaces like three or four dimensions.It works very poorlyif your input space is twenty, twenty-five,or maybe one hundred dimensions.So don't trust nearest neighbor to do a good jobif your input and measure spaces are high.So congratulations.You've just learned a lot about machine learning.We focused on supervised machine learningwhich deals with situationswhere you have input vectorsand given output labelsand your goal is to predict the output labelfrom an input vector.And we looked into parametric modelslike Naive Bayesnon-parametric models.We talked about classificationwhere the output is discreteversus regression where the output is continuous and we looked at samples of techniquesfor each of these situations.Now obviouslywe just scratched the surface on machine learning.There's books written about itand courses taught about it.Machine learning is a super fascinating topic.It's the one within the artificial intelligenceI love the most. It's really great about the real worldas we gain more datalike the world wide webor medical data sets or financial data sets.Machine learning is poised to become more and more important.I hope that the things you learned in this class so farreally excite youand entice you to apply machine learningto problems that you face in your professional life.[Narrator] So welcome to the class on unsupervised learning.We talked a lot about supervised learningin which we are given data and target labels. In unsupervised learning we're just given data.So here's a data matrix of data items of N features each.There's M and total.So the task of unsupervised learning isto find structure in data of this type.To illustrate why this is an interesting problemlet me start with a quiz.Suppose we have 2 feature values.One over here, and one over here,and our data looks as follows.Even though we haven't been told aboutanything in unsupervised learning, I'd like toquiz your intuition on the following 2 questions:1. Is there structure? Or put differently do you think there's something to be learned about data like this,or is it entirely random?And second, to narrow this down,it feels that there are clusters of data the way I do it. So how many clusters can you see?And I give you a couple of choices, 1, 2, 3, 4, or none.[Narrator] The answer to the first question is yes, there is structure.Obviously these data are seen not to be completely random determinants. They seem to be for me 2 clusters.So the correct answer for the second question is 2.There's a cluster over here, and there's a cluster over here.So one of the tasks of unsupervised learning will be to recover the number of clusters, andthe center of these clusters, and the variances of these clusters in data of the type I've just shown you.[Narrator] Let me ask you a second quiz.Again, we haven't talked about any details.I would like to get your intuition on the following question.Suppose in a two dimensional space,all data lies as follows.This may be reminiscent of the question I asked you for housing prices and square footage. Suppose we have 2 axes, X1 and X2.I'm going to ask you 2 questions here.One is what is the dimensionality of this spacein which this data falls, and the second one is an intuitive question which is how many dimensions do you need to represent this data to capture the essence,and, again, this is not a clear crisp 0 or 1 type question,but give me your best guess.How many dimensions are intuitively needed?No subtitles...[Narrator] So to start with some lingo about unsupervised learning. If you look at this as a probabilist, you're given data, and we interpretively assume the data is IID,which means identically distributed and independently drawn from the same distribution.So a good chunk of unsupervised learningseeks to recover the underlying--the density of probability distribution that generated the data.It's called density estimation.As we find out our methods for clustering,our versions of density estimation using what's called mixture models.Dimensionality reduction is also a methodfor doing density estimation, and there are many others.Unsupervised learning can be applied to find structure and data.One of the fascinating ones that I believe exists is called blind signals separation.Suppose you are given a microphone, andtwo people simultaneously talk, and you're record the joint of both of those speakers.Blind source separation or blind signal separationaddresses the question of can you recoverthose two speakers and filterthe data into two separate streams.One for each speaker.Now this is a really complicated unsupervisedlearning task, but is one of the many thingsthat don't require target signals as unsupervised learning yet make for really interesting learning problems.This can be construed as an exampleof what's called factor analysis where each speaker is a factor in the drawing signal that your microphone records.There are many other examples of unsupervised learning.I will show you a few in a second.Here is one of my favorite examples of unsupervised learning--one that is yet unsolved.At Google, I had the opportunity to participate--in the building of Street View,which is a huge photographic database--of many, many streets in the world.As you dive into Street View--you can get ground imagery--of almost any location in the world--like this house here, that I chose at random.In these images, there is vast regularities.You can go somewhere else--and you'll find that the type of objects--visible in Street View--is not entirely random.For example, there is many images of homes--many images of cars--trees, pavement, lane markers--stop sign, just to name a few.So one of the fascinating, unsolved, unsupervised learning tasks is:Can you take hundreds of billions of images--as comprised in the Street View data set--and discover from it that there are concepts such as--trees, lane markers, stop signs, cars, and pedestrians?It seems to be tedious to hand label each image--for the occurrence of such objects.And attempts to do so--has resulted in very small image data sets.Humans can learn from data--even without explicit target labels.We often just observe.In observing, we apply unsupervised learning techniques.So one of the great, great open questions of artificial intelligence is:Can you observe many intersections and many streets and many roads--and learn from it what concepts are contained in the imagery?Of course, I can't teach you anything as complex in this class.I don't even know the answer myself.So let me start with something simple.Clustering. Clustering is the most basic form of unsupervised learning.And I will tell you about two algorithms that are very related.One is called k-means,one is called expectation maximization.K-means is a nice, intuitive algorithm to derive clusterings.Expectation maximization is a probabilistic-- generalization of k-means.They were derived from first principles.Let me explain k-means by an example.Suppose we're given the following data points in a 2-dimensional space.K-means estimates for a fixed number of k. Here k = 2.The best centers of clusters representing those data points.Those are found interatively by the following algorithm.Step 1: Guess cluster centers at random, as shown over here with the 2 stars.Step 2: Assign to each cluster center, even though they are randomly chosen, the most likely corresponding data points.This is done by minimizing Euclidian distance.In particular, each cluster center represents half of the space.And the line that separates the space between the left and right cluster center is the equidistant line, often called a Voronoi graph.All the data points on the left correspond to the red cluster,and the ones on the right to the green cluster. Step 3: Given now we have a correspondence between the data points and cluster centers, find the optimal cluster center that corresponds to the points associated with the cluster center. Our red cluster center has only 2 data points attached. So the optimal cluster center would be the halfway point in the middle.Our right cluster center has more than 2 points attached;yet it isn't placed optimally, as you can see as they move with the animation back and forth.By minimizing the joint quadratic distance to all of those points, the new cluster center has attained the center of those data points. Now the final step is iterate. Go back and reassign cluster centers.Now the Voronoi diagram has shifted, and the points are associated differently,and then reevaluate what the optimal cluster center looks like given the associated points.And in both cases we see significant motion.Repeat. Now this is the clustering. The point association doesn't change, and as a result, we just converged.You just learned about an exciting clustering algorithmthat's really easy to implement called k-means.To give you the algorithm in pseudocode, initially we select k cluster centers at random and then we repeat.In a corresponding step, we correspond all the data points to the nearest cluster center,and then we calculate the new cluster center by the mean of the corresponding data points.We repeat this until nothing changes any more.Now special care has to be taken if a cluster center becomes empty--that means no data point is associated.In which case, we just restart cluster centers at random that have no corresponding points.Empty cluster centers restart at random.This algorithm is known to converge to a locally optimal clustering of data ponts.The general clustering problem is known to be NP-hard.So a locally optimal solution, in a way, is the best we can hope for. Now let me talk about problems with k-means.First we need to know k, the number of cluster centers.As I mentioned, the local minimum. For example, for 4 data points like this and 2 cluster centers that happen to be just over here,with the separation line like this there would be no motion of k means.Even though moving one over here and one over there would give a better solution.There's a general problem of high dimensionality of the spacethat is not dissimilar from the way k-nearest neighbor suffers from high dimensionality.And then there's lack of a mathematical basis. Now if you're a partitioner, you might not care about a mathematical basis.But for the sake of this class, let's just care about it.So here's a first quiz for k-means.Given the following two cluster centers, C1 and C2, click on exactly those points that are associated with C1 and not with C2.And the answer is these 4 points over here.And the reason is, if you draw the line of equal distance between C1 and C2,the separation of these 2 cluster areas falls over here.C2 is down there. C1 is up here.So here's my second quiz. Given the association that we just derived for C1, where do you think the new cluster center, C1, will be found after a single step of estimating its best location given the associated points?I'll give you a couple of choices.Please click on the one that you find most plausible. And the answer is, over here. These 4 data points are associated with C1, so we can safely ignore all the other ones.This one over here would be at the center of the 3 data points over here,but this one pulls back this data point drastically towards it.This is about the best trade-off between these 3 points over here that all have a stringattached and pull in this direction, compared to this point over here.Any of the other ones don't even lie between those points,and therefore won't be good cluster centers.The one over here is way too far to the right.In our next quiz let's assume we've done one interation,and the cluster center of C1 moved over there and C2 moved over here. Can you once again click on all those data points that correspond to C1?And the answer is now simple. It's this one over here.This one, this one, and this one.And the reason is, the line separating both clusters runs around here.That means all the area over here is C2 territory, and the area over here is C1 territory.Obvioulsy as we now iterate k-means, these clusters that have been moved straight over here will be able to stay, whereas C2 will end up somewhere over here.So, let's now generalize k-means into expectation maximization.Expectation maximization is an algorithm that uses actual probability distributionsto describe what we're doing, and it's in many ways more general,and it's also nice in that it really has a probabilistic basis.To get there, I have to take the discourse and tell you all about Gaussians,or the normal distribution, and the reason is so far, we've just encountered discrete distributions, and Gaussians will be the first example of a continuous distribution.Many of you know that a Gaussian is described by an identity that looks as follows,where the mean is called mu, and the variance is called sigma or sigma squared.And for any X along the horizontal access, the density is given by the following function:1 over square root of 2 pi times sigma, and then an exponential functionof minus ½ of x - mu squared over sigma squared.This function might look complex, but it's also very, very beautiful.It peaks at X = mu where the value in the exponent becomes 0.And towards plus or minus infinity, it goes to 0 quickly.In fact, exponentially fast.The argument inside is a quadratic function.The exponential function makes it exponential.And this over here is a normalizer to make sure that the area underneathsums up to one, which is characteristic of any probability density function.If you map this back to our discrete random variables, for each possible X, we can now assign a density value, which is the function of this, and that's effectivelythe probability that this X might be drawn.Now, the space itself is infinite, so any individual value will have a probability of 0,but what you can do is you can make an interval, A and B, and the area underneath this function is the total probability that an experiment will come up between A and B.Clearly, it's more likely to generate values around muthen it is to generate values in the periphery summary over here.And just for completeness, I'm going to give you the formula for what's called the multi-variate Gaussianwhere multi-variate means nothing else but we have more than one input variable.You might have a Gaussian over a 2-dimensional space or a 3-dimensional space.Often, these Gaussians are drawn by what's called level sets, sets of equal probability.Here's one in a 2-dimensional space, X1 and X2.The Gaussian itself can be thought of as coming out of the paper towards mewhere the most likely or highest point of probability is the center over here.And these rings measure areas of equal probability.The formula for a multi-variate Gaussian looks as follows:N is the number of dimensions in the input space.Sigma is a covariance matrix that generalizes the value over here.And the inner product inside the exponentialis now done using linear algebra where this is the difference betweena probe point and the mean vector mu transposed sigma to the minus 1 times X - mu.You can find this formula in any textbook or web page on Gaussians or multi-variate normal distributions.It looks cryptic at first, but the key thing to remember isit's just a generalization of the 1-dimensional case.We have a quadratic area over here as manifested by the product of this guy and this guy.We have a normalization by a variance or covariance as shown by this number over here or the inverse matrix over here.And then this entire thing is an exponential form in both cases,and the normalizer looks a little more different in the multi-variate case,but all it does is make sure that the volume underneath adds up to 1to make it a legitimate probability density function.For most of this explanation, I will stick with 1-dimensional Gaussians,so all you have to do is to worry about this formula over here,but this is given just for completeness.I will now talk about fitting Gaussians to data or Gaussian learning.You may be given some data points, and you might worry aboutwhat is the best Guassian fitting the data?Now, to explain this, let me first tell you what parameters characterizes a Gaussian.In the 1-dimensional case, it is mu and sigma squared.Mu is the mean. Sigma squared is called the variance.If we look at the formula of a Gaussian, it's a function over any possible input X,and it requires knowledge of mu and sigma squared.And as before, I'm just restating what I said before.We get this function over here that specifies any probabilityfor a value X given a specific mu and sigma squared.Suppose we wish to fit data, and our data is 1-dimensional, and it looks as follows.Just looking at this diagram makes me believe that there's a high density of data points over hereand a fading density of data points over there,so maybe the most likely Gaussian will look a little bit like thiswhere this is mu and this is sigma.They are really easy formulas for fitting data to Gaussians,and I'll give you the result right now.The optimal or most likely mean is just the average of the data points.There's M data points, X1 to Xm.The average will look like this.The sum of all data points divided by the total number of data points.That's called the average, and once you calculate the average,the sigma squared is obtained by a similar normalizationin a slightly more complex sum.We sum the deviation from the mean and compute the average deviation to the square from the mean,and that gives us sigma squared.So, intuitively speaking, the formulas are really easy.Mu is the mean, or the average.Sigma squared is the average quadratic deviation from the mean, as shown over here.Now I want take a second to convince ourselves this is indeed the maximum likelihood estimateof the mean and the variance. Suppose our data looks like this--There's "M" data points.And the probability of those data points for any Gaussian model--mu and sigma squaredis the product of any individual of data likelihood--x,i. And if you plug in our Gaussian formula, you get the following--This is the normalizer multiplied "M" times where the square root is now drawn into the half over here,and here is our joint exponential.We took the product of the individual exponentialsand moved it up straight in here where it becomes a sum. So the best estimates for mu and sigma squaredare those that maximize this entire expression over herefor given data set X1 to Xm. So we seek to maximize this over the unknown parametersmu and sigma squared.And now I will apply a trick. Instead of maximizing this expression,I will maximize the logarithm of this expression.The logarithm is a monotonic function.So let's maximize instead the logarithmwhere this expression over here resolves to this expression over here.The multiplication becomes a minus sign from over here,and this is the argument inside the exponentwritten slightly differently,but pulling the 2 sigma squared to the left.So let's maximize this one instead. The maximum was obtained where the first derivative is zero.If we do this for our variable mu, we take the "log f" expression andcomplete the derivative for spectrum mu,we get following--This expression does not depend on mu at all, so it falls out.And we can still get this expression over here, which we've set to zero.And now we can multiply everything by sigma squared next to zero,and then bring the Xi to the right and the mu to the left.The sum over all "E" of the mu is mu equals sum over i, xi.Hence, we proved that the mean is indeed the maximum likelihood estimatefor the Gaussian.This is now easily repeated for the variance.If you compute the derivative of this expression over herewith respect to the variance,we get minus "m" over sigma, which happens to be the derivativeof this expression over here.Keep in mind that the derivative of a logarithm stresses internal argumenttimes by chain rule--the derivative of the internal argument,which if you work out becomes this expression over here.And this guy over here changes signs but becomes the following.And again, you move this guy to the left side,multiply by sigma cubed, and divide by "m".So we get the following result over here.You might take a moment to verify these steps over here,I was a little bit fast,but this is relatively straight forward mathematics.And if you will verify them,you will find that the maximum likelihood estimatefor sigma squared is the averagedeviation of data points from the mean mu.This gives us a very nice basis to fitGaussians to data points.So keeping these formulas in mind, here's a quick quiz,which I ask you to actually calculate the mean and variance for a data sequence. So suppose the data you observe is 3, 4, 5, 6, and 7. There is 5 data points. Compute for me the mean and the varianceusing the maximum likelihood estimator I just gave you.So the mean is obviously 5,it's the middle value over here. If I add those things together, I get 25 and divide by 5.The average value over here is 5.The more interesting case is sigma square,and I do this in the following steps--I subtract 5 from each of the data pointsfor which I get -2, -1, 0, 1, and 2.I square those differences,which gives me 4, 1, 0, 1, 4.And now I compute the mean of those square differences.To do so, I add them all up, which is 10.10 divided by 5 is 2, and sigma square equals 2.Here is another quiz--Suppose my DATAlooks as follows--3,9,9,3.Compute for me mu and sigma squaredusing the maximum likelihood estimator I just gave you.And the answer is relatively easy.3 + 9 + 9 + 3 = 24 divided by m = 4 is 6so the mean value is 6subtracting the mean from the data gives us -3, 3, 3, and -3squaring those gives us 9, 9, 9, 9and the mean of 4 nines equals 9.I now have a more challenging quiz for youin which I give you multivariant datain this case 2-dimensional data.So suppose my data goes as follows.In the first column I get 3, 4, 5, 6, 7these are 5 data points and this is the first featureand the second feature will be 8, 7, 5, 3, 2.The formulas for calculating Mu and the covariance matrix Sigmageneralize the ones we studied beforeand they are given over here.So what I would like you to compute is the vector Muwhich now has 2 valuesone for the first and one for the second columnand the variance Sigmawhich now has 4 different valuesusing the formula shown over here.Now the mean is calculated as beforeindependently for each of the 2 features here.Three, 4, 5, 6, 7, the mean is 5.Eight, 7, 5, 3, 2, the mean is 5 again.Easy calculation. If you subtract the meanfrom the data we get the following matrixand now we just have to plug it in.For the main diagonal elements you get the same formula as before.You can do this separately for each of the 2 columns.But for the off-diagonal elements you just have to plug it in.So this is the result after plugging it inand you might just want to verify it using a computer.So this finishes the lecture on Gaussians.You learned about what a Gaussian is.We talked about the fit from dataand we even talked about multivariate Gaussians.But even though I asked you to fit one of thosethe one we are going to focus on right nowis the one-dimensional Gaussian.So let's now move back to the expectation maximization algorithm.It is now really easy to explain expectation maximizationas a generalization of K-means.Again, we have a couple of data points hereand 2 randomly chosen cluster centers.But in the correspondence step instead of making a hard correspondence we make a soft correspondence. Each data point is attracted to a cluster centerin proportion to the posterior likelihoodwhich we will define in a minute.In the adjustment step or the maximization stepthe cluster centers are being optimized just like beforebut now the correspondence is a soft variableand they correspond to all data points in different strengthsnot just the nearest ones.As a result, in EM the cluster centerstend not to move as far as in K-means.Their movement is smooth away.A new correspondence over here gives us different strengthas indicated by the different coloring of the linksand another relaxation step gives us better cluster centers.And as you can see over time, graduallythe EM will then converge to about the same solution as K-means.However, all the correspondences are still alive.Which means there is not a 0, 1 correspondence.There is a soft correspondence which relates to a posterior probability, which I will explain next.The model of expectation maximizationis that each data pointis generated from what's called a mixture.The sum of all possible classesor clusters, of which there are Kwe draw a class at randomwith a prior probability of p of the class C = iand then we draw data point Xfrom the distribution correspondent with its class over here.The way to think about this if there is K different cluster centers shown over hereeach one of those has a generic Gaussian attached.In the generative version of expectation maximizationyou first draw a cluster centerand then we draw from the Gaussian attached to this cluster center.The unknowns here are the prior probabilities for each cluster centershould we call P-i and the Mu-i and in the general case Sigma-ifor each of the individual Gaussian.Where i = 1 all the way to K.Expectation maximization iterates 2 steps just like K-means.One is called the E-step or expectation stepfor which we assume that we know the Gaussian parameters and the P-i.With those known values calculating the sum over hereis a fairly trivial exercise.This is our known formula for a Gaussianwe just plug that in and this is a fixed probability.The sum of all possible classes.So you get for e-ijthe probability that the j-th data pointcorresponds to cluster center number iP-i times the normalizertimes the Gaussian expression.Where we have a quadratic of Xj minus Mu-itimes Sigma-i to the -1 times the same thing again over here.These are the probabilitiesthat the j-th data pointcorresponds to the i-th cluster centerunder the assumption that we do knowthe parameters P-i, Mu-i, and Sigma-i.In the M-step we now figure out where these parameters should have been.For the prior probability of each cluster centerwe just take the sum over all the e-ijs, over all data pointsdivided by the total number of data points.The mean is obtained by the weighted mean of the x-jsnormalized by the sum over e-ijsand finally the sigma is obtained as a sumover the weighted expression like thisand this is the same expression as beforeand now again we are normalizing over the sum over all e-ijs.And these are exactly the same calculationsas before when we fit a Gaussian but just weighted by the soft correspondence of a data point to each Gaussian.And this weighting is relatively straightforward to apply in Gaussian fitting.Let's do a very quick quiz for EM.Suppose we're given 3 data points and 2 cluster centers.And the question is, does this point over herecalled X1 correspond to C1 or C2 or both of them?Please check exactly one of these 3 different check boxes here.[Thrun] And the answer is both of them,and the reason is X1 might be closer to C2 than C1,but the correspondence in EM is soft,which means each data point always corresponds to all cluster centers.It is just that this correspondence over here is much stronger than the correspondence over here.[Thrun] Here is another EM quiz.For this quiz we will assume a degenerative case of 3 data points and just 1 cluster center.My question pertains to the shape of the Gaussian after fitting,specifically M1 sigma.And the question is, is sigma circular, which would be like this,or elongated, which would be like this or like this?[Thrun] And the answer is, of course, elongated.As you look over here, what you find is that this is the best Gaussian describing the data points,and this is what EM will calculate.[Thrun] This is a quiz in which I compare EM versus K-means.Suppose we are giving you 4 data points, as indicated by those circles.Suppose we have 2 initial cluster centers, shown here in red,and those converge to possible places that are indicated by those 4 squares.Of course they won't take all 4 of them; they will just take 2 of them.But for now I'm going to give you 4 choices.We call this cluster 1, cluster 2, A, B, C, and D.In EM will C1 move towards A or will C1 move towards B?And in contrast, in K-means will C1 move towards A or will C1 move towards B?This is just asking about the left side of the diagram.So the question is will K-means find itself in the more extreme situation,or will EM find itself in the more extreme situation?[Thrun] And the answer is that while K-means will go all the way to the extreme, A,which is this one over here, EM will not.And this has to do with the soft versus hard nature of the correspondence.In K-means the correspondence is hard.So after the first situation, only these 2 data points over here correspond to cluster center 1,and they will find themselves straight in the middle where A is located.In EM, however, we find that there will still be a soft correspondenceto these further away points which will then lead to a small shift of the cluster centerto the right side, as indicated by B.That means K-means and EM will converge at different models of the data.[Thrun] One of the remaining open questions pertains to the number of clusters.So far I've assumed it's simply constant and you know it.But in reality, you don't know it.Practical implementations often guess the number of clusters along with the parameters.And the way this works is that you periodically evaluate which data is poorly coveredby the existing mixture, you generate new cluster centers at random near unexplained points, and then you run the algorithm for a whileto see whether the existence of your clusters is still justified.And the justification test is based on a memorization of a criterionthat combines the negative log likelihood of your data itselfand a penalty for each cluster.In particular, you're going to minimize the negative log likelihood of your datagiven the model plus a constant penalty per cluster.If we look at this expression, this is the expression that EM already minimizes.We maximized the posterior probability of datalogarithmic is a monotonic function, and I put a minus sign over hereso the optimization problem becomes a minimization problem.This one over here, we have a constant cost per cluster is new.If you increase the number of clusters, you would pay a penaltythat is in the way of your attempted minimization.Typically, this expression balances out at a certain number of clusters,and it is generically the best explanation for your data.So the algorithm looks as follows.Guess an initial K, run EM, remove unnecessary clustersthat will make this quote over here go up,create some new random clusters, and go back and run EM.There is all kinds of variants of this algorithm.One of the nice things here is this algorithm also overcomes local minima problemsto some extent.If, for example, 2 clusters end up grabbing the same data,then your tests would show you that 1 of the clusters can be omitted;thereby the score can be improved.That cluster can later be restarted somewhere else,and by randomly restarting clusters, you tend to get a much, much better solutionthan if you run EM just once with a fixed number of clusters.So this trick is highly recommended for any implementation of expectation maximization.[Thrun] This finishes my unit on clustering,at least so far.I just want to briefly summarize what we've learned.We talked about K-means, and we talked about expectation maximization.K-means is a very simple almost binary algorithmthat allows you to find cluster centers.EM is a probabilistic generalization that also allows you to find clustersbut also modifies the shapes of the clusters by modifying the covariance matrix.EM is probabilistically sound, and you can prove convergence in a log likelihood space. K-means also converges.Both are prone to local minima.In both cases you need to know the number of cluster centers, K.I showed you a brief trick how to estimate the K as you go,which also overcomes local minima to some extent.Let's now talk about a 2nd class of unsupervised learning avenues that are called dimensionality reduction.We're going to start with a little quiz, in which I will check your intuition.Suppose we're given a 2-dimensional data field, and our data lines up as follows.My quiz is: How many dimensions do we really need?The key is the word really,which means we're willing to tolerate a certain amount of error in accuracy,because we're going to capture the essence of the problem.The answer is obviously 1.This is the key dimension over here.The orthogonal dimension in this direction carries alomst information,so it suffices, in most cases, to project the data onto this 1 dimensional space.Here is a quiz that is a little bit more tricky.I'm going to draw data for you like this.I'm going to ask the same question.How many dimensions do we really need?This answer is not at all trivial, and I don't blame you if you get it wrong.The answer is actually 1, but the projection itself is nonlinear.I can draw, really easily, a nice 1-dimensional space that follows these data points.If I am able to project all the data points on this 1-dimensional space,I capture the essence of the data.The trick, of course, is to find the nonlinear 1-dimensional space and describe it.This is what's going on in the state-of-the-art in dimensionality reduction research.For the remainder of this unit,I am going to talk about linear dimensionality reduction.Where the idea is that the given data points like this, and we seek to find a linear subspace in which to perfect the data.In this case, I would submit this is probably the most suitable linear subspace.So we remap the data onto the space over here, with x1 over here and x2 over here.Then we can capture the data in just 1 dimension.The algorithm is amazingly simple.Number 1: Fit a gaussian; we now know how this works.The gaussian will look something like this.Number 2: Caluclate the eigenvalues and eigenvectors of this gaussian.In this gaussian this would be the dominant eigenvector,and this would be the 2nd eigenvector over here.Step 3 is take those eigenvectors whose eigenvalues are the largest.Step 4 is to project the data onto the subspace of eigenvectors you chose.Now to understand this, you have to be familiar with eigenvectors and eigenvalues.I give you an intuitive familiarity with those.This is standard statistics material, and you will find this in many linear algebra classes.So let me just go through this very quickly and give you an intuition how to do linear dimensionality reduction.Suppose you're given the following data points:Your axes are 0, 1, 2, 3, and 4, 4 x1, and 1.9, 3.1, 4, 5.1, and 5.9.These are essentially 2, 3, 4, 5, 6,but slightly modified to define actual variance over this dimension.So I draw this in here.What I get is a set of points that doesn't quite fit a line, but almost.There is a little error over here, a little error over here, and here and here.The mean is easily calculated; it's 2 and 4.The covairance matrix looks as follows.Notice the slightly different covairance for the 1st variable, which is exactly 2,to the 2nd variable, which is 2.008.The eigenvectors happen to be 0.7064 and 0.7078 with an eigenvalue of 4.004, and the 2nd one is orthogonal with an eigenvalue much smaller.So obviously this is the eigenvector that dominates the spread of the data points.If you look at this vector over here, it is centered around the mean, which sits over here, and is exactly this vector shown over here.Where this one is the orthogonal vector shown over here.So this single dimension with a large weight explains the data relative to any other dimension, which is a very small eidenvalue.I should mention why these numerical examples might look confusing.This is very standard linear algebra.When you estimate covariance from data and try to understand which direction they point,this kind of eigenvalue anylysis gives you the right answer.The dimensionality reduction looks a little bit silly when you go from 2 dimensions to 1 dimension.But in truly high-dimensional space it has a very strong utility.Here's an example that goes back to MIT several decades ago on something called eigenfaces.These are all well-aligned faces.The objective in eigenface research has been to find simple ways to describe different people in a parameter space,in which we can easily identify the same person again.Images like these are very high-dimensional statistics.If each image is 50 by 50 pixels,each image itself becomes a data point in a 2500 dimensional feature space.Now obviously, we don't have random images.We don't fill the space of 2500 dimensions with all face images.Instead, it is reasonable to assume that all the faces live on a small subspace in that space.Obviously, you as a human can easily distinguish what is a valid image of a faceand what is a valid image of a non face, like a car or a cloud or the sky.Therefore, there are many, many images that you can represent with 2500 pixels that are not faces.So research on eigenfaces has appliedprinciple component analysis and eigenvalues to the space of faces.Here is a database in which faces are aligned.A researcher at Santiago Serrano extracted from it the average face after alignment on the right side.The truly interesting phenomenon occurs when you look at the eigenvalues.The face on the top left, over here, is the average face, and these are the variations,the eigenvectors that correspond to the largest eigenvalues over here.This is the strongest variation.You see a certain amount of different regions in and around the head shapeand the hair that gets excited.That's the 2nd strongest one, where the shirt gets more excited.As you go down, you find more and more interesting variations that can be used to reconstruct faces.Typically a dozen or so will suffice to make a face completely reconstructable, which means you've just mapped a 2500 dimensional feature space into a, perhaps, 12 dimensional feature spaceon which we can now learn much, much easier.In our own reserch, we also have applied eigenvector decompositionto relatively challenging problems that don't look like a linear problem at the surface.We scanned a good number of people with different physiques:Some thin, some not so thin, some tall, some short, some male, some female.We also scanned them in 3-D in different body postures:The arms down, the arms up, walking, throwing a ball, and so on.We applied eigenvector decomposition of the type I've just shown youto understand whether there is a latent low-dimensional spacethat is sufficient to represent the different physiques that people have,like thin or thick, and the different postures people can assume, like standing and so on.It turns out if you apply eigenvector decomposition to the space of all the formations of your body,you can find relatively low dimensional linear spaces,in which you can express different physiques and different body postures.For the space of all different physiques it turns only 3-dimensions sufficedto explain different heights, different thicknesses or body weights,and also different genders.That is, even though our surfaces themselves are representiveof tens of thousands of data points, the underlying dimensionalitywhen scanning people is really small.I'll let you watch the entire movie.Please enjoy.[SCAPE: Shape Completion and Animation of People]We present a method named SCAPE for simultaneously modelingthe space of all human shapes and poses.Further, we demonstrate the method's usefulnessfor both shape completion and animation.The model is computed from an example set of surface meshes.We require only a limited set of training data:Examples of posed variation from a single subjectand examples of the shape variation between subjects.The resulting model can represent both articulated motionand, importantly, the nonrigid muscle deformationsrequired for natural appearance in a wide variety of poses. The model can also represent a wide variety of different body shapes, spanning both men and women.Because SCAPE incorporates both shape and posewe can jointly vary both shape and pose to create people who never existed and poses that were never observed.We demonstrate the use of this model 1st for shape completion of scanned meshes.Even when a subject has only been partially observed, we can use the model to estimate a complete surface.In this case, the entire front half of the subject has been synthesized.Note that the synthesized data both conforms to the individual subject's specific shape and faithfully representsthe nonrigid muscle deformations associated with a specific pose.Mesh completion is possible even whenneither the person or the pose exists in the original training set. None of the women in our example setlook similar to the woman in this sequence.Shape completion can also be used to synthesize completeanimated surface meshes.Starting from a single scanned mesh of an actorand a timed series of motion capture markerswe can treat the markers themselvesas a very sparse sampling of surface geometryand complete the surface which best fits the available data at each point in time.Using this method, animated surface modelsfor a wide variety of motions can be created with relative ease.In addition, the target identity of the surface model can easily be changedsimply by replacing the subject portion of our factorized model with a different vector.The new identity need not be present in our training set or even correspond to a real person.An artist is free to alter the identity arbitrarily.[Thrun] In modern dimensionality reduction, the trick has been to define nonlinear,sometimes piece-wise linear, subspaces on which data is being projected.This is not dissimilar from K nearest neighbors,where local regions are being defined based on local data neighborhoods.But here we need ways to interpret leveraging neighborsto make sure that the subspace itself becomes a feasible subspace.Common methods include local linear embedding, or LLE, or the Isomap method.If you're interested in this, check the Web.There's tons of information on these methods on the World Wide Web.We now talk about spectral clustering.The fundamental idea of spectral clusteringis to cluster by affinity.And to understand the importance of spectral clustering,let me ask you a simple intuitive quiz.Suppose you are given data like this,and you wish to learn that there's 2 clusters--a cluster over here and a cluster over here.So my question is, from what you understand,do you think that "EM" or "K" meanswe would do a great job finding those clustersor do you think they will likely fail to find those clusters? So what were the questions--Do "EM" or "K"mean succeed in finding the 2 clusters?There is a likely yes and a likely no.And the answer is likely no.The reason being that these aren't clustersdefined by a center of data points,but they're clusters define by affinity,which means they're defined by the presence of nearby points.So take for example the area over here, which I'm going to circle,and ask yourself, what's the best cluster center?It's likely somewhere over here where I drew the red dot.This is the cluster center for this cluster,and perhaps this is the cluster center for the other cluster.And these points over here will likely be classified as belonging to the cluster center over here.So, "EM" will likely do a bad job.So let's look at this example again--let me redraw the data.What makes these clusters so differentis not the absolute location of each data point,but the connectedness of these data points.The fact that these 2 points belong together is likely because there's lots of points in-between. In other words, it's the affinitythat defines those clusters, not the absolute location.So spectral clustering gets annotation of affinityto make clustering happen.So let me look at the simple example for spectral clusteringthat would also work for K-means or EM,but they'll be useful to illustrate spectral clustering.Let's assume there's 9 data points as shown over here,and I've colored them differently in blue, red, and black.But to clustering algorithms, they all come with the same color.Now the key element of spectral clustering is called the affinity martrix,which is a 9 by 9 matrix in this case,where each data point gets graphedrealtive to each other data point.So let me write down all the 9 data pointsinto the different rows of this matrix--the red ones, the black ones, and the blue ones.And in the columns, I graphed the exact same 9 data points.I then calculate for each pair of data points their affinity,where I use for now affinity as the quadratic distance in this diagram over here.Clearly, the red dots to each other have a high affinity,which means a small quadratic distance. Let me indicate this as follows--But realtive to all the other points, the affinity is weak.So there's a very small value in these elements over here.Similarly, the affinity of the blackdata points to each other is very high,which means that the following block diagonalin this matrix will have a very large value.Yet the affinity to all the other data points will be low.And of course, the same is true for the blue data points.The interesting thing to notice nowis that this is an approximately rank-deficient matrix.And further, the data points that belong to the same class--like the 3 red dots or the 3 black dots, have a singular affinitive vector to all the other data points. So this vector over here is similar to this vector over here.It's similar to this vector over here,but it's very different to this vector over here,which then itself is similar to the vector over here,yet different to the previous ones.Such a situation is easily addressed by what's called principal component analysis, or PCA.PCA is a method to identify vectors that are similarin an approximate rank-deficient matrix. Consider once again our affinity matrixwith prinicple component analysis,which is a standard linear trick,we can re-represent this matrixby the most dominant tivectors you'll find there.And the first one, might look like this.The second one, which would be orthogonal, may look like this. The third one, like this.These are called eigenvectors, and the principle componentnow is each eigenvector has an item of valuethat states how prevalent this vector is in the original data.And for these 3 vectors, you're going to find a large eigenvaluebecause there's a number data points that represent these vectors quite prevalently like the first 3 does for this guy over here.There might be additional eigenvectors like something like this,but such eigenvectors will have a small eigenvaluesimply because this vector isn't really required to explain the data over here.It might just be explaining some of the noisein the affinity matrixthat I didn't even dare draw in here.Now if you take the eigenvectors with the largest eigenvalues--3 in this case,you first discover that the dimensionality of the underlying data space.The dimensionality equals the number of large eigenvalues.Further, if you re-represent each data vectorusing those eigenvectors, you'll find a 3 dimensional space where original data falls into a varity of different places.And these places are easily told apart by conventional clustering.So in summary, spectral clustering builds an affinity matrix of the data points.It strikes the eigenvectors with the largest eigenvalues,and then re-map those vecotrs into a new space with the data points easily clustering the conventional way. This is called affinity-based clustering or spectral clustering.Let me illustrate this once again with thedata set that has a different spectral clusteringthan a conventional clustering. In this data set, the different clusters belong together because they're affinity is similar.These 2 points belong togetherbecause there is a point in-between.If we now draw the affinity matrix for those data points, you find that the first and second data points are close togetherand the second and the third, but not the first and the third.Hence these 2 off diagonal elements here have remained small.Similarly for the red points as shown herewith these 2 elements over here relatively small.And also for the black pointswhere these 2 elements over here are small. And interestingly enough, even though these aren't blocked diagonal, your first 3 largest eigenvectorswill still look the same as before.I find this quite remarkablethat even though these aren't exactly blocks,those vecotrs still represent the 3 most important vectors for which to recover the data using principle component analysis.So in this case, spectral clustering would easilyassign those guys and those guys and those guysto the respective same cluster,which wouldn't be quite as easily the case for expectation-maximization or k-means.So let me ask you the following quiz.Suppose we have 8 data points.How many elements will the affinity matrix have?And the answer is 64.There's 8 data points--8 times 8 is 64.My second question is, how many large eigenvalues will PCA find?Now I understand this doesn't have a unique answer,but in the best possible casewhere spectral clustering works well,how many large eigenvalues do you find?And the answer is 2.There's a cluster over here and a cluster over here.And while it might happen that it's as many as 8,if you adjust you're affinity matrix well,those 2 should correspond with the 2 larger eigenvalues. So, congratulations.You just made it through the unsupervised learning section of this class.I think you've learned a lot.You learned about K-means, you learned about expectation maximization,about dimensionality reduction and even spectral clustering.The first 3 items--K-means, EM, and dimensionality reduction--are used very frequently, and spectral clustering is a rarer used methodthat shows some of the most recent research going on in the field.I hope you have fun applying these methods in practice.I'd like to say a few final words about supervised versus unsupervised learning.In both cases you're given data, but in 1 case you have labeled data,in another you have unlabeled data.The supervised learning paradigm is the dominant paradigm in machine learning,and there are a vast amount of papers being written about it.We talked about classification and regressionand different methods to do supervised learning.The unsupervised paradigm is much less explored,even though I think it's at least equally important--possibly even more important.Many systems can collect vast amounts of data such as web crawlers,robots, I told you about street view,and getting the data is cheap, but getting labels is hard.So to me, unsupervised is the method of the future.It's one of the most interesting open research topicsto see whether we can make sense out of large amounts of unlabeled or poorly labeled data.In between, there are techniques that do both: supervised and unsupervised.They are called semi-supervised or self-supervised,and they use elements of unsupervised learning and pair them with supervised learning.Those are fascinating by their own rights.Our robot Stanley, for example, that won the DARPA Grand Challengeused its own sensors to produce labels on the fly to other data.And I'll talk about this when I talk about robotics in more detail.But for the time being, understand that the paradigms supervised and unsupervisedspan 2 very large areas of machine learning, and you learn quite a bit about it.Welcome to the third homework assignment covering topics of machine learning.[Thrun] This question is about naive Bayes and Laplacian smoothing.Our training data is a set of movie titles: A Perfect World,My Perfect Woman, and Pretty Woman.We also have a song class of song titles: A Perfect Day, Electric Storm,Another Rainy Day.Suppose we get a new title, the query Perfect Storm,and we wish to know whether Perfect Storm is more likely a movie or a song.Compute for me the following model probabilities:the probability for movie class and song class,the probability of the word "perfect" conditioned on the movie class,the probability of the word "perfect" conditioned on the song class,and the same for the word "storm."Please use Laplacian smoothing for this with K equals 1.Don't compute the maximum likelihood estimate.[Thrun] Remember in Laplacian smoothing our best estimate is the count of the occurrence of the words divided by N,but we add our Laplacian smoother over here, and down here we add K times number of classes.For the movie prior we have 3 examples of movie titles over 6 total titles,which gives us 3 over 6.We add our Laplacian prior, 1 over here.There's 2 classes, movie and song, 2 over here.We get 4 over 8, which is a half.The same is the case for song.It gets more interesting for this probability over here.In our movie class there's 2 occurrences of the word "perfect" out of 8 words,so we get 2 over 8.But in adding the Laplacian prior, 1 over hereand 1 number to add down here,the number of classes here is the size of the vocabulary.In total for this model there is 11 different words.There are 16 total words in both titles,but because of repetition there's only 11 distinct words:a, perfect, world, my, woman, pretty, day, electric, storm, another, rainy.So we add the number of classes over here, which is 11.We obtain 3 over 19.For the song class there's 1 occurrence of perfect.Adding 1 we get 2 over 19.There's no occurrence of storm in the movie class.However, our Laplacian prior gives us 1 over 19.And there's 1 occurrence of storm over here, which gives us 2 over 19.[Thrun] For the same example I now would like to know the probability of movie title for my query.So please write this into the following box.[Thrun] As usual, we can resolve this using Bayes' rule.Probability of Perfect Storm given movie times P of moviedivided by the same expression plus this expression for the opposite class, song.Here I simply write 3 dots for the text Perfect Storm.Plugging in the values over here and assuming conditional independence,as is the case when I use Bayes, we get the probably of "perfect" given movie,which is 3/19 and "storm" given movie, 1/19, times the prior over half,and we divide this by the same number plus probability of "perfect" given song,which is 2/19, and the probability of "storm" given song, which is 2/19 times the prior of half.Now, all the enumerators fall out, and we get 3 over 3 plus 2, which is 3 over 7.[Thrun] I would now like to ask the exact same questionfor the maximum likelihood estimator.So let's not assume we have Laplacian smoothingand instead use the maximum likelihood estimator.Simply compute for me the probability of movie for the title Perfect Storm.[Thrun] And the answer is simply 0, without much math.The word "perfect" occurs in movie, but the word "storm" has never been seen before.Therefore, the maximum likelihood estimate we will assign a 0 probability to the word "storm,"which will make the total product of the various factors involved in "storm" just 0.That is not the case for song.There is a non-zero probability for "perfect" and a non-zero probability for "storm."Hence, it will have a non-zero probability.After normalization this will become 1 and this will become 0.So without much math I can calculate the correct posterior under the maximum likelihood model, which of course is disappointingbecause Perfect Storm is actually a movie title.[Thrun] In this question I quiz you about linear regression.Given the following data, my first question is, can this data be fit exactly using a linear function that maps from X to Y?Yes or no.[Thrun] And the answer is no.To see, let's look at the slope of the linear function if it existed.From 0 to 1 we increment Y by 3.We go from 3 to 6.Therefore, the slope of it must be 3.However, from 1 to 2 we only increase the function by 1, from 6 to 7.Therefore, it can't be fit linearly.We can see the same if we plot the linear points.Over here we could fit a linear function, but it's very shallow,whereas those points over here have a much steeper situation.So any linear function would probably miss these points in between.[Thrun] I would now like to ask you to perform linear regression on these data pointsand calculate for me W0 and W1.As defined in this class, we might have to go backand look over the exact formula from the lecture that I taught on linear regression.[Thrun] For answering these questions, let me restate the essential formulas.W1 is obtained by M times sum of XY minus sum of X times sum of Yover M times sum Xi square minus sum of Xi in brackets square.And if you plug in these numbers over here for M equals 5because there's 5 training examples, we get 5 times 88 minus 10 times 35over 5 times 30 minus 100, which is 1.8.That is the correct answer for W1.W0 was obtained by 1 over M times sum over Ys minus W1 over M times sum over X.And plugging in the table over here gives us 1/5 times 35 minus 1.8 over 5 times 10,and that is 3.4, which would have been the correct answer over here.And again here are the data points with the solution.So if you take the axis where X equals 0,the Y value is actually 3.4, and the slope is 1.8.It's a little smaller than if you just click at the end points,which gave us a slope of 2, because there is a residual arrow over here,residual arrow over here, residual arrow over here, and a residual arrow over here.The resulting linear function ends up splitting in a quadratically optimal waythe arrows between these different data points.In my next question I would like to ask you about K-nearest neighbors.Consider the following data setwhere plus indicates a positive traning exampleand minus a negative training example in this 2-dimensional space.I want you, for the following places,to check for those boxes over herewhether they will be plus for K=5.Only check those boxes for which the label will be positive.And the answer would be this box over hereand this box over here, nothing else.This guy has clearly 4 positive nearest neighbors,so no matter what the 5th one is we stay positive.Similarly, this guy over here,when you draw a circle,has probably these 4 guys as nearest neighbors,perhaps this one as well, but it is a little further away.With those 4 ones, it already has 3 pluses, so whatever the 5th one is it can't overturn, it must be positive.All of these are negative, even this one over here has just 2 pluses as neighborsthat are positive, and those guys over here are all negative.Similarly, over here there are possibly 2 plusesin the 5 nearest neighbors, but these guys over here are all negative, and this guy issurrounded by negative examples, so they will just be negative.Here's another nearest neighbor example, and now I'm going to ask a different question.Given all the black data points,I want to make sure that the red ones are classified as indicatedand I am free to choose a different value for K.Say I can choose K to be 1, 3, 5, 7, or 9.Check any or all of the K valuesfor which you believe these 3 data points are classified correctly relative to the black training data set.And the answer is just 5.If you look carefully for K=1,this guy will be mis-classified.It's closer to a plus than a minus.Similarly, for K=3, this guy has 2 nearby pluses and 1 minus,so it would be positive.For K=5, to get the correct answer,the 5 nearest neighbors of this guy are those 3 minuses plus perhaps those 2 pluses over here.This guy has in his 5 neighborhood these 3 pluses over here, plus a minus, plus a plus.This data point over here has 2 pluses with 3 of the surrounding minuses,and they are all classified correctly.For K=7, this minus data point will have4 pluses, 3 minuses over here,and then everything in the vicinity becomes positive,so it must be 4 plus and it will be mis-classified.The same is true for K=9.The minus over here will have 1, 2, 3 minuses,5 pluses, and the minus over here,which makes 4 minuses.It will be classified as positive.So K=5 would have been the only correct answer.But nobody asked about the perceptron algorithm, suppose you have the following 2 dimensional linear set where plus indicatesa positive class label and minus a negative class label, and my first question is, "Are these data linearly separable?" I'd also like to know if we start perceptron with an initial separating plane like this,will it actually converge?Please check the appropriate boxes yes or no, and yes or no.[Narrator] And the answer is yes in both cases.There is a linear separation that goes along here that separates the positive class from the negative class,and it's been shown in the 60s,field perceptron algorithm always converges after finally many stepsif such linear separator exists. I'm not going to prove this, and I did prove thisin class, but I clearly said this in class.Congratulations! You just finished the third homework assignment.Welcome back.So far we've talked about AI as managing complexity and uncertainty.We've seen how a search can discover sequences of actions to solve problems.We've seen how probability theorycan represent in reason with uncertainty.And we've seen how machine learningcan be used to learn and improve.AI is a big and dynamic fieldbecause we are pushing against complexityin at least 3 directions.First, in terms of agent design,we start with a simple reflex-based agentand move into goal-based and utility-based agents.Secondly, in terms of the complexity of the environment, we start with simple environments and then start looking at partial observability,stochastic actions at multiple agents, and so on.And finally, in terms of representation, the agents model of the world becomes increasingly complex.And this unit will concentrate on that third aspect of representation,showing how the tools of logiccan be used by an agent to better model the world.The first logic we will consider is called propositional logic.Let's jump right into an example, recasting the alarm problem in propositional logic.We have propositional symbols B, E, A, M, and Jcorresponding to the events of a burglary occurring, of\ the earthquake occurring,of the alarm going off, of Mary calling, and of John calling.And just as in the probabalistic models, these can be either true or false,but unlike improbability, our degree of belief in propositional logic is not a number. Rather, our belief is that each of these is either true or false or unknown.Now, we can make logical sentences using these symbolsand also using the logical constants true and falseby combining them together using logical operators.For example, we can say that the alarm is truewhenever the earthquake or burglary is true with this sentence.(E V B) E or B implies A.So that says whenever the earthquake or the burglary is true, then the alarm will be true.We use this V symbol to mean or and a right arrow to mean implies. We could also say that it would be true that both John and Mary call when the alarm is true. We write that as A implies (J ^ M)and we use this symbol ^ to indicate an and,so that this upward-facing wedge looks kind of like an A with the crossbar missing, and so you can remember A is for "and"where with this downward-facing V symbol is the opposite of and,so that's the symbol for or.Now, there's 2 more connectors we haven't seen yet.There's a double arrow for equivalent, also known as a biconditional,and a not sign for negation,so we could say if we wanted to that John calls if and only if Mary calls.We would write that as J <=> M.John is equivalent to Mary--when one is true, the other is true;when one is false, the other is false. Or we could say that when John calls, Mary doesn't, and vice versa. We could write that as John is equivalent J<=> to not Mary,and this is the not sign.Now, how do we know what the sentences mean?A propositional logic sentence is either true or false with respect to a model of the world.Now, a model is just a set of true/false values for all the propositional symbols,so a model might be the set B is true, E is false, and so on.We can define the truth of the sentence in terms of the truth of the symbols with respect to the models using truth tables. [Male narrator] Here are the truth tables for all the logical connectives.What a truth table does is list all the possibilities for the propositional symbols,so P and Q can be false and false, false and true, true and false, or true and true.Those are the only 4 possibilities,and then for each of those possibilities, the truth table lists the truth value of the compound sentence. So the sentence not P is true when P is false and false when P is true. The sentence P and Q is true only when both P and Q are true and false otherwise. The sentence P or Q is true when either P or Q is trueand false when both are false. Now, so far, those mostly correspond to the English meaning of those sentenceswith one exception, which is that in English, the word "or" is somewhat ambiguousbetween the inclusive and exclusive or,and this "or" means either or both.We translate this mark into English P implies Q; or as if P, then Q,but the meaning in logic is not quite the same as the meaning in ordinary English.The meaning in logic is defined explicitly by this truth tableand by nothing else, but let's look at some examples in ordinary English.If we have the proposition O and have that mean 5 is an odd numberand P meaning Paris is the capital of France, then under the ordinary model of the truth in the real world, what could we say about the sentence O implies P?That is, 5 is an odd number implies Paris is the capital of France.Would that be true or false?And let's look at one more example.If E is the proposition that 5 is an even numberand M is the proposition that Moscow is the capital of France,what about E implies M?5 is an even number implies Moscow is the capital of France.Is that true or false?[Male narrator] The answers are first, the sentence if 5 is an odd number, then Paris is the capital of France, is truein propositional logic.It may sound odd in ordinary English, but in propositional logic, this is the same as true implies trueand if we look on this line--the final line for P and Q, P implies Q is true.The second sentence, 5 is an even number, implies Moscow is the capital of France.That's the same as false implies false, and false implies false according to the definition is also true.[Male narrator] Here's a quiz.Use truth tables or whatever other method you wantto fill in the values of these tables.For each of the values of P and Q--false/false, false/true, true/false, or true/true--look at each of these boxes and click on just the boxesin which the formula for that column will be true.So which of these 4 boxes, if any, will this formula be true, and this formula and this formula?[Male narrator] Here are the answers.For P and P implies Q, we know that P is truein these bottom 2 cases, and P implies Q, we saw the truth table for P implies Qis true in the first, second, and fourth case.So the only case that's true for both P and P implies Q is the fourth case. Now, this formula, not the quantity, not P or not Q, can work that out to be the sameas P and Q, and we know that P and Q is true only when both are true, so that would be true only in the fourth case and none of the other cases.And now, we're asking for an equivalent or biconditional between these 2 cases.Is this one the same as this one?And we see that it is the same because they match up in all 4 cases.They're false for each of the first 3 and true in the fourth one,so that means that this is going to be true no matter what.They're always equivalent, either both false or both true, and so we should check all 4 boxes. [Male narrator] Here's one more example of reasoning in propositional logic. In a particular model of the world, we know the following 3 sentences are true.E or B implies A,A implies J and M,and B.We know those 3 senetences to be true, and that's all we know.Now, I want you to tell me for each of the 5 propositional symbols, is that symbol true or false, or unknown in this model,and tell me for the symbols E, B, A, J, and M.The answer is that B is true. And we know that because it was one of the 3 sentences that was given to us.And now, according to the first sentence, says that if E or B is true then A is true.So now we know that A is true.And the second sentence says if A is true then J and M are true. What about E? That wasn't mentioned.Does that mean E is false? No. It means that it is unknown that a model where E is true and a model where E is falsewould both satisfy these 3 sentences. So we mark E as unknown.Now for a little more terminology.We say that a valid sentence is one that is true in every possible model,for every combination of values of the propositional symbols.And a satisfiable sentence is one that is true in some models, but not necessarily in all the models. So what I want you to do is tell me for each of these sentences,whether it is valid, satisfiable but not valid, or unsatisfiable, in other words, false for all models.And the sentences are P or not P, P and not P, P or Q or P is equivalent to Q, P implies Q or Q implies P.And finally, Food implies Party or Drinks implies party implies Food and Drinks implies Party.The answers are P and not P is valid.That is, it's true when P is true because of this, and it's true when P is false because of this clause.P and not P is unsatisfiable.A symbol can't be both true and false at the same time.P or Q or P is equivalent to Q is valid. So we know that it's true when either P or Q is true, so that's 3 out of the 4 cases.In the fourth case, both P and Q are false, and that means P is equivalent to Q.And therefore, in all 4 cases, it's true.P implies Q or Q implies P, that's also valid.Now in ordinary English that wouldn't be valid. If the 2 clauses or the 2 symbols P and Q were irrelevant to each other we wouldn't say that either one of those was true. But in logic, one or the other must be true, according to the definitions of the truth tables.And finally, this one's more complicated,if Food then Party or if Drinks then Party implies if Food and Drinks then Party.You can work it all out and both sides of the main implication work out to be equivalent to Not Food or Not Drinks or Party.So that's the same as saying P implies P, saying one side is equivalent to the other side.And if they're equivalent, then the implication relation holds.Propositional logic. It's a powerful language for what it does.And there are very efficient inference mechanisms for determining validity and satisfiability, alhough we haven't discussed them.But propositional logic has a few limitations.First, it can only handle true and false values.No capability to handle uncertainty like we did in probability theory.And second, we can only talk about events that are true or false in the world. We can't talk about objects that have properties,such as size, weight, color, and so on.Nor can we talk about the relations between objects.And third, there are no shortcuts to succinctly talk about a lot of different things happening.Say if we had a vacuum world with a thousand locations, and we wanted to say that every location is free of dirt. We would need a conjunction of a thousand propositions.There's no way to have a single sentence saying that all the locations are clean all at once.So, we will next cover first-order logic which addresses these two limitations.[Norvig] I'm going to talk about first order logicand its relation to the other logics we've seen so far--namely, propositional logic and probability theory.We're going to talk about them in terms of what they say about the world,which we call the ontological commitment of these logics,and what types of beliefs agents can have using these logics,which we call the epistemological commitments.So in first order logic we have relations about things in the world,objects, and functions on those objects.And what we can believe about those relations is that they're true or false or unknown.So this is an extension of propositional logic in which all we had was facts about the world and we could believe that those facts were true or false or unknown.In probability theory we had the same types of facts as in propositional logic--the symbols or variables--but the beliefs could be a real number in the range 0 to 1.So logics vary both in what you can say about the worldand what you can believe about what's been said about the world.Another way to look at representationis to break the world up into representations that are atomic,meaning that a representation of the state is just an individual statewith no pieces inside of it.And that's what we used for search and problem solving.We had a state, like state A,and then we transitioned to another state, like state B,and all we could say about those states was are they identical to each other or notand maybe is one of them a goal state or not.But there wasn't any internal structure to those states.In propositional logic, as well as in probability theory,we break up the world into a set of facts that are true or false,so we call this a factored representation--that is, the representation of an individual state of the worldis factored into several variables--the B and E and A and M and J, for example--and those could be Boolean variables or in some types of representations--not in propositional logic--they can be other types of variables besides Boolean.Then the third type--the most complex type of representation--we call structured.And in a structured representation, an individual state is not just a set of values for variables,but it can include relationships between objects,a branching structure, and complex representations and relationsbetween one object and another.And that's what we see in traditional programming languages,it's what we see in databases--they're called structured databases,and we have structured query languages over those databases--and that's a more powerful representation,and that's what we get in first order logic.[Norvig] How does first order logic work? What does it do?Like propositional logic, we start with a model.In propositional logic a model was a value for each propositional symbol.So we might say that the symbol P was trueand the symbol Q was false,and that would be a model that corresponds to what's going on in a possible world.In first order logic the models are more complex.We start off with a set of objects.Here I've shown 4 objects, these 4 tiles,but we could have more objects than that.We could say, for example, that the numbers 1, 2, and 3were also objects in our model.So we have a set of objects.We can also have a set of constants that refer to those objects.So I could use the constant names A, B, C, D, 1, 2, 3,but I don't have to have a one-to-one correspondence between constants and objects.I could have 2 different constant names that refer to the same object.I could also have, say, the name C that refers to this object,or I could have some of the objects that don't have any names at all.But I've got a set of constants, and I also have a set of functions.A function is defined as a mapping from objects to objects.And so, for example, I might have the Number Of functionthat maps from a tile to the number on that tile,and that function then would be defined by the mapping from A to 1and B to 3 and C to 3 and D to 2,and I could have other functions as well.In addition to functions, I can have relations.For example, I could have the Above relation,and I could say in this model of the world the Above relation is a set of tuples.Say A is above B and C is above D.So that was a binary relation holding between 2 objects.Say 1 block is above another block.We can have other types of relations.For example, here is a unary relation--vowel--and if we want to say the relation Vowel is true only of the object that we call A,then that's a set of tuples of length 1 that contains just A.We can even have relations over no objects.Say we wanted to have the relation Rainy, which doesn't refer to any objects at allbut just refers to the current situation.Then since it's not rainy today, we would represent that as the empty set.There's no tuples corresponding to that relation.Or, if it was rainy, we could say that it's represented by a singleton set,and since the arity of Rainy is 0, there would be 0 elements in each one of those tuples.So that's what a model in first order logic looks like.[Man] Now let's talk about the syntax of first order logic,and like in propositional logic, we have sentences which describe facts that are true or false.But unlike propositional logic, we also have termswhich describe objects. Now, the atomic sentences are predicates corresponding to relations,so we can say vowel (A) is an atomic sentenceor above (A, B).And we also have a distinguished relation--the equality relation.We can say 2 = 2 and the equality relation is always in every model,and sentences can be combined with all the operators from propositional logicso that's and, or, not, implies, equivalent, and parentheses.Now, terms, which refer to objects, can be constants, like A, B, and 2. They can be variables. We normally use lowercase, like x and y.And they can be functions, like number of A,which is just another name or another expression that refers to the same object as 1,at least in the model that we showed previously.And then, there's 1 more type of complex sentencebesides the sentences we get by combining operators,that makes first order logic unique, and these are the quantifiers.And there are two quantifiers for all, which we write with an upside-down Afollowed by a variable that it introducesand there exists, which we write with an upside-down E followed by the variable that it introduces. So for example, we could say for all x, if x is a vowel, then the number of (x) is equal to 1,and that's the valid sentence in first order logic. Or we could say there exists in x such that the number of (x)is equal to 2,and this is saying that there's some object in the domain to which the number of function applies and has a value of 2,but we're not saying what that object is. Now, another note is that sometimes as an abbreviation, we'll omit the quantifier, and when we do that, you can just assume that it means for all; that's left out just as a shortcut.And I should say that these forms, or these sentences are typical,and you'll see these form over and over again, so typically, whenever we have a "for all" quantifier introduced, it tends to go with a conditional like vowel of (x) implies number of (x) =1, and the reason is because we usually don't want to say something about every object in the domain, since the objects can be so different,but rather, we want to say something about a particular type of object, say, in this case, vowels. And also, typically, when we have an exists an x, or an exists any variable, that typically goes with just a form like this, and not with a conditional, because we're talking about just 1 object that we want to describe.[man] Now let's go back to the 2-location vacuum world and represent it in first order logic. So first of all, we can have locations.We can call the left location A and the right location Band the vacuum V, and the dirt--say, D1 and D2.Then, we can have relations.The relation loc, which is true of any location;vacuum, which is true of the vacuum; dirt, which is true of dirt; and at, which is true of an object and a location.And so if we wanted to say the vacuum is at location A, we just say at (V, A).If we want to say there's no dirt in any location, it's a little bit more complicated. We can say for all dirt and for all locations, if D is a dirt, and L is a location, then D is not at L.So that says there's no dirt in any location. Now, note if there were thousands of locations instead of just 2, this sentence would still hold, and that's really the power of first order logic. Let's keep going and try some more examples. If I want to say the vacuum is in a location with dirt without specifying what location it's in,I can do that. I can say there exists an L and there exists a Dsuch that D is a dirt and L is a location and the vacuum is at the location and the dirt is at that same location. and that's the power of first order logic. Now one final thing.You might ask what "first order" means.It means that the relations are on objects, but not on relations, and that would be called "higher order."In higher order logic, we could, say, define the notion of a transitive relation talking about relations itself, and so we could say for all R, transitive of R is equivalent to for all A, B, and C; R of (A, B) and R of (B, C) implies R (A, C).So that would be a valid statement in higher order logic that would define the notion of a transitive relation,but this would be invalid in first order logic. [Man] Now let's get some practice in first order logic. I'm going to give you some sentences, and for each one, I want you to tell me if it is valid--that is, O is true--satisfiable, but not valid; that is, there's some models for which it is true; or unsatisfiable, meaning there are no models for which it is true. And the first sentence is there exists an x and a y such that x = y. Second sentence: there exists an x such that x = x, implies for all y there exists a z such that y = z. Third sentence: for all x, p of x or not p of x. And fourth: there exists an x, P of x. [Man] The answers are the first sentence is valid. It's always true. Why is that? Because every model has to have at least 1 objectand we can have both x and y refer to that same object,and so that object must be equal to itself. Second, let's see. The left-hand side of this implication has to be true.X is always equal to x, and the right-hand side says for every y, does there exist a zsuch that y equals z? And we can say yes, there is. We can always choose y itself for the value of z, and then y = y, so true implies true.That's always true. Valid. Third sentence: for all x, P of x or not P of x, and that's always true because everything has to be either in the relation for P or out of the relation for P, so that's valid. And the fourth: there exists an x, P of x, and that's true for the models in which there is some x that is a member of P, but it doesn' t necessarily have to be any at all.P might be an empty relation, so this is satisfiable. True in some models, but not true in all models. [Man] Now I'm going to give you some sentences or axioms in first order logic,and I want you to tell me if they correctly or incorrectly represent the English that I'm asking about.So tell me yes or no, are these good representations? And the first, I want to represent the English sentence "Sam has 2 jobs," and the first order logic sentence is there exists an x and y such that job of Sam x and job of Sam y and not x = y. And so tell me yes, that correctly represents Sam has 2 jobs, or no, there's a problem. And secondly, I want to represent the idea of set membership. Now, assume I've already defined the notion of adding an element to a set.Can I define set membership with these 2 axioms?For all x and s, x is a member of the result of adding x to any set s,and for all x and s, x is a member of s implies that for all y, x is a member of the set that you get when you add y to s. And third, I'm going to try to define the notion of adjacent squareson, say, a checkerboard, where the squares are numbered with x and y coordinatesand we want to just talk about adjacency in the horizontal and vertical direction.Can I define that as follows? For all x and y, the square x, y is adjacent to the square +(x,1), y, and the square (x, y) is adjacent to the square (x, +(y, 1)and assume that we've defined the notion of + somewhere and that the character set allows + to occur as the character for a function.Tell me yes or no, is that a good representation of the notion of adjacency?[Man] The first answer is yes, this is a good representation of the sentence "Sam has 2 jobs."It says there exists an x and y, and one of them is a job of Sam. The other one is a job of Sam, and crucially, we have to say that x is not equal to y.Otherwise, this would be satisfied and we could have the same job represented by the variables x and y.Is this a good representation of the member function? No. It does do a good job of telling you what is a member,so if x is a member of a set because it's one memberand then we can always add other members and it's still a member of that set,but it doesn't tell you anything about what x is not a member of.So for example, we want to know that 3 is not a member of the empty set,but we can't prove that with what we have here. And we have a similar problem down here.This is not a good representation of adjacent relation.So it will tell you, for example, that square (1,1) is adjacent to square (2,1)and also to square (1,2).So it's doing something right, but one problem is that it doesn't tell you in the other direction.It doesn't tell you that (2,1) is adjacent to (1,1)and another problem is that it doesn't tell you that (1,1) is not adjacent to (8,9)because again, there's no way to prove the negative. And the moral is that when you're trying to do a definition, like adjacent or member, what you usually want to do is have a sentence with the equivalent or the biconditional signto say this is true if and only if rather than to just have an assertion or to have an implication in one direction.[Narrator] Hi, and welcome back.This unit is about planning.We defined AI to be the study and process of finding appropriate actions for an agent.So in some sense planning is really the core of all of AI.The technique we looked at so farwas problem solving search over a state space using techniques like A star. Given a state space and a problem description,we can find a solution, a path to the goal.Those approaches are great for a variety of environments, but they only work when the environment is deterministic and fully observable.In this unit, we will see how to relax those constraints. [Narrator] You remember our problem-solving work?We have a state space like this, andwe're given a start space and a goal to reach, and then we'd search for a pathto find that goal, and maybe we findthis path.Now the way a problem-solving agent would work is first it does all the workto figure out the path to the goaljust doing by thinking,and then it starts to execute that pathto drive or walk, however you want to get there,from the start state to the end state,but think about what would happenif you did that in real life; if you did all your planning ahead of time, you had the complete goal,and then without interacting with the world,without sensing it at all,you started to execute that path.Well this has, in fact, been studied.People have gone out andblindfolded walkers, put them in a field and told them to walk in a straight line,and the results are not pretty.Here are the GPS tracks to prove it.So we take a hiker, we put him at a start location, say here,and we blindfold him so that he can't see anything in the horizon,but just has enough to see his or her feetso that they won't stumble over something,and tell them execute the plan of going forward.Put one foot in front of each other and walk forward in a straight line,and these are the typical paths we see.They start out going straight for awhile but then go in loop de loops and end up not at a straight path at all.These ones over here, starting in this location,are even more convoluted.They get going straight for a little bitand then go in very tight loops.So people are incapable of walking a straight linewithout any feedback from the environment.Now here on this yellow path, this one did much better, and why was that?Well it's because these paths were on overcast days,and so there was no input to make sense of.Whereas on this path was on a very sunny day,and so even though the hiker couldn't see farther than a few feet in front of him, he could see shadows and say,"As long as I keep the shadows pointing in the right direction then I can go in a relatively straight line."So the moral is we need some feedback from the environment. We can't just plan ahead and come up with a whole plan.We've got to interleave planning and executing.[Narrator] Now why do we have to interleaveplanning and execution?Mostly because of properties of the environment that make it difficult to deal with.The most important one isif the environment is stochastic. That is if we don't know for sure what an action is going to do.If we know what everything is going to do,we can plan it our right from the start, but if we don't, we have tobe able to deal with contingencies of say I tried to move forward, and the wheels slipped, and I went someplace else,or the brakes might skid, or if we're walking our feet don't go 100% straight,or consider the problem of traffic lights.If the traffic light is red, then the result of the action of goforward through the intersectionis bound to be different than if the traffic light is green.Another difficulty we have to deal with is multi-agent environments. If there are other cars and people that can get in our way,we have to plan about what they're going to do, and we have to react when they do something unexpected,and we can only know that at execution time, not at planning time.The other big problem is with partial observability.Suppose we've come up with a planto go from A to S to F to B.That plan looks like it will work,but we know that at S,the road to F is sometimes closed,and there will be a sign theretelling us whether it's closed or not,but when we start off, we can't read that sign.So that's partial observability.Another way to look at it is when we start off we don't know what state we're in.We know we're in A, but we don't know if we're in A in the state wherethe road is closed or if we're in A in the state where the road is open, and it's not until we get to Sthat we discover what state we're actually in, and then we know if we can continue along that route or if we have to take a detour south. Now in addition to these properties ofthe environment, we can also have difficulty because of lack of knowledge on our own part.So if some model of the world is unknown, that is, for example, we have map or GPS software that's inaccurate or incomplete,then we won't be able to executive a straight-line plan,and, similarly, often we want to deal witha case where the plans have to be hierarchical. And, certainly, a plan like this is at a very high level.We can't really execute the action of going from A to S when we're in a car.All the actions that we can actually executeare things like turn the steering wheel a little bit to the right, press on the pedal a little bit more.So those are the low-level steps of the plan,but those aren't sketched out in detail when we start,when we only have the high-level parts of the plan, and then it's during execution that we schedule the rest of the low-level parts of the plan.Now most of these difficulties can beaddressed by changing our point of view. Instead of planning in the space of world states,we plan in the space of belief states.To understand that let's look at a state.[Narrator] Here's a state space diagram for a simple problem.It involves a room with 2 locations.The left we call A, and the right we call B,and in that environmentthere's a vacuum cleaner, and there may or may not be dirt in either of the 2 locations,and so that gives us 8 total states.Dirt is here or not, here or not, andthe vacuum cleaner is here or here.So that's 2 times 2 times 2 is 8 possible states, and I've drawn here the states based diagram with all the transitions for the 3 possible actions, and the actions are moving right.So we'd go from this state to this state. Moving left, we'd go from this state to this state, and sucking up dirt, we'd go from this state to this state for example, and in this state space diagram, if we have a fully deterministic,fully observable world, it's easy to plan.Say we start in this state, and we want to be--end up in a goal state where both sides are clean.We can execute the suck-dirt action and get here and then move right, and then suck dirt again,and now we end up in a goal statewhere everything is clean.Now suppose our robot vacuum cleaner'ssensors break down, and so the robot can no longer perceive either which location its in or whether there's any dirt.So we now have an unobservable or sensor-less world rather than a fully observable one, and how does the agent then represent the state of the world?Well it could be in any one of these 8 states,and so all we can do to represent the current state is draw a big circle or box around everything, and say,"I know I'm somewhere inside here."Now that doesn't seem like it helps very much.What good is it to know that we don't really know anything at all?But the point is that we can search in the state space of the least states ratherthan in the state space of actual spaces.So we believe that we're in 1 of these 8 states,and now when we execute an action,we're going to get to another belief state.Let's take a look at how that works.[Narrator] This is the belief state spacefor the sensor-less vacuum problem.So we started off here.We drew the circle around this belief state.So we don't anything about where we are,but the amazing thing is, if we execute actions, we can gain knowledgeabout the world even without sensing.So let's say we move right, then we'll know we're in the right-hand location.Either we were in the left, and we moved rightand arrived there, or we were in the right to begin with, and we bumped against the walland stayed there.So now we end up in this state.We now know more about the world.We're down to 4 possibilities rather than 8,even though we haven't observed anything,and now note something interesting,that in the real world, the operations of going left and going right areinverses of each other, but in the belief state world going right and going left are not inverses.If we go right, and then we go left,we don't end up back where we werein a state of total uncertainty, rathergoing left takes us over herewhere we still know we're in 1 of 4 statesrather than in 1 of 8 states.Note that it's possible to form a plan that reaches a goal without ever observing the world. Plans like that are called conform-it plans.For example, if the goal is to be in a clean location all we have to do is suck.So we go from one of these 8 statesto one of these 4 states and, every one of those 4,we're in a clean location.We don't know which of the 4 we're in,but we know we've achieved the goal.It's also possible to arrive at a completely known state.For example, if we start here,we go left; we suck up the dirt there.If we go right and suck up the dirt,now we're down to a belief state consisting of 1 single state that iswe know exactly where we are.Here's a question for you:How do I get from the state where I know my current square is clean, but know nothing else, to the belief statewhere I know that I'm in the right-hand sidelocation and that that location is clean?What I want you to do is click on thesequence of actions, left, right, or suckthat will take us from that start to that goal.[Narrator] And the answer is that the stateof knowing that you're current square is cleancorresponds to this state.This belief state with 4 possible world states,and if I then execute the right action, followed by the suck action,then I end up in this belief state,and that satisfies the goal.I know I'm in the right-hand-side locationand I know that location is clean.[Narrator] We've been considering sensor-less planning in a deterministic world. Now I want to turn our attention to partially observable planning but still in a deterministic world.Suppose we have what's called local sensing,that is our vacuum can see what location it is in and it can see what's going on in the current location, that is whether there's dirt in the current location or not, but it can't see anything about whether there's dirt in any other location.So here's a partial diagram of the-- part of the belief state from that world,and I want it to show how the belief state unfolds as 2 things happen.First, as we take action, so we start in this state, and we take the action of going right, and in this case we still go from 2 world states in our belief state to 2 new ones, but then, after we do an action,we do an observation, and we have the actprecept cycle, and now, once we get the observation, we can split that world, we can split our belief state to say, "If we observe that we're in location B and it's dirty, then we knowwe're in this belief state here,which happens to have exactly 1 world state in it,and if we observe that we're cleanthen we know that we're in this state,which also has exactly 1 in it.Now what is the act-observe cycle do to the sizes of the belief states?Well in a deterministic world, each of the individual world states within a belief state maps into exactly 1 other one.That's what we mean by deterministic,and so that means the size of the belief state will either stay the same or it might decreaseif 2 of the actions sort of accidentally end up bringing you to the same place.On the other hand, the observation works in kind of the opposite way.When we observe the world, what we're doing is we're taking the current belief state and partitioning it up into pieces.Observations alone can't introduce a new state--a new world state into the belief state.All they can do is say, "Some of them go here and some of them go here."Now maybe that for some observationall the belief states go into 1 bin,and so we make an observationthat we don't learn anything new, but at least the observation can't make us more confused than we were before the observation. [Norvig] Now let's move on to stochastic environments.Let's consider a robot that has slippery wheelsso that sometimes when you make a movement--a left or a right action--the wheels slip and you stay in the same location.And sometimes they work and you arrive where you expected to go.And let's assume that the suck action always works perfectly.We get a belief state space that looks something like this.Notice that the results of actions will often result in a belief statethat's larger than it was before--that is, the action will increase uncertaintybecause we don't know what the result of the action is going to be.And so here for each of the individual world states belonging to a belief state,we have multiple outcomes for the action, and that's what stochastic means.And so we end up with a larger belief state here.But in terms of the observation, the same thing holds as in the deterministic world.The observation partitions the belief state into smaller belief states.So in a stochastic partially observable environment,the actions tend to increase uncertainty, and the observations tend to bring that uncertainty back down.Now, how would we do planning in this type of environment?I haven't told you yet, so you won't know the answer for sure,but I want you to try to figure it out anyways, even if you might get the answer wrong.Imagine I had the whole belief state from which I've diagrammed just a little bit hereand I wanted to know how to get from this belief state to one in which all squares are clean.So I'm going to give you some possible plans, and I want you to tell me whether you think each of these plans will always workor maybe sometimes work depending on how the stochasticity works out.Here are the possible plans.Remember I'm starting here, and I want to know how to get to a belief statein which all the squares are clean.One possibility is suck right and suck, one is right suck left suck,one is suck right right suck,and the other is suck right suck right suck.So some of these actions might take you out of this little belief state here,but just use what you knew from the previous definition of the state spaceand the results of each of those actions and the fact that the right and left actions are nondeterministic and tell me which of these you think will always achieve the goalor will maybe achieve the goal.And then I want you to also answer for the fill-in-the-blank plan--that is, is there some plan, some ideal plan, which always or maybe achieves the goal?And the answer is that any plan that would work in the deterministic world might work in the stochastic worldif everything works out okayand all of these plans meet that criteria.But no finite plan is guaranteed to always workbecause a successful plan has to include at least 1 move action.And if we try a move action a finite number of times,each of those times, the wheels might slip, and it won't move,and so we can never be guaranteed to achieve the goalwith a finite sequence of actions.Now, what about an infinite sequence of actions?Well, we can't represent that in the language we have so farwhere a plan is a linear sequence.But we can introduce a new notion of plans in which we do have infinite sequences.In this new notation, instead of writing plansas a linear sequence of, say, suck, move right, and suck,I'm going to write them as a tree structure.We start off in this belief state here, which we'll diagram like this.And then we do a suck action.We end up in a new state.And then we do a right action,and now we have to observe the world,and if we observe that we're still in state A,we loop back to this part of the plan.And if we observe that we're in B,we go on and then execute the suck action.And now we're at the end of the plan.So, we see that there's a choice point here,which we indicate with this sort of tieto say we're following a straight line, but now we can branch.There's a conditional, and we can either loop,or we can continue on, so we see that this finite representationrepresents an infinite sequence of plans.We could write it in a more sort of linear notationas S, while we observe A, do R, and then do S.Now, what can we say about this plan?Does this plan achieve the goal?Well, what we can say is that if the stochasticity is independent, that is, if sometimes it worksand sometimes it doesn't, then with probability 1 in the limit,this plan will, in fact, achieve the goal,but we can't state any bounded number of stepsunder which it's guaranteed to achieve the goal.We can only say it's guaranteed at infinity.Now, I've told you what a successful plan looks like,but I haven't told you how to find one.The process of finding it can be done through searchjust as we did in problem solving.So, remember in problem solving,we start off in a state, and it's a single state, not a belief state.And then we start searching a tree,and we have a big triangle of possible statesthat we search through, and then we findone path that gets us all the way to a goal state.And we pick from this big tree a single path.So, with belief states and with branchingplan structures, we do the same sort of process,only the tree is just a little bit more complicated.Here we show one of these trees,and it has different possibilities.For example, we start off here, and we have one possibilitythat the first action will be going right,or another possibility that the first actionwill be performing a suck.But then it also has branches that are part of the plan itself.This branch here is actually part of the planas we saw before.It's not a branch in the search space.It's a branch in the plan, so what we dois we search through this tree.We try right as a first action.We try suck as a first action.We keep expanding nodesuntil we find a portion of the treelike this path is a portion of this search tree.We find that portion which is a successful planaccording to the criteria of reaching the goal.Let's say we performed that search.We had a big search tree, and then we threw outall the branches except one, and this branch of the search treedoes itself have branches, but this branch of the search treethrough the belief state represents a single plan,not multiple possible plans.Now, what I want to know is, for this single plan,what can we guarantee about it?So, say we wanted to know is this plan guaranteed to find the goalin an unbounded number of steps?And what do we need to guarantee that?So, it's an unbounded solution.Do we need to guarantee thatsome leaf node is a goal?So, for example, here's a plan to go through,and at the bottom, there's a leaf node.Now, if this were in problem solving,then remember, it would be a sequence of stepswith no branches in it, and we know it's a solutionif the one leaf node is a goal.But for these with branches, do we need to guaranteethat some leaf is a goal, or do we need to guarantee that every leaf is a goal,or is there no possible guaranteethat will mean that for sure we've got a solution,although the solution may be of unbounded length?Then I also want you to answerwhat does it take to guaranteethat we have a bounded solution?That is, a solution that is guaranteed to reach the goalin a bounded, finite number of steps.Do we need to have a plan that hasno branches in it, like this branch?Or a plan that has no loops in it,like this loop that goes back to a previous state?Or is there no guarantee that we have a bounded solution?And the answer is we have an unbounded solutionif every leaf in the plan ends up in a goal.So, if we follow through the plan, no matter what pathwe execute based on the observations--and remember, we don't get to pick the observations.The observations come into us, and we follow one path or anotherbased on what we observe.So, we can't guide it in one direction or another,and so we need every possible leaf node.This one only has one, but if a plan had multiple leaf nodes,every one of them would have to be a goal.Now, in terms of a bounded solution,it's okay to have branches but not to have loops.If we had branches and we ended up with one goal hereand one goal here in 1, 2, 3, steps,1, 2, 3, steps, that would be a bounded solution.But if we have a loop, we might be 1, 2, 3, 4, 5--we don't know how many steps it's going to take.Now, some people like manipulating treesand some people like a more--sort of formal--mathematical notation.So if you're one of those, I'm going to give you another way to think about whether or not we have a solution;and let's start with a problem-solvingwhere a plan consists of a straight line sequence.And we said one way to decide if this is a plan that satisfies the goalis to say, "Is the end state a goal state?"If we want to be more formal and write that out mathematically, what we can say is--what this plan represents is--we started in the start state,and then we transitioned to the state that is the result of applying the action of going from A to S, to that start state; and then we applied to that, the result of starting in that intermediate stateand applying the action of going from S to F.And if that resulting state is an element of the set of Goals, then this plan is valid; this plan gives us a solution. And so that's a mathematical formulation of what it means for this plan to be a Goal. Now, in stochastic partially observable worlds, the equations are a little bit more complicated. Instead of just having S Prime is a result of applying some action to the initial state, we're dealing with belief states, rather than individual states. And what we say is our new belief state is the result of updating what we get from predicting what our action will do; and then updating it, based on our observation, O, of the world. So the prediction step is when we start off in a belief state;we look at the action, we look at each possible result of the action--because they're stochastic--to each possible member of the belief state,and so that gives us a larger belief state;and then we update that belief state by taking account of the observation--and that will give us a smaller--or same size--belief state. And now, that gives us the new state. Now, we can use this to predict and update cycleto keep track of where we are in a belief state. Here's an example of tracking the Predict Update Cycle;and this is in a world in which the actions are guaranteed to work, as advertised--that is, if you start to clean up the current location,and if you move right or left, the wheels actually turn; and you do move.But we can call this the kindergarten world because there are little toddlers walking around who can deposit Dirt in any location, at any time. So if we start off in this state, and execute the Suck action, we can predict that we'll end up in one of these 2 states.Then, if we have an observation--well, we know what that observation's going to be because we know the Suck action always works, and we know we were in A;so the only observation we can get is that we're in A--and that it's Clean--so we end up in that same belief state. And then, if we execute the Right action--well, then lots of things could happen;because we move Right, and somebody might have dropped Dirt in the Right location, and somebody might have dropped Dirt in the Left location--or maybe not. So we end up with 4 possibilities,and then we can update again when we get the next observation--say, if we observed that we're in B and it's Dirty, then we end up in this belief state. And we can keep on going--specifying new belief states-- as a result of success of predicts and updates. Now, this Predict Update Cycle gives us a kind of calculus of belief statesthat can tell us, really, everything we need to know. But there is one weakness with this approach--that, as you can see here, some of the belief states start to get large; and this is a tiny little world.Already, we have a belief state with 4 world states in it. We could have one with 8, 16, 10, 24--or whatever. And it seems that there may be more succinct representations of a belief state,rather than to just list all the world states. For example, take this one here:If we had divided the world up--not into individual world states,but into variables describing that state, then this whole belief state could be represented just by: Vacuum is on the Right. So the whole world could be represented by 3 states--or 3 variables:One, where is the Vacuum--is it on the Right, or not? Secondly, is there Dirt in the Left location?And third, is there Dirt in the Right location? And we could have some formula, over those variables, to describe states. And with that type of formulation, some very large states--in terms of enumerating the world states--can be made small, in terms of the description. [Norvig] I want to describe a notation which we call classical planning,which is a representation language for dealing with states and actions and plans,and it's also an approach for dealing with the problem of complexityby factoring the world into variables.So under classical planning, a state space consists of all the possible assignmentsto k Boolean variables.So that means they'll be 2 to the k states in that state space.And if we think about the 2 location vacuum world,we would have 3 Boolean variables.We could have dirt in location A, dirt in location B, and vacuum in location A.The vacuum has to be in either A or B.So these 3 variables will do, and there will be 8 possible states in that world,but they can be succinctly represented through the 3 variables.And then a world state consists of a complete assignment of true or falsethrough each of the 3 variables.And then a belief state.Just as in problem solving, the belief state depends on what type of environment you want to deal with.In the core classical planning, the belief state had to be a complete assignment,and that was useful for dealing with deterministic fully observable domains.But we can easily extend classical planning,and we can deal with belief states that are partial assignments--that is, some of the variables have values and others don't.So we could have the belief state consisting of vacuum in A is trueand the others are unknown, and that small formula represents 4 possible world states.We can even have a belief state which is an arbitrary formula in Boolean logic,and that can represent anything we want.So that's what states look like.Now we have to figure out what actions look like and what the results of those actions look like.These are represented in classical planning by something called an action schema.It's called a schema because it represents many possible actions that are similar to each other.So let's take an example of we want to send cargo around the world,and we've got a bunch of planes in airports, and we have cargo and so on.I'll show you the action for having a plane fly from one location to another.Here's one possible representation.We say it's an action schema, so we write the word Actionand then we write the action operator and its arguments,so it's a Fly of P from X to Y.And then we list the preconditions,what needs to be true in order to be able to execute this action.We can say something like P better be a plane.It's no good trying to fly a truck or a submarine.And we'll use the And formula from Boolean propositional logic.X better be an airport.We don't want to try to take off from my backyard.And similarly, Y better be an airport.And, most importantly, P better be at airport X in order to take off from there.And then we represent the effects of the action by sayingwhat's going to happen.Once we fly from X to Y, the plane is no longer at X,so we say not at P,X--the plane is no longer at X--and the plane is now at Y.This is called an action schema.It represents a set of actions for all possible planes, for all X and for all Y,represents all of those actions in one schemathat says what we need to know in order to apply the action and it says what will happen.In terms of the transition from state spaces, this variable will become falseand this one will become true.When we look at this formula, this looks like a term in first order logic,but we're actually dealing with a completely propositional world.It just looks like that because this is a schema.We can apply this schema to specific ground states, specific world states,and then P and X would have specific values,and you could just think of it as concatenating their names all together,and that's just the name of one variable.The name just happens to have this complex form with parentheses and commas in itto make it easier to write one schema that covers all the individual fly actions.[Norvig] Here we see a more complete representation of a problem solving domainin the language of classical planning.Here's the Fly action schema.I've made it a little bit more explicit with from and to airportsrather than X or Y.We want to deal with transporting cargo.So in addition to flying, we have an operator to load cargo, C, onto a plane, P, at airport A--you can see the preconditions and effects there--and an action to unload the cargo from the planewith preconditions and effects.We have a representation of the initial state.There's 2 pieces of cargo, there's 2 planes and 2 airports.This representation is rich enough and the algorithms on it are good enoughthat we could have hundreds or thousands of cargo planes and so onrepresenting millions of ground actions.If we had 10 airports and 100 planes, that would be 100, 1,000, 10,000 different Fly actions.And if we had thousands of pieces of cargo,there would be even more Load and Unload actions,but they can all be represented by the succinct schema.So the initial state tells us what's what, where everything is,and then we can represent the goal state:that we want to have this piece of cargo has to be delivered to this airport,and another piece of cargo has to be delivered to this airport.So now we know what actions and problems of initial and goal state looks likein this representation, but how do we do planning using this?[Norvig] The simplest way to do planning is really the exact same waythat we did it in problem solving.We start off in an initial state.So P1 was at SFO, say, and cargo, C1, was also at SFO,and all the other things that were in that initial state.And then we start branching on the possible actions,so say one possible action would be to load the cargo, C1, onto the plane, P1, at SFO,and then that would bring us to another state which would have a different set of state variables set,and we'd continue branching out like that until we hit a state which satisfied the goal predicate.So we call that forward or progression state space searchin that we're searching through the space of exact states.Each of these is an individual world state,and if the actions are deterministic, then it's the same thing as we had before.But because we have this representation, there are other possibilities that weren't available to us before.[Norvig] Another way to search is called backwards or regression searchin which we start at the goal.So we take the description of the goal state.C1 is at JFK and C2 is at SFO, so that's the goal state.And notice that that's the complete goal state.It's not that I left out all the other facts about the state;it's that that's all that's known about the state is that these 2 propositions are trueand all the others can be anything you want.And now we can start searching backwards.We can say what actions would lead to that state.Remember in problem solving we did have that option of searching backwards.If there was a single goal state, we could say what other arcs are coming into that goal state.But here, this goal state doesn't represent a single state;it represents a whole family of states with different values for all the other variables.And so we can't just look at that,but what we can do is look at the definition of possible actions that will result in this goal.So let's look at it one at a time.Let's first look at what actions could result at C1, JFK.We look at our action schema, and there's only 1 action schema that adds an At,and that would be the Unload schema.Unload of C, P, A adds C, A.And so what we would know is if we want to achieve this,then we would have to do an Unload where the C variable would have to be C1,the P variable is still unknown--it could be any plane--and the A variable has to be JFK.Notice what we've done here.We have this representation in terms of logical formulathat allows us to specify a goal as a set of many world states,and we also can use that same representation to represent an arrow herenot as a single action but as a set of possible actions.So this is representing all possible actions for any plane, P, of unloading cargo at the destination.And then we can regress this state over this operatorand now we have another representation of this state here.But just as this state was uncertain--not all the variables were known--this state too will be uncertain.For example, we won't know anything about what plane, P, is involved,and now we continue searching backwards until we get to a statewhere enough of the variables are filled in and where we match against the initial state.And then we have our solution.We found it going backwards, but we can apply the solution going forwards.[Norvig] Let's show an example of where a backwards search makes sense.I'm going to describe a world in which there is one action,the action of buying a book.And the precondition is we have to know which book it is,and let's identify them by ISBN number.So we can buy ISBN number B, and the effect is that we own B.And probably there should be something about money, but we're going to leave that out for now to make it simple.And then the goal would be to own ISBN number 0136042597.Now, if we try to solve this problem with forward search, we'd start in the initial state.Let's say the initial state is we don't own anything.And then we'd think about what actions can we apply.If there are 10 million different books, 10 million ISBN numbers,then there is a branching factor of 10 million coming out of this node,and we'd have to try them all in order until we happened to hit upon one that was the right one.It seems very inefficient.If we go in the backward direction, then we start at the goal.The goal is to own this number.Then we look at our available actions, and out of the 10 million actionsthere's only 1 action schema, and that action schema can match the goal in exactly one way,when B equals this number, and therefore we know the action is to buy this number,and we can connect the goal to the initial state in the backwards direction in just 1 step.So that's the advantage of doing backwards or regression search rather than forward search.[Norvig] There's one more type of search for plans that we can do with the classical planning languagethat we couldn't do before, and this is searching through the space of plansrather than searching through the space of states.In forward search we were searching through concrete world states.In backward search we were searching through abstract statesin which some of the variables were unspecified.But in plan space search we search through the space of plans.And here's how it works.We start off with an empty plan.We have the start state and the goal state, and that's all we know about the plan.So obviously, this plan is flawed. It doesn't lead us from the start to the goal.And then we say let's do an operation to edit or modify that planby adding something in new.And here we're tackling the problem of how to get dressedand put on all the clothes in the right order,so we say out of all the operators we have, we could add one of those operators into the plan.And so here we say what if we added the put on right shoe operator.Then we end up with this plan. That still doesn't solve the problem, so we need to keep refining that plan.Then we come here and say maybe we could add in the put on left shoe operator.And here I've shown the plan as a parallel branching structurerather than just as a sequence.And that's a useful thing to do because it captures the fact that these can be done in either order.And we keep refining like that, adding on new branches or new operatorsinto the plan until we got a plan that was guaranteed to work.This approach was popular in the 1980s, but it's faded from popularity.Right now the most popular approaches have to do with forward search.We saw some of the advantages of backward search.The advantage of forward search seems to be that we can come up with very good heuristics.So we can do heuristic search, and we saw how important it was to have good heuristicsto do heuristic search.And because the forward search deals with concrete plan states,it seems to be easier to come up with good heuristics.[Norvig] To understand the idea of heuristics, let's talk about another domain.Here we have the sliding puzzle domain.Remember we can slide around these little tiles and we try to reach a goal state.A 16 puzzle is kind of big, so let's show you the state space for the smaller 8 puzzle.Here is just a small portion of it.Let's figure out what the action schema looks like for this puzzle.We only need to describe one action, which is to slide a tile, T, from location A to location B.The precondition: the tile has to be on location A and has to be a tileand B has to be blank and A and B have to be adjacent.This should be an And sign, not an A.So that's the action schema. Oops. I forgot we need an effect, which should be that the tile is now on Band the blank is now on A and the tile is no longer on A and the blank is no longer on B.We talked before about how a human analyst could examine a problemand come up with heuristics and encode those heuristics as a functionthat would help search do a better job.But with this kind of a formal representationwe can automatically come up with good representations of heuristics.For example, if we came up with a relaxed problemby automatically going in and throwing out some of the prerequisites--if you throw out a prerequisite, you make the problem strictly easier--then you get a new heuristic.So for example, if we crossed out the requirement that B has to be blank,then we end up with the Manhattan or city block heuristic.And if we also throw out the requirement that A and B have to be adjacent,then we get the number of misplaced tiles heuristic.So that means we could slide a tile from any A to any B, no matter how far apart they were.That's the number of misplaced tiles.Other heuristics are possible.For example, one popular thing is to ignore negative effects,to say let's not say that this takes away the blank being in B.So if we ignore that negative effect, we make the whole problem strictly easier.We'd have a relaxed problem, and that might end up being a good heuristic.So because we have our actions encoded in this logical form,we can automatically edit that form.A program can do that, and the program can come up with heuristicsrather than requiring the human to come up with heuristics.[Norvig] Now I want to talk about 1 more representation for planningcalled situation calculus.To motivate this, suppose we wanted to have the goal of moving all the cargofrom airport A to airport B, regardless of how many pieces of cargo there are.You can't express the notion of All in propositional languages like classical planning,but you can in first order logic.There are several ways to use first order logic for planning.The best known is situation calculus.It's not a new kind of logic;rather, it's regular first order logic with a set of conventionsfor how to represent states and actions.I'll show you what the conventions are.First, actions are represented as objects in first order logic,normally by functions.And so we would have a function like the function Flyof a plane and a From Airport and a To Airportwhich represents an object, which is the action.Then we have situations, and situations are also objects in the logic,and they correspond not to states but rather to paths--the paths of actions that we have in state space search.So if you arrive at what would be the same world state by 2 different sets of actions,those would be considered 2 different situations in situation calculus.We describe the situations by objects, so we usually have an initial situation,often called S0,and then we have a function on situations called Result.So the result of a situation object and an action object is equal to another situation.And now instead of describing the actions that are applicable in a situation with a predicate Actions of S,situation calculus for some reason decided not to do thatand instead we're going to talk about the actions that are possible in the state,and we're going to do that with a predicate.If we have a predicate Possible of A and S, is an action A possible in a state?There's a specific form for describing these predicates,and in general, it has the form of some precondition of state Simplies that it's possible to do action A in state S.I'll show you the possibility axiom for the Fly action.We would say if there is some P, which is the plane in state S,and there is some X, which is an airport in state S,and there is some Y, which is also an airport in state S,and P is at location X in state S,then that implies that it's possible to fly P from X to Y in state S.And that's known as the possibility axiom for the action Fly.[Norvig] There's a convention in situation calculus that predicates like At--we said plane P was at airport X in situation S--these types of predicates that can vary from 1 situation to another are called fluents,from the word fluent, having to do with fluidity or change over time.And the convention is that they refer to a specific situation,and we always put that situation argument as the last in the predicate.Now, the trickiest part about situation calculus is describing what changesand what doesn't change as a result of an action.Remember in classical planning we had action schemas where we described 1 action at a time and said what changed.For situation calculus it turns out to be easier to do it the other way around.Instead of writing 1 action or 1 schema or 1 axiom for each action,we do 1 for each fluent, for each predicate that can change.We use the convention called successor state axioms.These are used to describe what happens in the state that's a successor of executing an action.And in general, a successor state axiom will have the form of sayingfor all actions and states, if it's possible to execute action A in state S,then--and I'll show in general what they look like here--the fluent is true if and only if action A made it trueor action A didn't undo it.So that is, either it wasn't true before and A made it be true,or it was true before and A didn't do something to stop it being true.For example, I'll show you the successor state axiom for the In predicate.And just to make it a little bit simpler, I'll leave out all the For All quantifiers.So wherever you see a variable without a quantifier, assume that there's a For All.What we'll say is it's possible to execute A in situation S.If that's true, then the In predicate holds between some cargo C and some plane in the state, which is the result of executing action A in state S.So that In predicate will hold if and only if either A was a load action--so if we load the cargo into the plane, then the result of executing that action Ais that the cargo is in the plane--or it might be that it was already true that the cargo was in the plane in situation Sand A is not equal to an unload action.So for all A and S for which it's possible to execute A in situation S,the In predicate holds if and only if the action was a loador the In predicate used to hold in the previous state and the action is not an unload.[Norvig] So I've talked about the possibility axioms and the successor state axioms.That's most of what's in situation calculus, and that's used to describe an entire domain like the airport cargo domain.And now we describe a particular problem within that domain by describing the initial state.Typically we call that S0, the initial situation.And in S0 we can make various types of assertionsof different types of predicates.So we could say that plane P1 is at airport JFK in S0, so just a simple predicate.And we could also make larger sentences, so we could say for all C, if C is cargo, then that C is at JFK in situation S0.So we have much more flexibility in situation calculus to say almost anything we want.Anything that's a valid sentence in first order logic can be asserted about the initial state.The goal state is similar.We could have a goal of saying there exists some goal state Ssuch that for all C, if C is cargo, then we want that cargo to be at SFO in state S.So this initial state and this goal says move all the cargo--I don't care how much there is--from JFK to SFO.The great thing about situation calculus is that once we've described thisin the ordinary language of first order logic,we don't need any special programs to manipulate it and come up with the solutionbecause we already have theorem provers for first order logicand we can just state this as a problem,apply the normal theorem prover that we already had for other uses,and it can come up with an answer of a path that satisfies this goal,a situation which corresponds to a path which satisfies thisgiven the initial state and given the descriptions of the actions.So the advantage of situation calculus is that we have the full power of first order logic.We can represent anything we want.Much more flexibility than in problem solving or classical planning.So all together now, we've seen several ways of dealing with planning.We started in deterministic, fully observable environmentsand we moved into stochastic and partially observable environments.We were able to distinguish between plans that can or cannot solve a problem,but we had 1 weakness in all these different approaches.It is that we weren't able to distinguish between probable and improbable solutions.And that will be the subject of the next unit.In this exercise, I'm going to write some logical expressionsin propositional logic and ask you if these expressions are always true or always falseor if their truth value depends on the values of the propositional variables.The first sentence is smoke implies fireis equivalent to smoke or not fire.Is that true or false, or does it depend on the values of smoke and fire?The second sentence, again, smoke implies fireis equivalent to not smoke implies not fire.The third sentence, smoke implies fireis equivalent to not fire implies not smoke. The fourth sentence, big or dumbor big implies dumb.The final sentence, big and dumbis equivalent to not, not big or not dumb.For each of these, tell me if they're always true regardless of the values of the variables,always false or sometimes true and sometimes false.In this exercise, I'm going to write some logical expressionsin propositional logic and ask you if these expressions are always true or always falseor if their truth value depends on the values of the propositional variables.The first sentence is smoke implies fireis equivalent to smoke or not fire.Is that true or false, or does it depend on the values of smoke and fire?The second sentence, again, smoke implies fireis equivalent to not smoke implies not fire.The third sentence, smoke implies fireis equivalent to not fire implies not smoke. The fourth sentence, big or dumbor big implies dumb.The final sentence, big and dumbis equivalent to not, not big or not dumb.For each of these, tell me if they're always true regardless of the values of the variables,always false or sometimes true and sometimes false.Here are the answers. The first sentence is true half the time and false half the time.It would have been true all the time if we had written fire or not smokerather than smoke or not fire on the right-hand side.The second sentence is false when fire is true and smoke is falseand otherwise true.The third sentence is always true, and this is called the contrapositive.Smoke implies fire is the same thing as not fire implies not smoke.The fourth sentence is always true, and you can figure that outby writing out the full truth tables or by reasoning about the variable big.When big is true, the whole sentence is true because big is one of the disjuncts, and when big is false, it's true because big implies dumb is truewhenever the antecedent is false.And the final sentence is also always true,and this is known as de Morgan's law.In this exercise, I'm going to give you some English sentences and then some first-order logic sentencesand ask you does the first-order logic sentencecorrectly encode the English sentence, does it incorrectly encode it,or is it just an error that is not a legitimate sentencein first-order logic?The first English sentence is "Paris and Nice are both in France."Here's one possible translation.Paris and Nice are in France.Here's another. Paris is in France, and Nice is in France.Tell us if each of these is a correct encoding of English,incorrect, or if it's erroneous first-order logic syntax.The second sentence in English is "There is a country that borders Iran and Syria."Here are the possible translations. There exists a c, and we're going to use the predicate capital C to mean C when the argument is a country.So, there exists a c such that C of c, and we're going to use the predicate B to mean 2 objects border each other. So, c borders Iran, and c borders Syria.That's one translation. Here's the other translation.There exists a c if C is a country,then c borders Iran and c borders Syria.And the final English sentence is no 2 bordering countries can have the same map color, and we're going to use the predicate MC for map color.Here's one possibility for all x and y.X is a country, and y is a country.And x and y border each other.That implies it's not the case that the map color of x equals the map color of y.And I should say we're using map color here as a function, not as a predicate.Here's another possibility.For all x and y, it's not the case that x is a country,or it's not the case that y is a country,or it's not the case that x and y border,or it's not the case that the map color of xis equal to the map color of y. The answers are the first sentence has erroneous syntax.We're using an and here between 2 terms, but you can't do that.An and can only be used between sentences and predicates in first-order logic.The second sentence does correctly encode the English sentenceParis and Nice are both in France.Similarly, the third sentence does correctly encodethere's a country that borders Iran and Syria,but the fourth one incorrectly encodes it.Here we have an existential. There exists a C.And then an implication, and that's usually the wrong thing.The problem here is not if C represents a country, but what if C represents something that's not a country, say my dog.My dog is not a country, so there does exist a c, which is my dog, such that this implication is truebecause whenever the antecedent of an implication is false,my dog is not a country, then the whole thing is true.For the final sentence in English, no 2 bordering countries can have the same map color, both of these are correct encodings.The first one seems more obvious, and the second one we've just manipulated things a little bit.We know that A implies B is the same thing as saying not A or B,so here we've just taken the left-hand side and negated itand then put those all together with an or,so these 2 sentences represent the same thing,and they're both correct.This problem is about planning in belief space.We have the 2 room vacuum world, and we've represented various belief states here.Now, in this version, there are no sensors and so no percepts. The actions are all deterministic. A right or left or suck action will always do what it's supposed to do,and the environment is static.That is, dirt stays put until it's cleaned up.Now, in the start, you know nothing about the environment.You have no input. You don't know what location you're in.You don't know where the dirt is, and your goal is to be in the leftmost of the 2 squaresand have both squares cleaned up, and what I want you to dois click on the sequence of actions, an action like this one or this one or this oneor this one, that constitute a path from the start state to the goal state.And then I want you to click on yes if that path is guaranteedto always reach the goal, and no if the path only sometimes reaches the goal.The answer is that we start off knowing nothing,so we're in this belief state here where any of the 8 possible states are possibilities.Then our path to the goal is to move right, and we arrive in this belief state,and then suck up the dirt there.Then move left, and then suck up the dirt there,and we end up in a belief state with only a single world state,and that's one that reaches the goal where we're on the leftand both squares are clean.And yes, that is guaranteed to reach the goal.In this problem, we're again in the 2-location vacuum world,but this time around, we have local sensing,meaning at each turn, we get input of what location we're at,the left or the right, and whether there's dirt in that location.But we don't know what's going on in the other location.We have a dynamic world where dirt can appear anywhere.As we move around, dirt can spontaneously appearin the location we left or in the location we're going to visit.However, if we're sucking, the dirt can't appear, because if it did appear there, we would successfully suck it up.And now in addition, the right and left movesare stochastic in that they don't always succeed.Sometimes when you try to go right, you do successfully go right,and sometimes you stay in the same location, same for left.The suck action is always successful.It will always clean up dirt in the current location.Now, when we start out, we get the percept saying that we're in the leftmost locationand that location is clean, and that means our belief stateis that we're in either 5 or 7.Now, the first thing I want you to answer is if we decide to move right,what do we predict the possible belief state will be,the possible set of states in our belief state will be after we execute the right movement?The answer is the right movement is stochastic,so it may fail, so that means we may stay on the left, and we may move to the right.And the world is dynamic, which means dirt may appearin either the left or the right location,and we didn't know for sure if there was dirt or not in the right,so that means any of the 8 possible states belong to the belief statefor the prediction of moving right.That means any of the 8 states belong to the belief statefor the prediction of moving right.Now we get a percept from the world, and we've observedthat we're in the rightmost square, so the action worked,and that square is dirty.Now we want to update our belief stateand click on all the states that belong to the belief state nowas we update due to this percept.The answer is state 6 and state 2.Those are the 2 states in which the vacuum is on the right and that state is dirty.The other state we can't observe, and it could have been in any state before,so now it can be either clean or dirty.Now our belief state contains 2 and 6,and we decide we want to execute the suck action.Now tell me, by clicking on the appropriate states, what states belong to the belief stateafter we make a prediction for what's going to happen after the suck action.The answer is states 4 and 8.We know that the suck action will make it cleanin our current location, but we don't know what's going on in the other location.Now, we make the observationright, clean, and I want you to update our belief state by clicking on the statesthat belong to the new belief state nowafter taking that observation into account.The answer is nothing has changed.Our belief state is still 4 and 8.We didn't really get any new information from that inputbecause we knew that the result of the suck actionwas going to have to clean up locally, and we still didn't know anythingabout the other non-local state.This is a famous problem called the monkey and bananas problem,described in the language of classical planning.There are six actions. The monkey can go from location x to y.It can push some object from x to y. It can climb up an object. It can grab something.It can climb down from an object, and it can un-grab something. Initially, the monkey is at location A. The bananas are at location B.The box is at C, and the monkey is at a low height, as is the box,but the bananas are at a high height,but the box is pushable and climbable.Now, assuming that we execute this plan--go from A to C, push the box from C to B,climb up on the box, grasp the bananas, and climb down from the box.What I want you to do is look at these definitions of actions, tell me how the state unfolds from this initial state here to the final state,and then click off all of these instances that are going to be true in the final state.The answers are, yes, the monkey has the bananas.No, the box is not at C. It has been pushed to B.Yes, the monkey is at B. Yes, the bananas are at B.No, the height of the monkey is not high, because he climbed down,which means that that the effect was that he is at height low.But, yes, the height of the bananas is high, according to these definitions.You would think once the monkey grasped the bananas and climbed down that the height of the bananas should be low,but if we look at the operator for climb down, it doesn't say that.It refers to the monkey, but it doesn't refer to anything that the monkey is holding.That kind of thing is difficult to express in the language of classical planning.You could say that's a weakness in the definition of climb down.[Norvig] The final problem involves situation calculus.In the domain I want to describe, we have a combination lock with 4 digits,and the correct combination that will open the lock we'll call X.There are 2 actions you can perform.One is to dial any combination on the dial,and if you dial the correct one, X, then the lock will open.And the other action you can perform is to press a lock button,and if you press that button, then the lock will be locked,whether it was open before or not.I'm going to describe some axioms, and I want you to tell me whether these axioms are correct for the domain or not.First the possibility axioms.One choice is the possibility axiom that saysif C equals X, then it's possible to dial C in situation S.And here I'm assuming that all variables are scopedso that we say an implicit for all C and for all S here.And X is not a variable. This is a constant, referring to the correct combination.The other possible axiom is for all C if C is greater than or equal to 0and less than or equal to 9999,then it's possible to dial C in any situation S.So tell me which, if any or both, of these axioms you think correctly encode the situation.Next we'll look at the possibility axioms for the lock action.Here's one.We can say if the safe is open in situation S,then it's possible to execute the lock action in S.Or maybe we should say if the safe is not open in S,then it's possible to execute Lock in S.Or maybe we should say if true, then it's possible to execute the lock action in situation S.And tell me which, if any, of those represents a correct representation of the problem.And finally we need successor state axioms for all the fluents,but there's really only one fluent, and that's whether or not the safe is open.So here's one example of a successor state axiom.We could say for any situation and action,if it's possible to execute that action in the situation,then the Open fluent is going to be true in the result of executing that actionif and only if the action is dialing the correct combination, X, or if the safe was already open in S and the action is not equal to Lock.That's one option.And the other option is the same thing on the left-hand side,and on the right-hand side it's open if and only if the action is dialing the correct combinationand the action is not equal to Lock.So tell me which, if any or all, of these are accurate representations of the problem.In each case I want you to tell me if each of these axioms are good as they stand alone.I don't want you to look at any combinations of axiomsbut just go through each one and check the box if you think that the axiom on that line aloneis a correct representation of the problem.[Norvig] The answers are, in this case, only the second is a correct representation.Any combination is possible to be dialed.It's not the case that it's only possible to dial the correct combination.Now, here we said that the lock button works at any point.Whether it's open or not, the lock button will always lock it.And so that's represented by the third option.True implies it's possible to lock.In this case the first one is a correct representation of the successor state axiom for Open,and the second one is not, because note what it says.If we already have the lock open and then we execute some dialing actionthat's not dialing the correct combination, X, we want it to remain open.But this second axiom would make it be closed, which is not what we want.In this exercise, I'm going to give you some English sentences and then some first-order logic sentencesand ask you does the first-order logic sentencecorrectly encode the English sentence, does it incorrectly encode it,or is it just an error that is not a legitimate sentencein first-order logic?The first English sentence is "Paris and Nice are both in France."Here's one possible translation.Paris and Nice are in France.Here's another. Paris is in France, and Nice is in France.Tell us if each of these is a correct encoding of English,incorrect, or if it's erroneous first-order logic syntax.The second sentence in English is "There is a country that borders Iran and Syria."Here are the possible translations. There exists a c, and we're going to use the predicate capital C to mean C when the argument is a country.So, there exists a c such that C of c, and we're going to use the predicate B to mean 2 objects border each other. So, c borders Iran, and c borders Syria.That's one translation. Here's the other translation.There exists a c if C is a country,then c borders Iran and c borders Syria.And the final English sentence is no 2 bordering countries can have the same map color, and we're going to use the predicate MC for map color.Here's one possibility for all x and y.X is a country, and y is a country.And x and y border each other.That implies it's not the case that the map color of x equals the map color of y.And I should say we're using map color here as a function, not as a predicate.Here's another possibility.For all x and y, it's not the case that x is a country,or it's not the case that y is a country,or it's not the case that x and y border,or it's not the case that the map color of xis equal to the map color of y. This problem is about planning in belief space.We have the 2 room vacuum world, and we've represented various belief states here.Now, in this version, there are no sensors and so no percepts. The actions are all deterministic. A right or left or suck action will always do what it's supposed to do,and the environment is static.That is, dirt stays put until it's cleaned up.Now, in the start, you know nothing about the environment.You have no input. You don't know what location you're in.You don't know where the dirt is, and your goal is to be in the leftmost of the 2 squaresand have both squares cleaned up, and what I want you to dois click on the sequence of actions, an action like this one or this one or this oneor this one, that constitute a path from the start state to the goal state.And then I want you to click on yes if that path is guaranteedto always reach the goal, and no if the path only sometimes reaches the goal.In this problem, we're again in the 2-location vacuum world,but this time around, we have local sensing,meaning at each turn, we get input of what location we're at,the left or the right, and whether there's dirt in that location.But we don't know what's going on in the other location.We have a dynamic world where dirt can appear anywhere.As we move around, dirt can spontaneously appearin the location we left or in the location we're going to visit.However, if we're sucking, the dirt can't appear, because if it did appear there, we would successfully suck it up.And now in addition, the right and left movesare stochastic in that they don't always succeed.Sometimes when you try to go right, you do successfully go right,and sometimes you stay in the same location, same for left.The suck action is always successful.It will always clean up dirt in the current location.Now, when we start out, we get the percept saying that we're in the leftmost locationand that location is clean, and that means our belief stateis that we're in either 5 or 7.Now, the first thing I want you to answer is if we decide to move right,what do we predict the possible belief state will be,the possible set of states in our belief state will be after we execute the right movement?Now we get a percept from the world, and we've observedthat we're in the rightmost square, so the action worked,and that square is dirty.Now we want to update our belief stateand click on all the states that belong to the belief state nowas we update due to this percept.Now our belief state contains 2 and 6,and we decide we want to execute the suck action.Now tell me, by clicking on the appropriate states, what states belong to the belief stateafter we make a prediction for what's going to happen after the suck action.Now, we make the observationright, clean, and I want you to update our belief state by clicking on the statesthat belong to the new belief state nowafter taking that observation into account.This is a famous problem called the monkey and bananas problem,described in the language of classical planning.There are six actions. The monkey can go from location x to y.It can push some object from x to y. It can climb up an object. It can grab something.It can climb down from an object, and it can un-grab something. Initially, the monkey is at location A. The bananas are at location B.The box is at C, and the monkey is at a low height, as is the box,but the bananas are at a high height,but the box is pushable and climbable.Now, assuming that we execute this plan--go from A to C, push the box from C to B,climb up on the box, grasp the bananas, and climb down from the box.What I want you to do is look at these definitions of actions, tell me how the state unfolds from this initial state here to the final state,and then click off all of these instances that are going to be true in the final state.[Norvig] The final problem involves situation calculus.In the domain I want to describe, we have a combination lock with 4 digits,and the correct combination that will open the lock we'll call X.There are 2 actions you can perform.One is to dial any combination on the dial,and if you dial the correct one, X, then the lock will open.And the other action you can perform is to press a lock button,and if you press that button, then the lock will be locked,whether it was open before or not.I'm going to describe some axioms, and I want you to tell me whether these axioms are correct for the domain or not.First the possibility axioms.One choice is the possibility axiom that saysif C equals X, then it's possible to dial C in situation S.And here I'm assuming that all variables are scopedso that we say an implicit for all C and for all S here.And X is not a variable. This is a constant, referring to the correct combination.The other possible axiom is for all C if C is greater than or equal to 0and less than or equal to 9999,then it's possible to dial C in any situation S.So tell me which, if any or both, of these axioms you think correctly encode the situation.Next we'll look at the possibility axioms for the lock action.Here's one.We can say if the safe is open in situation S,then it's possible to execute the lock action in S.Or maybe we should say if the safe is not open in S,then it's possible to execute Lock in S.Or maybe we should say if true, then it's possible to execute the lock action in situation S.And tell me which, if any, of those represents a correct representation of the problem.And finally we need successor state axioms for all the fluents,but there's really only one fluent, and that's whether or not the safe is open.So here's one example of a successor state axiom.We could say for any situation and action,if it's possible to execute that action in the situation,then the Open fluent is going to be true in the result of executing that actionif and only if the action is dialing the correct combination, X, or if the safe was already open in S and the action is not equal to Lock.That's one option.And the other option is the same thing on the left-hand side,and on the right-hand side it's open if and only if the action is dialing the correct combinationand the action is not equal to Lock.So tell me which, if any or all, of these are accurate representations of the problem.In each case I want you to tell me if each of these axioms are good as they stand alone.I don't want you to look at any combinations of axiomsbut just go through each one and check the box if you think that the axiom on that line aloneis a correct representation of the problem.So today is an exciting day.We'll talk about planning under uncertainty,and it really puts together from the materialwe've talked about in past classes.We talked about planning,but not under uncertainty, and you've had many, many classes of under uncertainty,and now it gets to the point where we can make decisions under uncertainty.This is really important for my own research field like robotics where the world is full of uncertainty, and the type of techniques I'll tell you about todaywill really make it possible to drive robotsin actual physical roles andfind good plans for these robots to execute.[Narrator] Planning under uncertainty.In this class so far we talked a good deal about planning.We talked about uncertainty and probabilities,and we also talked about learning,but all 3 items were discussed separately.We never brought planning and uncertainty together,uncertainty and learning, or planning and learning.So the class today, we'll fuse planning and uncertaintyusing techniques known as Markov decision processes or MDPs,and partial observer Markov decision processes or POMDPs.We also have a class coming up on reinforcement learningwhich combines all 3 of his aspects, planning, uncertainty, and machine learning.You might remember in the very first classwe distinguished very different characteristics of agent tasks, and here are some of those.We distinguished deterministic was the casting environments,and we also talked about photos as partial observable.In the area of planning so far all of our evidence falls into this field over here,like A*, depth first, right first and so on.The MDP algorithms which I will talk about firstfall into the intersection of fully observable yet stochastic, and just to remind us what the difference was,stochastic is an environment where the outcome of an action is somewhat random.Whereas an environment that's deterministicwhere the outcome of an action is predictableand always the same.An environment is fully observable if you can see the state of the environment which means if you can make all decisions based on the momentary sensory input.Whereas if you need memory,it's partially observable.Planning in the partially observable caseis called POMDP, and towards the end of this class,I'll briefly talk about POMDPs but not in any depth.So most of this class focuses on Markov decision processesas opposed to partially observable Markov decision processes.So what is a Markov decision process?One way you can specify a Markov decision process by a graph.Suppose you have states S1, S2, and S3,and you have actions A1 and A2.In a state transition graph, like this,is a finite state machine,and it becomes Markov if the outcomes of actions are somewhat random.So for example if A1 over here, with a 50% probability, leads to state S2 but with another 50% probability leads to state S3.So put differently, a Markov decision process of states, actions, a state's transition matrix,often written of the following formwhich is just about the same as a conditional state transition probabilitythat a state is primeis the correct posterior stateafter executing action A in a state S,and the missing thing is the objective for the Markov decision process.What do we want to achieve?For that we often define a reward function,and for the sake of this lecture, I will attach rewards just to states.So each state will have a function R attachedthat tells me how good the state is.So for example it might be worth $10to be in the state over here,$0 to be in the state over here,and $100 to be in a state over here.So the planning problem is now the problem which relies on an action to each possible state.So that we maximize our total reward.[Narrator] Before diving into too much detail,let me explain to you why MDPs really matter.What you see here is a robotic tour guide that the University of Bonn, with my assistance, deployed in the German museum in Bonn,and the objective of the this robot was to navigate the museum and guide visitors,mostly kids, from exhibit to exhibit.This is a challenging planning problem because as the robot moves it can't really predict its action outcomesbecause of the randomness of the environmentand the carpet and the wheels of the robot.The robot is not able to really follow its own commands very well,and it has to take this into consideration during the planning processso when it finds itself in a location it didn't expect,it knows what to do.In the second video here, you see a successor robotthat was deployed in the Smithsonian National Museum of American History in the late 1990swhere it guided many, many thousands of kidsthrough the entrance hall of the museum,and once again, this is a challenging planning problem.As you can see people are often in the way of the robot.The robot has to take detours.Now this one is particularly difficult becausethere were obstacles that were invisiblelike a downward staircase.So this is a challenging localization problemtrying to find out where you are,but that's for a later class.In the video here, you see a robot being deployed in a nursing homewith the objective to assist elderly peopleby guiding them around, bring them to appointments,reminding them to take their medication, andinteracting with them, and this robot has been active for many, many yearsand been used, and, again, it's a very challenging planning problemto navigate through this elderly home.And the final robot I'm showing you here.This was built with my colleague Will Whittaker at Carnegie Melon University.The objective here was to explore abandoned mines.Pennsylvania and West Virginiaand other states are heavily mined.There's many abandoned old coal mines,and for many of these mines,it's unknown what the conditions are and where exactly they are.They're not really human accessible. They tend to have roof fall and very low oxygen levels.So we made a robot that went insideand built maps of those mines.All these problems have in common that they have really challenging planning problems.The environments are stochastic.That is the outcome of actions are unknown,and the robot has to be able to react toall kinds of situations, even the ones that it didn't plan for.[Narrator] Let me give a much simpler exampleoften called grid world for MDPs,and I'll be using this insanely simple example over here throughout this class.Let's assume we have a starting state over here,and there's 2 goal states whoare often called absorbing stateswith very different reward or payout.Plus 100 for the state over here,minus 100 for the state over here,and our agent is able to move about the environment,and when it reaches one of those 2 states,the game is over and the task is done.Obviously the top state is much more attractive than the bottom state with minus 100.Now to turn this into an MDP, let's assume actions are somewhat stochastic.So suppose we had a grid cell, and we attempt to go north.The deterministic agent would always succeedto go to the north square if it's available,but let's assume that we only have an 80% chanceto make it to the cell in the north.If there's no cell at all,there's a wall like over here,we assume with 80% chance, we just bounce back to the same cell,but with 10% chance, we instead go left.Another 10% chance, we go right.So if an agent is over here and wishes to go north,then with 80% chance, it finds itself over here,10% over here, 10% over here.If it goes north from here,because there's no north cell,it'll bounce back with 80% probability,10% left, 10% right.In a cell like this one over here, it'll bounce back with 90% probability,80 from the top and 10 from the left, but it still has a 10% chance of going right.This is a stochastic state transition whichwe can equally define for actions,south, west and east, andnow we can see a situation like thisconventional planning is insufficient.So for example if you're plan a sequence of actions starting over here,you might go north, north, east, east, eastto reach our plus 100 absorbing or final state,but with this state transition model over here,even with the first step it might happen with 10% chance do you find yourselves over here,in which case conventional planning would not give us an answer.So we wish to have a planning method that provides an answer no matter where we are and that's called a policy.A policy assigns actions to any state.So for example a policy might look as follows:for this state, we wish to go north, north, east, east, east,but for this state over here, we wish to go north, maybe east over here, and maybe west over here.So each state, except for the absorbing states, we have to define an action to define a policy.The planning problem we have becomes one of finding the optimal policy[Narrator] To understand the beautyof a policy, let me look into stochastic environments,and let me try to apply conventional planning.Consider the same grid I just gave you,and let's assume there's a discrete start state,the one over here, and we wish to find an action sequence that leads us to the goal state over here.In conventional planning we would create a tree.In C1, we're given 4 action choices,north, south, west, and east.However, the outcome of those choices is not deterministicSo rather than having a single outcome,nature will choose for us the actual outcome.In the case of going north, for example, we may find ourselves in B1,or back into C1.Similarly for going south, we might find ourselves in C1, or back in C2, and so on.This tree has a number of problems.The first problem is the branching factor.While we have 4 different action choices,nature will give us up to 3 different outcomeswhich makes up to 12 different things we have to follow.Now in conventional planning we might have to follow just 1 of those,but here we might have to follow up to 3 of those things.So every time we plan a step ahead, we might have to increase the breadth of the search of tree by at least a factor of 3.So one of the problem is the branching factor is too large. [Narrator] To understand the branching factor,let me quiz you on how many states you can possibly reachfrom any other states, and as an examplefrom C1, you can reach under any action choice B1, C1, and C2, but itwill give you an affective branching factor of 3. So when I ask you what's the affective branching factor in B3?What is the maximum level of states you can reach under any possible action from B3?So how many states can we reach from B3 over here?[Narrator] And the answer is 8.If you go north, you might reach this state over here,this one over here, this one over here.If you go east, you might reach this state over here,or this one over here, or this one over here, or this one over here.When you put it all together, you can reach all of those 8 states over here.[Narrator] There are other problems withthe search paradigm. The second one is that the tree could be very deep,and the reason is we might be able to circle forever in the area over here without reaching the goal state, and that makes for a very deep tree, and untilwe reach the goal state, we won't even know it's the best possible action.So conventional planning might have difficulties with basically infinite loops.The third problem is that many states recur in the search.In a star, we were careful to visit each state only once,but here because the actions might carry you back here to the same state,C1 is, for example, over here and over here. You might find that many states in the tree might be visited many, many different times.Now if you had a state it doesn't really matter how you got there.Yet, the tree doesn't understand this, and it might expand states more than once.These are the 3 problems that are overcome by our policy method,and this motivates in part by calculating policies is so much better of an idea than using conventional planning and still casting environments. So let's get back to the policy case.[Narrator] Let's look at the grid world, again,and let me ask you a question. I wish to find an optimal policy for all these states that with maximum probability leads me to the absorbing state plus 100, and as I just discussed, I assume there's 4 different actions, north, south, west, and eastthat succeed with probability 80% providedthat the corresponding grid cell is actually attainable.I wish to know what is the optimal actionin the corner set over here, A1,and I give you 4 choices, north, south, west, and east.[Narrator] And the answer is east.East in expectation transfers you to the right side, and you're one closer to your goal position. [Narrator] Let me ask the same question for the state over here, C1,which one is the optimal action for C1?[Narrator] And the answer is north. It gets you one step closer.There is 2 equally long paths, but over hereyou risk falling into the minus 100; therefore, you'd rather go north.[Narrator] The next question is challenging.Consider state C4, which one is the optimal actionprovided that you can run around as long as you want.There's no costs associated with steps, butyou wish to maximize the probability of ending up in plus 100 over here.Think before you answer this question.[Narrator] And the answer is south.The reason why it's south is if we attempt to go south,an 80% probability we'll stay in the same cell.In fact, a 90% probability because we can't go south and we can't go east. In a 10% probability, we find ourselves over here which is a relatively safe state because we can actually go to the left side.If we were to go just west which is the intuitive answer, then there's a 10% chance we end up in the minus 100 absorbing state.You can convince yourself if you go south,find ourselves eventually in state C3, and then go west, west, north, north, east, east, east.You will never ever run risk of falling into the minus 100, and that argument is tricky and to convince ourselves let me ask the other hard question:so what shall we do in state B3 that's the optimal action?[Narrator] And the answer is west.If you're over here, and we go east,we'd likely end up with minus 100.If you go north, which seems to be the intuitive answer,there's a 10% chance we fall into the minus 100.However, if we go west, then there's absolutely no chance we fall into the minus 100.We might find ourselves over here.We might be in the same state. We might find ourselves over here, but from these states over here,there's safe policies that can safely avoid the minus 100.[Narrator] So even for the simple grid world, the optimal control policy assuming stochastic actionsand no costs of moving, except for the final absorbing costs, is somewhat nontrivial. Take a second to look at this.Along here it seems pretty obvious, but for the state over here, B3, and for the state over here, C4,we choose an action that just avoids falling into the minus 100, which is more important than trying to make progress towards the plus 100.Now obviously this is not the general case of an MDP,and it's somewhat frustrating they'd be willing to run through the wall,just so as to avoid falling into the minus 100,and the reason why this seems unintuitive is because we're really forgetting the issue of costs.In normal life, there is a cost associated with moving.MDPs are gentle enough to have a cost factor,and the way we're going to denote costs is by defining our award function over any possible state.We are reaching the state A4, gives us plus 100, minus 100 for B4,and perhaps minus 3 for every other state,which reflects the fact that if you take a step somewherethat we will pay minus 3.So this gives an incentive to shorten the final action sequence.So we're now ready to state the actual objectiveof an MDP which is to minimize notjust the momentary costs, but the sumof all future rewards,but you're going to write RT to denote the fact that this reward has received time T, and becauseour reward itself is stochastic,we have to complete the expectation over those, and that we seek to maximize.So we seek to find the policy that maximizes the expression over here. Now another interesting caveat is a sentence people put a so called discount factor into this equation with an exponent of T, where a discount factor was going to be 0.9,and what this does is it decays future reward relative to more immediate rewards, and it's kind of an alternative way to specify costs.So we can make this explicit by a negative reward per state or we can bring in a discount factorthat discounts the plus 100 by the number of steps that it went by before it reached the plus 100. This also gives an incentive to get to the goal as fast as possible.The nice mathematical thing about discount factor isit keeps this expectation bounded.It easy to show that this expression over herewill always be smaller or equal to 1 over 1 minus gamma times the absolute reward maximizing value and which in this case would be plus 100.The definition of the expected sum of futurepossible discounted reward that it has given youallows me to define a value function.For each status, my value of the stateis the expected sum of future discounted rewardprovided that I start in state S,then I execute policy pi.This expression looks really complex,but it really means something really simple,which is suppose we start in the state over here,and you get +100 over here, -100 over here.And suppose for now, every other state costs you -3.For any possible policy that assigns actions to the non-absorbing states, you can now simulate the agent for quite a while and compute empiricallywhat is the average reward that is being receiveduntil you finally hit a goal state.For example, for the policy that you like,the value would, of course, for any state depend on how much you make progress towards the goal,or whether you bounce back and forth.In fact, in this state over here, you might bounce downand have to do the loop again.But there's a well defined expectationover any possible execution of the policy pithat is generic to each state and each policy pi.That's called a value.And value functions are absolutely essential to MDP,so the way we're going to plan is we're going to iterateand compute value functions, and it will turn outthat by doing this, we're going to find better and better policies as well.Before I dive into mathematical detail aboutvalue functions, let me just give you a tutorial.The value function is a potential functionthat leads from the goal location--in this case, the 100 in the upper right--all the way into the space so that hill climbingin this potential function leads you on the shortest path to the goal.The algorithm is a recursive algorithm.It spreads the value through the space, as you can see in this animation,and after a number of iterations, it converges,and you have a grayscale valuethat really corresponds to the best way of getting to the goal.Hill climbing in that function gets you to the goal.You can simplify.Think about this as pouring a glass of milkinto the 100th state and having the milk descend through the maze, and later on, when you go in the gradient of the milk flow,you will reach the goal in the optimal possible way.Let me tell you about a truly magical algorithm called value iteration.In value iteration, we recursively calculate the value functionso that in the end, we get what's called the optimal value function.And from that, we can derive, look up, the optimal policy.Here's how it goes.Suppose we start with a value function of 0 everywhereexcept for the 2 absorbing states, whose value is +100 and -100.Then we can ask ourselves the question is, for example,for the field A3, 0 a good value.And the answer is no, it isn't. It is somewhat inconsistent.We can compute a better value.In particular, we can understand thatif we're in A3 and we choose to go east,then with 0.8 chance we should expect a value of 100.With 0.1 chance, we'll stay in the same state,in which case the value is -3.And with 0.1 chance, we're going to stay down here for -3.With the appropriate definition of value,we would get the following formula,which is 77.So, 77 is a better estimate of valuefor the state over here.And now that we've done it, we can ask ourselves the questionis this a good value, or this a good value, or this a good value?And we can propagate value backwards in reverse order of action executionfrom the positive absorbing state through this grid worldand fill every single state with a better value estimatethen the one we assumed initially.If we do this for the grid over here and run value iterationthrough convergence, then we get the following value function.We get 93 over here. We're very close to the goal.89, 85, 81, 77, 73, 70, over here.This state will be worth 68, and this state is worth 47, and the reason why these are not so good is becausewe might stay quite a while in thosebefore we'll be able to execute an actionthat gets us outside the state.Let me give you an algorithm that defines value iteration.We wish to estimate recursively the value of state S.And we do this based on a possible successor stateas prime that we look up in the existing table.Now, actions A are non-deterministic.Therefore, we have to go through all possible as primesand weigh each outcome with the associated probability.The probability of reaching S prime given that we started state Sand apply action A.This expression is usually discounted by gamma,and we also add the reward or the costs of the state.And because there's multiple actions and it's up to usto choose the right action, we will maximize over all possible actions.See, we look at this equation, and it looks really complicated,but it's actually really simple.We compute a value recursively based on successor valuesplus the reward and minus the cost that it takes us to get us there.Because Mother Nature picks a successor state for us for any given action,if you compute an expectation over the value of the successor stateweighted by the corresponding probabilities which is happening over here,and because we can choose our action,we maximize over all possible actions.Therefore, the max as opposed to the expectation on the left side over here.This is an equation that's called backup.In terminal states, we just assign R(s),and obviously, in the beginning of value iteration, these expressions are different, and we have to update.But as Bellman has shown a while ago,this process of updates converges.After convergence, this assignment over here is replaced by the equality sign,and when this equality holds true,we have what is called a Bellman equality or Bellman equation.And that's all there is to know to compute values.If you assign this specific equation over and over again to each state,eventually you get a value function that looks just like this,where the value really corresponds to what's the optimal futurecost reward trade off that you can achieve if you act optimally in any given state over here.Let me take my example world and apply value iteration in a quiz.As before, assume the value is initialized as+100 and -100 for the absorbing statesand 0 everywhere else.And let me make the assumption that our transition probability is deterministic.That is, if I execute the east action of this state over herewith probability 1 item over here,if I assume the north action over here, probability 1, I will find myself in the same state as before.There is no uncertainty anymore.That's really important for now, just for this one quiz.I'll also assume gamma equals 1, just to make things a little bit simpler.And the cost over here is -3 unless you reach an absorbing state.What I'd like to know, after a single backup,what's the value of A3?And the answer is 97.It's easy to see that the action east is the value maximizing action.Let's plug in east over here. Gamma equals 1.This is a single successor state of 100,so if we have 100 over here minus the 3 that it costs us to get there, we get 97.Let's run value iteration again, and let me askwhat's the value for B3, assuming that we already updatedthe value for A3 as shown over here.And again, making use of the observation that ourstate transition function is deterministic, we get 94, and the logic is the same as before.The optimal action here is going north, which we will succeed with the probability 1.Therefore, we can use the value recursively from A3to deflect back to B3.97 - 3 gives us 94.And finally, I would like to know what's the value of C1, the figure down here, after we ran value iterationover and over again all the way to a convergence.Again, gamma equals 1. State transition function is deterministic.And the answer is easily obtained if you just subtract -3 for each step.We get 88 and 85 over here.We could also reach the same value going around here.So, 85 would have been the right answer,and this will be the value function after convergence.It's beautiful to see that the value function is effective the distance to the positive absorbing state times 3subtracted from 100.So, we have 97, 94, 91, 88, 85 and so on.This is a degenerate case.If we have a deterministic state transition function,it gets more tricky to calculate for the stochastic case.Let me ask the same question for the stochastic case.We have the same world as before,and actions have stochastic outcomes.The probability 0.8, we get the action we commanded,otherwise we get left or right.And assuming that the initial values are all 0, calculate for me for a single backup the value of A3.This should look familiar from the previous material.It's 77, and the reason is in A3,we have an 80% chancefor the action going east to reach 100.But the remaining 20%, we either stay in place or go to the field down here,both of which have an initial value of 0.That gives us 0, but we have to subtract the cost of 3, and that gives us 80 - 3 = 77.It's also easy to verify that any of the other actions have lower values.For example, the value of going west will be0 in all possible outcomes given the current value functionminus 3, so the value of going west would right now be estimated as -3, and 77 is larger than -3.Therefore, we'll pick 77 as the action that maximizesthe updated equation over here.And here's a somewhat non-trivial quiz.For the state B3, calculate the value functionassuming that we have a value function as shown over hereand all the open states have a value of assumed 0,because we're still in the beginning of our value update.What would be our very first value function for B3that we compute based on the values shown over here?And the answer is 48.6.And obviously, it's not quite as trivial as the calculation beforebecause there's 2 competing actions.We can try to go north, which gives us the 77but risks the chance of falling into the -100.Or we can go west, as before, which gives us a much smaller chanceto reach 77, but avoids the -100.Let's do both and see which one is better.If we go north, we have a 0.8 chance of reaching 77.There's now a 10% chance of paying -100and a 10% chance of staying in the same location,which at this point is still a value of 0.We subtract our costs of 3, and we get 61.6- 10 - 3 = 48.6.Let's check the west action value.We reach the 77 with probability 0.1 with 0.8 chance we stay in the same cell,which has the value of 0, and with 0.1 chance, we end up down here,which also has a value of 0.We subtract our costs of -3,and that gives us 7.7 - 3 = 4.7.At this point, going west is vastly inferiorto going north, and the reason is we already propagated a great value of 77 for this cell over here,whereas this one is still set to 0.So, we will set it to 48.6.So, now that we have a value backup functionthat we discussed in depth, the question now becomeswhat's the optimal policy?And it turns out this value backup function defines the optimal policy as completely oppositeof which action to pick, which is just the action that maximizes this expression over here.For any state S, any value function V,we can define a policy,and that's the one that picks the action under argmaxthat maximizes the expression over here.For the maximization, we can safely draw up gamma and R(s).Baked in the value iteration function was alreadyan action choice that picks the best action.We just made it explicit.This is the way of backing up values,and once values have been backed up, this is the way to find the optimal thing to do.I'd like to show you some value function after convergenceand the corresponding policies.If we assume gamma = 1 and our cost for the non-absorbing stateequals -3, as before, we get the following approximate value functionafter convergence, and the corresponding policy looks as follows.Up here we go right until we hit the absorbing state.Over here we prefer to go north.Here we go left, and here we go north again.I left the policy open for the absorbing statesbecause there's no action to be chosen here.This is a situation wherethe risk of falling into the -100 is balanced bythe time spent going around.We have an action over here in this visible state here that risks the 10% chance of falling into the -100.But that's preferable under the cost model of -3to the action of going south.Now, this all changes if we assume a cost of 0 for all the states over here, in which case,the value function after convergence looks interesting.And with some thought, you realize it's exactly the right one.Each value is exactly 100,and the reason is with a cost of 0,it doesn't matter how long we move around.Eventually we can guarantee in this case we reach the 100,therefore each value after backups will become 100.The corresponding policy is the one we discussed before.And the crucial thing here is that for this state,we go south, if you're willing to wait the time.For this state over here, we go west, willing to wait the time so as to avoidfalling into the -100.And all the other states resolveexactly as you would expect them to resolveas shown over here.If we set the costs to -200,so each step itself is even more expensivethen falling into this ditch over here,we get a value function that's strongly negative everywherewith this being the most negative state.But more interesting is the policy.This is a situation where our agent tries to end the gameas fast as possible so as not to endure the penalty of -200.And even over here where it jumps itself into the -100'sit's still better than going north and taking the excess of 200 as a penaltyand then leave the +100.Similarly, over here we go straight north, and over here we go as fast as possibleto the state over here. Now, this is an extreme case.I don't know why it would make sense to set a penalty for lifethat is so negative that even negative death is worse than living,but certainly that's the result of running value iteration in this extreme case.So, we've learned quite a bit so far.We've learned about Markov Decision Processes.We have fully observable with a set of statesand corresponding actions where they have stochastic action effectscharacterized by a conditional probability entity of P of S primegiven that we apply action A in state S.We seek to maximize a reward functionthat we define over states.You can equally define over states in action pairs.The objective was to maximize the expectedfuture accumulative and discounted rewards,as shown by this formula over here.The key to solving them was called value iterationwhere we assigned a value to each state.There's alternative techniques that have assigned valuesto state action pairs, often called Q(s, a),but we didn't really consider this so far.We defined a recursive update ruleto update V(s) that was very logicalafter we understood that we have an action choice,but nature chooses for us the outcome of the actionin a stochastic transition probability over here.And then we observe the value iteration convergedand we're able to define a policy if we're assuming the argmax under the value iteration expression,which I don't spell out over here.This is a beautiful framework.It's really different from planning than before because of the stochasticity of the action effects.Rather than making a single sequence of states and actions,as would be the case in deterministic planning,now we make an entire field a so-called policythat assigns an action to every possible state.And we compute this using a technique called value iterationthat spreads value in reverse order through the field of states.So far, we talked about the fully observable case,and I'd like to get back to the more general caseof partial observability.Now, to warn you, I don't think it's worthwhile in this classto go into full depth about the type of techniquesthat are being used for planning and uncertaintyif the world is partially observable.But I'd like to give you a good flavorabout what it really means to plan in information spacesthat we reflect the types of methods that are being brought to bearin planning and uncertainty.Like my Stanford class, I don't go into details here eitherbecause the details are much more subject to more specialized classes,but I hope you can enjoy the type of flavor of materialsthat you're going to get to see in the next couple of minutes.So we now learned about fully observable environments,and planning in stochastic environments with MDPsI'd like to say a few words about partially observable environments,or POMDPs--which I won't go into in depth; the material is relatively complex. But I'd like to give you a feeling for why this is important, and what type of problemsyou can solve with this, that you could never possibly solve with MDPs. So, for example, POMDPs address problems of optimal exploration versus exploitation, where some of the actions might be information-gathering actions;whereas others might be goal-driven actions. That's not really possible in the MDPs because the state space is fully observableand therefore, there is no notion of information gathering. I'd like to illustrate the problem, using a very simple environmentthat looks, as follows:Suppose you live in world like this;and your agent starts over here,and there are 2 possible outcomes. You can exit the maze over here--where you get a plus 100--or you can exit the maze over here, where you receive a minus 100.Now, in a fully observable case, and in a deterministic case, the optimal plan might look something like this;and whether or not is goes straight over here or not, depends on the details. For example, whether the agent has momentum or not. But you'll find a single sequence of actions and states that might cut the corners,as close as possible, to reach the plus 100 as fast as possible. That's conventional planning.Let's contrast this with the case we just learned about,which is the fully observable case or the stochastic case. We just learned that the best thing to compute is a policythat assigns to every possible state, an optimal action; and simplified speaking, this might look as follows:Where each of these arrows corresponds to a sample control policy. And those are defined in part of the state space that are even far away. So this wouuld be an example of a control policy where all the arrows gradually point you over here. We just learned about this, using MDPs and value iteration. The case I really want to get at is the case of partial observability--which we will eventually solve, using a technique called POMDP.And in this case, I'm going to keep the location of the agent in the maze observable.The part I'm going to make unobservable is where, exactly, I receive plus 100and where I receive minus 100.Instead, I'm going to put a sign over here that tells the agent where to expect plus 100,and where to expect minus 100.So the optimum policy would be to first move to the sign, read the sign;and then return and go to the corresponding exit, for which the agent now knows where to receive plus 100.So, for example, if this exit over here gives us plus 100, the sign will say Left. If this exit over here gives us plus 100, the sign will say Right.What makes this environment interesting is that if the agent knew which exit would have plus 100, it will go north, from a starting position. It goes south exclusively to gather information. So the question becomes: Can we devise a method for planningthat understands that, even though we'd wish to receive the plus 100 as the best exit, there's a detour necessary to gather information. So here's a solution that doesn't work: Obviously, the agent might be in 2 different worlds--and it doesn't know. It might be in the world where there's plus 100 on the Left sideor it might be in the world with plus 100 on the Right side,with minus 100 in the corresponding other exit. What doesn't work is you can't solve the problem for both of these casesand then put these solutions together--for example, by averaging. The reason why this doesn't work is this agent, after averaging, would go north. It would never have the idea that it is worthwhile to go south,read the sign, and then return to the optimal exit. When it arrives, finally, at the intersection over here,it doesn't really know what to do. So here is the situation that does work--and it's related to information space or belief space. In the information space or belief space representation you do planning,not in the set of physical world states, but in what you might know about those states. And if you're really honest, you find out that there's a multitude of belief states.Here's the initial one, where you just don't know where to receive 100.Now, if you move around and either reach one of these exits or the sign, you will suddenly know where to receive 100. And that makes your belief state change-- and that makes your belief state change. So, for example, if you find out that 100 is Left, then your belief state will look like this--where the ambiguity is now resolved. Now, how would you jump from this state space or this state space?The answer is: when you read the sign, there's a 50 percent chance that the location over here will result in a transition to the location over here--50 percent because there's a 50 percent chance that the plus 100 is on the Left.There's also a 50 percent chance that the plus 100 is on the Right,so the transition over here is stochastic;and with 50 percent chance, it will result in a transition over here. If we now do the MDP trick in this new belief space, and you pour water in here, it kind of flows through here and creates all these gradients--as we had before.We do the same over here and all these gradients are being created point to this exit on the Left side.Then, eventually, this water will flow through here and create gradients like this;and then flow back through here, where it creates gradients like this. So the value function is plus 100 over here, plus 100 over herethat gradually decrease down here, down here;and then gradually further decrease over here-- and even further decrease over there, so we've got arrows like these.And that shows you that in this new belief space, you can find a solution.In fact, you can use value iteration--MDP's value iteration--in this new space to find a solution to this really complicated partially observable planning process.And the solution--just to reiterate--we'll suggest: Go south first, read the sign, expose yourself to the random position to the Left or Right world in which you are now able to reach the plus 100 with absolute confidence. So now we have, learned pretty much, all there is to know about Planning Under Uncertainty. We talked about Markov Decision Processes.We explained the concept of information spaces; and what's better, you can actually apply it. You can apply it to a huge number of problems where the outcome of states are uncertain. There is a huge legislation about robot motion planning.Here are some examples of robots moving through our environmentsthat use MDP-style planning techniques; and these methods have become vastly popular in artificial intelligence--so I'm really glad you now understand the basics of those and you can apply them yourself. Hi--welcome back. You just learned how Markov Decision Processescan be used to determine an optimal sequenceof actions for an agent in a stochastic environment. And that is, an agent that knows the correct model of the environmentcan navigate, finding its ways to the positive rewards and avoiding the negative penalties.But it can only do that if he knows where the rewards and penalties are. In this Unit, we'll see how a techniquecalled reinforcement learningcan guide the agent to an optimal policy,even though he doesn't know anything about the rewards when he starts out. No subtitles...For example, in the 4 by 3 GridWorld, what if we don't know where the plus 1 and minus 1 rewards are when we start out? A reinforcement learning agent can learn to explore the territory,find where the rewards are, and then learn an optimal policy.Whereas, an MDP solver can only do that once it knows exactly where the rewards are. Now, this idea of wandering around and then finding a plus 1 or a minus 1is analogous to many forms of games, such as backgammon--and here's an example: backgammon is a stochastic game;and at the end, you either win or lose. And in the 1990s, Gary Tesauro at IBMwrote a program to play backgammon. His first attempt tried to learn the utility of a Game state, U of S, using examples that were labelled by human expert backgammon players. But this was tedious work for the experts,so only a small number of states were labelled. The program tried to generalize from that, using supervised learning, and was not able to perform very well. So Tesauro's second attempt used no human expertise and no supervision.Instead, he had 1 copy of his program play against another;and at the end of the game, the winner got a positive reward,and the loser, a negative. So he used reinforcement learning; he backed up that knowledge throughout the Game states, and he was able to arrive at a function that had no input from human expert players,but, still, was able to perform at the level of the very best players in the world. He was able to do this, after learning from examples of about 200,000 games. Now, that may seem like a lot--but it really only covers about 1 trillionth of the total state space of backgammon. Now, here's another example: This is a remote controlled helicopter that Professor Andrew Ng at Stanford trained, using reinforcement learning;and the helicopter--oh--oh, sorry--I made a mistake--I put this picture upside downbecause--really, Ng trained the helicopter to be able to fly fancy maneuvers--like flying upside down. And he did that by looking at only a few hours of training data from expert helicopter pilots who would take over the remote controls, pilot the helicopter--and those would all be recorded--and then, you would get rewards from when it did something good, or when it did something bad;and Ng was able to use reinforcement learrning to build an automated helicopter pilot, just from those training examples. And that automated pilot, too, can perform tricks that only a handful of humans are capable of performing. But enough of this still picture--let's watch a video of Ng's helicopters in action. [Stanford University Autonomous Helicopter][sound of helicopter flying] [Chaos][Stanford University Autonomous Helicopter]Let's stop and review the 3 main forms of learning. We have supervised learning, in which the training setis a bunch of input/output pairs--X1,Y1; X2, Y2; et cetera--in which we try to produce a function: y equals f of x--and so the learning is producing this function, f. Then we have unsupervised learning,in which we're given just a set of data points--X1, X2, and so on--and each of these points, maybe, has many dimensions, many features. And what we're trying to learn is some patterns in that--some clusters of these data--or you could just say what we're trying to learn is a probability distributionor what's the probability that this random variable will have particular values;and learn something interesting from that. In this Unit, we're introducing the third type of learning--reinforcement learning--in which we have a sequence of action and state transitions.So: state and action, state and action--and so on. And at some point, we have some rewards associated with these. So there's a reward, and maybe not a reward for this state;and then another reward for this state--and the rewards are just scalar numbers, positive or negative numbers. What we're trying to learn here is:at optimal policy, what's the right thing to do in any of the states?Let's show some examples of machine learning problemsand I want you to tell me, for each one, whether it's best addressed with supervised learning, unsupervised learning, or reinforcement learning. And the first example is speech recognition--where I have examples of voice recordings,and then the transcript's intermittent text for each of those recordings;and from them, I try to learn a model of language. Is that supervised, unsupervised or reinforcement? Next example is analyzing the spectral emissions of stars and trying to find clusters of stars in dissimilar typesthat may be of interest to astronomers.Would that be supervised, unsupervised or reinforcement?The data here would just consist of:for each star, a list of all the different emission frequencies of light coming to earth.Next example is lever pressing. So--I have a rat who is trained to press a leverto get a release of foodwhen certain conditions are met. Is that supervised, unsupervised or reinforcement learning? And finally, the problem of an elevator controller.Say I have a bank of elevators in a buildingand they have to have some program--some policy--to decide which elevator goes upand which elevator goes downin response to the percepts, which would be the button presses at various floors in the building. And so, I have a sequence of button presses, and I have the wait time that I am trying to minimize--so after each button press, the elevator moves; the person waiting is waiting for a certain amount of time, and then gets picked up,and the algorithm is, given that amount of wait time. Would that be supervised, unsupervised or reinforcement?The answers are that speech recognitioncan be handled quite well by supervised learning.That is, we have input/output pairs;the input is the speech signal,and the output is the words that they correspond to.Analyzing the spectral emissions of stars is an example of unsupervised clusteringwhere we're taking the input data--we have data for each star, but we don't have any label associated with it. Rather, we're trying to make up labels by clustering them together,giving them to scientists, and then letting the scientists see:Do these clusters make any sense?Lever pressing is a classic example of reinforcement learning. In fact, the term "reinforcement learning" was used for a long time in animal psychology,before it was used in computer science.And elevator controllers is another area that has been investigated, using reinforcement learningand, in fact, very good algorithms--better than the previous state of the art--have been made, using reinforcement learning techniques. So the input, again, is a set of state/action transitions;and then the reinforcement is--in this case, it's always a negative numberbecause there's always a wait time.And so that's the penalty--we're trying to minimize that penalty,but all we get is the amount of wait time that we're trying to minimize. Now, before we get into the math of reinforcement learning, let's review MDPs--which are, of course, Markov Decision Processes.An MDP consists of a set of states--S is an element of the state, S;a set of actions--A is an element of the actions that are available in each of the states, S. And we're going to distinguish a Start state,which we'll call S-zero,and then we need a transition function that says:How does the world evolve as we take actions in the world? And we can denote that by the probability that we get a Result state, S prime--given that we start in state, S, and apply action, A. That's a probability distribution because the world is stochastic.The same result doesn't happen every time,when we do the same action,so we have this probability distribution. In some notations, you'll see:T of S, A, S--for the transition function. And then, in addition to the transition,we need a reward function--which we'll denote R.Sometimes that's over the whole triplet--the reward that you get from starting in one state,taking an action, and arriving at another state;sometimes we only need to talk about the result state. So in the 4 by 3 Grid World, for example, we don't care how you got to this state--it's just, when you get to one of the states in the upper right, you get a plus 1 or minus 1 reward. And similarly, in a game like backgammon--when you win or lose, you get a positive or negative reward. It doesn't matter what move you took to win or lose. And so that's all there is to MDPs. Now to solve an MDP,we're trying to find a policy--pi of S--that's going to be our answer. The pi that we want--the optimal policy--is the one that's going to maximizethe discounted, total Reward. So what we mean is:we want to take the sum over all Times into the future of the Rewardthat you get from starting out in the state that you're in, in time T--and then applying the policy to that state, and arriving at a new state, at time T plus 1.And so we want to maximize that sum--but the sum might be infiniteand so, what we do is we take this value, Gamma, and raise it to the T power, sayingwe're going to count future Rewards less than current Rewards--and that way, we'll make sure that the sum total is bounded. So we want the policy that maximizes that result. If we figure out the utility of the stateby solving the Markov Decision Process, then we have: the utility of any state, S, is equal to the maximum over all possible actions that we could take in Sof the expected value of taking that action. And what's the expected value? Well, it's just the sum over all resulting statesof the transition model--the probability that we get to that state,given from the start state, we take an actionspecified by the optimal policytimes the utility of that resulting state. So--look at all possible actions;choose the best one--according to the expected, in terms of probability utility. Now here's where reinforcement learning comes into play:What if you don't know R--the Reward function? What if you don't even know P--the transition model of the world? Then you can't solve the Markov Decision Processbecause you don't have what you need to solve it. However, with reinforcement learning, you can learn R and P by interacting with the world or you can learn substitutes that will tell you as much as you know, so that you never actually have to compute with R and P.What you learn, exactly, depends on what you already know and what you want to do. So we have several choices. One choice is we can build a utility-based agent.So we're going to list agent types, based on what we know,what we want to learn,and what we then use once we've learned. So for a utility-based agent, if we already know T, the transition model,but we don't know R, the Reward model,then we can learn R--and use that, along with P, to learn our utility function;and then go ahead and use the utility function just as we did in normal Markov Decision Processes. So that's one agent design. Another design that we'll see in this Unitis called a Q-learning agent.In this one, we don't have to know P or R;and we learn a value function, which is usually denoted by Q.And that's a type of utilitybut, rather than being a utility over states, it's a utility of state action pairs--and that tells us:For any given state and any given action, what's the utility of that result--without knowing the utilities and rewards, individually? And then we can just use that Q directly.So we don't actually have to ever learn the transition model, P,with a Q-learning agent. And finally, we can have a reflex agent where, again, we don't need to know P and R to begin with;and we learn directly, the policy, pi of S;and then we just go ahead and apply pi. So it's called a reflex agent because it's pure stimulus response:I'm in a certain state, I take a certain action. I don't have to think about modeling the world, in terms of: What are the transitions--where am I going to go next? I just go ahead and take that action. Now, the next choice we have in agent design revolves around how adventurous he wants to be. One possibility is what's called the passive reinforcement learning agent--and that can be any of these agent designs, but what passive means is that the agent has a fixed policy and executes that policy. But it learns about the reward function, R, and maybe the transition function, P, if it didn't already know that.It learns that while executing the fixed policy.So let me give you an example.Imagine that you're on a ship in uncharted waters and the captain has a policy for piloting the ship. You can't change the captain's policy. He or she is going to execute that, no matter what. But it's your job to learn all you can about the uncharted waters. In other words, learn the reward function, given the actions and the state transitionsthat the ship is going through. You learn, and remember what you've learned,but that doesn't change the captain's policy--and that's passive learning. Now, the alternative is calledactive reinforcement learning--and that's where we change the policy as we go. So let's say, eventually, you've done such a great jobof learning about the uncharted waterthat the captain says to you,"Okay--I'm going to hand over control and as you learn, I'm going to allow youto change the policy for this ship. You can make decisions of where we're going to go next."And that's good, because you can start to cash in early on your learning and it's also good because it gives you a possibility to explore.Rather than just say: What's the best action I can do right now?--you can say: What's the action that might allow me to learn something--to allow me to do better in the future? Let's start by looking at passive reinforcement learning. I'm going to describe an algorithm called Temporal Difference Learning--or TD. And what that means--sounds like a fancy name, but all it really means is we're going to move from one state to the next;and we're going to look at the difference between the 2 states,and learn that--and then kind of back upthe values, from one state to the next. So if we're going to follow a fixed policy, pi,and let's say our policy tells us to go this way, and then go this way. We'll eventually learn that we get a plus 1 reward thereand we'll start feeding back that plus 1, saying:if it was good to get a plus 1 here, it must be somewhat good to be in this state,somewhat good to be in this state--and so on, back to the start state. So, in order to run this algorithm, we're going to try to build up a table of utilities for each state and along the way, we're going to keep track of the number of times that we visited each state. Now, the table of utilities, we're going to start blank--we're not going to start them at zero or anything elsewhere they're just going to be undefined. And the table of numbers, we're going to start at zero, saying we visited each state a total of zero times. What we're going to do is run the policy,have a trial that goes through the state; when it gets to a terminal state, we start it over again at the start and run it again;and we keep track of how many times we visited each state, we update the utilities, and we get a better and better estimate for the utility.And this is what the inner loop of the algorithm looks like--and let's see if we can trace it out. So we'll start at a start state,we'll apply the policy--and let's say the policy tells us to move in this direction.Then we get a reward here, which is zero; and then we look at it with the algorithm, and the algorithm tells us if the state is new--yes, it is; we've never been there before--then set the utility of that state to the new reward, which is zero. Okay--so now we have a zero here;and then let's say, the next step, we move up here. So, again, we have a zero;and let's say our policy looks like a good one,so we get: here, we have a zero.We get: here, we have a zero. We get: here--now, this state,we get a reward of 1, so that state gets a utility of 1.And all along the way, we have to think about how we're backing up these values, as well. So when we get here, we have to look at this formula to say:How are we going to update the utility of the prior state?And the difference between this state and this state is zero. so this difference, here, is going to be zero--the reward is zero, and so there's going to be no update to this state. But now, finally--for the first time--we're going to have an actual update.So we're going to update this state to be plus 1, and now we're going to think about changing this state. And what was its old utility?--well, it was zero.And then there's a factor called Alpha,which is the learning rate that tells us how much we want to move this utilitytowards something that's maybe a better estimate. And the learning rate should be such that, if we are brand new, we want to move a big step;and if we've seen this state a lot of times, we're pretty confident of our numberand we want to make a small step.So let's say that the Alpha function is 1 over N plus 1. Well, we'd better not make it 1 over N plus 1, when N is zero.So 1 over N plus 1 would be ½;and then the reward in this state was zero;plus, we had a Gamma--and let's just say that Gamma is 1, so there's no discounting; and then we look at the difference between the utility of the resulting state--which is 1--minus the utility of this state, which was zero. So we get ½, 1 minus zero--which is ½.So we update this;and we change this zero to ½.Now let's say we start all over againand let's say our policy is right on track; and nothing unusual, stochastically, has happened.So we follow the same path, we don't update--because they're all zeros all along this path.We go here, here, here; and now it's time for an update. So now, we've transitioned from a zero to ½--so how are we going to update this state?Well, the old state was zeroand now we have a 1 over N plus 1--so let's say 1/3.So we're getting a little bit more confident--because we've been theretwice, rather than just once. The reward in this state was zero,and then we have to look at the difference between these 2 states. That's where we get the name, Temporal Difference;and so, we have ½ minus zero--and so that's 1/3 times ½--so that's 1/6.Now we update this state. It was zero; now it becomes 1/6.And you can see how the results of the positive 1 starts to propagate backwards--but it propagates slowly. We have to have 1 trial at a time to get that to propagate backwards. Now, how about the update from this state to this state?Now, we were ½ here--so our old utility was ½;plus Alpha--the learning rate--is 1/3.The reward in the old state was zero; plus the difference between these two,which is 1 minus ½.So that's ½ plus 1/6 is 2/3.And now the second time through, we've updated the utility of this state from 1/2 to 2/3.And we keep on going--and you can see the results of the positive, propagating backwards.And if we did more examples through here, you would see the results of the negative propagating backwards. And eventually, it converges to the correct utilities for this policy. Now here are some results from running the passive TD algorithm on the 4 by 3 maze. On the right, we see a graph of the average error in the utility function--average across all the states. So it starts off--for the first 5 or so trials,the error rate is very high--it's off the charts. But then it starts to settle down, through 10, 20, 40; and up to about 60 or so, it's still improving;and then it gets to a final steady stateafter about 60 trials of about .05 in the average error in utility. So that's not too bad, but not really converging all the way down to no rate of error. And on the left, you see the utility estimatesfor various different states; and, as we see--as we get out to 500 trials, they're starting to converge a little bit,close to their true values. But we see in the first 100 or so trials-- they were all over the map, and so it wasn't doing very well.It took awhile for it to converge to something close to the true values. Now I want to do a little quiz,and ask you: True or False,which of the following are possible weaknesses in this TD learning with a passive approach to reinforcement learning?One: Is it possible that we would have a long convergence time--that it might take a long time to converge to the correct utility values? Secondly, are we limited by the policy that we choose?So remember: in passive reinforcement learning, we choose a fixed policyand execute that policy; and any deviance from the policy results from the stochasticity. We may visit different squaresbecause the environment is stochastic,but not because we made different choices. So there's that elementation. Third, can there be a problem with missing states? That is, could there be some states that have a zero count--that we never visited, and never got a utility estimate?And fourth, could there be a problem with a poor estimate for certain states?So could it be that, even though a state didn't have a count of zero, it had a low count, and we weren't able to get a good utility estimate for that state? An answer is that every one of these is a potential problem for passive reinforcement learning. So every problem won't show up in every possible domain. It'll depend on what the environment looks like. But it is a possibility that you could get bitten by any of these problems. And they all stem from the same cause, from the fact that passive learning stubbornly sticks to the same policy throughout. We have a policy, pi of S, and we always execute that policy. So if the policy here was to go up and then go right, then we would always stick to that;and the only time we would explore any other state is when those actions failed. If we tried to go up from this state--because that's what the policy said;but, stochastically, we slipped over to this state--then we wouldn't do something else, according to the policyand so we'd get a little bit of exploration, but we'd only vary from the chosen path because of that variationand we wouldn't intentionally explore enough of the space. So let's move on to Active Reinforcement Learningand, in particular, let's examine a simpleapproach called a Greedy Reinforcement Learner. And the way that works is it uses the same passive TD learning algorithm that we talked about,but, after each time we update the utilitiesor maybe after a couple of updates--you can decide how often you want to do it--after the change to the utilities, we recompute the new optimal policy, pi.So we throw away our old pi, pi1,and replace it with a new pi, pi2--which is a result of solving the MDP described by our new estimates of the utiliities. Now we have a new policy, and we continue learning with that new policy. And so, if the initial policy was flawed, the Greedy algorithm would tend to move away from the initial policy, towards a better policy--and we can show how well that works. Here's the result of running the Greedy agent over 500 trials. And I've graphed 2 things here: One is the error; and you see, over the top-- over the first 40 or so trials--the error was very high--way up here. But then, suddenly, it jumped down to a lower level, and stayed along that level all the way through to 500. I've also graphed, with a dotted line, the policy loss. What does that mean?--so that's the difference between the policy that the agent has learned and the optimal policy. So if it had learned the optimal policy, the policy loss would be zero, down here. It doesn't quite get to zero.It was high, up here, and then at around step 40, it learned something important. What did it learn?--well, here's the final policy that it came up with. Maybe it started originally going in this direction in hitting the minus 1; and then it flipped and learned a new policy that went in a better direction. But it still hasn't learned the optimal policy.And we can see--for example, this looks like a mistake here. In state 1-2, it's policy is moving down and then following this path, which it learned, towards the goal.But really, a better route would be to take the northern route, and go through this path. But it hasn't learned that. Because it was Greedy, it found something that seemed to be doing good for it, and then it never deviated from that. So the question, then, is: How do we get this learner out of its rut?It improved its policy for awhile, but then it got stuck in this policy where we go here, go up and then go right. Most of the time, that's a perfectly good policy. But if a stochastic error makes us slip into the minus 1, then it hurts us. We'd like to be able to say we're going to stop doing that and somehow find this route. But in order to find that new route, we'd have to spend some time executing a policy which was not the best policy known to us. In other words, we'd have to stop exploiting the best policy we'd found so far--which is this one--and start exploring, to see if maybe there's a better policy. And exploring could lead us astrayand cause us to waste a lot of time.So we have to figure out: what's the right trade-off?When is it worth exploring to try to find something better for the long term--even though we know that exploring is going to hurt us in the short term?Now, one possibility is, certainly, random exploration. That is, we can follow our best policysome percentage of the time,and then randomly, at some point,we can decide to take an action which is not the optimal action. So we're here, the optimal action would be to go east;and we say, "Well, this time we're gong to choose something else--let's try going north. And then we explore from thereand see if we've learned something. So that policy does, in fact, work--randomly making moves with some probability--but it tends to be slow to converge.In order to get something better, we have to really understand what's going on with our exploration, versus exploitation. So let's really think about what we're doing when we're executing the active TD learning algorithm. First, we're keeping track of the optimal policy we've found so far;and that gets updated as we go, and replaced with new policies. Secondly, we're keeping track of the utilities of states--and those, too, get updated as we go along. And third, we're keeping track of the numberof times that we visited each state.And that gets incremented on each trial. Now, what could happen? What could go wrong? There are really 2 reasons why our utility estimates could be off. First, we haven't sampled enough. The end values are too low for that stateand the utilities that we got were just some random fluctuations and weren't a very good, true estimate. And secondly, we could get a bad utility because our policy was off. The policy was telling us to do something that wasn't really the best thing,and so the utility wasn't as high as it could be. So let's do a little quiz. I want you to tell me, for the 2 sources of possible error--too little sampling and wrong policy-- I want you to tell me, is it True or False--each of these statements:One: Could the error--either the sampling error or the policy error-- could that make the utility estimates too low? And secondly, could it make utility too high? And third, could it be improved with higher N values--that is, more trials? And here are the answers: For the error introduced by a lack of enough sampling, all these problems are true. If you don't have enough samples, it might make the utility too high; it might make the utility too low--and it could certainly be improved by taking more trials. But with the differences due to having not quite the right policy, The answers aren't the same. So yes, if you don't have the right policy, that could make the utilities too low--if you're doing something silly, like starting in this state and the policy says, "Drive straight into the minus 1"that could make the utility of this state lower than it really should be. But it can't make the utility too high. So we really have a bound on the utility here.The bound is: what does the optimal policy do? And no matter what policy we have, it's not going to be better than the optimal policy; and so we can only be making things worsewith our policy, not making them better. And finally, having more N won't necessarily improve things. It will decrease the variance, but it won't decrease or improve the mean. Now what that suggests is the design for an exploration agent that will be more proactive about exploring the world when it's uncertain,and will fall back to exploiting the optimal policy--or whatever policy it has as close to optimal--when it becomes more certain about the world. And what we can do is go through this normal cycle of TD learning--like we always did.But when we're looking for the estimate of the utility of the state, what we can do is say:The utility of the state estimate will be some large value, plus R--say, plus 1--in the case of this example--the largest reward we can expect to get.In every case, when the number of visits to the stateis less than the sum threshold, E, the exploration threshold.And when we've visited a state E times, then we revert to the learned probabilitiesor the learned utilities, rather. So when we start out, we're going to explore from new states;and once we have a good estimate of what the true utility of the state actually is, then we stop exploring and we go with those utilities. And here we have the result of some simulations of the exploratory agent.We see it's doing much better than the passive agent or than the Greedy agent. So I'm graphing here; and we only had to go through 100 trials. We didn't have to go through 500--so it's converging much faster. And it's converging to much better results. So the policy loss and the dotted lines started off high; but after only 20 trials, it's come down to perfect.So it learned the exact, correct policy after 20 trials. The error in the utilities--so you can have the perfect policy, while not quite having the right utilities for each state--and the errors in utility comes down, and that, too, comes down to a level that's lower than the previous agent's--but still, not quite perfect.And we see here that it, in fact, learns the correct policy. Now, let's say we've done all this learning, we've applied our agent, and we've come up with a utility model; and we have the estimates for the utility for every state. Now what do we do when we want to act in the world?Well, we now have out policy for the state, which is determined by the expected value and we compute the expected value of each state by looking at the utility, which we just learned. But then, we have to multiply by the transition possibilities.What's the probability of each resulting state that we have to look up the utility of? And so, we need to know that--and in some cases, we're given the transition model, and so we know all these probabilities. But in other cases, we don't have it;and so if we haven't learned it, we can'tapply our policy, even though we know the utilities. I want to talk, briefly, about this alternative methodcalled Q Learning, that I mentioned before. Where in Q Learning, we don't learn U direclty,and we don't need the transition model.Instead, what we learned is a direct mapping, Q, from states and actionsto utilities and so then, once we've learned Q, we can determine the optimal policy of he state, just by taking the maximum overall possible actions of this Q of S, A values. Now, how do we do Q Learning? Well, we start off with this table of Q values--and notice that there's more entries in thistable than there were in the utility table.So for each state, I've divided it up into different actions--so here's the action of going north, south, east or westfrom this particular state. They all start out with utility--or rather Q utility, at zero. But as we go, we start to update,and we have an update formula that's very similar to the formula for TD learning.It has the same learning rate, Alpha,and the same discount factor, Gamma;and we just start applying that.So we start tracking through the state space, and when we get a transition--say we go east from here, and then east and then north and then north; and then east-- and then we would back up this value;and depending on what the values of Alpha and Gamma were, we might update this to .6 or something;and then the next time through, we might update that to .7, and update this one to .4, and so on. In each case, we'd be updating only the action we took,associated with that state, not the whole state. We'd keep repeating that process until we had values filled in for all the action state pairs. Now, in some sense, you've learned all you need to know about reinforcement learning. Yes, it's a huge field, and there's a lot of other details that we haven't coveredbut you've seen all the basics. The theory is there and it works. But in another sense, we haven't gone very farbecause what we've done works for these small 4 by 3 Grid Worlds,But it won't work very well for larger problems: dealing with flying helicopters or playing backgammon--because there's just too many statesand we can't visit every one of the states and build up the correct utility values, or Q values, for all the billions or trillions or quadrillions of states we would need to represent. So let's go back to a simpler type of example. Here's a state in a Pacman gameand we can see that this is a bad state, where Pacman is surrounded by 2 bad guys,and there's no place for him to escape. And so reinforcement learning could quickly learn that this is bad.But the problem is that that state has no relation whatsoever to this state.Where conceptually, it's the same problem--that the Pacman is stuck in a cornerand there are bad guys own either sides of him. But in terms of a concrete state, the 2 are completely different. So what we want to be able to do is find some generalization, so that these 2 states look the same, and what I learn for this state--that learning can transfer over into this state. And so, just as we did in supervised machine learning, where we wanted to takesimilar points in the state and be able to reason about them, together, we want to be able to do the same thing for reinforcement training. And we can use the same type of approach.So we can represent a state, not by an exhaustive listing of everything that's true in the state--every single dot, and so on. But rather, by a collection of important features. So we can say that a state is this collection of Feature 1, Feature 2, and so on. And what are the features? Well, they don't have to be the exact position of every piece in the board. They could be things like the distance to the nearest Ghost or maybe the square of the distance--or the inverse square;or the distance to a dot or food-- or the number of Ghosts remaining.And then we can represent the utility of a state,or let's go with a Q value, of a state action pairand represent that as the sum over some set of waits times the value of each feature. And then our task, then, is to learn good values of these waits--how important is each feature, whether they're positive or negative, and so on. This formulation will be good to the extent that similar states have the same value. So if these 2 states have the same value, that would be goodbecause we could learn that, in both cases, Pacman is trapped. It would be bad, to the extent that dissimilar states have the same value-- say, if we're ignoring something important. So, for example, if one of the features was: Is Pacman in a tunnel? It would probably be important to know: is that tunnel a dead end or not? And if we represented all tunnels the same, we'd probably be making a mistake. Now, the great thing is that we can make a small modification to our Q learning algorithmwhere, when we were updating, the Q of S, A got updated in terms of a small change to the existing Q of S, A values. We can do the same thing with the wait's sub-i values. We can update them as we make each change to the Q values. And they're both driven by the amount of error. If the Q values are off by a lot, we have to make a big change; if they're not, we make a small change-- the same thing with the Wi values. And that looks just like what we did when we used supervised machine learning to update our waits. So we can apply that same process, even though it's not supervised. It's as if we're bringing our own supervision to reinforcement learning. In summary, then, we've learned how to do a lot with MDPs--especially using reinforcement learning. If we don't know what the MDP is, we know how to estimate it and then solve it. We can estimate the utility for some fixed policy, pi;or we could estimate the Q values for the optimal policy while executing an exploration policy. And we saw something about how we can make the right trade-offsbetween exploration and exploitation.So reinforcement learning remains one of the most exciting areas of AI.Some of the biggest surprises have come out of reinforcement learning--things like Tesauro's backgammon player or Andrew Ng's helicopter; and we think that there's a lot more that we can learn. It's an exciting field, and one where there's plenty of room for new innovation. The answer is we're transitioning from this state to this state.We get a reward of zero in the old state.Then we get the Q value of 100 minus the Q value of zero,and the discount rate is 90.That's a difference of 90.Then the alpha, the learning rate, is 1/2. That gives us 45.We apply that 45 to the state action pair.We were in this state, and we executed the action north,so the 45 goes here.Notice it doesn't go over here. We did end up going to the east, but we didn't execute the action of going to the east.All the other actions remain unchanged.This problem involves the Q-learning agent who is currently situated at this squarecalled (3,3), and executes the NORTH action trying to go up, but because the environment is stochastic, it actually ends up arriving at this terminal statewith value 100.And what I want you to answer is how should the Q-values be updated for this state,and I want you to enter the Q-values over here because we don't want you to mess up the original, and we'll use the formula below which I should point out isfrom the Sarsa version of Q-learning, and in this formula, the parameter alpha--the learning rate--will take on the value of 1/2, andgamma--the discount rate--will be 0.9,and all the rewards for moving from one state to the next are 0 with the exception of moving into the terminal state,and this Q of S prime, A prime--that means what goes on in the next state, so here we were in this S, and we took the action of going NORTH,and we transferred into this state, and in that state, no matter what action you takethe Q value is always 100, so this value here will always be 100.To work out the answer, let's look at the individual features for each of the states.For this state up here, the values for F1, F2, and F3 would be 2, 1, and 1.That is, the distance from the agent to the goal is 2,the distance to the closest bad guy is 1, and the distance of the bad guy to the goal is 1.Now this state here also has values 2, 1, and 1.That would be indistinguishable under either functions F or G.This state here has values 2, 1, and 3.The 2 and the 1 are the same, so that would be indistinguishable under F,but would be different under G.And this state has values 2, 3, and 1, and the 2 and 3 are different than 2 and 1,so that would be different under either F or G.Now the question which is a more useful function--the answer is G is more useful, because G can actually distinguish between these 2 states.In this state the agent is surround by bad guys, so that's a bad situation.In this state the agent has a clear path to the goal, so that's a good situation.You'd want a function that says that those two are different rather than one that says they're the same.G says they are different whereas F says they're the same.This question involves function generalization in reinforcement learning,and we're operating in a 1-dimensional environment of squares,and we're going to consider a state generalization function, that is a function that takes a state such as this and condenses it into some featuresto represent that state.The first function we're going to consider F has these features--f1 is the distance from the Agent represented by A to the goal represented by G, and f2--the distance from the Agent to the closest Bad guy which is represented by a B.So that's the function F, and we also want to consider the function G which has the same 2 features--f1 and f2--and adds a third feature which is the distance of the closest Bad guy to the goal.That is distance from the goal to the Bad guy--the minimum of that overall possible Bad guys,and now I want you to say which of the states below--these 3 states--have the same value as the state above--this state--under the functions F and G.And click off the ones that have the same, and then I want you to answer for me--In this world, agents and Bad guys can move one Square at a time, and the agent tries to get to the goal without encountering Bad guys,and for the agent to do that, which is a more useful generalization function to use over these states--F or G?The answer is according to the policy the agent would prefer to follow this straight line,because it is the most direct, and it is the longer goal.Now, at any point he might slip off to one of these squares.Those would all potentially be explored,but if he did he would go back down onto the road.Likewise, he might fall off onto any of of these squares,but if he did, he would also go back towards the road.That's certainly true under this situation, when he's off road,but it also turns out to be true here and here,because the closest way to get to the goal would be to go in the north direction.Therefore, these three rows could all potentially be explored,but the bottom two rows would never be explored under any conditions no matter what happens stochastically as long as the agent is following this fixed policy.In this problem, a passive TD-reinforcement learning agent starts at S and moves to G under a fixed policy which says first, make moves that get closest to G.So if we started here, we'd want to go in this direction because it's closest to G,and (b) stay on these gray squares which represent roads, and (c) if you do happen to go off the road, then move it back onto the road immediately.The actions are stochastic, and they may go in the intended direction,or they may go 90 degrees off, so if we were here, we'd plan to start under this policygoing in this direction. We might end up there, but we might end up here or here.And if we did end up here, then we'd immediately head back towards the road,so we'd aim back down in this direction.And what I want you to do is click on all the squares that would never be exploredby this reinforcement learning agent following this passive fixed policy.Today I have the great, great pleasure to teach you about hidden Markov models and filter algorithms.The reason why I'm so excited is in pretty much all of my scientific career,hidden Markov models and filters have played a major role.There's no robot that I program today that wouldn't extensively use hidden Markov modelsand things such as particle filters.In fact, when I applied for a job at Stanford University as a professor many years ago,my job talk that I used to market myself to Stanford was extensively about a version of hidden Markov models and particle filtersapplied to robotic mapping.Today I will teach you those algorithms so you can use them in many challenging problems.I can't quite promise you that once you have mastered the materialyou will get a job at Stanford, but you can really, really apply themto a vast array of problems in places such as finance, medicine, robotics,weather prediction, time series analysis, and many, many other domains.This is going to be a really fun class.[Thrun] Hidden Markov models, or abbreviated HMMs, are used to analyze or to predict time series.Applications include robotics, medical, finance,speech and language technologies, and many, many, many other domains.In fact, HMMs and filters are at the core of a huge amount of deployed practical systemsfrom elevators to airplanes.Every time there is a time series that involves noise or sensors or uncertainty,this is the method of choice.So today I'll teach you all about HMMs and filtersso you can apply some of the basic algorithms in a wide array of practical problems.[Thrun] The essence of HMMs are really simply characterizedby the following Bayes network.There's a sequence of states that evolve over time,and each state depends only on the previous state in this Bayes network.Each state also emits what's called a measurement.It is this Bayes network that is the core of hidden Markov modelsand various probabilistic filters such as Kalman filters, particle filters, and many others.These are words that might sound cryptic and they might not mean anything to you,but you might come across them as you study different disciplines of computer scienceand control theory.The real key here is the graphical model.If you look at the evolution of states, what you'll find is that these states evolve as what's called a Markov chain.In a Markov chain, each state only depends on its predecessor.So for example, state S3 is conditioned on S2 but not on S1.It's only immediate through S2 that S3 might be influenced by S1.That's called a Markov chain, and we're going to study Markov chains quite a bitin this class to understand them well.But what makes it a hidden Markov model or hidden Markov chain, if you wish,is the fact that there is measurement variables.So rather than being able to observe the state itself, what you get to see are measurements.Let me put this to perspective, showing you several of the robots I've builtthat possess hidden state.And where I only get to observe certain measurements,let me infer something about the hidden state.[Thrun] What's shown here is the tour guide robot that I showed you earlier,but now I'll talk about the what's called localization problem--the problem of finding out where in the world this robot is.This problem is important because to find its way around the museumand to arrive at exhibits of interest, it must know where it is.The problem with this problem is that it doesn't have a sensor that tells us where it is.Instead, it's given what's called range finders.These are sensors that measure distances to surrounding objects.It's also given the map of the environment, and it can compare these range finders measurements with the map of the environmentand infer from that where it might be.The process of inferring the hidden state of the robot's location from the measurements,the range sensor measurements, that's the problem of filtering.And the underlying model is exactly the same I showed you before.It's a hidden Markov model where the state is the sequence of locationsthat the robot assumes in the museumand the measurements is the sequence of range measurements it perceiveswhile it navigates the museum.A second example is the underground robotic mapping robotwhich has pretty much the same problem--finding out where it is--but now it is not given a map; it builds the map from scratch.What this animation here shows you is a so-called particle filter applied to robotic mapping.Intuitively--what you see is very simple--as the robot transcends into a mine, it builds a map.But the many black lines are hypotheses on where the robot might have beenwhen building this map.It can't tell because of the noise in its motors and in its sensors.As the robot reconnects and closes the loop in this map,one of these black what we call particles in the trade--one of these black hypotheses are being selected as the best one,and by virtue of having maintained many of those, the robot is able to build a coherent map.In fact, this animation was a key animation in my job talkwhen I applied to become a professor at Stanford University.Here is one final example I'd like to discuss with you which is called speech recognition.If you have a microphone that records speechand you want to make your computer recognize the speech,you will likely come across hidden Markov models.This is a typical speech signal over here.It's an oscillation for the words "speech lab" which I borrowed from Simon Arnfield.And if you blow up a small region over here, you'll find that there is an oscillation,and this oscillation in time is the speech signal.What speech recognizing systems do is they transform this signal over hereback into letters like "speech lab."And you can see it's not an easy task.There is some signal here. The E, for example, is a certain shape.But different speakers speak differently, and there might be background noise,so decoding this back into speech is challenging.There's been enormous progress in the fieldmostly due to hidden Markov models that have been researched for more than 20 years.And today's best speech recognizers all use variants of hidden Markov models.So once again, I can't teach you everything in this class, but I'll teach you the very basicsthat you can apply to things such as speech signals.[Thrun] So let's begin by taking the hidden out of the Markov modeland study Markov chains.We're going to use an example for which I will quiz you.Suppose there are 2 types of weather--rainy, which we call R,and sunny, which we call S--and suppose we have the following state transition diagram.If it's rainy, it stays rainy with a 0.6 chance while with 0.4 it becomes sunny.Sunny remains sunny with 0.8 chance but moves to rainy with 0.2 chance.This is obviously a temporal sequence so the weather at time 1 will be called R1 or S1,at time 2, R2 or S2.Suppose in the beginning we happen to know it is rainy,which means R times 0 when we begin.We have the probability of rain equals 1 and the probably of sun, S times 0 equals 0.I'd like to know from you what's the probability of rain on day 1, the same for day 2,and the same for day 3.[Thrun] And the answer will be 0.6, 0.44, and 0.376.It's really an exercise applying probability theory.In the very beginning we know to be in state R,and the probability of remaining there is 0.6, which is directly the value on the arc over here.On the second state we know that the probability of R is 0.6and therefore, the probability of sun is 0.4,and we compute the probability of rain on day 2 using total probability.The probability of rain on day 2 given rain on day 1 times the probability of rain on day 1 plus the probability of rain on day 2given it was sunny on day 1 times the probability of sun on day 1.And if you plug in all these values,we get 0.6 times 0.6 plus rain following sun which is this arc over here, 0.2,times 0.4 as the prior, and this results in 0.44.We can now do the same with the probability of rain on day 3,which is the same 0.6 over here, but now our prior is different--it's 0.44--plus the same 0.2 over here with the prior of 0.56, which is 1 minus 0.44.And when you work this all out, it is 0.376 as indicated over here.So what we really learned here is that this is a temporal Bayes networkof which we can apply conventional probabilities such as the total probabilitywhich was also known as variable elimination in the Bayes network lecture.All these fancy words aside, it's really easy to evaluate those.So if you want to do this and you ask yourself given the probability of the certain time steplike 1, what's it related to time step 2,you ask yourself what's the durations that I encounter in time step 1.There are usually 2 in this case.What are the transition probabilities that lead me to the desired state in time step 2like the 0.6 if you started in R and 0.2 if you started in S,and you add all these cases up and you just get the right number.It's really an easy piece of mathematics if you think about it.[Thrun] Let's practice this again with another 2-state Markov chain.States are A and B.A has a 50% chance of transitioning to B,and B always transitions into A.There is no loop from B to itself.Let's assume again at a time, 0, we know with certainty to be in state A.I would like to know the probability of A at time 1, at time 2, and at time 3.[Thrun] And again the solution follows directly from the state diagram over here.In the beginning we do know we're in state Aand the chance of remaining in A is 0.5.This is the 0.5 over here. We can just read this off.For the next state we find ourselves to be with 0.5 chance to be in Aand 0.5 chance to be in B.If we're in B, we transition with certainty to A.That's because of the 0.5.But if we're in A, we stay in A with a 0.5 chance. So you put this together.0.5 probability being in A times 0.5 probability of remaining in Aplus 0.5 probability to be in B times 1 probability to transition to A.That gives us 0.75.Following the same logic but now we're in A with 0.75 times a 0.5 probabilityof staying in A plus 0.25 in B, which is 1 minus 0.75,and the transition's uncertainty back to A as 1, we get 0.625.So now you should be able to take a Markov chain and compute by handor write a piece of software the probabilities of future states.You will be able to predict something. That's really exciting.[Thrun] So one of the questions you might ask for a Markov chain like this iswhat happens if time becomes really large?What happens for the probability of A1000?Or let's go extreme. What about in the limit, A infinity, often written as the limits of timegoing to infinity of any P of At.That's like the fancy math notation, but what it really means is we just wait a long, long time.What is going to happen to the Markov chain over here? What is that probability?This probability is called a stationary distribution,and a Markov chain settles to a stationary distributionor sometimes a limit cycle if the transition is alternativistic(?), which we don't care about.And the key to calculating this is to realize that the probability for any tmust be the same as the probability 1 times (?)This can be resolved as follows.We know that P of At is P of At given At minus 1 times P of At minus 1plus P of At given Bt minus 1times probability of Bt minus 1.This is just the theorem of total probability or forward propagation rule applied to this case over here, so nothing really new.But if you call this guy over here X, then we now have X equals probability of At given At minus 1 is 0.5times--and this is the same X as this one over here because you're looking for the stationary distribution, so it's X again.This probability over here, A following B, is 1 in this special case,and the probability of Bt minus 1 is 1 minus At minus 1.And if you plug this in, that's the same as 1 minus X.And we can now solve this for X.Let me just do this.X equals, if you put these 2 Xs together we get minus 0.5 plus 1or, differently, 1.5X equals 1.That means X equals 1 over 1.5, which is 2/3.So the answer here is the stationary distribution will have A occurring with 2/3 chanceand B with 1/3 chance.It's still a Markov chain--it flips from A to B--but these are the frequencies at which A occursand this is the frequency at which B occurs.[Thrun] To see if you understood this, let me look at the rain-sun Markov chain again,and let me ask you for the stationary distribution or the limit distributionfor rain to be the case after infinitely many steps.[Thrun] And the answer is 1/3,as you can easily see if you call X the probability of rain in time Tand also the probability of rain, T minus 1.These 2 must be equivalent because we're looking for the stationary distribution.Then we get, by virtue of our expansion of the state at time T,the probability of transitioning from rain to rain is 0.6,the probability of having it rain is X again, the probability of transitioning from sun to rain is 0.2, and the probability of having sun before is 1 minus X,so we get X equals 0.4X plus 0.2.Or, differently, we have 0.6X equals 0.2,and when we work this out, X is 1/3,which is the probability of rain in the asymptote if you wait forever.One of the interesting things to observe hereis that the stationary distribution does not depend on the initial distribution.In fact, I didn't even tell you what the initial state was.Markov chains that have that property, which are pretty much all Markov chains,are called ergodic. You can safely forget that word again, but people in the field use this wordto express Markov chains that mix.And mix means that the knowledge of the initial distribution fades over timeuntil it disappears in the end.The speed at which it gets lost is called the mixing speed.[Thrun] You can also learn the transition probabilities of a Markov chain like thisfrom actual data.Suppose you look out of the window and see sequences of rainy daysfollowed by sunny days followed by rainy daysand you wonder what numbers to put here, here, here, and here.Let me assume you see a sequence rain, sun, sun, sun, rain, sun, and rain.These are, in total, 7 different days,and we wish to estimate all those probabilities over here,including the initial distribution for the first day using maximum likelihood.You might remember all this work with Laplace smoothing,but for now we keep it simple, just maximum likelihood.We find for day 0 we had rain, and maximum likelihood would just saythe probability for day 0 is 1.That's the most likely estimate.Then for the transition probability we find we transition from rain to something else twice here.We sometimes transition to sun and sometimes stay in rain.In both of the transitions we go from rain to sun. There is no instance of rain to rain.So maximum likelihood gives us over here a 1 and this over here 0.And finally, we can also ask the question what happens from a sunny state.We transition to a new sunny state or a rainy state,and those distributions are easily calculated.We have 4 transitioning out of a sunny state to something else--this one, this one, this one, and this one.Twice it goes to sunny over here and over here,twice it goes to rainy over here and over here,so therefore the probability for either transition is 0.5.So we have 0.5 over here, 0.5 over here, 1 over here, and 0 over herefor the transition probabilities.[Thrun] So in this quiz please do the same for me.Here is our sequence. There's a couple of sunny days--5 in total--a rainy day, 3 sunny days, 2 rainy days.Calculate using maximum likelihood the prior probability of rainand then the 4 transition probabilities as before.Please fill in those numbers over here.[Thrun] The initial probability for rain is 0because we are just encountering 1 initial day and it's sunny.The maximum likelihood estimate is therefore 0.We transition 8 times out of a sunny state--1, 2, 3, 4, 5, 6, 7, 8--twice into a rainy state, and therefore 6 times we remain in a sunny state,so the probability of sun to sun is ¾,whereas sun to rain is ¼.From a rainy state we have 2 outbound transitioning,1 to a sunny state and 1 to a rainy state.The last R over here has no outbound transition,so it doesn't really count in our statistic.The maximum likelihood therefore is 0.5 or ½ for each of those.[Thrun] One of the oddities of the maximum likelihood estimator is overfitting.So for example, we observed that we always have a single first day,and this becomes our prior probability.So in this case the prior probability for rain on day 0 would be 1,which kind of doesn't make sense, really.It should be more like the stationary distribution or something like that.Well, you might remember the work on Laplacian smoothing.This is a great moment where I can test whether you really thinklike an artificial intelligence person.I'm going to make you apply Laplacian smoothing in this new contextof estimating the parameters of this Markov chainusing the smoother of K = 1.You might remember you add something to the numerator, like 1,and something to the denominator to make sure things normalize,and then you get different probabilities than you would get with the maximum likelihood estimator.So I'm going to ask you a quiz here, even though I haven't completely shown youthe application of Laplacian smoothing in this context.But if you understood Laplacian smoothing, you might want to give it a try.What's the probability of rain on day 0, and what are its conditional probabilities?Sun goes to sun, sun goes to rain, rain goes to sun, and rain stays in rain.The way probabilities work, as you surely know, these 2 things over herehave to add up to 1, and these 2 things over here have to add up to 1.[Thrun] So in Laplacian smoothing we look at the relative counts.We know there is 1 instance of rain at time 0.Normally it would be 1.But we add 1 to the numerator and 2 to the denominator, and we get 2/3.Let's look at these numbers again.The count that we have is 1 out of 1 is rain and 1 out of 1 would give us 1under the maximum likelihood estimator.But because we're smoothing, we're adding a pseudocount, which is 1 rainy day and 1 sunny day,and we have to compensate for the 2 additional counts with a 2 over hereand therefore we get 2/3.So our probability under the Laplacian smoother is 2/3 for the rainy day to be the first day,which is really different from 1.Applying the same logic over here, we transition 3 times out of a sunny state--1, 2, 3--and each time it's a sunny state.So maximum likelihood would say 3 times out of 3 it's sunny into sunny.We add a pseudo observation of 1, and then there's 2 possible outcomes;hence, we have to count 2 over here.So it's 4/5.And the missing 1/5 shows up over here.We can do the same math as before.Zero with 3 transitions from a sunny day resulted in a rainy day.In fact, they were all sunny.But we add 1 pseudo observation over here and 2 of the normalizer, 1/5.These 2 things surely add up to 1.The last one is analogous. We have 1 transition of a rainy state and it led to a sunny state, so 1/1,but we add 1 over here and 2 on the denominator so you get 2/3.And if you do the math over here, you get 1/3.I really want you to remember Laplacian smoothing.It's applicable to many estimation problems,and it will be important going forward in this class.Here we applied it to the estimation of a Markov chain.Please take a moment and study the logic so you'll be able to apply those things again.[Thrun] So now let's return to hidden Markov models.Those are really the subject of this class.Let's again use the rainy and sunny example just to keep it simple.These are the transition probabilities as before.Let's assume for now that the initial probability of rain is 0.5;hence, the probability of sun at time 0 is 0.5.The key modification to go to hidden Markov model is that this state is actually hidden.I cannot see whether it's raining or it's sunny.Instead I get to observe something else.Suppose I can be happy or grumpyand happiness or grumpiness is being caused by the weather.So rain might make me happy or grumpy,and sunshine makes me happy or grumpybut with vastly different probabilities.If it's sunny, I'm just mostly happy, 0.9.There's a 0.1 chance I might still be grumpy for some other reason.If it's rainy, I'm only happy with 0.4 probability and with 0.6 I'm grumpy.In fact, living in California I can attest that these are actually not wrong probabilities.I love the sun over here.Suppose I observe that I'm happy on day 1.A question that we can ask now is what is the so-called posterior probabilityfor it raining on day 1 and what's the posterior probability for it being sunny on day 1?What's the probability of rain on day 1 given that I observed that I was happy on day 1?This is being answered using Bayes rule,so this is the probability of being happy given that it rainstimes the probability that it rains over the probability of being happy.We know the probability of rain at day 1 based on our Markov state transition model.In fact, let's just calculate it.The probability of rain on day 1 is the probability it was rainy on day 0and it led to a self transition from rain to rain from day 0 to day 1plus the probability it was sunny on day 0 times the probability that sun led to rain over here.If you can plug in all these numbers to obtain 0.4,you can just easily verify this.So we know this guy over here is 0.4.This guy over here is 0.4 again, but now it's this 0.4 over here.The probability of being happy on a rainy day is 0.4.This guy over here resolves to 0.4 times 0.4plus the same situation with sunny in time 1where the prior is 0.6 and the happiness factor is 0.9.And that gives us the entire expression is 0.229.Let's interpret the 0.229 in the context of the question we asked.We know that at time 0 it was raining with half a chance.If you look at the state transition diagram, it's more likely to be sunny afterwardsbecause it's more likely to flip from rain to sun than sun to rain.In fact, we worked out that the probability of rain at a time step later was only 0.4,so it was 0.6 sunny.But now that I saw myself being happy, my probability of rain was further loweredfrom 0.4 to 0.229.And the reason why the probability went down is if you look at happiness,happiness is much more likely to occur on a sunny day than it is to occur on a rainy day.And when you work this in using Bayes rule and total probability,you would find just the fact that it was at happiness at time 1makes your belief of it being rainy go down from 0.4 to 0.229.This is a wonderful example of applying Bayes rule in this really relatively complicated hidden Markov model.[Thrun] So let me use exactly the same hidden Markov model where we have rain and sunand happiness and grumpiness with 0.4 and 0.6and 0.9 and 0.1 probabilities.The only change I will apply is I will tell you that for probability 1 it's raining on day 0;hence, the probability of sunny at day 0 is 0.I now observe another happy face on day 1,and I'd like to know the probability of it raining on day 1 given this observation.This is the same as before with the only difference that we have a different initial probability,but all the other probabilities should just be the same.[Thrun] Once again let's calculate the probability of rain on day 1.This one is easy because we know it is raining on day 0, so it's 0.6, the 0.6 over here.This expression over here is expanded by a Bayes rule as applied over here.Probability of happiness during rain is 0.4,the probability of rain was said to be just 0.6,and we divide by 0.4 times 0.6 plus 0.9 times 0.4, which is 1 minus 0.6.And that resolves simply to 0.4 if you work it all out.So the interesting thing here is if you were just to run the Markov chain,on day 1 we have a 0.6 chance of rain,but the fact that I observed myself to be happy reduces the chance of rain to 0.4.[Thrun] So if you got those questions right, I'm in awe with you--wow--because you understand the very basics of using a hidden Markov model for 2 things now.One is prediction, and one is called state estimation.In state estimation that's a really fancy word for just computing the probabilityof the internal or hidden state given measurements.In prediction we predict the next state, and you might also predict the next measurement.[Thrun] I want to show you a little animation of hidden Markov modelsused for robot localization.This is obviously a little toy robot over here that lives in the grid world,and the grid world is composed of discrete cells where the robot may be located.This robot happens to know where north is at all times.It's given 4 sensors, a wall sensor to the left, to the right, to the top and the bottom over here, and it can sense whether in the adjacent cell there's a wall or not.Initially this robot has no clue where it is. It faces what we call a global localization problem.It now uses its sensors and its actuators to localize itself.So in the very first episode the robot senses a wall north and south of itbut none west or east.And look what this does to the probabilities.The posterior probability is now increased in places that are consistent with this measurement,like all of those places have a wall in north and east, like these guys over here,and free space in the left and the right,yet they have been decreased in places that are inconsistent, like this guy over here.These states over here are interesting. They are shaded gray and lighter gray.What this means is they still have a significant probability but yet not as much as over here,the reason being that this measurement over here would be characteristic for the state over here if there had been exactly 1 measurement error--if the bottom sensor had erred and erroneously detected a wall.Errors are less likely than no errors, and as a result, the cell over herewhich is completely consistent ends up to be more likely than the cell over here,yet you can see the HMM does a nice job in understanding the posterior probability.Let's assume the robot moves right and senses againand gets the exact same measurement.Of course it has no clue that it is exactly over here.It can see the probability as being decayed.Interestingly enough, this guy over here has a lower probability,and the reason is by itself it is very consistent with the most recent measurement,but it's less consistent with the idea of having moved right and measured beforea wall to the north and the south.And similarly, these places over here become less consistent.The only ones that are completely consistent are these 3 states over hereand the 3 states over here.The robot keeps moving to the right,and now we get to the point where the sequence of measurementreally makes 2 states equally likely--the ones over here.They are equally likely with symmetry.Those are still pretty likely, and those are gradually and likely over here to the left.As the robot now moves, it moves into a distinguishing state.It sees a wall in the north but free space in the 3 other directions,and that renders the state over here relatively unlikely,and now it has localized itself.[Thrun] We discussed specific incidents of hidden Markov model inference or filteringin our quizzes.Let me now give you the basic math.We all know hidden Markov model is a chain like thisof hidden states that are Markovian and measurements that only depend on the corresponding state.We know that this Bayes network entailed certain independencies.For example, given X2 the past, the future, and the present measurement are all conditionally independent given X2.The nice thing about this structure is it makes it possible to efficiently do inference.I'll give you the equations we used before here in a more explicit form.Let's look at the measurement side, and suppose we wish to know the probabilityof an internal state variable given a specific measurement,and that by Bayes rule becomes P of Z1 given X1 times P of X1 over P of Z1.When you start doing this, you'll find that the normalizer doesn't depend on the target variable X; therefore, we often write a proportionality sign and get an equation like this.This product over here is the basic measurement update of hidden Markov models.And the thing to remember when you apply it, you have to normalize.We already practiced all of this, so you know all of this.The other equation is the prediction equation,so let's go from X1 to X2.This is called prediction even though sometimes it has nothing to do with prediction.It's the traditional term, but it comes from the fact that we might want to predictthe distribution of X2 given that we know the distribution of X1.Here we apply total probability.The probability of X2 is obtained by checking all states we might have come from in X1and calculating the probability of going from X1 to X2.We also practiced this before.Any probability of X2 being in a certain state must have come from another state, X1,and then transitioned into X2, so we sum over all of those and we get the posterior probability of X2.These 2 equations together form the math of a hidden Markov modelwhere the next state distribution and the measurement distributionand the initial state distribution are all given as the parameters of a hidden Markov model.[Thrun] Here is the application of HMM to a real robot localization example.This robot is in a world that's 1-dimensional and it is lost.It has initial uncertainty about where it is,and it is actually located next to a door but it doesn't know.It's also given a map of the world,and the distribution of all possible states, here noted as s, is given by this histogram.We bin the world into small bins, and for each bin we assign a single numerical probabilityof the robot being there.The fact they have all the same height means that the robot is maximally uncertainas to where it is.Let's assume this robot is going to senseand it senses to be next to a door.The red graph over here is the probability of seeing a door for different locations in the environment.There are 3 different doors, and seeing a door is more likely herethan it is in between.It might still see a door here, but it's just less likely.We now apply Bayes rule.We multiply the prior with this measurement probability to obtain the posterior.That was our measurement update. It's that simple.So you can see how all these uniform values over here become nonuniform values over heremultiplied by this curve over here.The story progresses by the robot taking an action to the right,and this is now the next state prediction part, the what we call convolution partor state transition part, where these little bumps over here get shifted along with the robotand they are flattened out a little bit just because robot motion has used uncertainty.Again, it's a really simple operation.You shift those to the right and you smooth them out a little bitto account for the control noise in the robot's actuators.And now we get to the point that the robot senses again,and this robot senses a door again.And see what happens. It multiplies.It's now a nonuniform prior over here with the same measurement probability as before,but now we get a distribution that's peaked over here and has smaller bumps at various other places,the reason being the only place where my prior has a higher probabilityand my measurement probability is also high probability is the second door,and as a result of our distribution over here, it assumes a much larger value.If you look at that picture, that is really easy to implement,and that's what we did all along when we talked about rain and sun and so on.It's really a very simple algorithm.Measurements are multiplications, and motion become essentially convolutionswhich are shifts with added noise.[Thrun] This is a great segue to one of the most successful algorithmsin artificial intelligence and robotics called particle filters.Again, the topic here is robot localization,and here we're dealing with a real robot with actual sensor data.The robot is lost in this building.You can see different rooms, and you can see corridors,and the robot is equipped with range sensors.These are sound sensors that measure the range to nearby obstacles.Its task is to figure out where it is.The robot will move along the black line over here, but it doesn't know this.It has no clue where it is.It has to figure out where it is.The key thing in particle filters is the representation of the belief.Whereas before we had discrete worlds like our sun and rain exampleor we had a histogram approach where we cut the space into small bins,particle filters have a very different representation.They represent the space by a collection of points or particles.Each of these small dots over here is a hypothesis where the robot might be.It's a concrete value of its X location and its Y location and its heading directionin this environment.So it's a vector of 3 values.The sum or set of all those vectors together form the belief space.So particle filters approximate a posteriorby many, many, many guesses,and the density of those guesses represents the posterior probabilityof being at a certain location.To illustrate this, let me run the video.You can see in a very short amount of time the range sensors,even though they're very noisy, force the particles to collect in the corridor.There's 2 symmetrical point dots--this one over here and this one over here--that come from the fact that the corridor itself is symmetric.But as the robot moves into the office, the symmetry is broken.This office looks very different from this office over here,and those particles die out.What's happening here?Intuitively speaking, each particle is a representation of a possible state,and the more consistent the particle with the measurement,the more the sonar measurement fits into the place where the particle says the robot is,the more likely it is to survive.This is the essence of particle filters.Particle filters use many particles to represent a belief,and they will let those particles survive in proportion to the measurement probability.And the measurement probability here is nothing else but the consistencyof the sonar range measurements with the map of the environment given the particle place.Let me play this again.Here's the maze. The robot is lost in space.Again, you can see how within very few steps the particles consistent with the range measurements all accumulate in the corridor.As the robot hits the end of the corridor, only 2 particle clouds survivedue to the symmetry of the corridor, and the particles finally die out.This algorithm is beautiful, and you can implement it in less than 10 lines of program code.So given all the difficulty of talking of probabilities and Bayes networkand hidden Markov models, you will now find a way to implement one of the most amazing algorithms for filtering and state estimationin less than 10 lines of C code. Isn't that amazing?[Thrun] Here is our 1-dimensional localization example again,this time with particle filters.You can see the particles initially spread out uniformly.This 1-dimensional space of forward locations you're going to use as an exampleto explain every single step of particle filters.In the very first step, the robot senses a door.Here is its initial particles before sensing the door.It now copies these particles over verbatim but gives them what's called a weight.We call this weight the importance weight,and the importance weight is nothing else but the measurement probability.It's more likely to see a door over here than over here.The red curve over here is the measurement probability,and the particles over here are the same as up here,but they now attached an importance weight where the height of the particleillustrates the weight.So you can see the place over here, the place over here, and the place over herecarry the most weight because they're the most likely ones.This robot moves and it moves by using its previous particlesto create a new random particle set that represents the posterior probabilityof being at a new location.The key thing here is called resampling.The algorithm works as follows.Pick a particle from the set over here and pick it in proportion to the importance weight.Once you've picked one--and sure enough, you pick those more frequentlythan those over here--add the motion to it plus a little bit of noiseto create a new particle.Repeat this procedure for each particle.Pick them with replacement.You're allowed to pick a particle twice or 3 or 4 times.Sure enough, you pick these more frequently.These are being forward moved to over here, these to over here.You see a higher density of particles over here and over here,than you see, for example, over here.That's your forward prediction step in particle filters.It's really easy to implement.The next step is another measurement step,and here I'm illustrating to you that indeed this nonuniform set of particlesleads to a reasonable posterior in this space.We now have a particle set as nonuniform.We have increased density over here, over here, and over here.You can see how multiplying these particles with the importance weight,which is copying them over verbatim but attaching a vertical importance weightin proportion to the measurement probability,yields a lot of particles over here with big weights,some over here with big weights, lots of particles over here with low weights.They got copied over, but the measurement probability here is low and so on and so on.And if you look at this set of particles, you already understand why the majority of importance and weights resides in the correct locationgiven that we had a measurement of a door and motion to the rightand another measurement of the door.The nice thing here is that particle filters work in continuous spaces,and, what's often underappreciated, they use your computational resourcesin proportion to how likely something is.You can see that almost all the computation now resides over here,almost all the memory resides over here,and that's the place that's likely.Stuff over here requires less memory, less computation, and guess what? It's much less likely.So particle filters make use of your computational resources in an intelligent way.They're really nice to implement on something with low compute power.Let me move on to explain the next motion.Here you see our robot moving to the right again,and now the same what we call resampling takes place.We pick, with replacement, particles from over here.Sure enough, these are the ones we pick the most often.And then we add the motion command plus some random noise.If you look at this particle set over here, almost all the particles sit over here.It doesn't really show it very well on this computer screen,but the density of particles over here is significantly higher than anywhere else.There's occurrences over here and over here that correspond with these guys over hereand these guys over here and over here, correspond to this guy over here,but the vast majority of probability mass sits over here.So let's dive into how complicated this algorithm really is.[Thrun] So here is our algorithm particle filter.It sets as an input a set of particles with associated important weights,a control, and a measurement vector,and it constructs a new particle set as prime and in doing so it also has an auxiliary variable, eta.Here is the algorithm.Initially we go through all new particles of which there are nand we sample in index j according to the distributiondefined by the importance weights associated with the particle set over here.Put differently, we have a set of particles over hereand we have associated importance factors which we will construct a little bit later on,and now we pick one of these particles with replacementwhere the probability of picking this particle is exactly the importance weight, w.For this particle we now sample a possible successor stateaccording to the state transition probability using our controlsand that specific particle as an input. We call it sj over here.We also compute an importance weight, which is the measurement probabilityfor that specific particle over here.This gives us a new particle, and this gives us a new non-normalized importance weight.For now we just add them into our new particle set as prime and we reiterate.The only thing missing now is at the very end we have to normalize all the weights.For this we keep our running counter, eta,and we have a For loop in which we take all the weights in the set over hereand just normalize them accordingly.This is the entire algorithm.We feed in over here particles with associated important weights and a control and a measurement,and then we construct the new set of particles by picking particles from our previous setat random with replacement but in accordance to the importance weights,so important particles are picked more frequently.We guess for this particle this will be a state.We guess what a new state might be by just sampling it,and we attach it an importance weight which we later normalizethat is proportional to the measurement probability for this thing over here.So you're going to upweigh the particles that look consistent with the measurementsand downweigh the ones that are non-consistent.We add all of these things back to our particle sets and reiterate.I promised you it would be an easy algorithm.You can look at this, and you could actually implement this really easily.Just remember how much difficulty we introducedwith talking about Bayes networks and hidden Markov models and all that stuff.This is all there is to implement particle filters.[Thrun] Particle filters are really easy to implement.They have some deficiencies. They don't really scale to high-dimensional spaces.That's been recognized because the number of particles you need to fill a high-dimensional space tends to grow exponentially with the dimensionality of the space.So for 100 dimensions it's hard to make work.But there are extensions.They go under really fancy names like Rao-Blackwellized particle filtersthat can actually do this, but I won't talk about them in any detail here.They also have problems with degenerate conditions.For example, they don't work well if you only have 1 particle or 2 particles.They tend not to work well if you have no noise in your measurement modelor no noise in your controls.You need this kind of to remix things a little bit.If there is very little noise, you have to deviate from the basic paradigm.But the good news is they work really well in many, many applications.For example, our self-driving cars use particle filters for localization and for mappingand for a number of other things.And the reason why they work so well is they're really easy to implement,they're computationally efficient in the sense that they really put the computational resourceswhere they are needed the most, and they can deal with highly non-monotonic and very complex posterior distribution that have many peaks.And that's important. Many other filters can't.So particle filters are often the method of choice when it comes to building quicklyan estimation method for problems where the posterior is complex.[Thrun] Wow! You learned a lot about hidden Markov models and particle filters.Particle filters is the most used algorithm in robotics today when it comes to interpreting sensor data,but these algorithms are applicable in a wide array of applicationssuch as finance, medicine, behavioral studies, time series analysis, speech,language technologies, anything involving time and sensors or uncertainty.And now you know how to use them.You can apply them to all these problems if you listened carefully.It's been a pleasure teaching you with this class.As I told you in the beginning, this is a topic very close to my heart,and I hope it's going to empower you to do better stuff in any domain involving time series and uncertainty.Here is a sequence of MDP questions.We're given a maze environment with 8 fields where we receive +100 over here and -100 in the corner over here.Our agent can go north, south, west, or east, but actions may fail at random.With probability "P," and P is a number between 0 and 1, the action succeeds,and with 1 - P we go into reverse.For example, if we take the action go east into this state over here,with P probability we find ourselves over there.With 1 - P we find ourselves right over here in the exact opposite direction.Here is the east action again. With P we go to the right, and with 1 - P we go to the left.Of course, if you bounce into a wall we stay where we are.For my first question, I'll assume P equals 1.There is no uncertainty in action outcome, and there is no failure.The state transition function is deterministic.I want you to fill in for each state the final value after running value iteration to completion,and please assume the cost is -4 and we use gamma equals 1 as the discount factor.Please fill in those missing six values over here.And the answer is obtained by looking at the nearest value -4, which gives us 96 over here, 92, 88, and 84.Let us now assume that P = 0.8, which means actions fail with probability 0.2.Again, the cost is -4, and gamma equals 1.I want you to run exactly one value calculation for the state in red up here.Assuming that the value function is initialized with 0 everywhere,what will be the value after a single value iteration for the state up here.This is the state a4.The answer is 76.The value of over here is maximized for the south action,which we reach with 0.8 chance with 0.2 chance we'll stay in the same state for which inital value was 0,and we subtract the action cost of 4, which is 80 + 0 - 4 = 76.Now using the same premise as before, P equals 0.8, cost equals -4 and gamma equals 1,I'd like you to run value iteration to completion.For the one state over here, a4, I'd like to know what is it's final value.Now you might be tempted to write a piece of computer software,but for this specific state, it's actually possible to do it with a relatively simple peice of math.It's not trivial, but give it a try.What is the value of a4 after convergence?The answer is 95. To see, we observe the dominant equation suggests thateach new iteration doesn't change the value.We kind of know that the optimal policy is to go south over here, so just really the value iteration for going south.Let's call it the value of x.We know that x is updated by 0.8 times 100 plus 0.2 of saying in the same state,whose value is x minus the cost.This invariance must hold true after convergence. We can now resolve it for x.We get 0.8x equals 76, so 76 divided by 0.8 is 95.Finally, I'd like to ask you what is the optimal policy for the parameters we just studied.I'm listing here all states--a1, a2, a3, a4, a5, b2, and b3--and I'd like you to tell me whether you would like to go north, south, west, or eastin any of those six states over here.For each of those there is exactly one correct answer.It is easy to see that these three states over here, a1 to a3, you want to go east.In the state a4 up here, you wish to go south.In b2 you with to go north so as to not risk the 0.2 probability of falling into -100.In b3 it's perfectly fine to go east. In the worst case you find yourself in b2, in which case you can safely escape the -100 by going north, turn right again, and go down over here.This is the correct set of answers over here.In this question I would like you to check all the boxes that are true.First, there exists at least one environment in which every agent is rational,by which I mean optimal.For every agent, there exists (at least) one environment in which the agent is rational.To solve the sliding-tile 15-puzzle, an optimal agent that searches will usually requireless memory than an optimal table-lookup reflex agent.By "usually" I mean there are always extreme cases that he can constructwhere this isn't the case. I ask about the common case.Here is the sliding-tile 15-puzzle, and you can see this is a puzzle where you move these pieces arounduntil all these numbers are in order.It's a somewhat combinatorial search problem.Finally, to solve the sliding-tile 15-puzzle, an agent that searches will always do better--that means it will always find shorter paths--than a table-lookup reflex agent.This question is about A* search for the heuristic function h,which is indicated in the graph over here.An action costs 10 per step. Enter into each node of this graph the order when the node is expanded.That is the same as removed from the queue in A*.Start with a "1" in the start state over here at the topand enter "0" if the node will never be expanded. This is a graph where we have a whole number a nodes,and the heuristic function is indicated over here.I'm also asking you is the heuristic h admissible?Here's an easy question.For coin X, we know that the probability of heads is 0.3.What is the probability of tails?In this probability question, we study a potentially loaded or unfair coin, which we flip twice.Say the probability for it coming up heads both times is 0.04.These are independent experiments with the same coin.I wonder what is the probability it comes up tails twice if we flip the same coin twice.We now have two coins--one fair coin for which the probability of heads is 0.5,and a loaded coin for which the probability of heads is 1.This might be a coin where heads is on both sides.We now pick a coin at random with 0.5 chance,and we don't quite know which coin we've picked,but we do flip this coin, and we see "heads.What is the probability that this is the loaded coin?We now flip this coin again (the same coin), and we see "heads" again for a second time.What is now the probability that this is the loaded coin?Here's a Bayes network question. Consider the following Bayes network with variables A all the way to I.A and B connects in to E. C and D connects into F.E connects into G and H, and F also connects into H but also into I.I'm asking the question whether A is independent of B,A is conditionally independent of B given E.A is conditionally independent of B given G.A is conditionally independent of B given F,and A is conditionally independent of C given G.Check out this Bayes network over here,which is defined by the following conditional probability table:P of A equals 0.5. A connects into B and C.P of B given A is equal to 0.2. P of B given not A is also 0.2.P of C given A is equal to 0.8.P of C given no A is 0.4.You'll find an interesting oddity in this table if you look very carefully.I'd like to ask you what is the probability of B given C,and what is the probability of C given B?In this question, we apply naive Bayes with Laplacian smoothing--the same as we have learned in class.We have now 2 classes of movies. One is called "old" and one is called "new."There are titles in here. There's three old movies--Top Gun, Shy People, Top Hat.Two new movies--Top Gear, Gun Shy.Use Laplacian smoothing with k=1 to compute the probability of a movie being old--this is a prior probability, which is just based on class counts--the probability of the word "top" as a title word in the class of old movies,and the probability that a new movie that we look at--by new I mean a movie we've never seen before--that is called "top," the probability this movie that corresponds to the old movie class with the new movie class.I recommend you use a single dictionary for smoothing,so look at all the words and see how large the dictionary is.Top occurs here in two different ways.One is a word over here, but one also is a movie title over here.Don't pay too much attention to it, just don't get confused by it.Again, use Laplacian smoothing with k=1.In this question, I'm giving a set of data points--some positive, some negative--and a query point.Given the following labeled data set, I'd like to find the minimum of "k"for which the query point over here becomes negative.Enter "0" if this is impossible. Ties are broken at random, and I'd suggest trying to avoid them,because you might not be able to guarantee that the class is negative.In linear regression we are given a data set where the x's go 1, 3, 4, 5, and 9,and the y's 2, 5.2, 6.8, 8.4, and 14.8.I'd like to find the formula y = w1x = w0 by minimizing the residual quadratic erroras we learned in class in linear regression.What will be w1, and what will be w0?K-Means Clustering.We're given a data set indicated by the solid dots over here.There's a total of 9 dots if you count carefully.We have 2 initial cluster centers: C1 and C2 as indicated by those stars.I'd like to run K-Means to completion and wonder what the final location of C1 will be after running K-Means.Ignore the A, B, C, or D over here.Here is our logic question.I would like you to mark each sentence as "valid," which means it is always true--you can't make it untrue--"satisfiable," which means it is sometimes truebut could also be false depending on the variable values, or "unsatisfiable," which means you cannot possibly make it true.The first statement is not A.The second is A or not A.The third one is (A and not A) implies (B implies C).The fourth one is (A implies B) and (B implies C) and (C implies A).The next one is (A implies B) and not (not A or B).The final one is ((A implies B) and (B implies C)) equivalent to (A implies C).Remember, you might use truth tables to find out.The planning question might be a bit hard to read, so let me read the text for you.In the state space below, shown over here, we can travel between locationsS, A, B, and G along the roads as shown.For example, SA means we go from S to A.But the world is partially observable and stochastic.There may be a stoplight somewhere between B and G that can prevent passing from B to G.The action might fail, and there might be a flood that sits between A and G,and the flood also makes the action going from A to G fail.If the flood occurs, it will always remain flooded.If the stoplight is red, it will flip green at some point, but we can't predict when.The flood is only visible at A and the stoplight only visible at B.I want you to check all these plans over here and see what the outcome is.There are 3 potential outcomes.One is it always reaches the goal state and does so in a bounded number of steps.By bounded I mean in advance you can tell me a maximum number of steps--not after the fact.If you cannot do this after the fact, it's really not bounded.The second possibility is always reaches the goal state,but the number of steps cannot be bounded in advance.The third one is it might actually fail to reach the goal state.Look at the following plans: SA followed by AG, SB, step 2 if we can't move go back to 2,then finally proceed to BG, and so on and so on.See which of these plans fall into which category over here.Here is an MDP question. We have a deterministic environment, which means the state transitions are deterministic.There is no probablistic or stochastic outcome of actions.The cost of motion is -5. The terminal state is worth 100 as indicated.We have four actions: north, south, west, and east.The shaded state can't be entered.Please fill in the final values after value iteration converges.In this final question, I'd like you to learn the position parameters of Markov chains.There's a Markov chain over here with two states.There is an initial state distribution for the time step 0.Then there is conditional state distribution from time T to time T + 1.You might go from A and stay in A.You might go from A to B.You might go from B and stay in B and go from B to A.What we observe is the sequence A, A, A, A, B.This is our sample for the initial state and all these transitinos over here are samples for the state transitions in this Markov chain.I want you to compute all the parameters, which is the initial distribution,and the transition distribution out of state A and out of state B.However I'd like to do this with Laplacian smoothing with k=1.It is not maximum likelihood.It is Laplacian smoothing, which can be applied just exactly the same way we saw it in class in various contexts.I'd like you to learn these parameters of the Markov chain from the observed sequence.Again, the only sample for the initial state is the very first measurement observation in this sequence over here.This unit is about games. Why games?Well, for one, games are fun. They've captured the imagination of people for thousands of years.They form a well-defined subset of the real worldin that they have rules, which we understand and write down, and they are self-contained.They're not as messy as driving a car or flying an autonomous planeand having to worry about everything in the world.In that sense, they form a small-scale model of a specific problem.Namely, the problem of dealing with adversaries.Along the way we've seen a lot of different technologies in this classand a lot of different techniques, that are focused at different parts of the agentand environment mix and different difficulties there.Here we have a quiz, and what I want you to tell me is for each of these technologieswhat do they most address? Some of them address more than one, but give the best answer for each line.Do they address the problem of a stochastic environment--that is one where the results of actions can vary?Do they address the problem of a partially observable environment--one where we can't see everything?Do they address the problem of an unknown environment--one where we don't even know what the various actions are and what they do?Do they address computational limitations--that is problems of dealing with a very large problem rather than a small oneand making approximations to deal with that?Or do they deal with handling adversaries who are working against our goals?And I want you to answer that for MDPs, Markov decision processes; POMDPs, partially observable Markov decision processes and belief space;for reinforcement learning, and for A* algorithm, heuristic function, and Monte Carlo techniques.The answer is the MDPs are designed to do stochastic control.POMDPs are designed to deal with partial observability.Reinforcement learning deals with an unknown environment,and the heuristic function and A* search and Monte Carlo techniquesare used to deal with computational limitations.Monte Carlo techniques gives us an approximation.The heuristic function, if we use the right one, still gives us the right answer,but deals with the computational complexity.We don't as yet have any technology that's specifically designed to deal with adversaries.What is a game?The philosopher Wittgenstein said that there is no single set of necessary and sufficient conditions that define all games.Rather games have a set of features, and some games share some of them, and other games share others of them.It's a complex overlapping set rather than a simple criteria.Here I've listed six different games and in some cases sets of games like Chess and Go are similar, Robotic Soccer, Poker, hide-and-go-seek played in the real world,Cards Solitaire, and Minesweeper, the computer solitaire game.I want to ask you, for each one, which of these properties they exhibit.Are they stochastic? Are they partially observable? Do they have an unknown environment? Are they adversarial?For each game tell me all that apply.Let me add that your answers may not be the same as mine, because these very terms are not that precise.Sometimes you can analyze a problem in two different ways and flip from one of these attributes to another, depending on how you analyze it.Now, I've chosen to say that only robotic soccer and hide-and-go-seek are stochastic.By that I mean if you have an action like go forward 1 meter,the result of that action stochastic. You may not go forward exactly 1 meter.You could also analyze games like poker and cards and say that they're stochasticin that the next car is random, and so the action of flipping over the next card is stochastic.You don't know how that action is going to result.I've chosen to model that as partial observability.What I've said is it's not that you pick randomly from the next card,it's that the cards are already arranged in some order.It's just that you don't know what that order is.There's partial observability that gives you the next card.Partial observability also shows up in the real world sportsor of robot soccer and hide-and-go-seek.Obviously, that's kind of the point of hide-and-go-seek that it's partially observable.Now, in terms of unknown, I've said that only hide-and-go-seek satisfies that.In everything else, the world is well-defined.Even in the real world in an environment like robot soccer,you only have the known field to deal with.Whereas in hide-and-go-seek, someone could be hiding anywherein a room or location that you don't know about yet.Notice that many games are adversarial, but some games are not.Solitaire games are not adversarial.You could mark that down as saying, well, I'm playing against the game itself,but we don't count that as adversarial, because the games itself is not trying to defeat you.The game itself is passive.Whereas in these games and what adversarial has come to mean is that the opponent is taking into account what you are thinking when the opponent does their own thinking and tries to defeat you that way.Here's a game that we've seen before. We call this a single-player deterministic game.We know how to solve this.We use the techniques of search through a state space--the problems solving techniques.We draw a search tree through the state space, and I'm going to draw the nodes like this with triangles rather than with circles.In any position--in this position here--there are three moves I can make.I can slide this tile, this tile, or this tile.So I have 3 moves, and that gives me 3 more states.I keep on expanding out the states going farther and farther down until I reach onethat's a goal state, and then I have a path through there that gets me to a solution.What does it take to describe a game?Well, we have a set of states S, including a distinguished start state S0.We have a set of players P that can be our one player, as in this game, or two or more.We have a function that gives us the allowable actions in a state,and sometimes we put in a second argument, which is the player, in that state-making action,and sometimes it's explicit in the state itself whose turn it is to move.We have a transition function that tells us the result of,in some state, applying an action giving us a new state.And we have a terminal test to say is it the end of the game.That's going to be true or false.Finally, we have terminal utilities saying that for a given state and a given player there is some number which is the value of the game to that player.In simple games that number is a win or a loss, a one or a zero.Sometimes it's denoted as a +1 and a -1.In other games there can be more complicated utilities of you win twice as much or four times as much or whatever.Now let's consider games like chess and checkers,which we define as deterministic, two-player, zero-sum games.The deterministic part is clear. The rules of chess say you make a move, take a piece, and that's it. There's no stochasticity.It's two players, one against another, and zero sum means that the sum of the utilities to the two players is zero.If one player gets a +1 for winning the game, the other player gets a -1 for losing.How do we deal with these types of games?Well, we use a similar type of approach.We have a state-space search. We have a starting state.There are some moves available to player one.Then in the next state there are moves available to player two.We're going to draw them like this, and we're going to give names to our players.The first player we're going to call Max, because it's a nice name,and because player one is trying to maximize the utility to player one.The next player, who operates at this level, we draw with a downward-pointing triangle.We call that player Min, because Min is trying to minimize the utility to Max,which is the same thing as trying to maximize the utility to himself or herself.Then we have a game tree that continues like that, alternating between Max and Min moves.Now, the search tree keeps going and let's say we get to a point where one player,and let's say it's Max, has a choice, and there are two states,and these, rather than being states where it's Min's turn, are states that are terminal.We'll draw them with a square box.Let's say one of them results in +1, a win for Max,and one of them results in -1, a loss for Max.Now if Max is rational, of course, Max is going to make this choice to the +1.What we're going to do now is show we can determine the value of any state in the tree, including the start state up here in terms of the values of the terminal nodes.The tree keeps on going. We assume it's a finite game.After a finite number of moves, every path leads to a terminal state.Then we look at each state and say whose turn is it to make the decision.In this state Max is making the decision, and Max, being rational, will choose the maximum value, saying, "I'd rather have a +1 than a -1,so I'll get a +1 here."We start going back up the tree, and maybe we get up to a point here where Min has a choice, and we've used this type of process to go up the tree,and Min has a choice between a +1 and a -1.Min is going to choose the minimum and will have a -1 here.If we go through all the possibilities, let's say these all result in -1, but this move results in a +1. Then Max will take that move.He'll say, "Out of my four possibilities, I know this is the best one. I'll take that move."Now we've done two things.One, we've assigned a value to every state in the search tree,and secondly, we backed that all the way up the top.Now we've worked out a path through that state to say,if all players are rational, here's the choices they would make.The important point here is that we've taken the utility function, which is defined only on terminal states.Here's a state here. The utility of that state was +1.Here's a state here. The utility of that state was -1.We've used those utility values in the definition of available actions to back those utilities up and tell us the utility of every state, including the start state.Now let's define a function value of S, which tells us how to compute the value for a given state,and therefore will allow us to make the best possible move.If S is a terminal state, then the value is just the utility of the state given by the definition of the game.If S is a maximizing state, then we'll return something called max value of S,and if S is a minimizing state, then we'll return min value of S.Now we can define max value to just iterate over all the successorsand figure out the values of each of those.We'll initialize a value m equals minus infinity,and then we'll say for all pairs of actions and successors states in successors of S,we'll say the value is--and let's call this S-prime so we don't get confused--the value of S-prime and M for keeping track of the maximum so far and the new value.Then when we're all done we return the M with the maximum value.This will compute the maximum at a maximum node over all the states that we have from all the possible moves.The definition for min value is roughly equivalent but just reversed, taking the minimum instead.With these three recursive routines--value, max value, and min value--we can determine the value of any node in the tree.Now to do that efficiently, you'd want a little bit of bookkeepingso you aren't recomputing the same thing over and over again,but conceptually, this will answer any two-player, deterministic, finite game.Now we know we have an algorithm that can solve any game tree, that can propagate the terminal values back up to the topand tell us the value for any position.It's theoretically complete, but now we need to know the complexity of the algorithm to figure out if it's practical.Let's look at an analysis of how long it's going to take.Let's say that the average branching factor--the number of possible moves or actions coming out of a position--is b.Here b would be 4.And let's say that the depth of the tree is m, so b wide and m deep.Now what I want you to tell me is what would be the computational complexityof searching through all the paths and backing the values up to the top.Would it be of the order of b times m or the order of be to the mth poweror the order of m to the b power? Chose one of these.The answer is we have b choice at the top level,and for each of those b we have another b at the next level.That would be b squared, b cubed and so on, and all the way to b to the mth power.Now the next thing I want you to tell me is the space complexity.That was the time complexity.The space complexity is how much storage do we need to be able to search this tree.Remember that the value and max value and min value routines that we have definedare doing a depth-first search.Which of these would correctly represent the amount of storage that we would need--the space complexity?The answer is that we only need b times m space in order to do the search.Even though the entire tree is order b to the mth power of nodes,on any individual path through the tree we only need to look one path at a timein order to do the depth-first search.We generate these b nodes, store them away, look at the first one, generate b more,store those away, and so we're saving only b nodes at each level for m times levelfor total of b times m total storage space required.The next question is let's look at the game of chess for which the branching factor is somewhere around 30. It varies from move to move.The length of a game is somewhere around 40.Certainly some games are much longer, but that's an average length of a game.Now let's imagine that you have a computer system, and you want to search through this whole tree for chess,and let's assume that you can evaluate a billion nodes a second on one computer.Let's also say that for the moment somebody lent you every computer in the world.If you have all the computers and they can each do a billion evaluations a second,how long would it take you to search through this whole tree?Would it be on the order of seconds, minutes, days, years, or lifetimes of the universe. Tell me which of these.The answer is that it would take many lifetimes of the universe. Even though you have a lot of computing power at your disposal,30 to the 40th power is just such a huge numberthat there is no chance of searching through the entire tree for chess.Now our question is how do we deal with the complexity of having a tree with branching factor b and depth m.Here are some possibilities, and I want you to tell me which of these are good approaches.We have the problem of dealing with b to the m.Could we reduce b somehow, that is, reduce the branching factor,reduce m, the depth of the tree, or convert the tree into a graph in some way?Tell me which, if any or all, of these would be good approaches to dealing with the complexity.The answer is that all three are useful approaches, and we'll look at each of them.Let's review just for a second.This is called the minimax routine for evaluating a game tree.Given a particular state we look and see is it a terminal state? Is it a maximizing state? It is a minimum state?In each case we look up the utility from the game.We do the max value routine, or we do the min value routine, which is similar.That gives us the value of each state.Then the action that the agent would take would be just to take the action that results in the maximum state--the state with the best value.Now let's try to apply the minimax routine to this game tree.This is a small game in which Max has three options for his moves,and then Min has three options for its moves, and then the game is over.Here are the terminal values for these states in terms of Max's score.What I want you to do is use minimax to fill in the values of these intermediate states.What are the values of these three states for Min to move,and what is the value of this state for Max to move?The answer is that these are minimizing nodes.The minimum of 3, 12, and 8, is 3.Here the minimum is 2.Here it's 1.Then this is a maximizing move.The max is 3.That means that if both players played rationally, then Max would take this move.Then Min would take this move, and the value of the game would be 3.Now I want to get at the idea of reducing b, the branching factor.How is it that we can cut down on the number of nodes that we expandin the horizontal direction while still getting the right answer for the evaluation of the tree?Let's go back and consider that during our evaluation, if we get to this point,we've expanded these three nodes, we figured out that the value of this one is 3,we looked at this one so far and found its value was 2,and now, without looking at these, what can we say about the value of this node?Well, it's a minimizing node, so the least it could be is 2.If these are less than 2, it'll be less than that, and if these are more, it'll end up being 2.We can say that the value of this node is less than or equal to 2.Now if we look at it from Max's point of view, Max will have this choice here of choosing either this, this, or this,and if this one is 3 and this one is less than or equal to 2,then we know Max will always choose this one.What that tells us is that it doesn't matter what the value is of this node and this node.No matter what those values are this is still going to be less than or equal to 2,and is not going to matter to the total evaluation, because we're going to go this way anyway.We can prune the tree, chop off these nodes here, and never have to evaluate.Now, with this particular case, that doesn't save us very much,because these are terminal nodes, but these could have been large branches--big parts of the tree, and we still wouldn't have to look at them.We've made a potentially large pruning without effecting the value.We still get the exact correct value for the value of the tree.Now I want you to tell me over here which, if any or all, of the three nodes can be pruned away by this procedure.The answer is when we see the 14 we're not sure what this value is.It has to be less than or equal to 14,which means it might be the right path or it might not.Once we see the one then we know that the value is less than or equal to one,and we know that we have a better alternative here, so we can stop at that point.Then we can prune off the 8.Out of the three, only this node, the right-most, would be the one pruned away.Now I'm going to look at the issue of reducing m, the depth of the tree.Here, I've drawn a game tree and left out some bits, but the idea is that is that it keeps on going and going.There'll be too many nodes for us to evaluate at all. What can we do?The simplest approach is to just by fiat cut off the search at a certain depth.We'll say we're only going to search to level three,and when we get down to level three, we're going to pretend that these are all terminal nodes.We'll draw them as the square boxes for terminals rather than the max nodesand cut off the search at that point.Now, of course, they aren't terminal, so according to the rules of the game,we haven't either won or lost at this particular point.We can't say for sure what the value is for each of these nodes, but we can estimate it using something called an evaluation function,which is given a state S and returns an estimate of the final value for that state.What do we want out of our evaluation function and how do we get it?We want the evaluation function to be stronger for positions that are stronger and weaker for positions that are weaker.We can get it one way from experience--from playing the games before and seeing similar situations and figuring out what their values are.We can try to break that down into components by using experience with the game.For example, in the game of chess it is traditional to say that a pawn is worth 1 point,a knight 3 points, a bishop 3 points, a rook 5, and a queen 9.You could add up all those points.So we could have an evaluation function of S which is equal to this weighted sum of the various weights times the various pieces--positive weights for your pieces and negative weights for the opponent's pieces.We've seen this idea before when we did machine learningwhere we have a set of features, which could be the pieces, and they could be other features of the game as well.For example, in chess it's good to control the center,it's good not to have a double pawn, and so on.We could make up as many features as we can think of to represent each individual stateand then use machine learning from examples to figure out what the weight should be.Then we have an evaluation function.We apply the evaluation function to each state at the cutoff pointrather than doing a long search.Then we have an estimate, and we back those values up just as if they were terminal values.Now let's see how we can compute the value of a state using thesetwo innovations to work on b and m.I've modified our routine for value in two ways--one, I've introduced a new line that says if we decide to cut off the search at a particular depth then apply the evaluation function to the state and return that.Then I've also added some bookkeeping variables.One for the current depth, which will get increased as we go along,and then two values called alpha and beta, which are the traditional names,where alpha is the best value found so far for Max along the path that we are currently exploring, and beta is the best value found so far for Min.Then since we have these extra parameters when we start out,we would make the call value of our initial state S0 and we're currently at depth zero in the search tree, and we haven't found the best for Max yet so that would be minus infinity,and the best for Min similarly we haven't found anything there so that would be plus infinity.We call that and then each node we would chose one of these four cases.Here's the new definition of maxValue taking the depth and the alpha and beta parameters into account.It's similar to what we had before.We go through all the successors.We take the maximum, and in this case we're incrementing the depth as we call recursively for the value of each node.We get the cutoff here if we exceed beta,and otherwise we retain alpha as the maximum value to Max so far.Then we return the final value.Now we said we have three ways to reduce this exponential b to the m--reducing the branching factor b, reducing the depth of the tree m, and converting the tree to a graphLet's see how each of those fare.First, for reducing b we came up with this alpha-beta pruning technique.In fact, that is very effective.That takes us from a regime where we're in order b to the m to one where,if we do a good jog, we can get to order b to the m/2.Now what do I mean by doing a good job?Well, we get different amounts of pruning depending on the order in which we expand each branch from a node.If we expand the good nodes first, then we get a lot of pruning,because we do a good job of getting to the cutoff points.If we expand the poor nodes first, then we don't do any pruning, because we don't get to that cutoff point until later.But if we can do well, then we get to the square root of the number of nodes.In other words, we get to search twice as deep into the search tree.That's all 100% perfect in terms of not changing the result.We'd still get the exact evaluation.We just stop doing work that we didn't have to do.Now for the tree to the graph, we haven't talked that yet.In fact, it depends on the particular game, but in many games it can be very useful.In games like chess, we have opening books.That is, we look at the past openings and we just memorize those positions and what are the good moves.It doesn't matter how we get to those positions.We can get to them in multiple paths through a tree,and we can just consider it a single graph.We also have closing books, where we can memorize all the positions with five or fewer pieces and know exactly what to do.In the midgame when there are too many positions to memorize all of them,we can still search through a graph if we want to or if we want we can just do part of that.One thing that has proven effective in games like chess is called the killer-move heuristic.What that says is if there's one really good move in part of a search tree,then try the other move in the sister branches for that tree.In other words, if I try making one move and I find that the opponent takes my queen,then when I try making another move from that same position,I should also check if the opponent has that response of taking my queen.Converting from a tree to graph, also doesn't lose information.It can just help us make the search go faster.The third possibility was reducing m, the depth of the tree,by just cutting off search and going to an evaluation function.That is imperfect in that it is an estimate of the true value of the treebut won't give you the exact value. We can get into trouble. Let me show you an example of that.Here's a search tree for a version of Pacman in which there's only four squares.There's a little Pacman guy who can move around,and there are food dots that the Pacman can eat.Maybe someplace else in the maze there are opponents,but we're not going to worry about them right here.We're just going to consider the Pacman's actions.He has two actions--to go left or rightIf he goes left, he goes over here and eats that food particleand then moves back right--that's his only move from that position.Or if he moves right then he has two other moves.Now let's assume that we cut off the search at this depth,and we want to have an evaluation function,and the goal is for Pacman to eat all the food.The evaluation function will be the number of food particles that he's eaten so far.What I want you to do is tell me in these boxes what the evaluation should be for each of these three states.The answer is here he's eaten 1, here he's eaten 0, and here he's eaten 1.That's fine. The problem arises when we start backing up these numbers.If these are max nodes, we've skipped the opponent's moves, which are the min nodes.We're only looking at the maxes.The max of 1 is 1, so this would also get an evaluation of 1.The max of 0 and 1 is 1, so this would also get an evaluation of 1.This final node would be the max of 1 and 1, so that's also 1.But now when we go to apply the policy, if we're in this position, using these evaluation functions, both of these moves are equally good.The Pacman might choose this one, choosing at random or choosing by some predefined ordering.Then he'd end up in this state. So far he hasn't eaten anything.But this state is just as good because he knows in two moves he can eat one particlegoing this way just as well as in two moves he can eat one particle going this way.Now he's in this state, but notice that this state is symmetric to this one.On his next turn, if we did another depth-two search, he might just as well go back one position.He would be stuck going back and forth between these two states,because either one of those, if you look ahead only two, is equally good.You have to look ahead one, two, three, four moves to know that one of them is better than the other.This is known as the horizon effect.The idea is that when we cut off search we're specifying a horizon beyond which the agent can't see.If a good thing or a bad thing happens beyond the horizon, we don't see that.All we see is whatever is reflected in the evaluation function.If the evaluation function is imperfect, we don't see beyond the horizon, and we can make mistakes.There is one more thing to deal with when we have to talk about games--and that's chance.We want to move from purely deterministic games to stochastic games like backgammon or other games that introduce dice or other parts of random action.That means that the actions that an agent takes are not specifically specified to have a single result.Let's see how we can deal with stochastic games by looking at our value function and modifying it to allow for this.Here we have our valuation function, and we're dealing with four types of nodes--one, nodes that we decide to cut off on our own, because we reached a certain depth;second, nodes that are terminal according to the rules of the game;and third, max to move and min to move.Now I'm going to add one more type, which is a chance node.We say if the state is a chance node, then we want to return the expected value of S and carry along these bookkeeping variables.What we're saying here is if it's at the point of the game where it's time to roll the dice,then we're going to role the dice, and we're going to take the expected value of all the possible results rather than the max or the min.Here we have a schematic diagram for a stochastic game--a game with dice.We start out. The chance node or the dice-rolling node is first. The dice is rolled--one of six possibilities.Then the next player gets his move.In this case, we've let Min move first, and Min has various moves possible to make.For each one, there is then another role of the dice, and then Max gets to make his move.Here's the game tree for another stochastic game.This game involves flipping a coin.The chance nodes have two results: heads or tails.Then the player Max has two possible moves, A and B,and the player Min has two possible moves, C and D.This game is too small to have any alpha-beta pruning involved,but I've listed all the terminal values for the terminal states of the game.What I want you to do is fill in the non-terminal values for the chances nodes, the max nodes, and the min nodesaccording to the rules of minimum value and maximum value and expected value.I should say that the probability of the coin flip is 50% heads and 50% tails.To evaluate the game tree, we work from the bottom up.Let's start over here. This is a min node.Min chooses the minimum, which will be 1.In this position, Min would choose 2, the minimum of 2 and 4.Over here Min would choose 0, the minimum of 0 and 10.Now we have some chance nodes, so we have to choose the expected value.Chance, the flip of the coin, doesn't get the choice of one direction or the other.Rather both of them are possibilities.So we just average the results, since the probability of heads and tails are equal.So 7 and 1 is 8, divided by 2 is 4, and 8 and 2 is 10 over 2 is 5 is the expected value there.The expected value of 0 and 6 is 3, and the expected value of 0 and 4 is 2.Now we have a maximizing node. The max of 5 and 4 would be 5.The max of 3 and 2 would be 3, and finally, we have another chance node.The average of 5 and 3 would be 4, and that's the value of the final state.Now one more question for this same game tree.I want you to click on all the terminal states that are possible outcomes for this game if both players play rationally.No subtitles...One more quick game tree to evaluate.Here we have terminal values. We have chance nodes where the two options are equiprobable.We have a max node. The two actions A and B.I want you to fill in the values for all the nodesand click on which action, A or B, is the rational action for Max.The average of these two is 2. That's a chance node.The average of these is 2.5.Max will choose the better of 2 and 2.5, which is 2.5.Therefore, B will be the rational action for Max.Now we know if this game if these were terminal nodes,then that would be the right action for the game, and there was nothing to argue about.But what if instead of having these be terminal nodes, these were cutoff nodes,and these were evaluation values for those nodes?Furthermore, if it's an evaluation function, then it's an arbitrary function.Suppose if instead of coming up with these values, we used a different evaluation function,which squared these values, and so we came up with evaluations of 0, 16, 4, and 9.With that function, I want you to repeat the problem of filling in the values for each of these nodes and tell me what the rational action is for Max.The answer is this is a chance node so we take the average of 0 and 16.That's no longer 2. It becomes 8.We take the average of 9 and 4. That's no longer 2.5. It becomes 6.5.Notice what's happened now is Max now chooses 8 over 6.5.and now the rational action has shifted from B to A. What's gone on here?We notice that just by making a change to the evaluation function, we changed the rational action.Let's summarize what we've done so far.We've built up this valuation function that tells us the value of any state,and therefore we can choose the best action in a state.We started off just having terminal states and max value states.That's good for one-player, deterministic games,and we realized that that's just the same thing as searches we've seen beforewhere we had A* search or depth-first search.Then we added in an opponent player for two-player or multiplayer games,which is trying to minimize rather than maximize. We saw how to do that.Then we optimized by saying at some point we may not be able to search the whole tree,so we're going to have a cutoff depth and an evaluation function.We recognized that that means that we're no longer perfect in terms of valuating the tree. We now have an estimate.We also tried to be more computationally effective by throwing in the alpha and beta parameters,which keep track of the best value so far for Max and Minand prune off branches of the tree that are outside of that rangethat are provably not part of the answer for the best value.We kept track of those through these bookkeeping parameters.Then finally we introduced stochastic games,in which there is an element of chance or luck or rolling of the dice.We realized that in order to valuate those nodes,we have to take the expected value rather than the minimum or the maximum value.Now we have a way to deal with all the popular types of games.The details now go into when we figure out to cut off and what's the right evaluation function.Those are a complex area. A lot of research in AI is being done in that,but it's being done for specific games rather than for the theory in general.Hey, welcome back.Hope you enjoyed the last unit. You guys have been doing great.You've been doing amazing work, getting a lot done,doing a really good job of answering the questions.I've been looking at this book here.This is a book from my father's collection.It's called "Introduction to the Theory of Games" by McKinsey.It was published in 1952, 4 years before the start of artificial intelligence.And so game theory and AI have kind of grown up together.They've taken different paths, and now they've begun to merge back together.We've talked about games already in a previous unit.We talked about mostly turn-taking gameswhere 1 player moves and then another moves,and the trick is how to work against an adversarywho's trying to maximize his own utility and thus minimize your utility.Game theory handles those types of games, but it also really focuseson games where the 2 moves are simultaneous,or another way to think about them is 1 player movesand then the other moves, but the second player doesn't knowwhat choice the first player made, so it's partially observable.And it's this back and forth of trying to figure out what should I movegiven what I think he's going to move and what does he think aboutwhat I'm going to move that gives game theory its special status.We're going to talk about how that works for AI,and 2 problems are studied.The first is agent design. That is, given a game, find the optimal policy.And the second is mechanism design.That is, given utility functions, how can we design a mechanism so thatwhen the agents act rationally the global utility will be maximized in some way?Let's take a look.We're going to talk about game theory,which is the study of finding an optimal policywhen that policy can depend on the opponent's policy and vice versa. And let's look at 1 of the most famous games of all,a game called the "Prisoner's Dilemma."And the story is that there are 2 criminals, Alice and Bob,who have a working relationship, and they're both caughtat the scene of a crime, but the police don't quite have enough evidenceto put them away.They offer each independently a deal saying"If you testify against your cohort,we'll give you a better deal and give you a reduced sentence time."And Alice and Bob both understand what's going on.They're both perfectly rational, and to understand what the situation is,we draw up a matrix in which we have possible outcomesand possible strategies for each side.For Alice, she has 2 strategies.1 is to testify against Bob,and the other is to refuse to testify.And Bob has the same choices, to testify against Alice or to refuse.In general, different agents may have different actions available to them.And now we show the payoff to each agent.Sometimes those payoffs are opposite,as in a game like chess where if 1 player gets a +1,the other gets a -1.In this game, the payoffs are not opposite,so it's a non-zero-sum game.And if they both refuse to testify against each other,then neither can be convicted of the major crime,but the police will get them for a lesser crime.And let's say they each serve 1 year in jail,so that's a -1 for each of them.If Alice testifies and Bob refuses,then the police are grateful to Alice,and she gets off with nothing, and Bob gets the book thrown at him and gets a -10 score.Likewise, if the roles are reversedand if both testify against each other, then they're both guilty,and they split the penalty.Now, the question that both Alice and Bob have to faceis what is the strategy going to be?And the first concept we want to talk about is the concept of a dominant strategy.A dominant strategy is one for which a playerdoes better than any other strategyno matter what the other player does.And now the question is, does either Alice or Bobhave a dominant strategy?If Alice has a dominant strategy, I want you to check that off, either testify or refuse,and similarly, if Bob has a dominant strategy,check that off.The answer is for Alice,testify is a dominant strategy.Let's see. We have to compare it against all possible strategies for Bob.If Bob does testify,then Alice gets -5 here and -10 here,so testify is better.And if Bob does refuse,then Alice gets 0 here and -1 here, so testify is better.Testify is better for Alice no matter what,and by similar reasoning, testify is better for Bob no matter what, so testify is a dominant strategy for both players.The next concept I want to talk aboutis the concept of a pareto optimal outcome.So, this is talking about outcomes rather than strategies.The strategies are in the margins.The outcomes are in the matrix, and the pareto optimal outcomeis one where there's no other outcomethat all players would prefer.And this is named after the economist Pareto.What I want you to answer isis there a pareto optimal outcome in this game?Is there an outcome such thatthere's no other outcome that all players would prefer?And the answer is that this outcome, A = -1, B = -1, is Pareto optimal because there's no other outcomethat all the players would prefer.Sure, B would prefer being up here,and A would prefer being over here,but none of them that both players can agree on.Now, the third concept is the concept of equilibrium.An equilibrium is an outcome such that no playercan benefit from switching to a different strategy,assuming that the other players stay the same.And there was a famous result from John Nash, economist,who was shown in the movie and book "A Beautiful Mind"proving that every game has at least 1 equilibrium point.The question here is which, if any, of these outcomesare equilibriums in this game?And the answer is only this outcome, with A = -5, B = -5, is an equilibrium pointbecause if A switches, it gets -10.If B switches, it gets -10.Neither player wants to switch away from keeping with that strategy.Over here, the Pareto optimal solution is not an equilibrium pointbecause if B switches, it will do better,and A will do worse.This is where the game turns out to be a dilemmabecause there's an equilibrium point that it seems likeif both players are rational, they're bound to end upin this outcome, whereas the Pareto optimal solution is over here in the other corner.And yet, being rational, neither Alice nor Bob can see a wayto get to this preferred outcome.Let's try another example.This one is called the Game Console Game,and the story is that there is a game console manufacturer called Acme,and it has to decide whether its next consoleis going to play Blu-ray discs or DVD discs.And then there's a game manufacturer called Best,and they similarly have to decide whether to put out their next gameon Blu-ray discs or DVD discs.And the payoffs are if they're both on Blu-ray,A gets a +9, and B is also a +9.If they both choose to go with DVD, it's not quite as lucrative.A gets a +5. B gets a +5.And if they disagree, then they'll be in trouble, and they'll take losses.A gets a -4, and B gets a -1, while here A = -3 and B = -1.The first question is is there a dominant strategy?And is there one for A?Click here if yes.And is there one for B?Click here if yes, and if there's none at all, click here.There may be both A and B. It's your choice.And then the next question is is there an equilibrium?Click on any of these 4 outcomesto indicate whether there's an equilibrium.The answers are that there's no dominant strategybecause for each player, what's best depends on what the other player does.They do best if they match,and so you can't figure out what your own best strategy isunless you know what the other player is going to play.In terms of equilibrium, there's 2 equilibrium points, the +9/+9 and the +5/+5.Both of them are equilibriums because neither of the playerscan benefit from switching to the other strategy.And now the next question isis there 1 or more Pareto optimal outcomes?Click on any of the outcomes that you think are Pareto optimal.And the answer is that there's just 1.The +9/+9 is Pareto optimal.Both players would rather be there than anyplace else.And so it seems that if both players are rationalthey'll both know that there are 2 equilibrium points,but only 1 of them is Pareto optimal.And even though there isn't a dominant strategy,they can both arrive at that happy conclusion.So, we've seen that it's easy to figure out the solution to a gameif there's a dominant strategy or if there's a Pareto-optimal equilibrium.Now let's look at a harder game for which such solutions don't exist.This game is called Two Finger Morra,and it's a betting game, and we're going to show a simplified version of it.Again, we have a simple 4-state outcome matrix,and there are 2 players called even and odd.And they both simultaneously show either1 or 2 fingers.And then if the result of the total number of fingers is even, then the even player wins that many dollars from the odd player.And if the total number of fingers is odd,then the odd player wins that number of dollars from the even player.So, if 1 and 1 is 2, so that's even, so even gets +2, and we won't bother writing odd getting -2because it's a zero-sum game, and it will always be the opposite. Similarly, 2 and 2 is 4, so even gets +4 and odd gets -4.2 and 1 is 3, so even loses 3 dollars and pays it to odd and similarly up here.Now, there's no dominant strategy, and it seems kind of trickyto figure out what the right strategy is.We're going to need more complicated techniques,and it turns out that there is no single movethat's the best strategy for either player.But there is what's called a mixed strategy,so a single strategy of always playing one or the otheris called a pure strategy, and a mixed strategyis when you have a probability distribution over the possible moves.Now, since it seems complicated to solve this game in this form,one way we can address it is to change from this matrix forminto the familiar tree form.We'll move this over here,and we'll draw it as a game tree.Max will be the even player, and min will be the odd player,and for the moment, let's look at the gameof what would happen if max had to go firstrather than having them move simultaneously.So, max would make a move either 1 or 2.And then min--so max is even and min is O--would also make the move, 1 or 2, 1 or 2. And then the outcome in terms of E would be 2 here-3 here, -3 here and 4 here.And now what does min do? Well, try to minimize.So, we choose 2 here, so this node would be -3.We'd choose 1 here, so this node would be -3,and then E tries to maximize.It doesn't matter what he chooses,and we get a -3 up here.So, that's giving E the disadvantage of having to revealhis or her strategy first.What if we did it the other way around?Let's take a look at that.What if O had to go first and reveal a strategy of 1 or 2and then E as the maximizing player goes secondand does a 1 or 2?And then we have these 4 terminal states here,and I want you to fill in the values of the 4 terminal statestaken from the table and the intermediate statesor the higher up states in the tree as well.And the answer is 1 + 1 is 2, and so that's even, so it'd be a positive payoff to E.1 + 2 is 3, that's odd, so it'd be a -3.Similarly, 2 +1 is 3, which is odd.So, -3, 2 + 2 is 4. That's a positive payoff.Now E is maximizing, so E would prefer 2 hereand would prefer 4 here.And now O is minimizing, so O would prefer 2 here.And notice what we've done here is thatwe're trying to figure out what the utility of the game isto E, and the true game, both players move simultaneously.Over here, we've handicapped E.And over here, we handicapped O.The true value of the game must be somewhere in between there,so we can say that the utility to E must beless than or equal to 2, which is the value here,and greater than or equal to -3, which is the value here. We've narrowed it down to some degree, but we still haven't nailed downexactly what the utility of the game is.Now, 1 reason there's such a wide discrepancy in the outcomesof these 2 versions of the game is thatwe handicapped E and O so severelythat here E had to reveal his entire strategy,whether he's going to play 1 or 2 all the time,and the same thing for O over here.What if we could think of a way where we didn't handicap them quite as much,where they weren't giving away quite as much information?Let's look at a way to do that.Let's look at the situation where E goes firstand has to reveal the strategy, but instead of having to reveal my strategy is to play 1 or to play 2, what if E says "Well, my strategy iswith probability P, I'm going to play 1.""And with probability 1 - P, I'm going to play 2."And that's called a mixed strategy.So, E would announce that strategy for some number P.And there could be an infinite number of possibilities,so we should be drawing an infinite number of branchesout of this decision point for all the possibilities for values of P that E would come up with.But instead, I'm just going to sort of parameterize that and just draw 1 line coming out.And now O as the minimizing playerhas to make a choice between 1 and 2, and what are the outcomes?Well, if P was 1, then 1 + 1 is 2, so with probability P, we get an outcome of 2.That's 2P, but if we choose 2,the probability 1 - P, then 2 + 1 is 3, so with probability 1 - P, we get a -3.So, 2P - 3 times 1 - Pwould be the outcome for this day.And then the outcome over here would be-3P + 4 times 1 - P.That's the parameterized outcome given the parameterized strategy.And we could do the same thing on the other side.What if O had to go first? With probability Q, O plays 1, and with probability 1 - Q plays 2.Then even is the maximizer and we get 2Q - 3(1 - Q)and -3Q + 4(1 - Q).Now, what value should E choose for P?Remember, you've got an infinite number of choices for any value for P.Well, if E chose a value of P such that this value here was larger than this value here,then O would know to always play 1, and similarly, if this value was larger,then O would know to always play 2.So, it seems that what E wants to dois choose the value of P such that these 2 are equal.So, how much is that? Well, let's see.This is 2P - 3 - P.That's 5P - 3, and we want to set that equal to -3 + 4 - P.That's -7P + 4.And let's gather the terms together.So, that would be 12P = 7or P = 7/12.So, if E chooses the value of P = 7/12,so 7/12 of the time play 1, 5/12 of the time play 2,then O doesn't know what to do.No matter whether he chooses 1 or 2, he gets the same result.And you can do the same calculation over here,and it turns out that Q also equals 7/12.Now, let's take this strategy of P = 7/12, 1,and feed it back into the matrix for the game,and if E plays this strategy of 7/12, 1, 5/12, 2, then no matter what the strategy O plays, the value of the game to E, the utility to E is -1/12.And then we can do the same computation over here.If Q has the strategy 7/12, 1, and 5/12, 2,then we plug that back into here, and no matter what strategy E chooses,the value there is also -1/12.And so now we've shown that the utility to Eis greater than or equal to -1/12and less than or equal to -1/12.In other words, the utility to Eis exactly -1/12, so we've solved the game.Now, the introduction of mixed strategybrings us some curious philosophical problemsrelated to the idea of randomness, secrecy, and rationality.We said that sometimes the rational strategycan be a mixed strategy.That is, ones with probability in it.Probability P, I do action A, and with probability 1 - P I do action B. And that suggests that we need some secrecyso that our opponent doesn't know which of these random choices we're making.The curious thing is that that's only trueat the extent of an individual play, not to the extent of the strategy itself.So, if this is the optimal strategy,a mixed strategy, it's okay for us to reveal that strategy to our opponent because our opponentcan also compute that that's our rational strategy,and so we won't do any worse by revealing to the opponentexactly what our strategy is.However, the actual implementation of that strategy, that is, this is the grand strategy, that in this situation,whenever we're faced with playing this game, this is what we'll do,that part can be revealed, but the actual choicethat this time we're going to choose A or we're going to choose B,of course, that has be to kept secret.If we reveal that, if our opponent can somehow discoverwhich choice we're going to make based on this random choice,then our opponent can get an advantage over us.Now, with respect to rationality,we said that a rational agent is one that does the right thing,and that's still true.However, it turns out that there are games in which you can do better if your opponent believesyou are not rational, and that has been said about various politicians throughout history, and I won't pick on one or another.But sometimes it has been said that they are intentionally cultivating an image of being crazyso that they can gain an advantage when faced with certain games with opponents.For example, suppose 1 action available to a leader is to go to war,but both sides realize that the strategy of going to waris dominated by other strategies and thus would be irrational.So, a leader who is perceived to be rational and makes a threatof "Give me this concession, or I'll go to war against you,"that's not a credible threat.The leader's threat would be dismissed, and it would have no effect.However, if the leader can convince the opponentthat he is irrational or crazy, then the threat suddenly becomes credible.And so note that being irrational doesn't help,but appearing irrational can gain you an advantage.Now, let's give you a chance to solve a game.This will be another 2x2 game,and let's just go ahead and call the players max and min,and they each have 2 moves,and we'll make this be a zero-sum game.We'll just show the value to max, and the value to min will be the negation of that.And what I want you to do to solve the gameis tell me what the strategy should bein terms of fill in these blanks.What are the percentages that min should play 1 or 2?And in these blanks, the percentages for max to play 1 or 2.And then tell me the final value for the game.The utility or expected value to max equals what?We see in this game each player has a dominant strategy.For max, if he plays strategy 1, then that's better than playing strategy 2.If min plays 1, then strategy 1 is also better than strategy 2.If min plays 2, so strategy 1 is the dominant strategy,and that should have probability 1.Strategy 2 should have probability 0.And it's the same thing for min, that 2 minimizes better than 1 in both cases. So, strategy 2 should have probability 1, and strategy 1 should have probability 0,and that means we're always going to end up with this outcome,and the value of the game is 3.Now, that last one was easy. Let's do one more.Here we have a game, and the payoff to max is 3, 6, 5 and 4.And I want you to tell me what the strategy is,whether it's pure or mixed.What are the probabilities that max should play 1 and 2,and what are the probabilities that min should play 1 and 2?And what is the value of the game to max?So, in this case, there's no dominant strategy,so we'll have to go to a mixed strategy.And we'll start by looking at max and sayinghe has a mixed strategy with a probability P of playing 1, so then if min chooses 1,then we'll have the outcome 3P + 5(1 - P).And we want to set that equal to the outcomeif min plays 2, which is 6P + 4(1 - P).And we solve that, and that works out to P = 1/4.So, P, that was a probability of max playing 1.That should be 1/4, which leaves 3/4 over here.And now we go at it from min's direction,and if min has a probability of Qof playing 1, then we want to set 3Q + 6 (1 - Q)equals 5Q + 4 (1 - Q).And you solve that, and you get Q = 1/2.So, 1/2 and 1/2.And then the utility of the game is the expected value,so we look at all the outcomes and the probability of each outcome.So, 3 times 1/8, because it's 1/4 times 1/2 would be the probability, so 3 times 1/8+ 6 times 1/8 + 5 times 3/8+ 4 times 3/8, and that works out to 4.5.Here's a geometric interpretation that may help you understand a little better what's going on.Here we've gone back to the two-finger Morra game.Now, remember we looked at the two possibilities of E going firstand revealing a strategy of playing one with probability P and two with probability of 1 - P.Now O has a choice of what to do, and O wants to minimize.If O chooses one, he'll be somewhere along this linecorresponding to this strategy for different values of P.This graph here is showing the utility to E for different values of P that P chose in P's strategy.Now since O can achieve the minimum since O gets to chose the strategy of doing one or two,O can be anywhere on this frontier.It makes sense to E to push that up at high as possible since H is the maximizer.E will choose this point here, which turns out to be at P = 7/12.The same argument going on this side.Here O has gone first and chosen the strategy, q:one and (1 - q):two.Now E has a choice of what to do. E can either choose two and be along this line,depending on what the value of q is,or can chose one, which will be along this line.It would make sense for O, who is trying to minimize, to get this frontier down as low as possible.O would choose the value of q that puts us right here at this point.It turns out that that also is q = 7/12.We see that each side is trying to maximize or minimize,and we end up at a distinguished point that's the intersection of their two strategies.Now so far we've dealt only with games that take a single turn--that is there are two players, they both simultaneously reveal their move, and the game is over.But game theory can also deal with more complex games that have multiple rounds of turn taking.Here I'm describing a simple game of poker, the simplest type of poker you've probably ever seen.The deck only has four cards.One card is dealt to each player.There are two rounds.In the first, player one has a choice to either raise--to bet a dollar--or to check.Then in the second round, the second player has the chance to call--to say I want to see what's up--or to fold.Now this format begins to look very much like the game tree that we talked about in the previous unit.It starts out and there's a chance node.This corresponds to dealing the cards with the 1/6 that the first player gets an Aceand the second player gets an Ace.One-third that the first player gets and Ace and the second Player gets a Kind, and so on.There there are maximizing nodes and minimizing nodes.What this format, which is known as the sequential game format,is especially good at is keeping track of the belief states of the possibilitiesof what each agent knows and doesn't know.The tree as a whole describes everything that's going on,but each agent doesn't know at which point in the tree they are.So if you're agent number one, you know that you have an Ace,so you know you're in one of these two states denoted by the dotted lines.You're either in the state where you have an Ace and the other player has an Ace,or in the state where you have an Ace and the other player has a King,But you don't know which one you're at.Similarly, over here there is confusion for the second player as to what state they're in.Now, we can solve this game using this game tree approach,and it's not quite the same as the max and the min approach,because where you are in the states, what you know about the partial information,affects your strategy in a way that we haven't dealt with before.One possibility for how you can evaluate again like thisis just to convert it to the other form.The form we've seen before is called the normal form or matrix form.This is the sequential game in extensive form.If we convert the extensive form, we get something like this.Here for each player, we've denoted by a two-letter strategy what you should do when you have an Ace and what you should do when you have a King.So we end up with an exponentially large search space,but here the game was so simple, that it ends up being rather small,and the game is rather trivial, and you can solve it.It turns out that there are two equilibrium corresponding to the strategy for player two,which is he should check when he has an Ace, and he should fold when he has a King,and strategy for player one is it doesn't matter if he raises or checks when has an Ace,but he should check when he has a King.That would give the game a value of zero.Now this works fine for the simple version of poker.For real poker, this table would have about 10^18 states, and it would be impossible to deal with.So we need some strategies for getting back down to a reasonable number of states.One of the best strategies is to try abstraction.Instead of dealing with every single possible state of the game,we can take similar states and deal with them as if they were the same.For example, in poker one abstraction that works pretty well is to eliminate the suits.If no player is trying to get a flush, then we can treat all four Aces as if they were identicalrather than treating the four of them as being differentand similarly with all the other face values.Another thing we can do is lump similar cards together.Rather than saying that 2, 3, 4, and 5 are all different values,if I know that I'm holding a pair of 10s then I can think of the other players' cardsas being equal to 10, lower than 10, or higher than 10.Otherwise, lump the same.Similarly, I can lump bets together.Rather than thinking of every dollar amount of a bet from $1 to the upper limit,I can lump the bets into small, medium, and large.Then finally another way to do abstraction is rather than considering every possible dealof all the cards, I can just consider a small subset of the deals to do Monte Carlo sampling over the possible deals,rather than considering them all.This approach extensive games can handle quite a lotin terms of dealing with uncertainty, dealing with partial observability,dealing with multiple agents, stochastic, sequential, dynamic.But there's a few things they can't handle very well.They aren't very good at unknown actions.We need to know what all the actions are for either player before we can define the game.Game theory doesn't deal very well with continuous actions,because we have this matrix-like form.It doesn't deal very well with irrational opponents.We can know that we're going to do the best we possibly can against a rational opponent,but it doesn't tell us how to exploit our opponent's weakness if he turns out to be irrational.Then finally, it doesn't deal with unknown utilities.If we don't know what it is we're trying to optimize, game theory isn't going to tell us how to do it.This exercise describes a game played between the federal reserve board and politicians.Now the politicians have a choice whether they want to contract fiscal policy,expand it, or do nothing, and the Fed has the same three choices.Each party has preference for what outcome they would like to see.Here we've ranked them for each party from 1 being the worst outcometo 9 being the best outcome.What I want you is find the equilibrium point for this game.There will be one equilibrium point. I want you to find it.The equilibrium point defines a pure strategy for each player.Tell me the pure strategy for the Fed.It is contract, do nothing, or expand?Click on the right box here.Similarly, for the politicians, click on the right box for their strategy,which leads to the equilibrium point.Then tell me the outcome for the game for each player for that equilibrium point.Then tell me if the equilibrium is Pareto optimal.Now, we could determine the equilibrium point by examining all 9 of the outcomes and checking each one to see if both parties do no better by switching.But instead, I'm going to show an alternative method to analyze games, which is to look for dominated strategies.There are no dominant strategies here, but there are dominated strategies.For example, for the politician, the strategy of contracting is domininatedby the strategy of doing nothing.To the politician, 2 is greater than 1, 5 is greater than 4, and 9 is greater than 6.We can say that this strategy is dominated, and we can take it out of consideration.Now, how does that help?Well, now in the other direction, we do have a dominant strategy that we didn't have before.Now for the Fed, the option of contracting gives them 8, which is better than 5 or 4,or 3 which is better than 2 and 1.This a dominant strategy for the Fed, and we can mark that off.Now for the politicians, they know they're going to be in this column,and they have a choice of getting a 2 or a 3, The 3 would be the strategy for the politicians. That leads us to this Nash equilibrium point,and the values of that outcome are 3 for each party.Is that Pareto optimal?Actually, it's more like Pareto pessimal in that this is worst total.Out of all these outcomes the total is only 6as opposed to every other one is better.To answer the question specifically is it Pareto optimal,the answer is no, because any of these four would be better for both parties.That may tell you something about our political system.Next time you get an outcome that you don't like,don't assume that the players are irrational.Just assume that that's the way the game was set up.Now let's switch to the other part of game theory,which remember we called mechanism design.It could really be called game design.The idea is that someone is going to be running a game that players are going to be participating in.We want to design the rules of the game such that we get a high outcomeor a high expected utility for the people that run the game,for the players who play the game, and for the public at large.Here's an example of a game.This is the advertising game.Here I've shown it on an Internet search engine, where you do a search,and then ads show up, sometimes at the top, sometimes at the right,sometimes at the bottom of the page, depending on the mechanism.This is also done at sites like eBay that sell items, and there's lots of places where auctions are run.The idea of mechanism design is to come up with the rules of the auctionthat will make it attractive to bidders and/or people who want to respond to the ads,and make a good result for all.Now, one property that you would like an auction to have is to attract more bidders to make it a more competitive market,and you could attract more if it's less work for them.It's easier for the bidders if they have a dominant strategy.You saw how hard it was to work out the value of a game when you didn't have a dominant strategy,and how easy it is to work it out if you did.If you want to save everybody a lot of trouble, design the gameso that dominant strategies exist.These strategies have various names in auctions.Sometimes you call it an auction strategy proof if you only need to know your own strategy.You don't have to think about what all the other people are going to be bidding.They also call that truth revealing or incentive compatible.Let's examine a type of auction called the second price option.This is popular in various internet search and auction sites.The way it works is that we have a line of possible prices--higher prices at the top--and bids come in.Different players can bid whatever they want,and whoever bids the highest is the winner,but the price that they pay is the price of the second highest bidder.Now let's say you're participating in this auction,and something is for sale, and you place a value on that.We'll call that value "V", and say V is here.Your bid we'll call "b", and the highest other bid we'll call "c."Now the payoff to you if your bid is higher than all the othersthen the payoff is you get the value of the auction,because you won the item, and you get V,but you have to pay the second highest price, which is c.You get b minus c. Otherwise, you lose the auction.You don't get anything, and you don't pay anything.The value to you of the auction is zero.What I want you to do is fill in this chart to look at different strategies for different possible bids.We'll say that the value to you of the item for sale is V equals 10.You have the option of bidding, say, 12, 10, or 8,and we'll consider the cases where the highest other bid is 7, 9, 11, or 13.What I want you to do is fill in this chart with the value to you of this game according to your strategy and the strategies of the other players.Tell me if one of these strategies is a dominant strategy.Then tell me is that dominant strategy, if there is one, a truth revealing strategy?I should have one note about dominance.When we talked about it before, we glossed over the possibility of ties.If some policy is better everywhere than any other policy,then we say that that policy strictly dominates the others.On the other hand, if there are some ties and some places where its better but none where it's worse, then we say it weakly dominates.Either way, it's a case of dominance.Now I'll do the first entry to get you started.If you bid 12 and the highest other bid is 7,then you have the high bid, so you win.It's a second-price auction, so you pay 7.The value of the goods is 10, so the total value of the outcome is 10 minus the cost of 7 for 3.I want you to fill in the rest.Here are the answers.We can see that the strategy of bidding 10, the true value, is weakly dominant.It's the same here, but it's better in these two cases.Let's look at these cases a little bit more carefully and figure out what's going on.If you bid 12--so if b was up here--and if c snuck in between the bid and the valuation,then you'd be paying too much. You'd be paying more than the goods are worth.You'd end up with a negative utility.So you don't want to bid more than what it's really worth to you.On the other hand, if you bid down here, and if c snuck in in betweenwhat your bid is and what the valuation is, then you've lost the auction, and you get a zero,but you should have won--or it would have been worth your while to win--because the price still would have been a bargain to you.That says that the rational strategy, the dominant strategy in second-price auction,is to bid your true value and that makes it a truth-revealing auction mechanism.This unit we'll return to the topic of planning, and we'll talk about 4 things that we left out last time we talked about planning.First is time.That is, rather than just saying whether an action occurs before an action or after it,We'll talk about actions that persists over a length of time.Second is resources necessary to do a task.Third is active perception--That is, taking the action of perceiving something.Fourth is hierarchical plans--that is, plans that consist of steps which have substeps.We'll start with time.We'll look at the problem of scheduling a series of tasks, each of which has duration.We'll show a task network which has a start and a finish date,and then has a sequence of tasks, which have to be completedand arrows to indicate precedence of which ones have to go before other ones.This task has to occur before this one,but there's nothing said about the relationship between this task and this task.We'll list for each task their duration.This one takes 30 minutes, 30, 10, 60, 15, and 10.Scheduling then is a process of figuring out a schedule under whichwe specified the times at which each of these tasks startssuch that we can finish as soon as possible.Now schedule is defined in terms of specifying for every task in the networkthe earliest start that, which we'll call "ES," and the latest possible start time,which we call "LS" for which it's possible to complete the task network in the shortest possible total amount of time.We can define these with a set of recursive formulas which can be solved by dynamic programming.The earliest start time of the start state is defined as being zero.The earliest start time of any state B is defined as being the maximum over all As which have an arrow leading into B--that is all As that are defined to be predecessors of B--of the earliest start time of A plus the duration of A.For example, the earliest start time of this state here would be the maximum over all the ones that are coming in, which is only this one.The maximum of its start time, which will be here, plus its duration, which would be 30.Then the latest start time is defined by saying the latest start time of the finish,is the same as the earliest start time of the finish,because the finish by itself has no duration--it's just there to give us a point to end at.The latest start time in general of any node Ais the minimum over all B which come after A,of the latest start time of B minus the duration of A.These formulas together define a unique schedule, which is the fastest possible schedule. What I want you to do is fill in for me in the upper left hand the earliest start time,in the upper right the latest start time for each of these nodes.Here I've zoomed in a bit just to give you a little bit more room to fill in the blanks.You can see the earliest and latest start times filled in for all the states.Here's an alternative method for visualizing this.Here I've given names to the various actions--the three on the top and the three on the bottom.They have a duration along a time line, and we can see the time line there.Notice that these three actions have no slack between them.One has to start after the other.We say that these are on the critical path in that if any of these slip in the schedule,then the whole schedule will slip.Whereas these three actions have some slack.The could occur anywhere within this gray window,and if this action slipped to the right, then the others would slip to the right without affecting the final schedule.I should say that over the years the field of scheduling has moved in and out of the artificial intelligence.Some people have worked on it, but most of the work on schedulinghas been done in the field of operations research-a closely related field to artificial intelligence.The next question I want to address is the one of resources.Resources are things like this pile of nuts and bolts that are used somewhere in a plan.Of course, resources could be handled just in the language of classical planning.Here we have a description of a problem domain in classical planning language.The goal is to get an assembly inspected,and in order to do that, we have the action of inspecting,which looks at an assembly which has five nuts and boltswhich each have to be fastened to each other.If that precondition is satisfied, then the effect is that the assembly is inspected.We have an action of fastening a nut and bolt to the assembly,which requires a nut and a bolt, and the result is that they're fastenedand that the nut and bolt are no longer available for use.Initially we have four nuts and five bolts.Now the question is with this description of this problem can we achieve the goal?Assuming that we have a depth-first tree search planner,how many paths would that planner have to consider?Would it be 1, 4, 5, 4 +5, 4 * 5, 4! + 5!, or 4! * 5!?The answer is we can't achieve the goal.We're just missing a nut so we can't do it.But we're going to have to consider 4 factorial times 5 factorial pathsbefore we discover that.The reason is because we start out, and we say in order to achieve inspected,we need the precondition of being fastened.In order to achieve fastened, we need some nut and some bolt.We can try N1 and B1, but then we would also, when we end up back tracking, have to try N2 against B1, N3 against B1, and so on, for all these and all these,and we'd have to do that at every step in the back track.So we end up trying all combinations of nuts and all combinations of bolts.That seems silly, and so the idea of resources is to make the nuts and boltsrather than make each one be distinct so we can handle them more efficiently.Here I've shown how to extend the language of classical planning to handle resources.We've added a new type of statement saying that there are resourcesand how many of each there are.We can say there's five nuts and four bolts,and we're also going to explicitly model inspectors, and we have one of them.Then the actions have two new types of clauses.The fasten action has a consume clause, saying it consumes resourcesand once it uses them, they're gone forever.Fastening is going to consume one nut and one bolt.The inspect action has a used clause, and that says it's going to use one of the resources,the inspector, while the action is going on.But once the action is completed then the inspector has returned to the pooland is available for use elsewhere.Keeping track of resources this way gets rid of that computational or exponential explosion of looking at different combinations by just treating all of the same resource identically.The final topic in this unit is called hierarchical planning.The idea here is we want to close the abstraction gap. What do I mean by that?Well, let's think about what you have to do to plan your own lifetime.You live about maybe a couple of billion seconds,and during that time you have a choice of actions to make,and you have maybe around 1,000 muscles, which you can operate maybe around 10 per second.You end up at a lifetime as somewhere around 10^13 actions give or take an order of magnitude or two.But there's a big gap between 10^13 and the 10^4 or so actionsthat current planning algorithms or programs can deal with.Part of the problem with such a big gap is that it's just difficult to deal at the level of an individual muscle movement.We'd rather deal with more abstract plans.We're going to introduce the notion of a hierarchical task network,and rather than having a plan be a sequence of individual steps,we can talk about higher order steps where maybe there's a smaller number,and the individual steps can correspond to multiple steps.This idea is called refinement planning.Here's how refinement planning works.In addition to regular actions, we have abstract actions like going from my home to the San Francisco airport.Then we have possible ways to take to refine these abstract options into concrete actions.Here one refinement is I can drive from home to long-term parkingand then take the shuttle to the airport.Another refinement is I can just take a taxi.Here's another example of an abstract action,which is if I'm at one point on a grid, ab, and I want to get to point xy,and if I know the grid is all connected,and I have this abstract action of just navigating from ab to xy.One refinement says if I'm already there I do nothing.Another refinement says I can start the journey by going left.Another refinement says I can start the journey by going right and so on.The idea is I can figure out a complex plan that involves navigating around picking up and object, doing something else,and do that planning just at the level of abstract actions like navigaterather than having to figure out a path from ab to xy.How do we know when we have a solution?A hierarchical task network achieves the goal if for every part, every abstract action,at least one of the refinements achieves the goal.We only need to at least one of them, because we're the planner.We get to make the choice.It's like an and/or search where we can make the best possible choices,and if any of the choices work, then the goal can be achieved.Now, in addition to doing an and/or search,sometimes we can solve an abstract hierarchical task network planning problemwithout going all the way down to the concrete steps.Let's talk about how to do that.Here we have a description of a state space.The start state is here, and the goal state is outlined in gray here.We have one abstract action, and we're shown a set of possible statesthat can be reached by that abstract action,if we refine the abstract action, using one concrete action or another.This is like when we were dealing with belief stateswhere we would move, because we had a stochastic action,from one state to several possible other states.Here we have several possible states that we'll end up with,not because the actions are stochastic, but because we haven't decided yet which refinement we're going to use.This would be a single step that would bring us to this belief state,and then when we add a second step, we get to this belief state.Now we can check to see if we can achieve the goal with this two-step planjust by checking if there is an intersection between the reachable state and the goal state.In this case, there is.We know that we've achieved the goal,and now if we want to find a refinement that actually works,the way to do it is to search backwards rather than forward.If we search forward we'd have a large tree of possibilities,but if we search backwards, we know the intersections here.What could have brought us to here? Only this refinement.And what could have brought is to this state? Only this refinement.That's the plan that is a refinement of this abstract plan that achieves the goal.Now sometimes it may be very difficult to specify exactly what states can be reachable by an abstract action,because the refinements are complicated.We can go with the notion of an approximate set of reachable states.That's what I've shown schematically here.For this abstract action, I've shown a lower bound and an upper bound on the states that are reachable.What do I mean by that?Consider the abstract action of going to the airport in San Francisco.Now, some things I know are going to be true about the resulting state.I know it's going to take, say, half an hour to get there no matter what way I go.That's always going to be true.Other things depend on which choice I make.I may consume some money if I take a taxi.I may consume some gas if I take a car,but I may not be able to specify exactly which of those combinations hold true.So we approximate the set of reachable states by this lower boundof things that we know we can get toand this upper bound of things that we might be able to get to,but we're not quite sure if all combinations of them will check out depending on the refinement.Here, similarly, there's another set of lower and upper bounds and here as well.These are the goals. What I want you to tell me is for each of these three actions,is it guaranteed, yes, that I can reach the goal state if I choose the right refinement,or is it never possible--no, that I'll never be able to reach the goal state--or is it uncertain yet because the description of upper and lower bounddoesn't tell us enough about whether we can reach the goal state.Answer that for this abstract action here, and for this abstract action here,and for this abstract action here.In the case of this abstract action, we know all the possible outcomes are somewhere within hereand none of those intersect with a goal, so there's nothing we can do to make this one work.For this abstract action, we see that there is an intersection,and there's an intersection even in the under estimate of the state.We know that we can reach someplace in here,and since we have the choice of where we want to go,we know we can reach there,so we know that we can always refine this abstract action to achieve the goal.Over here there's an intersection, but it's only in the over estimate--the outside part of the search space.So we're not quite sure.We have to look more carefully at the refinements to see if there isa combination of refinements that allow us to reach this state,or if the combination of refinements leave us somewhere over here,which is not inside that state.So that would be questionable or unknown yet.Here's one more topic. We're going to talk about how to extend classicial planning to allow active perceptionto deal with partial observability.Here's a problem description.There's a table and a chair, and there are two cans of paint.The table is within our field of view, and our goal is to have the chair and the table have the same color.Here's the actions.We can remove the lid from a can, making it open.We can paint one thing with that can if the can is open.We also have an active perception action,which is we can look at something.If it's in view, we can look at it,and then we're looking at that one thing, and we're no longer looking at whatever we were looking at before.Now, here's the big extension that in addition to actions,we now have percept schemas.Action schemas and perception schemas,and we can perceive the color of something if it's an object.Here the objects are declared to be the table and the chair,and if it's within our field of view.Notice that here we're introducing a new variable.We never did that before in planning.Before all the actions in planning, all the variables,were predefined by matching against the precondition.Here we're introducing a new variable.We're saying if these preconditions are true,then you can perceive something, and you'll learn something new.You'll learn the value of this variable. Here's a question. How can we achieve this goal?The first thing I want to ask is, without even thinking about the percepts,is there a conformant plan that is a that doesn't do sensingthat will allow us to achieve this goal?Is there that type of conformant plan?Tell me yes or no.The answer is, yes, there is.That is we can remove the lid from the can,and we can do that to either can, can one or can two,and then without knowing what color that can is and without knowing what color any of the furniture is,we can first paint the table, and then paint the chair.Then we know they'll both have the same color, and we'll have achieved the goal.Now one of the problems with this plan is say the chair and the table were already the same color.We would've wasted our time painting them when we didn't have to do it.Now the next question is, yes or no, is there a better sensory plan?That is a plan that uses perception and comes up with,in at least some cases, a smaller number of painting actions.We're going to allow these plans to have conditionals in them as well as having perception.The answer is, yes, there is.There's a variety of possibilities.I'll show you a somewhat complex one.This one says we can look at the table and look at the chair.Then if the color of the table and the color of the chair is the same, c,then do nothing. We're done. We don't have to do any painting.Otherwise, we can remove the lids from can one and look at it,remove the lid from can two and look at it.If the color of the table is the same as the color of the can,then we can paint the chair with that can.This is for any possible can, either can one or can two.Otherwise, if the color of the chair and the color of the can match,then we can paint the table, and otherwise we have to paint both of them.In this question I'd like you to fit a Markov model.We have 3 sequences of observation--A, B, C, A, B, C.This is the first state--state0--and these are the other states.Same for A, A, B, B, C, C and A, A, A, C, C, C.These are 3 different observation sequences. The first element here is state0.All the other ones are examples of transitions from state to xt-1 to xt.Using maximum likelihood, I'd like to know the initial properties for state0and all the transition probabilities. Those are indicated by the arrow.This is the probability that A at time t - 1 goes to A at time t.Here they are.For any of the symbols in the sequence, we see there are three possible outcomes--A-B-C, A-B-C, and A-B-C.Each has a probability. Obviously, these 3 over here have to add up to 1.These 3 over here have to add up to 1, and these 3 over here have to add up to 1.Initially there is only A,and the maximum likelihood we get the probability of A being in the first place is 1,and all the other ones are 0.There are 7 transitions out of A as indicated by the lines under the As over here.In 3 cases, A is followed by A--over here, over here, and over here.This gives us 3/7 for the maximum likelihood estimator.A flows into B in another 3 cases--over here, over here, and over here--again, 3/7.There is 1 instance where A moves into C--1/7.There are 4 transitions out of B--3 go to C, and 1 goes to B over here,which gives us 0, 1/4, and 3/4.Then there are 4 transitions out of C--one, two, three, four--3 of which result in C, but one goes into A over here.These are the results--1/4, 0, and 3/4.Here we're given a marchov chain between A and B,with the transition of A to itself is 0.9 with 0.1 chances transitions to B.B stays in B with 0.5 chance and transitions back into A.with 0.5 chance.I'd like to know the stationary distribution over here.So what's the probabability of A in the stationary distribution?Of course correspondingly, what's the probability of B in the stationary distribution?The answer is 5/6 for A and 1/6 for B. To see, let's call this property over here, X, and we can now solve for the equationthat in the stationary case, the property of A is 0.9 x the property of A before + 0.5, x the property of B which is 1- X.If you solve this, this gives us 0.4X + 0.5 or put differently,0.6X = 0.5. That is the same as saying X = 5/6, which is what I wrote over here.I am now asking a hidden markov model question.In the given, the following hidden markov models with 2 internal states,with a property of transitioning to either side is 0.5, and the probability of staying is therefore, 0.5.This Hidden Markov Model has 2 possible measurements or observations, X and Y.The probability of observing X and Y depends on what state the hidden markov model is in.For A, it's 0.1 for X and 0.9 for Y. For B, it's 0.8 for X and 0.2 for Y.Let's assume that the initial probability of distribution x 0 is 1/2 for either of the 2 states.I would like to know what's the [ ] probability of being A x 0 given that we observedX x 0 and then Y, what's the [ ] probability of state A x 1 given the observation of X x 0,No subtitles...This is a particle filter question.Suppose we had a word with 4 states.In these 2 states over here, we tend to observe A property 80%.The remaining 20%, we'll observe B.In these 2 states, we tend to observe B with 80% probabilityand with 20% probability, we observe A.Suppose we have 3 particles--1 over here, 1 over here, and 1 over here,and we observe A.Let's call this particle over here particle A. Lower caps a.Lower caps b.Lower caps c.What is the probability that the sample a, given that we just observed A,which means it will be more likely to be in 1 of these 2 states over here.What's the probability of sampling b? What's the probability of sampling c?The particle a will get an importance weight of 0.8, nonnormalized.You have to normalize in the 2nd.Particle b will get an importance weight of 0.2. Same for particle c.If you add those together, we get 1.2.Then we have to divide everything by 1.2--which is 2/3 for the probability of sampling a,and 1/6 each for b or c.Here's another particle filter question.Now we're looking at the state position,beginning with the same state position as before and the same 3 particles--1 over here, 1 over here, 1 over here,and to give the states names, you're going to call this one a1, a2, b1, and b2.Let's assume we take a single random particle with uniform distribution,and we emulate a next state.The state position works as follows: A particle will move with property 1 to an adjacent statebut adjacent is either north, south, east, or west, but not diagonal,because every particle has 2 adjacent states, it'll break ties at randomso you're going to pick 1 of the 2 with 50% probability.So with this 1 particle that you've drawn random, and it's in the random single position,what's the probability that finds itself in a1, a2, b1, and b2?If you look at the chances of the particles that can end up in a1, we find that only this particle can go up here, so let's call this a 1.To a2, there's 2 particles that can go to this. Call this a 2.There's 2 particles that can end up in b1. This one and this guy over here. Again, a 2,and one that can make it into b2.Now this is a total of 6. If you normalize, you get extra probabilities.a1 is worth 1/6.a2, 1/3, which is 2/6ths.b1, again 1/3,and b2 is 1/6.So here's a multiple choice question for particle filters.Say we implement a particle filter such as a mobile localization,and we use exactly 1 particle.Which one of the following statements are true?Check any or all of the following statements.Measurements will be ignored? Check this box if you believe that's the case.The result is generally poor.It cannot represent mulit-model distributions.The initial state, if known, is ignored.The state transitions are ignored.If none of the above applies, check the final box down here.And the answer is measurements are indeed ignored which has to do with the following:We do weigh particles by the measurement probability but we normalize them to 1,and if there's only 1 particle, it will always normalize itself back to 1,so the measurement probability has no effect.The results are generally poor. That is, a single particle is just insufficient to anything interesting, so that is absolutely the correct answer over here.Clearly, a single particle cannot represent multi-modal distributions because multi-modes look something like this over here because it's just 1 particle, so this is actually correct.The initial state, if known, is not necessarily ignored.You might actually place the particle at the initial stateand it might consider it in the filtering result.The state transitions are also not ignored because we will still propagate this particle forward according to the state transistion, and because 3 of them are true, the final one isn't.Another particle filter question.Check the following statements if they're true.They are usually easy to implement.They scale quadratically with the dimensionality of the state space.They can only be applied to discrete state spaces.Finally, if none of those applies, check the final check mark over here.And only the first answer is correct.They are usually easy to implement compared to any other filter.They do not scale quadratically, in fact, normally they scale exponetially with the dimensionality of the state space because you need exponentially many particles to fill up the state space.The filter that scales quadratically is called a common filter,but we didn't really talk about it in this class.They can only be applied to discrete state spaces is clearly wrong.We saw how to apply the robot localization which is a continuous of revalued state space,and because this first one is true, the last one, none of the above, is clearly false.For this exercise, I want you to solve this game.So it's a 2 x 2 zero-sum game.We're showing the utilities to the player Max, and so tell me what Max's strategy isby putting in probabilities in these 2 boxes for his 2 plays--1 and 2.Tell me what Min's strategy is by putting in probabilities in these boxes,and then tell me the value of the game, the expected utility to Max.The answer is both players have a rational mixed strategy.For Max, he plays 1--8/17ths of the time and 2--9/17ths of the time.Min would play 1--10/17ths and 2--7/17ths, and then the utility of the game to Max turns out to be 5/17ths.In this scheduling problem, we have a network of actions with the precedence relations between them and the duration of each actionshown in each box.What I want you to do is, for each action fill in the earliest start time in the upper leftand the latest start time in the upper right.Here we see the start times for each of the actions.Note that the critical path which the earliest and latest start time are the same goes straight down the center.Here's a game tree for a stochastic 2-player game.There are max nodes, min nodes, and chance nodes.What I want you to do is back up all the values, so fill in a value for the value of each of these nodes, and then check off all the nodes that could be pruned away by a procedure that's similar to alpha beta, but updated to handle chance nodes.So what I mean by that is, a node can be pruned away if evaluating the nodesis not necessary to figure out what the best moves are for max and min.For the chance nodes, all the possibilities are equally probably.So here, there's a 1/3 chance of each of these.Here there's a 1/2 chance of each of these.And in this game, the result of every game is either +1, -1, or 0,and all the players know that those are the only possible outcomes for the game.Therefore, the players can take that into account when trying to figure outwhich nodes to prune away.For backed up values for the min nodes, the backed up value is always the minimum.For the chance nodes, it's the expectation, or the average, and for the max node, it's the maximum.Here the nodes that can be pruned. This and this can be pruned because in each of these cases min has achievedthe best possible play that min can get, and therefore,doesn't need to consider any other possibilites.Once you know you can win the game or do the best you can, you don't need to find another way to do just as well.This node here can be pruned and thus, all the ones below it because at this point, when we're trying to evaluate this node, max knows he can get at least 1/3, and here once we know that this node is worth -1then we know that regardless of the value here,that has to be somewhere between -1 and +1.So therefore, the expectation has to be between -1 and 0, and if 0 is the best that this can be, max knows he already has 1/3 over here.Here's a 2-player game. Each player has 3 possible moves.What I want you to tell me is, first, does A have a dominant strategy? Yes or no.Second, does B have a dominant strategy? Yes or no.Third, click on all the boxes that are equilibrium points.A does not have a dominant strategy, but B does.If B plays the middle e, then in this case, B will get 5, which is more than 3 or 2.In this case, B will get 7, which is more than 2 or 4,and in this case, B will get 8, which is more than 7 or 5.So that makes this play the dominant strategy for B.Now if B is going to do that, then what should A do?Well, A should try to get the best possible value that A can,and that would be here, and that makes this square the lone equilibrium point.Hi again. It's great to see you again.We talked a lot about basic methods of AI,and from today on we'd like to go into applications.Specifically, today we'll talk about computer vision.Computer vision is a very bright field that concerns itself with making sense out of camera images or video.Many devices today are equipped with cameras, such as cell phones or cars,and making sense out of image data has become a really important subfieldof artificial intelligence.Today I'll teach you some of the very basics.It's not as deep as my graduate level class on computer vision,and I hope you get a chance to take that in the future,but I hope to enable you to apply some of the very basic methodsto, for example, use images and classify them using artificial intelligence technologythrough feature extraction and other techniquesand also to start doing some of the more 3D-oriented taskssuch as 3D constructions.So let's start with the very, very basics and ask ourselves what is a camera.Cameras come in all sizes and shapes.This is my beautiful Nikon D3 camera [shutter clicks],but I don't use it much because it's very heavy, even though it takes beautiful pictures.This is the camera I use the most. It's a cell phone camera.It's an 8 megapixel camera over here with a flash,and I can start it, and you get to see whatever is underneath,like this pen over here.I can also activate the front camera, and you get to see the way I've been recordingall those wonderful online lectures over all those weekswith this little camera over here.In all of those cameras there is a lens and there's a chip,and the light is captured from the environment and focused through the lens on the chip,which raises the question, how does a lens and a chip really work?[Thrun] The science of how images are created using cameras is called image formation,where formation just means the way an image is being captured.Perhaps the easiest model of a camera is called a pinhole camera.In a pinhole camera, the light from within the worldgoes through a various small hole--ideally it's a really, really small hole--to project into a camera chip that sits somewhere in the background.So for example, if you had an object that was a person over here,then this person would be projected as follows.The feet would be projected to over here and the head to over here,which gives us this inverted person on the projection plane or the camera chip.There is some very basic math that governs the geometry of a pinhole camera.If we call X the physical height of the object and small x the height of the projection,which I'll call -x because it points in the opposite direction as the original object,then we can also talk about other values such as the distance of the object to the camera planeand f, which is the focal distance of the camera,which is the distance between the pinhole and the projection plane over here.There's a simple piece of math that relates all of those 4 variables over here,and it's easily obtained by what's called equal triangles.In particular, it turns out if I map this triangle over here to right over here--so these are the same triangles, just flipped, where x is over here and f is over here--we get that the ratio of upper caps X to Z is the same as lower caps x to f.So I write this as follows.This is a result of equal triangles.So as you take a triangle of a certain shape,when you scale it up to larger triangles, those proportions are retained,so therefore, upper caps X divided by Z is the same as lower caps x divided by f.If we now transform this, I find that the projection of lower caps x,which I might care about, is upper caps X, the physical size of the object itself,times the quotient of the focal length over the distance.That's an interesting equation.The further an object is away, the smaller it appears.The larger the focal length of the camera, the larger the object in its projection.And of course the size of the object itself directly influences how big its image of the object really is.So let's see if you can practice that equation using a quiz.[Thrun] So here is our equation again.Let's say James is 2 meters tall.He is 10 meters away from the camera, and the focal length is 10mm.How large will be James's projection using a pinhole cameraon the camera chip with a focal length of 10mm?Please specify your answers in millimeters as a unit.[Thrun] And the answer is 2mm.James, even though he is 2 meters tall, will look like 2mm tall in the camera.The picture will be there's a 2 meter tall person over here who is 10 meters away,there's a pinhole, and the focal plane is only 10mm away from the pinhole.So this projection will be really, really small.Let's do this in math.The upper caps X is 2 meters, the 10 meters over here is the distance, Z,and the focal length, 10mm, is the thing over here,so 2 meters for X divided by 10 meters for Z times 10mmbecomes 0.2 times 10mm, which is 2mm.[Thrun] I have another quiz.We are looking at a building that is 10 meters tall,and our camera is 100 meters away.We also know that the projection of the building on the internal chip is 4mm in size.I want to know what is the focal length in millimeters.[Thrun] And the answer is 40mm.With f being the unknown, we can transform this equation as follows,and now we can plug in the things we know.The 10 meters tall is the upper caps X,the distance of 100 meters is the Z,and the 4mm projection goes over here.And if you work this all out, it's 40mm.[Thrun] So in this final quiz we're going to see we can use a camera as a range sensoror as a distance measuring device,provided that we know the size of the object we are looking at.Suppose you're looking at a car, and we happen to know that this car is 160cm tall.Imagine we take a picture of this car using a pinhole camerawith a focal length of 40mm,and in our projection the car is 2mm tall.My question is what is the range? How far is this car away?Please answer in centimeters.[Thrun] And the answer is 3200cm or 32 meters.And to see, we transform this equation over here so that the range, Z, is on the left side.With this new equation we can just plug in the known quantities.F is 40mm, x is 2mm, and upper caps X is 160cm.We work this out as 160cm by 20, which gives us 3200cm.[Thrun] So we just learned something really important, which is the central law of perspective projection,which basically says that in a pinhole camera, or in fact any camera,the projective size of any object scales with distance.So you have an object that's yea tall over herethat looks just about the same as an object yea tall over here.In math x is proportional to the size of the objectbut inverse proportional to the distance to the object,and the only constant that then governs that relationship is the focal length, f.So if we take an object and move it further away,it'll appear smaller, and we all know this.Just look at this object over here, how large it is.And as I move it away from the camera, it becomes smaller.Large...and small.And that's a function of distance.The law that governs the size change of this object in appearancerelative to the camera image is the perspective law we just saw.Actually Cal images have 2 dimensions--not just 1. Here's an X and a Y,and the perspective laws apply to both dimensions.The projection of the X-coordinate into the camera planeis governed by the perspective law over here. And the same is true for Y. In both cases, the appearance of an object, of size X and Y,is scaled inversely with the distance of the object to the camera plane. One of the interesting consequences of perspective projection isthat parallel lines in the world seem to result in vanishing points over hereso that these lines are parallel in the physical world.Because things shrink in inverse proportion to the distance, you'd find it far away. The perceived distance is much smaller, resulting all the way in a vanishing point that sits somewhere over here. So sometimes there's more than 1 vanishing point. In this specific instance, there's a vanishing point over here, and a vanishing point over here--and those correspond to parallel lines,like the curb over here or the house facadesthat all shrink, with distance, in their visual appearance. So here's my quiz on vanishing points. The question is: How many vanishing points may exist in a single image? Exactly 1 answer is correct here. We already encountered it in an example with 2. So perhaps there's 3, 4, 6--or even infinitely many.So what's the maximum number of vanishing points you may possibly encounter in an image? And the answer is infinitely many. If you thought it's 3, then yes--cubes might have 3 vanishing points, like this one over here, which is a cube under perspective projection. It actually has 3 vanishing points:No. 1, No. 2, and No. 3 over here. But that's because the cube has 3 faces. You've had different faces whose Z-distance to the camera varied. And those enclosing lines that might be parallel in the physical spacewould result in their own vanishing point. So you can theoretically make an object with infinitely many vanishing points. Let me comment on the idea of a lens. A fundamental limitation of a pinhole camera is that only very few rays of light hit the plane of the imager.So suppose we have an object over here and the object emits light in all directions. Then most beams get absorbed by the area outside the pinhole and a very small number of beam makes it through the pinhole. Now this is misfortunate because the total amount of light that hits the camera chipis small and its support is only applicable for very, very bright scenes. And further, as you make this gap smaller and smaller to increase your focus on the image plane, you will eventually run into what's called "Light Defraction,"which puts a limit on how small you can make this pinhole over here. Now if you use a lens, then all rays will make it to the same point in the image plane. So an example--a ray over here gets projected like this, and a ray over here might make it like this. So any ray in a good lens will eventually meet at the same point over here. The lens collects all the light that hits it, and projects it back to 1 point. Now this specific situation is characterized by only a small plane over here, for which everything is in complete focus. If you move your object back to over here, then what you find is the resulting projections don't match up. Therefore, when you have a camera with a large lens or a large aperture, you have to focus the camera to make sure that the distance between the image plane,the lens itself, and the opposite object are in tune. There is an equation that governs all of this,and it looks about as follows:1 over the focal length, f, for the lens. This would be the sum over the extrinsic distance.Plus 1 over the intrinsic distance, lower cap z. I won't revise this equation, but this is the fundamental equation that governs when things are in focus, for a lens. So this is great--we now learned a lot about cameras and images. We learned the Law of Perspective Projectionand we also know when things are in focus. Now the first law is really important. and the second law doesn't really matter that much--but I put it in so you understand what the implications of using a lens really is. Let's talk of when we're in Computer Vision and what type things we can do with images. One of the primary purposes is to extract information from these images, such as classify objects. So here is one of my son's favorite objects. And he might be interested in understanding that this is a car. A second purpose is 3D reconstruction. So you might care to take many images of this object, from different perspectivesor with multiple cameras, like a Stereo Camera Rig and ask yourself what is the 3D model that we can reconstruct from these 2D projections. Or you might care about motion analysis. This is a common problem in Computer Vision where things might move and are seen in the video of many imagesand you might, for example, care how do things move, over time. So I'd like to quiz you a number of times on something that's really essential to the problem of Object Recognition. In Object Recognition, you're given an image of an objectand you care to understand what the nature of the object really is--like, for example, you might look at the image of a plane and say it's a plane. You might look at the image of a person and say it's a person. A key concept in Object Recognition is called invariance, which means there is natural variations of the imagethat don't affect the nature of the object itselfand you wish to be invariant in your softwareto those natural variations. So what I will do is I'm going to run a couple of invariances by you and I'd like you to help me which invariance I'm referring to. And here are the possible invariances:Scale, Illumination, Rotation, Deformation, Occlusion, and View Point. I understand--none of these words you've seen before, so I am really appealing to your intuition hereand your sense of the English language. So here is the object,and we wish to recognize this object. And, as I'm illustrating, the object might vary in some important dimension. And I wonder what type of invariance you or I, with it, must possessto be able to recognize this object. So here is the list. And the answer here is Rotation. I rotated the object.So you wish to make sure that any recognition item is invariant to a rotation. Sometimes the object gets closer to the cameraand increases in size;and gets further away from the camera and, therefore, becomes smaller. Just watch how the object becomes larger, and smaller. What do you think? What type of invariance is this? And the answer is this is Scale Invariance. Scale means how large the image is, relative to the camera.This is governed by the Perspective Projection Law that we discussed before.The further objects are away, the smaller they appear.We wish any classifier to be invariant to scale so it can recognize objects nearby or really far away. This object over here is interesting because we can change its shape. So you can take this thing over hereand move it around, and what you actually see of the object depends on the angle of the rotors. So what do you think? What type of invariance is this? I would call this Deformation Invariancebecause the object is actually deformable as our many objects that surround us, like clothes and dollar bills and water glasses. This kind of deformation is really important in the recognition of objects. You wish to make sure that a helicopter can be recognized,no matter what angle it's rotor blades currently have. So here is one of my favorites. I'm holding in my hand a flashlight, and as I move the flashlight around, you can see that the appearance of the object changes a lot,based on where I hold my flashlight. So my question now is what type of invariance does this realize? And this is clearly an example of Illumination Invariance. Depending how the object's illuminated, it might appear very differently, even though its position to the camera might be identical. Sometimes objects are behind other objects. For example, I can partially cover up this object. You can probably still recognize it. Or I might move a pen in front of it, as shown over here. Now we've almost no choices left. Tell me what type of invariance this was. This is called Occlusion Invariance. Sometimes objects are partially occluded,yet you would wish to be able to recognize them even with a partial occlusion. And because there's only 1 invariance left, let me talk about View Point Invariance as the final invariance. So the appearance of this object depends on from what direction you look, what your view point is. And you can see it's very different, from different view points. So this looks fundamentally different from this, from this. That's called View Point or Vantage Point Invariance,and it's one of the hardest invariancesbecause the appearance of the object might really alter a lot, from different vantage points. [Thrun] The reason why I went through these different invariancesis because they are really crucial to computer vision.These and a number of other invariances really matter.When you want to recognize objects, you want to write softwarethat is invariant to scale and illumination and so onand that it retains the important information in the imageregardless of the present rotation and occlusion and deformation.If you succeed in eliminating the effects of those changesand build a truly invariant computer vision algorithm,I will be very impressed with you.You will have solved a major computer vision problem.[Thrun] So we learned a lot about invariances.Let's now take actual images and do something with these images.This is an image I took a while back in Amsterdam,and it's interesting because there's a lot of interesting featureslike these line features over here and possible corner features over there.In computer vision we don't use color images very much. We mostly use black and white images like this greyscale image over here,which misses information from the original image,but it turns out that greyscale is more robust to lighting variations than color is.That's a fairly common representation for images for computer vision.So a greyscale image is a matrix typically of several hundred rows and several hundred columnsthat has small values imprinted that correspond to the greyscale of a pixel.These values scale between 0 and 255, where 255 is white and 0 is black.You can see how this matrix is full of values that together compose the image.Here is a very small image of size 4 by 5,and based on the numbers I put in, it feels like there's a transition going on.At the top the image is relatively bright.These have values close to 255.And at the bottom it is relatively large.This is way too small an image to recognize anything.Picture a matrix much, much larger than this,yet an image that's still just a 2-dimensional matrix of singular brightness values.At least a greyscale image is a matrix like this.A color image would have 3 different values per pixel which correspond to red, blue, or green or some other encoding of the color itself.But for now we're going to be content with greyscale images.[Thrun] One of the most basic things we can do with computer visionis to extract features.For example, there is a very strong edge feature over hereand a strong corner feature right over here and right over here.Let me tell you how to do this.How can you find in an image like this whether there is an edge,or in an image like this where there is an edge from a bright area on the leftto a dark region on the right?Let us write a feature extractor that identifies transitions of this type,and let's start with horizontal transitions.The most obvious feature detector looks like this.We run this little 2-value matrix across the entire image over here,and we add whatever is on the left side and subtract whatever is on the right side.So if both sides are approximately in balance, like these points over here,adding an expression here is approximately 0.But if the left side is significantly larger than the right side,then adding and subtracting yields a very large value, like 212 - 7 over here.So this specific mask gives us edges that run from bright to dark.So here I'm taking the first value and subtract the second value from it.255 - 212 gives me 43. That's applying this mask over here.From 211 to 237 is -26 and so on.212 - 7 is 205.237 - 3 is 234 and so on.7 - 1 is 6.3 - 9 is -6 and so on.If you look at this result of applying the mask over here,you'll find that this column stands out.It is much, much larger in value than any of the adjacent columns,and that indicates that we have a high likelihood of a horizontal edge feature occurringat the ridge between this column and this column over here.So here we are applying that same trick to the original image, and this is the result.You can see that areas where the original image has a strong transitionyou get a strong response over here.This is actually showing the absolute value of the differencewhere we get rid of the minus sign, so you can see any transition from bright to darkor dark to bright horizontally shows up.Now, you can see these lines over here that are vertical show up very strongly.The lines over here don't, and the reason is the way we defined our kernel,it ran actually horizontal, so it finds horizontal edges and not vertical edges.Vertical edges require a different kernel, so let me get to this in a second.[Thrun] So in this quiz I've given you a very small image of 3 by 3,and I'd like you to apply a kernel that's about like the previous one,except I flipped the left and right side over here.And just apply this kernel to this image over here.We're going to receive a 3 by 2 image in this case.So please fill in all these 6 values over here.[Thrun] And here are the results.7 - 255 is -248.3 - 7 is -4.4 - 240 is -236.240 - 212 is 28.230 - 216 is 14.And 216 - 218 is -2.So this would be the image under that specific mask or kernel over here.Now, we already learned something really interesting,which is the special case of a linear filter.We took an image, and we applied a small kernel.The application of a kernel is often denoted with a special symbol over here.And we received a new imagethat was slightly smaller, and we don't really worryabout the fact that it's smaller.There's ways to keep it the same sizeby assuming everything around the original image is zero.But we did receive a new image that was part of the kernel over here.And the general math of the new image,for any pixel accorded x and y, is obtained by summingover all layers in the kernel, u and v,of the original image shifted by u and vtimes the kernel itself.Now, this will take some time to digest,but what it really does is it does exactly what we did before.We take our kernel, which in this case might be a 2 x 1 kernel.We go over both of these fieldsor any number of fields that exists over here.We look at the corresponding image field and shift it a little bit.We did this before. We shifted it by 0, 1 pixels.We multiply these 2 things.There was a +1 here and a -1 here before.And we add all these things up to arrive at the resulting image.Think for a moment to realize that this function over hereimplements what we just did.It's a nice and elegant function.It's called a linear filter.And the reason is the math inside this sum is linear.It's a multiplication.And so is the sum, and the convolution operation itselfis often called the linear operation.So, let me ask you another quiz.What type of filter do we needto find horizontal edges?And here are the choices.We have a filter like this, 1 and 1,horizontal filter 1, -1, and a vertical filter, 1, 1 and 1, -1.One of those is actually correct,so pick the one that is best suited to find horizontal edges.And the answer is this one over here.It takes a pixel and subtractsthe vertically next pixel from it.And if there's a horizontal edge,if we have an image where the values over here are large,the values over here are small,then the specific filter over here,when applied to the transition betweenthe large and small values will give you a large response.Here's another quiz.Given an image like this one over herewith pixel areas 12, 18 and 6, 2, 1, 7, 100, 140, 130,convolve this image with the vertical -1, +1 filterto arrive at the 6 missing values on the right side.And now the answers will be rather straightforward.100 - 2 is 98, 2 - 12 is -10,140 - 1 is 139, and so on and so on.This is the convolved image,and you can see a relatively large responsecorresponding to the position of larger areas over hereto smaller areas over here, so this filter is well suitedto find horizontal edges.And you can really see this in the results.So, this is the vertical mask applied to finding horizontal edges.These edges [s/l flare or throw] out really strongly.This one is nearly invisible.Compare this to the filter of vertical edgeswhere these things now light up,but these have gone missing, and here's the original image againwhere you'll see all the different edges.Again, the horizontal filter finds the vertical edges,and the vertical filter finds horizontal edges.Now, what I've just shown you is called a gradient image.The gradient image in the horizontal directionis the image convolved with this kernel over here.And the gradient image in the vertical directionis the original image convolved with this kernel over here.This notation should now make sensesince we practiced it a number of times.This is called, again, the convolution of the image.Now, if we wish to find edges in any direction,a really easy way to do this is to combine both of these gradient imagesinto a single edge image, and here's how it goes. We take our gradient image in direction x, and we square it.The same with y,and we take the square root.And this response over here tells usin any of the 2 directions how strong the gradient response is.Here is just that gradient image.Compare this to the original image,and you can see that wherever there is a strong transitionbetween a bright and dark color, the gradient [mack] into the image I just calculated has an edge.It has an edge vertically and an edge horizontally.And again, it's made of these 2 components, vertical edges and horizontal edges.By combining both of them, we get a gradient [s/l magnet?] image, and we have our very first feature detector,which is a feature detector of any edge in the image.Now, state-of-the-art edge detection is a little bit more advancedthan this one over here.This is called a Canny edge detector.You see much more crisp edges over here.What this does, in addition to the gradient magnitude,it traces areas and finds local maxima.And it tries to connect them in a way that there's always just the single edge.When multiple edges meet, the Canny edge detector has a hole,like the area over here or the area over here.But when edges are single edges, the Canny edge detector traces them very, very nicely.This is named after John Canny, a professor at UC Berkeley. And he did one of the most impressive pieces of workon early edge detection.There are a few other common masks in the [I'm going to share?].One is the Sobel mask, which is just like the edge detector I showed you,a little bit larger, and you can see it goes from left to right.There's about 8 of them. 2 are shown over here, including some diagonal ones.Something called the Prewitt masks, which is like the Sobelbut doesn't emphasize the center line.And the Kirsh mask, like the one over here.In fact, you can claim your own kernel.If you come up with a kernel that finds certain features,name it after yourself, and who knows?Maybe you'll get remembered like Mr. Sobel did or Mr. Prewitt.So, here's a mask that's a special case of a Prewitt mask,and I'd like to ask you a quiz.Will this find horizontal edges, vertical edges, corners, or none or all of the above?Please check exactly 1 of those 3 buttons.And the answer is horizontal edgesbecause it shifts negative mass on the left side, positive mass on the right side.That gives us a horizontal edge.It doesn't find any of the other ones.Now, linear filters can also be appliedto very different matrices.This is what's called a Gaussian kernel.You can see it over here.It's a matrix whose value is maximumat the center of the matrix and whose value fallsexponentially to the side of this matrix.It's a Gaussian in 2D, as you can see over here.So, what happens if we convolve an image with a Gaussian kernel?Let me ask your intuition on the following quiz,and it's completely okay if you get this wrong.If you convolve an image with a Gaussian kernel, what do we get?An edge detector, a corner detector,a blurred image, or none of the above?And the answer is a blurred image.A Gaussian kernel gives us a blurred image.Let me demonstrate this to you.Here is the original image,and this is the result of convolving with a Gaussian.You can see that the features are much blurred,and the reason is each of these pixelsis the sum of its weighted neighboring pixels.And the larger the neighborhood, the more the blurring effect.You can see clearly the difference in sharpness between these 2 images.So, why on earth would we ever want to blur an image?There are generally 2 reasons why you might want to do this.One is for down-sampling.If you have an image of super high resolution,maybe 5,000 x 5,000 pixels,and you'd like to go to a web image of much smaller resolution,it's better to blur by Gaussian before down-samplingthen picking each nth pixel.And the reason is called aliasing. If you pick each nth pixel without blurring,you sometimes get very, very funny effectsbecause each nth pixel might by chancecorrespond to something that's somewhat irregular.For example, if you have a checkerboard and you pick each nth pixel,you might only end up with black pixels.The second reason is called noise reduction.In noise reduction, you respond to pixel noisethat might otherwise make it hard to compute things like image gradients.If you blur the image first, you get a smoother result that isn't quite as pronouncedbut has much less noise in the image.Here's the original gradient magnitude image to find edges,and here's the same applied to the blurred image.And you can see the original one is much more succinct, but also it's more subject to noise.Take the area over here, which has lots of image noise,and compare this to the area over here, which has [s/l many few edges.]The same is true over here and over here.I wouldn't really claim this is a much better result.In fact, it looks kind of funny and very coarse,but it does have less noise.Just to complete the issue on blurring,what we just did is we took an image, we blurred it with a Gaussian kernel, and then we applied a gradient kernel.If you dive into the math of convolution,you'll find that convolution is associative,so you could apply this one to the image and then this one over here,or you can combine these 2 guys over here into a Gaussian gradient kerneland apply this Gaussian gradient kernel to the image.So, f convolved with g is this big maybe 9 x 9 Gaussian matrix convolved by a single+1, -1 kernel g.And here's what this Gaussian gradient kernel looks like.It's really interesting.It is the same gradient kernel we had beforebut smooth now and spread outby Gaussian.And it really responds to an area over here similar to a Sobel operatorthat might have a strong negative value.And the area over here on the right sidehas a strong positive value,so you can think of Sobel and many other kernelsas a combination of smoothing and taking a gradient.I find this really interesting becausewe can now devise a single, linear kernel that does both smoothingand find gradients at the same time.Sometimes you wish to find corners,as in this checkerboard over here.Corners have an advantage over edges.Edges aren't localizable.They could be anywhere on an edge.But a corner like this or a corner like thiscan be localized, which is useful in computer vision.What you see here is a Harris corner detectorapplied to a checkerboard pattern.And you can see all the points that define the checkerboardclearly found by a relatively simple algorithmwhich I'm just about to explain to you.The Harris corner detector is really a simple algorithm.Suppose you wished to find a corner just like this.Then in the small region over here where the corner resides,you will find a lot of horizontal gradientsand a lot of vertical gradients.Now, what's our trick of finding gradients?Well, we know about horizontal gradients.We know about vertical gradients.If those summed up over a small window--as shown right over here--are large, we have a corner. If only 1 of them is large and the other 1 is small, we likely have an edge.We already learned this before.It should be no surprise so far.Now, the Harris corner detector generalizes 2 images. We might have a corner like thisthat is rotated from the original corner.An image like this on a horizontal gradient isn't quite as pronounced as it is on the vertical gradient.But if you were to rotate our coordinate systemback into the correct orientation, we could reduce it back to the case over here.The trick that's being applied is to de-rotatethis image over here using eigenvalue decomposition.We use a matrix that slightly generalizes these 2 things over herewhere again we add our small windows.We plug in the statistic over here up here.The statistic over here down there.And here we have [s/l mixed strums] if we sum over the productof Ix and Iy in [ s/l after angle terms].If we apply eigenvalue decomposition to this matrix over here,we get 2 eigenvalues.And if both eigenvalues are large,we again say we have a corner.So, applying this eigenvalue decompositionto every positive pixel in the original imageand then taking the local maxima of that resultwhere both eigenvalues are large gives us exactlythe Harris corner detector in a very robust wayto find corners in an image.This is exactly what's being done over here,and you can see it's very robust even to small rotations of the image,and of course, to a scale of the image.It's a beautiful way to find stable, localizable features in contrast-rich images.Now, modern feature detectors extend Harris cornersinto much more advanced features.They are usually localizable, like corners are.They also have unique signaturesthat summarize the identity of a featurethat's typically invariant to lighting, orientation,translation and size variance, as you might find it in the image space.So, common methods that people use are called HOG,for histogram of oriented gradients, or SIFT, for scale invariant feature transform.All of these methods take cornersand reduce the various variants like rotational variantsby extracting statistics that are invariant to things likerotation and scale and certain perspective transformation.I took the liberty to apply SIFT featuresto the bridge image,and what you find here is a myriad of featuresthat are all very localizable.There's features over here, very large ones like the square over here,which is, I guess, very visible, another square over here,and very small, tiny features like the square over here and the square over herethat all have a unique signature and can easily be matched across images.This is called a SIFT feature extractor,and it's one of the state-of-the-art methods that are very commonly used.So, if you wish to extract features from an image, I recommend checking out HOG or SIFT.You can download software from the web.They are somewhat involved, and you can learn about themin advanced computer vision classes.So, now you know some of the very basics of computer vision.We talked about images, how images are being formed.We talked about perspective projection as a mathematical toolfor understanding how cameras perceive images.And we talked a whole bunch about features.We talked about invariances, the type of things that affectthe appearance of a feature in the camera image,and we went through methods for extracting edges,for extracting corners, and for extracting fairly sophisticated features like SIFT features.This is one of the basic processing methods in computer vision.Almost everyone who does computer vision preprocesses imagesby feature extraction, and now you knowquite a bit about how to process images in computer vision.This class is all about 3D vision.There's a scene somewhere here, and there's a camera over here,and the scene is projected into the camera plane.Obviously, the camera image is only 2D. The scene is 3D.3D vision vision attempts to recover the full 3D information.The most important missing thing is called the range, sometimes called depth or distance to the camera plan.Cameras are deficient in that they can only recover 2D information-a perspective projection of the image.The question right now is going to be can we possibly recover the full 3D informationabout the scene outside the camera just as single or modular camera images.This is our first quiz. Given a single image and given parameters of the camera like the focal lengthand all the other parameters, can we recover the depth or range of a scene?And I'll give you a couple of possible answers.Yes, always; sometimes; or never.We haven't even talked about this. This requires some thinking on your end.But give it your best try.The correct answer is sometimes. There are actually cases where we can do this.It's not always possible because, as we learned before, the camera doesn't really record the distance.If we look into an arbitrary scene, we normally can't recover the depth from a single image.But in certain cases it's possible. Here's a dollar bill.A dollar bill is a fixed size, and you can know the size. All dollar bills have the same size.And by understanding the internal projection size of this bill,you can actually really recover how far it is away, if you know things like focal length and so on.The answer is yes in cases we know the size of the objectand no in cases where you don't know the size.Therefore, sometimes was the correct answer here.One easy way to recover the depth with 3D vision is called Stereo.Humans use stereo all the time.We have two eyes--eye 1 and eye 2--and these eyes have a so-called displacement,which just means that one eye is further left than the other eye.We're looking at the scene from slightly different angles.Humans can actually recover the depth of the scene in many situations where objects are nearby.Let's look at this in more detail.In stereo vision, we're given two cameras--usually both with identical focal length.Here are the pinholes, and here are the image planes.An object in the scene is being seen by both cameras.If I draw the optical axes over there, which are the axes orthogonal to the image planes that go through the pinholes,you will see that the projection of this point depends on the displacement, or the baseline, of the so-called stereo rig.Clearly, these two images see the point at a different angle,and it reflects itself by different coordinateswhere this point is being projected onto the image plane.The idea of stereo is to screen objects and use the displacement, often called "paralax," of those two different projections to estimatethe depth or the range of the object.Let me just ask a simple quiz about stereo.Given two identical cameras for which we know things like focal length and all the other intrinsic parameters,and we also know the baseline,can we now recover the depth of a scene?Here are the answers: yes, always, no matter what the scene is.The second is sometimes, and the third one is never.As before, the answer is sometimes, although more often than before from a single camera image given we have two now.To give an intimation of why it's not always, let's look at two images where the object of interest is a vertical objectand another pair of two images where the object of interest is a horizontal feature,like this one over here.Now, in the vertical case, there would be displacement.This would be slightly further to the left than this guy over here,and we can use the displacement to recover depth in a way I'll tell you in a second.But for the horizontal, it's really hard.If this feature crosses all of the camera image, there is something called "aperture effect."What this really means is we can't really tell which of the little dots on this line correspond to which little dots on this line over here.In cases where the image lacks structure--or the worse one would be two images of fog. In fog, there is certainly a depth. Each water particle has a certain range, but we can't really recover how far away fog is, because, honestly, both images look alike.There are certain degenerate cases where stereo doesn't work.We are going to focus on this case over here right now where we do get information from the stereo sensor.Let's get back stereo rig.We have two pinholes with a known focal length f, and we wish to recover the depth z of a point p.We happen to know that the projection of p on the two image planes is somewhat different.Over here we call it x1 for the first imager.Over here we call it x2 for the second imager.The question is what is the formula that allows us to look at this rig over herewith two images with a known baseline b to recover the depth zfrom the relative displacements x1 and x2.There happens to be a relatively simple answer.If you look at this big triangle over here, that triangle has the same proportionsand the triangle put together by this little thing over here and this thing over here.You move these two triangles over here together into a single triangle.It looks like this.The proportions of this triangle over here are the same as the proportions of this triangle over here.Specifically, the length back here is x2 minus x1.This distance over here is f, the length over here in the baseline b,and this length over here is the unknown depth z.If we transform this and solve it for z,we get z equals f times b over x2 minus x1.If we look at the relative displacement of a point in these two different camera images,which is x2 minus x1, you'll find the the actual depth is inversely proportional,but in this case linearly with the focal length f and the baseline b.These are all things we know. The baseline and the focal length are constants.They're called intrinsics.These are measurements, and from this we can actually recover the real depth.Let's just try to practice this.So let me give you another quiz in which we have a stereo rig with baseline Bwith two measurements, x1 and x2, of the same point P in the scene.We know our focal length f, and I care about our depth z.Here is our formula again to make things a little bit easier.Here assume that my x2 equals 3 mm, my x1 is -1 mm, my focal length is 8 mm, and my baseline B is 20 cm.I'd like to know z in centimeters.The answer is 40.F is 8 mm over 3 minus -1 is 4 mm,which gives this guy over here a factor of 2 times B.Ten centimeters makes 40 cm for z.Using the same formula, we are going to write x2 minus x1 as delta x just to make it a little bit simpler.Let me see if we can recover other things.Let's assume the range is actually 10 m. We know about the physical world.We know our baseline 1 m. We know that our focal length is 30 mm.Can we possibly recover the delta x?I'd like you to give your answer in mm.The answer is absolute yes. It's going to be 3 mm.To see, we transform this equation over here to delta x to the left,B over z is 0.1 times 30 mm makes 3 mm over here.Let's now go to a difficult challenging casewhere I'd like to recover the focal length f from measurements of the type z, delta x, and B.Suppose I happen to know that an object is 100 m away, and my baseline is 0.5 m, which is 50 cm.And suppose my displacement x2 minus x1 is exactly 1 millimeters.What do you think f is expressed in mm?The answer is 200 mm for our focal length.We can transform this expression over here to bring f to the left side.And z over B equals 200, and delta x equals 1 mm.We get 200 mm as an answer for our question.I'd like to say a few words on the issue of correspondence, often also called data association.Supposing we have two camera images, as shown over here,and we seen an interesting point P in the left image.The question is where do you search in the right image?Everywhere? Along a line? Or can you already predict the point?So where do you search in the right image? Everywhere would be in 2D--the entire image.Along a line would be 1D, and a fixed point would be 0D.Please check the appropriate box.The answer is 1D. You can actually search along this line over here.You can't really know where along the line the point is,because where it is is a function of the depth of the scene, which you don't know,but it can't be the full image.To illustrate this, let me look a little bit from above.Here was have two image planes from the two cameras.There is a point over here that finds itself in the image plane over there.If we don't know the depth, we know that the point must lay on this ray over here,and each of the points on this ray get projected into this imager along a line.If the point is over here, it might be the projection over there,and as we go out to infinity, it might be the point over here.Now this camera array is a little bit more general than we talked about.The image plans aren't parallel anymore,but even if they're no parallel, each point in the left image corresponds to a potential lineof corresponding points in the right image.It makes the search for correspondences much, much easier.Let's talk a little bit more about correspondences.The general correspondence problem is given if there are two identical-looking points in the scene that have different depths.For example with P1 might reflect into the image over here,and P2 will reflect into the image as indicated by these red lines.Now we understand the correspondence of P1 in both images that this point corresponds to this point, we are well off,and we can estimate the depth of P1.If we get it wrong, if we correspond this point over here in the image to this guy over here,then what we will see is this point right over here--P1 prime.If we correspond this guy over here with this guy over here,we get P2 prime.These aren't really points in the action image, but they'll be phantom pointsthat occur because we got the correspondence wrong.It's really important when we look at two camera imagesto understand what is the actual correspondence.Here are actually two images from a stereo rig of a scene,and you can see there's a slight displacement. It's actually really hard to see.We're looking at this feature over here for now.I'd like to correspond it to something in the right image.We have already learned that the search will have to be along a line.Here is the green line, which is the corresponding line.It can't be that this point over here shows up somewhere in the sky over here,but even along the point, it's not completely obvious how to do correspondence--how to match this image over here to this image over there.So my question is how can we possibly find where this feature corresponds to a feature over here?How can we determine correspondence?By matching small image patches using some of the linear techniques we talked about inthe last class by just basically comparing how similar looking small image patches areor by matching features, and particularly edge features or corner featuresthat we might extract from the original image.Or maybe neither of those two. Please check any or all of those that apply.The answer is both. You can use image patches and features, and I'll talk about both.They are somewhat similar, and they're not without problems, but both are being used to estimate correspondence.Here is my pair of images again,and my scan line, and I'm extracting from it a very small little windowthat is the local image of the specific feature over here which happens to have a strong vertical structure,which is nice of localization.Now I'm comparing this little patch with my little patches in the right image,and I'm drawing a sum of square difference error,which is minimized when these two patches look alike.I'll tell you in a second how this looks like mathematically,but intuitively we have to pick the place along the random measured search spacethat has the smallest sum of square difference error,which is the one where these two patches just look mostly alike.This is a space of the scan line in which I search,often called disparity, and for one location this is actually being minimized right over here.Here's the basic algorithm for SSD minimization. We take two patches--one from the left image, one from the right image.We normalize, so the average brightness is zero.We then take the normalized image and take the difference.Then we square the difference. That gives us a sum-of-square image.Then we can sum up all the pixels to get a single value.This is our SSD value, our sum-of-square difference value.All of these operations are easily implemented using the material you already know.The smaller the SSD value, the closer these two images correspond.This is a very common technique for comparing what's called image templates,where your left image is a template, and you're searching the left image for the optimal template.As you vary the location of the right image, you can find different SSDs.You tend to get graphs for the right image.With an image template, it gives you certain errors.Sometimes you get a very small disparate error.That's the place you'll pick for the best, mostly likely alignment.Here is the result of such an operation.We have yet again a left image over here. The right one is missing.Here you can see what's called a disparity map, which the map of the best match.In the right image, the further the disparity, the more we have to assume the patch shifted.We extracted every possible patch from this image, did the search on the right image,and we find in the foreground, the disparity is much larger than the background.Sometimes we get a black spot, like over here,where the information itself is not good enough to make any decision.Or in the pathway over here, there are no real features.Same for the sky over here.But in most cases, we can see a nicely shaded gray that decreases with distancewhere the disparity decreases.This is a very typical stereo vision result.Here is a disparity map from driving in desert with our DARPA Grand Challenge car, Stanley.We equipped it with two cameras, one on the left and one on the right.You can see the two camera images, and on the right the disparity map.It's not that informative, because there is very little structure in the road surface itself,but by and large you can see things further away end up being darker.The big dominant thing here is lack of texture, which leads to certain areas in the disparity map just being black,which means we don't know.But where it registers it does a pretty find job.I'd like to talk a little bit more about correspondence.Specifically, we've learned that searching for correspondence meanswe search along a single scan line,but I'd like to ask the question whether it's optimal to correspond individual patches which are independent of each other.Would it make sense to look at the context of an entire scan line?Let's look at the following situation.We have a background that's black.We have a foreground that's red, and we have sides of the object that are both blue.In a left image, we might see black, black,and then there is this blue element that is only visible from the left camera,a couple of reds--3 of them--and then we see more blacks.From the right imager we might see black, black.We won't see the blue over here, because it's occluded,but we'll see a couple of reds followed by the blue over here,which is only visible from the right camera, followed by more blacks.When we look at the entire situation, the question is whether we can correspond red pixels to each otherirrespective of context or whether it makes sense to look at context.Specifically, take the mid red pixel over here--this guy over here--and let me ask you does it correspond to the left red, the center red, or the right red?Please check the corresponding box.The answer is it corresponds to the center red,which is the guy over here.Finding this is not easy.This is the fifth pixel on the left camera image, and it's the fourth pixel on the right camera image.To make that correspondence, we have to understand that the best match matches these 2 black pixels over here, followed by an occlusion pixel that's only visible on the left but not on the right,followed by the 3 corresponding red pixels,followed by another occlusion pixel,followed by 2 black pixels that basically correspond.I now want to look into algorithms that can take entire scan lines of the left sideand correspond them to entire scan lines on the right side.Let's look at the same problem again,and let me just draw the two scan lines--the left scan line and the right scan line.As before, we get to see red pixels, black pixels, and the occlusive blue pixels as indicated over here.Now we'll try to match the entire scan line on the top to the entire scan line on the bottom.so we can figure out what the exact correspondence is.We do this by minimizing the cost function.The cost comes in two different flavors.There is the cost of bad matches.Let's assume if the color matches perfect, we pay zero,but if the color matches very poor, we pay 20.There is also the cost of occlusion.If in the process of matching these lines we have to assume a pixel is occluded,we're just going to pay 10.The question now is optimal alignment of the top to the bottom under this cost function?Let's just go through this. Let me look at two different possible alignments.Here is one. We align those black pixels, and we align the red pixels.If we did this, what is the cost of the total match. Please put the answer over here.The cost is 20. The reason being that in this match over here, we get a perfect color match.Black matches to black. Red matches to red. But we have to assume this this pixel over here and the pixel over here,are both the result of occlusion. Each costs us 10.So the result is we pay 20 as the total cost.Let me now the same question again for a different alignment.Suppose we were to marry the red pixel over here to the blue one over here,the red one to the red one over here, and so on.What would now by the cost of the alignment?We don't have these diagonals, but match pixel by pixel over here.The answer would be in this case 40,because we pay a 20 penalty of a bad match over here.These are good matches. We end up paying another 20 penalty for a bad match over there.In total we get a penalty of 40.What this teaches us is that in matching pixels to pixels,we match an entire corresponding line in stereo.We can trade off the bad match cost with the occlusion cost.Sometimes it is cheaper to assume occlusion,and sometimes it is cheaper to assume a bad match.The result of this optimization is that it gives us the best association of the scan line over here to the scan line over here.The tricky part is how to compute the best possible alignment.It's usually done by dynamic programming.The recognition here is that in principle there are exponentially many ways to align pixels in the left and right image,but in practice you can get away with an n-squared algorithmwhere n is the number of pixels in the scan line.Let's write this as n-squared. It's a much, much faster algorithm.Here's the idea. Let's write down both scan lines as shown over here.And let's write down a matrix of size and square.The neat thing here is that any path from the top left to the bottom rightis a specific correspondence of pixels over here on the left scan lineto pixels over here on the right scan line.For example, if I take the path that's diagonal, that line pixels by each other.But the best possible path would assume that the first two pixels correspond,and there's a left occlusion afterwards.Then all the red guys correspond.So this red guy over here corresponds to this red guy over here.There's an occlusion over here.Then we go diagonal again.So any path that picks actions that go diagonal, down, or rightso that the top left is connected to the bottom right becomes a valid correspondence of the left scan line to the right scan line.How do we find the best one?Well, just like in MVPs we use the same methodology as an MVP.We define the value of any of these points in the grid to be the best, taking the value of getting there.The value of a point ij in the grid is the maximum of the match value if we chose diagonal, which is expressed over here to be the match of ijgiven that we chose the diagonal, which means add the value of i minus 1 and j minus 1,over the occlusion penalty plus any way we could have occluded for the left or the right.If we look at these three different things we maximize over here,then each value over here becomes the maximum of assuming we have no occlusionplus the corresponding match penalty or assuming we did have an occlusion,either from the top or the bottom, and then we just pay the occlusion penalty,and we assume the value over there.Now, that's not trivial. You have to think about this.Why does this give us the optimal path?But if you think about it and look at the optimal path,we pay no penalty over here because the match is perfect.We pay no penalty over here because the match is perfect again.So, again, the first clause in this formula.Over here we do pay a penalty.We pay a penalty of 10, which is the occlusion penalty,because we assume that between the blue pixel over here and the right scan imagethere's just no appropriate match. We're going to pay a penalty of 10 over here.Over here we pay no penalty, because the right corresponds perfectly to the red,and we assume it is a perfect match.The same over here and the same over here.Down here we pay a penalty of 10, because we assume an occlusion,and down here we just assume no penalty at all.Now with dynamic programming, it computes for every possible location.For example, this guy over here would have a best optimal path,which might assume we had a perfect match over here and two occlusions over there,but now the penalty is already 20 whereas the penalty over here is 10.So likely this point won't survive.By working out the value function in this really interesting grid over here,we find the value of the final point, which is 20,and we also find the best possible pathby tracing the way in which the value propagated through this grid.This becomes the best possible correspondence of the left and the right imageby aligning the entire left scan line and the entire right scan line simultaneously using dynamic programming.This is the state of the art in stereo computer vision.Let me see if you understand what I just talked about by the following quiz.Let's assume we have two scan lines of six pixels each.You get to observe the following that I've colorized:two blacks followed by three reds by one black for the left imageand one black followed by four reds by one black for the right scan line.Let us also assume the occlusion penalty is 5, and the bad match penalty is 20.Can you mark the location to which you'd like to correspond this specific red pixel over here in the right scan line by minimizing the total cost of occlusion and bad matches.This is a tricky question, because it turns out that the occlusion answer is better than the bad match answer.If you were to correspond each pixel in the left scan line straight to each pixel in the right scan line,you find that the total penalty will be 20, because there's one bad match between the black pixel over here and the red pixel here.However, you can also correspond as follows.You end up paying two occlusion penalties for this guy over here and this guy over here.Because the occlusion penalty is only 5, you pay only a total of 10 as a penalty,which is better than a single bad match penalty.As a result, this would've been the right answer over here.Let me discuss the second quiz here, in which we're given two scan lines again--a left and a right scan line.The pixels are for the left scan line black, red, black, black, black, blackand for the right scan line black, black, black, black, red, and black.Let's assume an occlusion now costs us 10, and a bad match costs us 20.So what pixel should be aligned with the black pixel over here?Check one of those boxes.The answer is this box over here.There are two points on the line where one may correspond each pixel on the left scan line to each pixel on the right scan line with the same index.So this guy corresponds to this guy and so on--just straight up.We're going to pay a penalty of 40, which is this red pixel over here as a bad match to the pixel over there,and the same is true over here.So it's a penalty of 40, which I conjecture to be the best.Let me show you the answer that I don't like,which corresponds this black pixel to this black pixel here,the red guy to the guy over here, and then black to black over here,in which case we pay an occlusion penalty for the following six pixels:the guys over here and the guys over here.Each occlusion value is 10, which makes a total occlusion penalty of 60,which is worse than the 40.So I conjecture that the straight up max like this is superior,and we should check the box over here.As you can see the optimal path for this diagram over here, which determines what is occlusion and what is a bad match,really is a function of those penalties, the costs that we associate with poor matches or the occlusion assumption.So running dynamic programming through this grid over here will give you the best alignment that gives you the best possible total cost that assumes an optimal trade off between occlusion costs and the cost of matching 2 pixels.This segment is my explanation of correspondence in stereo vision.It came a long way. There are a few things that don't work really well.For example, we have two cameras over here, and we have a big object over here with a foreground separate object.Then the order contraint is being opposed and dynamic programming doesn't hold.That is, an object over here might appear left of the object over here in the left imagerbut right of the object over here in the right imager.There are other cases where things go wrong.For example, suppose you were imaging a circular object with these two imagers here.Then the occlusion boundary of this object as viewed from the right imageris different from the occlusion boundary of the same object as viewed from the left imager.These are not corresponding points. They correspond to different points on the object.As a result, your stereo calculation will give you a poor result.A final instance where things might go wrong is reflective objects that have specular reflections.This ball over here reflects the ceiling lights, and obviously, where the ceiling lights are being reflected is a function of where an imager is positioned.For these specific features over here, we get a really lousy depth estimate for the object at hand.I'd like to say a few words about how to improve the results of stereo vision.Here is a vision assembly that James David built up of two cameras.In addition to having these two cameras, he also put a projector into the scenethat emitted a random light pattern.In fact, it emitted a striped pattern, shown over here on this frog,and by adding texture to the scene, you can making correspondence easier.This is a striped pattern of unequal distances.There's a coding over here, which makes certain stripes larger than others.If you run the same algorithm I just told you,you'll find that stereo vision becomes better, because we can now better disambiguate the correspondence of points. Here is the assembly used for imaging myself. This is me with a sweater on.That's my face.And you can see by emitting structured light, as it is called,you can enhance the performance of stereo and objects that otherwise have very poor texture.Another solution is called the Microsoft Kinect. You're probably familiar with it.It's a new gaming platform that's been sold at record pace.It uses a camera system, together with a laser.The laser adds texture to the scene,and by triangulation using the same method I showed you,it can recover depth.Here's my postdoc Christian using a Kinect-like sensor to do certain poses in front of a depth sensor.You can see in the screen how his pose is being perceived,and you can see Christian trying to do handstands and other acrobatic maneuvers.He's actually pretty good.That's all using effectively stereo vision.There is actually a whole bunch of different types of techniques for sensing range in computer vision.I'm just going to briefly talk about them.They're called laser range finders. They send off beams of light, and they measure the time until the light comes back into the sensor.They're being manufactured by many different companies.In our experiments using robots to drive through the desert and through traffic.We quite extensively used laser range finders as an alternative to stereo vision,because they give us very, very good range estimates.Here is a 3D model constructed by laser range finders of our neighborhood in Palo Alto,and it's easy to see how 3D points can making amazing 3D models,using techniques like stereo vision or like the laser range finders I just briefly talked about.[Thrun] This very final episode of the computer vision classesI will teach you about structure from motion.This is a really funny name for something much more intuitive,and it comes from the early days of computer visionwhere the structure referred to the 3D world.And of course it's impossible to capture the 3D world with the camera itselfbecause the camera only gives 2D projections of the 3D scene.Motion referred to the locations of the camera.So the idea was to take a handheld camera and move it around a 3D structureand be able to recover or estimate the 3D coordinates of all the features in the worldbased on many 2D images.So suppose you have a scene with 3 features--A, B, and C--and you're moving a camera around to different positions--1, 2, and 3.Then the different features get projected onto different points in the camera planes,as shown over here.And from the positions of those projected featuresit may be impossible to recover not just where the camera wasat the time these images were taken but also where in the world the features are.That's called structure from motion.So here is my first quiz.Is this possible?Given that we look at a number of features in the scene--maybe 1, maybe 2, maybe more--and given that we have 1 or more camera positions, can we always, sometimes, or never recover or calculate the 3D position of the featuresand the 3D position of the cameras simultaneously?Please check almost, sometimes, or never.[Thrun] And the answer is sometimes,and it's not entirely obvious whether this is the right answer.Clearly if there is only 1 point feature in the world and 1 image,then you can't recover where the feature is.You already learned this, because the camera can't estimate depth on itself.So it can't be almost,but it's also not never.There are cases in which we can actually recover the full sceneand all the camera positions, and we will ask ourselves in a minuteunder what situation this might be possible.[Thrun] Let me first give you a brief quiz to understand how the projection worksin structure from motion.Suppose we have 3 point features at the known location.We have a camera over here, camera A,which can see these 3 point features.We have a second camera over here, camera B,a pinhole camera which can see the same features.Suppose camera A on the left sees feature 1,at the center sees feature 3, and on the right side of the camera plane sees feature 2.I would like to know for camera B what will be on the camera plane on the left side,the center, or the right side.Which of the features 1, 2, 3 will be seen left, center, or right?[Thrun] And the answer is 3, 2, 1. Let's just go through this.Clearly the leftmost feature in camera A is 1, which corresponds to this point over here,the center will be 3, and the right one will be 2,as indicated in the table over here.If we now look into imager B, we find that the leftmost projection comes from feature number 3,the center projection from feature number 2, and the rightmost projection from feature number 1.This is not the full structure from motion problem,but it's a good exercise to understand how feature indices under known position A and Band under known locations of the target features map to each other,and it's good to understand the complexity of the structure from motion problem.[Thrun] Here is a very early example of structure from motion by Carlo Tomasiand Takeo Kanade.They used Harris corner detectors to find corners in the image of this toy 3D house,and they were able from a number of images to fully recover the 3D structureof every single corner point, as shown in this video.So as they then take this 3D data set and turn it in arbitrary directions,you can see the full 3D structure was recovered.This is work in 1992.It used principal component analysis to solve the problemand is one of the most amazing pieces of early computer vision research.Carlo, who used to be a Stanford professor for many years,then scanned his kitchen and with the same Harris corner detectorwas able to reconstruct a 3D structure of his kitchen, as shown over here.Again, this is one of the most impressive early computer vision research results I've seen.Here is a flight video of flying over the hills of Pennsylvania.As you can see, using the same technique he was able to recover the 3D structureof the outdoor terrain and build elevation maps, as shown over here.Marc Pollefeys, who presently teaches at ETH Zurich,came up with a beautiful solution to the structure from motion problem,here imaging different buildings in his hometown.From this video you can see multiple snapshots of a single buildingwhere the different perspective distortion has an effect on the appearance of the building,quite obviously.Using those images he was able to reconstruct the 3D shape of the building facade,as shown in this video.Again, at the time it was one of the most impressive results ever achievedin structure from motion.You can see amazing detail as he zooms in to his building model.He then moved on to map entire cities,and here is an example of a map that he produced from an entire city block.You can see how he reconstructs the building facades in unprecedented detail.There's also a lot of occlusion gaps where the original imager wasn't able to see anything.Those show up in black, and they look a little bit disturbing in this image over here.But in reality, your camera can't see everything.So even if you do a perfect job with structure from motion,it's really hard to reconstruct every single inch of the environment.Still, this stands out as one of the most impressive results everin what I would call the Holy Grail of 3D computer vision.[Thrun] The mathematics of the structure from motion problem are involved,and I don't want to go into detail here.Here is our perspective projection model with our well-known equation on the right.Under the assumption that the camera itself might be at a random locationand a random orientation, this equation becomes a really complicated compositionof original image points in 3D, 3 rotation matrices as shown over here,and an offset over here that relates to the camera coordinates.You do this for X divided by Z over here.This will be the projected camera input coordinates.This is the generative math that specifies how cameras work under arbitrary orientationsand arbitrary translations.If you now want to solve it, you can look at the observed measurementsminus the predicted measurements, minimize all this,and solve for the translations, the point locations, and the orientations simultaneously.This is entirely nontrivial, and nonlinear optimization techniqueshave been used extensively to solve this problem.They go by names like gradient descent, conjugate gradient,Gauss-Newton, Levenberg Marquardt, and other things like singular value decomposition.I won't go into detail--just to give you a flavor of the problem.Instead I'd like to ask you a question.[Thrun] I should warn you this question is hard.If you're new to computer vision, you likely won't get it.I'd like to ask it to you anyhow just to see how close you can getand whether you appreciate the answer I'll be giving you.Suppose we have m camera poses.That means m directions from which we take an image.That's called the motion.And suppose we have n 3D points, which is called the structure.Then it's quite obvious that we have 2 times m times n constraintssimply because in each of the images we see all the n pointsand we get an x and a y coordinate for each of the points, which makes 2-m-n constraints.We also have unknowns.Specifically, each camera position is a 6D unknown about the rotation and translation of the camera,and each point itself has a 3D coordinate.So the total number of unknowns is 6m plus 3n.At first glance, to solve the structure from motion problem you would want 6m plus 3nto be smaller or equal to 2mn.And of course if m and n is big enough, this equation will be satisfied.But my question is if you run the structure from motion problem,how many of these unknowns can you actually recover?Or, put differently, how many of those unknowns can you not recover?If you think about it, for example, you won't be able to really recoverthe absolute coordinates of our system, because you can move the entire system1 meter to the right and you'll still get the same answer.So there's going to be a number over here that I want you to enterthat specifies the number of parameters that cannot possibly be recoveredin this structure from motion problem.[Thrun] And surprisingly, the answer is 7.You cannot recover the absolute location and orientation of the coordinate system,which are 6 of those parameters,but you can also not recover scale.For example, take a situation like this where you have 3 points over hereand now make this situation twice as largewith the points spread out twice as widely.Because of perspective math, this over here will be the same answeras this guy over here.So this is 1 scale parameter that you can't recover,so you can only recover 6m plus 3n minus 7 parameters.And as long as this is smaller than 2mn, you have a solutionof the structure from motion problem.This was entirely nontrivial.If you got this wrong, I would have gotten this wrong if I hadn't known the solution.But it's fun to think about these things.A lot of computer vision people care about whether the problem is well posed,and you need a certain number of features and a certain number of imagesto make this equation hold true.[Thrun] Now, at this point I would love to go deeper into structure from motionand tell you more about how to solve it.But, unfortunately, this is the Introduction to Artificial Intelligence class,so I really want to leave it at a level that covers the typical material I cover at Stanford.If you're interested, take a computer vision class.It's a fascinating subject area.This finishes my survey of computer vision.Congratulations. You made it through the computer vision classes.I think you now understand the very basics of computer vision,you understand how images are being formed,how features are being extracted,and how we can do some very basic 3D inference about the world.This is just a teaser.The field of computer vision is of course much richer.I used to teach the class at Stanford,and I hope to be able to invite you in the near future to an actual online 3D computer vision class.Welcome to the homework assignment on computer vision.I will first ask you a few questions about perspective projectionin which you will exercise the math that we exploredthat relates the size of an object in the scene, uppercase X,with the depth for the range to the pinhole camera Z, the focal length f,and the size of the projection, small x.Remember from class the following equation.I'm literally dropping the minus sign. I want all numbers to be positive in this example.So please don't worry about the minus sign that might not occur in this specific version of the equation.I'll give you three values for X, Z, f,and would like you to understand what the missing value is.X is measured in meters and the same with Z.f is in millimeters and so is lowercase x.Here's my first question. Suppose X is 10 m in size.It's 100 meters away. Suppose our focal length is 10 mm.How large is our projection lowercase x in millimeters?Now we're asking you what is the focal length if an object of size 20 m that is 400 meters outif we observe it to be 1 mm on our projection surface.Suppose we have a 2 m sized object that with a 40 mm focal lengthappears to be 1 mm in size. What is the distance of this object to the camera?Finally, say a known object is 300 m away.Our focal length is now 100 mm, and the object is again 1mm.How far is this object away in meters?The answer can be seen directly from this formula over here.In the first case we plug in X equals 10.F over Z is 10 divided by 100.We multiply 10 by 0.1, and we get 1.It turns out all the units take care of themselves.No matter which way we pose the question, we can effectively ignore those units,because they're the same outside the camera as they are inside the camera.For the second question, we transform this equation over here as follows:We now plug in Z as 400, X as 20, lowercase x as 1,to give us a focal length of 20 mm.We can transform this equation further into this quotient over here.F over x is 40 times uppercase X of 2 mm is 80 meters.Finally we can write this expression over here, and we plug in x over f.We get 100th times 300 is 3 meters over here.In this question I'm going to ask you about whether certain image functions are linear.A function is linear if each resulting pixel of the processed image is a linear combination of input pixels.They could be rated by constants like plus 1 or minus 1,and they could be added up. Addition is linear.But for example, taking the square of a pizel isn't a linear operation.I realize this question goes beyond what we discussed in class,so please think a little bit about it and understand the difference between linear and nonlinear in trying to answer these questions.First is our gradient kernel here: minus 1, 1.Please check if it's linear or nonlinear.Again, the linearity of an output image is given if the output image is a linear function of the pixels in the input image.How about our Gaussian kernel that we discussed in class of size 5 by 5?Is the kernal linear or nonlinear?How would we take the absolute value of a pixel?If pixels are negative, we just ignore the negative sign and map back to the absolute value.Is it linear or nonlinear?We talked about the gradient magnitude kernel,which was defined over a square root of the squares of the image gradients.Is this a linear or nonlinear operation.Finally, if you were to calculate the absolute brightness of a grey-scale image,or let me call this the average brightness.We have an imager of a certain size and like to calculate just the average brightnessof all the individual image pixels. They are all in greyscale.Is this linear or nonlinear?The answer is every kernel convolution is linear.Each pixel becomes the linear sum of, in this case, 2 pixelsthat are weighted by plus 1 and minus 1, but in terms of the original variables,which is the original image, this resulting sum, minus the left pixel plus the right pixel,is a linear equation in the original pixel values.The same is true for the Gaussian kernel of size 5 by 5.It is a linear kernel because it just adds up all these values summed by the Gaussian kernel.Absolute value is nonlinear.The function that governs absolute value for input and output looks like this,and there is a nonlinear kink over here.The same is true for gradient magnitude.There are squares in there, which are nonlinear,and the square root makes it a nonlinear operation.The absolute brightness is a linear operation.It's just like a Gaussian kernel with a uniform mask.It just adds up all the values and divides them by the number of pixels.It is a linear function in all the input pixels.In this example, I'd like you to calculate a gradient image. I'm giving you a relatively simple image of size 3 by 3 with the following greyscale pixel values:, 2, 0, 2, 4, 100, 102, 242.And for the sake of this exercise, I'd like to retain another 3 by 3 image,so we'll assume that all the values outside the image are just zero.What I'm asking you is to compute a 3 by 3 matrixthat is the result of convolving this image with the following kernel:minus 1 on the left, zero, and 1.Then take the absolute value of each pixel, so you're going to ignore the minus sign,which is clearly an nonlinear operation.Please apply this kernel to the image over here.For each pixel, down here you get a linear combination of applying this kernelfrom the values over here, assuming these off image values are all zero.We then take the absolute--drop the minus sign--and please plug in the number over here.This kernel applied in this location over here will give me a minus 1 times zero plus 1 times zero is zero.Shift to the right you get minus 1 times 2 plus 2 times 1 to zero.The absolute value of this is zero.On the right side we get zero again. You get 100 over here, which is minus zero plus 100.We get 98 over here, which is minus 4 plus 102.And we get 100 over here, which is minus 100 over here plus zero,which gives minus 100. We take the absolute, so it's 100.You get 4 over here, which is minus zero plus 4 equals 4.Zero over here. These two balance each other out.Then another 4 over here. Minus 4 plus zero is minus 4.Taking the absolute we get 4 over here.I now have a stereo question.For a valid calibrated stereo rig, we're given two pinhole cameras whose displacement is B, and we observe a point out in the scene.The distance of this point to the image plane up uppercase Z.Our cameras have a focal length of f.Of course, this point is being projected into two different locations for the two different images--x2 and x1.For the sake of this question, we're going to just consider delta x, which is x2 minus x1--the displacement in the corresponding images.We measure delta x in millimeters, same with the focal length f.B is in meters and so is Z, and I use meters for B and Z so that the units effectively fall out.You don't really have to consider them.Suppose the measured delta x is 4 mmwith a focal length of 40, and our displacement is 0.1.How far away is the object in meters?Suppose we have a displacement of 0.05 mm for a focal length of 50.We now care about the baseline B, if we happen to know the object is 100 meters away.Next we have a displacement of 0.1 mm. We don't know our focal length.We know that the baseline is 0.2 mm, and the object is 50 meters away.Finally we don't know the displacement, but we do know that the focal length is 200 mm.We have a baseline of 1 m and the object is 50 meters away.Can you fill in the missing numbers?We answer this question by first writing down the fundamental equation here,which is delta x over f relates to B over Z.To see this, we find by equal triangles that this triangle over heredescribed by Z over B, is the same as these 2 things over here put together into a single triangle, which is delta x over f.This proportionality must be the case.This can now be transformed to solve for Z.Z equals f over delta x B.If we plug in f with delta x, we get 10 times 0.1 which is 1.We can resolve B is delta x over f times Z.We plug in delta x, 0.05, divided by 50 is 0.001 times 100 gives us 0.1.We can also resolve it for f, which is Z over B times delta x.Z over B is 10 times delta x as 0.1 gives us 1 over here.Finally, we can resolve it for delta x.B over Z times f. B over Z is 1/50 times f as 200 gives us 4 over here.All the units fall out by themselves.We will now talk about correspondence in stereo.You might remember our dynamic programming approach forresolving correspondence along an entire scan line.So I'll give you another scan line. This is the left scan line--red, red, blue, blue, blue, red.Then the right scan line we get to see the following.Obviously there is a shift going on.I'd like to ask you where this little pixel over here will go into the lead association.It can go into any of those pixels over here, so please check exactly one of those boxes.Let's assume the cost for a bad match, when we match 2 colors that don't correspond, is 20.The cost of an assumed occlusion or a disocclusion is 10.Try to find the optimal alignment, and then tell me where in the right scan line this 1 pixel corresponds to.Check the exact box to which it corresponds.Here is a second question I'd like to ask you.What if we changed the cost of occlusion to 100?Please answer the exact same question--where does the B over here go--under this different cost model.In the case where the occlusion costs are low,it is best to assume that those Bs over here correspond as indicated.That means we have an occlusion cost to pay over here and an occlusion cost to pay over here.Our total cost is 20 for 2 occlusions, but we have a perfect match.As a result, this B moves over here.However, there is another viable solution when the occlusion costs are large,because you would pay a total of 200 with the occlusion cost.If we match pixel one to one like this, then you get two mismatches--one over here and one over here.The cost of those in total are 40.That is still smaller than the 200 occlusion cost we had before.Therefore the B gets matched to this point over here.Notice how the different occlusion costs give different results for the correspondence program in this dynamic programming question.This final question is motivated by structure from motion,and it's not the full-blown structure from motion problem, which is hard to do on a piece of paper here.But it's a variant event for which I know the motion but not the structure.Suppose we are given two cameras, and we happen to know there are three features in the scene.All three features can be scene by both cameras.This camera will see a feature on the left, in the center, and on the right--L, C, R.This camera is camera A.One camera B, we also see a feature on the left, center, and right,but I don't know the identity of those features, which I will call 1, 2, and 3.Suppose in camera A we see from left to the right the following sequence: 1, 2, 3.These are the features numbers.So in the left camera we notice that the left-most visible feature is feature 1,the center feature is feature number 2, and the right feature is feature number 3.I'm going to ask what is the order of those pixels in camera B,assuming that the features are located as shown over here and so are the cameras.Please give the index of the features over here that clearly have to be 1, 2, and 3in some order which you have to determine.For a different configuration let's now assume we get to see 1, 2, 3 in camera B,and we care about the feature indices we see in camera A that corresponds to those features over here.Let's study the first case.We saw left feature number 1. The left one is over here, therefore this one will be feature number 1.Feature number 1 will be seen at the center of camera B.In the center of the left camera, we see feature number 2,which must be this guy over here, because it's the center feature to be seen.This will be projected to the right side of camera B.Even though it's left in the image plane, the way a I drew it you can see thatthe projection over here is on the right side of the camera chip.The same with feature number 3, which is seen in R over here on the right side.Hence, it'll project into the leftmost field over here.Using a different color now, if camera B sees feature number 1 on its L position,Then this must be feature number one, which will appear on the right position for camera A.And feature number 2 is in the center position--this guy over here--which shows up in the left position over here,and the remaining fits in over here. This is the correct answer.One of the things I've been working on for most of my professional career are self-driving cars.The vision is that in the future cars will drive themselves,and in doing so they can be significantly safer.We lose about a little over 1 million people per year in the entire world in traffic accidents.I believe most of these accidents can be avoided by making cars safer.If they drive themselves, they can drive disabled people.They can drive blind people, young children, aging people,and they could drive all of us while we do better things that staring at the road ahead.So one of my life passions has been to be develop self-driving cars.Today, I'd like to tell you about those, and also show you some of the basic techniquesso you can in principle program your own self-driving car.So for me the work on self-driving cars started in 2004 after the first DARPA Grand Challenge.This was a government-sponsored robot racein which autonomous robots were asked to drive through the Mojave desert from California to Nevadaalong 141 miles of really punishing desert terrain.Lots of teams competed from various universities, car companies,and also lots of hobbiests that were new to the field competed,and built this huge set of different cars. There were over 100 different entries into the first DARPA Grand Challenge.Despite all this work, most robots failed out of the starting gate, like this one over here flipped over less than 100 meters into the race.Some were very, very large. This is a major defense contractor who built this 35,000 pound vehicle,which on the course was rather timid,and some of the the teams had very small robots, like the next one by UC Berkeley,which was a motorcycle.So here we go.The first DARPA Grand Challenge came with $1 million of prize money,and despite this prize money, no team made it further than 5% of the total course.In fact, almost all cars stopped for something very stupid,some went up in flames,and the furthest any team made it was this car over here by Carnegie Mellon University,which made it about just below 8 miles of the total distance.So for many of us, this was a massive failure of robotic technology,which motivated me to get involved in this race.My own story is really simple.I started a class at Stanford, and I got about 20 students to work with meon what would become the Stanford racing team that would ultimately go and win this race.We modified a Volkswagen Toureg to put all kinds of sensors onto the roof and actuators into the car that could actuate the steering wheel, the gas pedal, and the brake.The sensors came in multiple versions.Some were related to localization, such as global positioning sensors,and some were related to understanding where obstacles are, like laser-range finders.We talked about computer vision in this class.The actuators were basically a motor on the steering wheel and on the brake pedals and on the gas pedal.Early on, we tested on Stanford's campus.This is the roof of the medical parking garage.Here you can see my students and I performed simple maneuvers.Now, I've got to tell you that this is usually a busy parking garage.It's the medical parking garage at Stanford Hospital,but as we practiced autonomous driving, people would come and pick up their car and ask us about, what we were doing, so we kept telling them, well, we're building a self-driving car.Within less than a week, people just chose not to park there anymore.Closer to the next version of the Grand Challenge, the second one in 2005,we had built a car that could drive competently on most desert tracks at speeds up to about 60 km per hour through dry river beds, through steep inclines and declines,and would be able to avoid obstacles like this little shrub on the right side over here.It was never really elegant, but it was insanely effective.Now, not all testing went smooth.This is imagery that the New York Times shot of us when we invited them for a test drive.During this day, we managed to crash into a tree and get stuck in the mud.It was pretty embarrassing.Here is imagery of our laser system mapping out the terrain ahead.We talked a little bit about lasers and range finders in this class.Here you can see all these systems work together on building 3D maps of the environmentthat our car, Stanley, uses to assess the driving situation.This shows work on machine learning autonomous driving,where we used the laser to identify driveable terrain at a short rangeand then extrapolate this out into the long range using a machine-learning techniqueapplied to computer vision.What you see here is a coloring, which is the output of a machine learning algorithmlthat identifies driveable terrain in the desert.So very briefly to tell you about the race, one with a lot of fame and $2 million.This race started early in the morning. The sun was basically still gone and was just rising.Our car was able to drive itself followed by a human-driven change vehicle and did quite well.It did so well that it actually passed the front-seated and first-running vehicle by Carnegie Mellon University.It had to navigate complicated and dangerous mountain trails where destruction lured on both sides of the car.On the left there was a cliff. On the right side there was a mountain.It is here followed by a human-driven chase vehicle.Our car very carefully ascended this route.You can see it here close before the finishing line,and after just about 7 hours it managed to do what no robot had every done before.It managed to really finish DARPA Grand Challenge, do this race, and won Stanford $2 million.We were insanely proud on this day.From this we moved onto build Junior, which competed in the DARPA Urban Challenge.Here you can see Junior's laser pursuing obstacles and being able to detect those,using basically range vision.We will talk today of localization.Junior was able to localize itself using particle filtersrelative to a given map of the environment, which is essential for navigating safely in traffic.It was able to detect other cars using particle filtersand estimate not just where they are and how far they are moving but also what size they are, how big they are.You can see on the left the detected cars. On the right side, you see our camera view of the same situation.Here again, you can see it detect cars.Here is how it looked like from an external observation point.You can see Junior, our vehicle, driving in a fairly busy city street with lots of cars passing.It has to wait for a gap to take a left turn.When the gap finally occurs, it confidently takes the turns and drives.Today in today's class I teach you how to basically program a car just like that.So this is footage from our Google self-driving car, which you might have heard about.This car was able to drive at speeds as high as a Prius can go.It drives seamlessly in traffic.In fact, we drove over 100,000 miles without anybody noticing that there were self-driving cars in our experiments.This is near Stanford University on University Street in Palo Alto.You can see how the vehicle yields by itself for pedestrians.Of course, there's also a human driver on board just for safety,but this car, you can take my word for it, is really driving itself in traffic.This is image footage from the car itself as it goes onto a highway.This is sped up, I should say.Driving through a toll booth, and driving in Los Angeles.You can see a lot of palm trees here. It's a beautiful environment to drive in.Here you can see some of the inner workings, where you can see a corridor that the vehicle attempts to go.We can see obstacles being flagged using machine-learning techniques,range vision, laser radar, and so on.You can see it is colored by its relation to our car and its nature,and you can see it drives fairly confidently.This is an attempt to drive down Lombard Street in San Francisco--the famous crooked street.It's very curvy, and while this is sped up it gives you a sense of the complexity that is involved in building cars like these. It's actually quiet amazing how far technology has come in such a short amount of time.Here is an experiment that my Stanford students did on south parking using machine learning,reinforcement learning for control,and you can see how agile and how capable these methods are.So today I really want to enable you to write software like this based on lots of what we learned before.We talked a little bit about machine learning, a lot about particle filters,and some about motion planning, which relates to the planning class that Peter taught you quite a while back.Welcome to my class on robotics.In many ways this is applying AI technology to the problem of robotics.You might remember that a robot agent takes in sensor data from its environment.Here is the environment and here is the robot agent.It processes it into controls and actionsthat it uses to manipulate its actuators.Robotics is the science of bridging the gap between sensor data and actions.Just for the fun of it, let me ask you a quiz that links back to the very first class.Robotics is partially observable, yes or no?It's continuous in its state and action spaces and its measurement, yes or no?The environment may be stochastic, yes or no,and it may be adversarial, yes or no?Specifically, something like the DARPA Grand Challenge might be adversarial or not.Please choose the answer that best fits.I understand there might be multiple choices possible here, but please go back to what fits the best.And very clearly robotics is partially observable.We'll talk about this today a little bit more when I apply particle filters.It is continuous--that is all measurements are continuous and all actions tend to be continuous.The environment is clearly stochastic. It's impossible to predict what's happening next with absolute certainty.Then you can argue back and forth whether it's adversarial or not.In most cases we don't treat it as adversarial. We don't think about it.But somethings about robotics is indeed adversarial and to some extent driving is as well.I want to say no for now, but I'm going to accept both answers over here, so you can write whatever you want,because I don't want us to think about robotics as adversarial.At least not in the case of driving cars.One of the fundamental things about robotics is called "perception."The story here is that you get sensor measurements, and you're trying to estimate an internal state such that the internal state is sufficient to determine what to do next.It's usually a recursive method. It's called a "filter." We talked about this at length.I'm going to ask you a few questions about the state itself. So here's a quiz.Suppose we have a mobile robot that is round and lives on a plane,and it can turn on the spot, but its location is given by a two-dimensional coordinate.It might face in a certain direction.We really care about what's called the kinematic state.That is, we care about where it is but not how fast it is actually moving.So what is the dimensionality of the state space for such a robot?I do realize we haven't really talked about this much yet.I'd like you to take a good guess, and I'll tell you the answer once you have made your guess.The answer of this specific example over here is 3,although you could argue it could be something else, but 3 is a convenient and common answer,which is this robot's state is determined by its xy location and by its heading direction,which is often called theta.Now, you could argue heading doesn't matter because it can turn itself on the spot,and in some examples that is actually correct.You might be able to get away with it with a two-dimensional state,but if you're going to predict what's happened next when you, for example, drive forward,the heading matters greatly.In that sense, it's actually a three-dimensional state, so 3 is the best answer.Let me ask you a few more questions about dimensionality of state spaces.Here is a car,and again we worry about the kinematic state.That is, where in the world is this robot irrespective of its current velocity?What do you now the right answer is for dimensionality?The correct answer is still 3, although you can argue it is more,because maybe the position of its steering wheel matters,but at first approximation is the same as before.There might be a center point for the car. It has a certain location in the global coordinate system,and it again has a heading direction in which it can go.So 3 is a convenient answer, although technically the steering wheel angle might also influential for what's happening next.Now let's talk about the dynamic state.The dynamic state includes the velocities of the vehicle itself.I'd like to understand what's a good number of dimensions to encode the dynamic state of this car.The common answer here is 5,although I realize many, many different answers are possible,and there is clearly no single answer that is really correct.But 5 is my favorite.It's the 3 kinematic state variables, and in addition we care about velocities,so there is formal velocity of the vehicle itselfand the faster the vehicle moves, the further it is going to advance in a max timestep.There is also something called a yaw rate.Yaw is one way to name the heading of the car,the orientation of the car, or the bearing as some people call it.The rate is the change over time.This car will not just move forward. It will also turn.That turn has a velocity, and the velocity is often called "yaw rate."We're going to talk about this in a few minutes.Before we do this, let me look into the state helicopter.Here's by best depiction of a helicopter.This helicopter can fly anywhere in the xyz space,and it can also point in any possible direction.What's the dimensionality of the kinematic state for such a vehicle?The answer is now 6.You can really see that this helicopter can assume any location in xyz space,but it also has 3 rotation degrees of freedom.It has a yaw, which is its bearing. It can tilt forward and backward.And it can roll left and right.If we look from above, here is the yaw. This is the tilt.If you look from the front, you find there is also a roll variablewhere the vehicle can turn around its own axis.So the total answer is 6.Here is my most difficult question for the helicopter.What's the dynamic state? What's the right dimensionality here?The answer is commonly 12,which is simply we can have 6 state variables,and in each of those the helicopter might have its own velocity.So for each of these variables, we have the state variable itselfand its velocity, which makes a total of 12.This is completely nontrivial and something you probably can't know.We just have to learn, but when you think about it, you realize this is the most general caseof a vehicle that can move in 3D at every possible location,every possible orientation at any velocity.That's called the dynamic state of a free-flying object.Let's talk about localization of a car like our DARPA Urban Challenge car Junior.This car uses a map of the environment--it knows it advance where the lay markers are--and uses probabilistic localization to keep track of where it is.The reason for that is it could use GPS, the global positioning system,but that has enormous errors, sometimes in order of 5 or more meters,which is very unsafe for driving.By localizing utilizing particle filters or histogram filters,our car can do the same with about 10 cm error,which means it can really understand where to stay in the lanejust by known where the lane is in advance and using localization techniqueslike the ones we'll discuss right now.Let's talk about particle filters for localizationthat is commonly called Monte Carlo localization.We learned in the particle filter lesson that the state is retained by a set of particles.Each particle is a three-dimensional vector here,comprising x,y, and the heading direction theta,as indicated by these little arrows that I'm going to just draw here.A set of particles like these would be a representation for the distribution at any point in time.Now let me look at the 2 main steps in particle filters.On is the prediction step, and one if the measurement step. Let's start with prediction.Just to make things simpler, let's assume our vehicle has only 2 wheels.It's called a differential-drive robot, and it can navigate by moving both wheels forward,but if 1 wheel moves faster than the other one, it'll turn.Let's understand how to apply a particle filter to a robot on that simplicity.This is simpler than a car, but not much simpler. It's about the same complexity.As I said, the state of this vehicle is given by the following 3 values: x, y, and θ.And to predict the outcome of an action, we need to write a function that predicts those values based on values Δt over here where Δt might be a 10th of a second.Now the math for this in first approximation is very simple.It turns out this approximation is good enough to do pretty much anything in robotics even though it is not very accurate.Let's assume the robot just keeps moving forward at a fixed velocity v.Then the new x is given by the old x plus the progress it makes along axis x with velocity v.So you get v times Δt, which is the total distance traversed,but the x portion of it is cos θ.Similarly, for the y coordinates, you get the old y plus the distance traversed--velocity times Δt times sin θ.This is a robot that doesn't really change heading directions,and it'll be sufficient for very small Δt to assume that robot doesn't change heading directions.These are actually good equations even if the robot is turning.However, to understand the change of heading direction, we also have to assume that there is an angular velocity, and we call this ω [omega], which is a Greek letter.So the new heading direction is the old one plus ω times Δt.These are really nice equations to model relatively complex smaller robots.They're really simple geometry. If you understand cosine and sine,you realize this is basically a robot that moves on a fixed straight trajectory.For time Δt it then applies the rotation and it moves again for fixed time Δt on a straight trajectory,which is an approximation to the actual curve the robot might be taking.Let's exercise these equations over here using the following example.Suppose in the beginning x equals 24, y is 18, and the orientation for now is going to be zero, just to make it simple,and suppose Δt is 1 second, our velocity is 5 units per second,and our rotation velocity is π over 8 seconds.Can you use this formula to calculate x', y' and θ' after Δt?This is a robot that points in the x direction, because θ equals zero.We have a coordinate system over here where this is 24.That's 18 in the y direction, and it moves forward for a while.In fact we have 1 second, and it moves at 5 units per second, so this is 5.Then it turns its heading into this direction over here, and this is π/8.Again, the real robot would take a curve, but in our approximation we assume it goes in a straight line and then finally does a very discrete turn, which is an approximation,but that's the question I have asked you here.When you plug these values in, you'll find that the x extends by 5,which is 29. Y doesn't change.The final turn is π/8, which is about 0.3927.Let me ask you a similar quiz,but this time let's say that x equals zero, y equals 10, and our heading direction is π/4,which is the same as 45 degrees.Again Δt equals 1 second, and we move at 5 units per second.Then we turn by -π/4 per second.Don't worry about the units, seconds. It's just hear to make it mathematically consistent.So please plug your best estimates into the boxes over here.This robot is located at 0, 10 and it points at 45 degrees.In doing so, it moves some right and some up.In fact the same ratio in the x dimension as it will be in the y dimension.Now cosine of this θ here, is about 0.7071 multiplied by 5 added to 0 and you get 3.5355.It so turns out that this is also the value of sin θ,so you can add the same value over here to the x value of 10, which gives us 13.5355.Finally, we find that within 1 second the initial heading is canceled out by the change of heading,so we'll be facing in this direction over here, and that angle is just zero.If you got those right, this was a somewhat tedious exercise on simple geometry,but this is the kind of math you need to implement in particle filter equations like those over here in robotics.Let's get back to Monte Carlo localization.Let's look at single particle that sits over here.There's an x, y, and a θ.Let's assume we happen to know that the robot is moving at velocity vand at angle velocity ω, which is the differential of its wheels.And after it moves so far, it will end up exactly over there with a heading pointing in this direction.Now that you worked the math, you know exactly how to implement this.In Monte Carlo localization, we don't predict the exact outcome. We add noise.We add noise to velocity v and to the heading direction ω.As we do so, we might find ourselves with lots of particles, all of which have a slightly different xy coordinate and a slightly different heading outcome.These particles together comprise our estimation after the motion command over here.So, a single particle over here, if drawn multiple times, gives a set of particles like the ones over here.They're kind of hard to see at this point,but you can imagine by varying v and ω with a little bit of noise that weadd or subtract from these values,we will get slightly different predictions where the robot might beand as a result get a particle cloud like this one over here.That's really important.We just implemented the prediction step of a particle filter in a real robotics example.This is exactly what's happening when we drive our Google self-driving car and our Stanford car.Now we have a set of predictions that might arise from a single particle,and the other important step in particle filtering is the measurement step.We need to understand at what rate will these particles survive,and that's usually done in proportion to the measurement probabilities.Let's talk about measurements. For the sake of this exercise, let's assume we only have 2 measurements.We would either see something bright or something dark.It does a certain response to whether it's on land marking.Just for simplicity, let's assume we have certain locations that have land markings,like this one over here and these over there.If a robot center is aligned with a lane marking, it should see a bright spotbecause lane markings tend to be bright.But if it's off the lane marking on the regular road, it should see a dark spot.Let's turn this into a probability that's called the measurement probability.The probability of seeing something bright is going to be large when it's on a lane marker, say 0.8.From that we can deduce that the probability of seeing something dark on a lane marker is 0.2.The probability of seeing something dark when off a lane marker is even higher at 0.9,and from that it follows that the probability of seeing something bright on the regular road with regular pavement is going to be 1 minus 0.9 equals 0.1.Here's my quiz for you. This is an entirely nontrivial quiz.If you get this right, you understand particle filters.Suppose we measure bright.The actual sensor told us it saw something bright underneath the robot.I'd like to know what is the importance weight of the particle over here,which we're going to call x1, and the particle over here, which I'll call x2.Tell me what's the weight w of x1 after I apply the measurement probability and the normalization that's common in particle filters.Please do the same for the particle x2 where x1 happens to be on the lane marker,and x2 happens to be off a lane marker.So please put in these two numbers of over here.It'll take a while to calculate those, so please take the time.I assure you if you get those right, you really understand particle filters.As promised, the answer is nontrivial.The importance weight for x1 will be 8/27, which is the same as 0.2963,and the one for x2 will be 1/27 or 0.037.How did we get there?Let's look at the non-normalized importance weights before normalization.The guys on the lane markings will all get a 0.8.The guys off the lane markings will get a 0.1. So the three guys over here.The reason is the probability of seeing bright, which is what we saw,off a lane marker is 1 minus 0.9. That's 0.1.Now we have 3 guys that are on the lane markings and 3 off the lane markings.The total weight over here is 2.4, and the total weight over here is 0.3.Our total weight for all particles, not normalized particles, will be 2.7 or 27 tenths.We have to really normalize the weight by dividing by 2.7.0.8 divided by 2.7 is 8/27 or this number over here.0.1 divided by 2.7 is 1/27, which is this value over here.That's how we get to these final weights.If you got this, you understand that the measurement probability effects the weight before normalization, which is multiplying in the measurement probability,and you did this correctly.Afterwards we have to normalize to make sure all the weights add up to 1.So we divide by the total weight of all particles,and we get out those probabilities over here.Put differently, this particle x1 that sits on a lane markeris being regenerated in the resampling phase with a probability of 0.2963.The same is true for the 2 other particles that sit on lane markers.For the 3 particles that are off lane markers like the one x2 over here,the resampling probability is a small as small as 0.037.That's true for x2, but it's true for all 3 particles.In total, these probabilities add up exactly to 1.Let's now apply the resampling where the on-lane-marker particles are being resampled for probability 0.2963,and the ones in the middle with probability 0.037.Let's draw with replacement 6 new particles.A typical outcome will be we draw this one over here twice,this one down here twice, and this one over here once.Perhaps we draw once over here.Clearly we draw the particles that sit on the lane markings much more frequently than the ones that sit off the lane markings for a total of 6 new particles.We now apply our resampling method whereby we draw twice from this particle over here.That might give us something over there and over here, given that we add a small amount of noise.The guy over here will be resampled to something over there. Same with this guy over here, and this guy might find itself over here.The set over here of 6 particles in total, will now be the new posterior.As you can see, this posterior is more consistent with the lane marking observationthan the one of not being on a lane marking by virtue of the fact thatwe saw a bright measurement before.Now we just repeat. We look at the next measurement. We weight particles accordingly.We resample. We do forward prediction.That's the basic particle filter algorithm.4Look at measurment, compute weights, sample, and predictwhere the prediction has a certain amount randomness.If you get that loop implemented, you've implemented an amazing algorithmthat's exactly what has driven many of my robots in the ability to localize themselves.Obviously they have more than just 1 pixel sensor that measures bright and dark.They might take an entire road image and use the road image to complete the measurement probability,but the basic mechanics is exactly the same as shown over here.So let me ask you, did you actually understand this. Yes or no?I just hope you answered "yes."If you answered "no," please go through the same sequence again,because the steps end up to be relatively straightforwardeven though they're somewhat uncommon.But if you understand it, you can now go and implement particle filtersfor the great range of robotics applications.Let's talk a bit about planning.One of the key problems is that these robots have to decide what to do next.I'll address the planning problem at multiple levels of abstraction.The easiest happens to be to look at a city level of abstraction.Suppose we have a road like this over here,and my car is located down here in the beginning.I wish to get to this point up here.Let me draw in an abstraction of the state space, and we just draw the states as shown with these red lines over here.So you see a maze with lots of discrete states.I'm going to ask you given that you traversing from red cell to red cell costs you 1,or -1 using the definition of the MDP lecture before.Suppose the goal state has a value of 100.What's the value of the start state assuming deterministic actions?You probably got it right. It's 86.The reason is the value of the goal is 100, 99, 98, 97.It turns out the start state is 14 steps away from the goal,so 100 minus 14 is 86.The exact same algorithm works beautifully for planning the shortest path to a single mission goal from any possible start location,and the only difference here is in this graph over here of an actual road graphwe are also incorporating the heading direction as measure of distance.Green corresponds to nearby to large values, red to far away.The reason why the area below the mission goal is green is because we expect the car to point up, to point north, at the time it reached the mission.So if it came from the north, it would point in the wrong direction.The state space is augmented correspondingly.Where if it comes from over here, it points in the correct direction.If you look at the circle over here, it's interesting.If we came from the left side over here, it could do a right turn,but over here it's forced on this one-way circle to do the entire loop to go around,and that increases the value as it comes over here.This is value iteration applied to the road graph where we keep track of headingand where the circle over here is a one-way circle.Let's look at dynamic programming again.Specifically let's look at environment that has a loop.Here's the environment.The environment possesses 14 states,and here is the loop as indicated, and there is a big intersection in the middle over here.Let's assume this is our start state in the very south, and we're facing north.We we reach to the goal state in the west, and here we will be facing west.Obviously, there are two ways to get to the goal.We can go north 3 steps and then turn left to the goal,or we can take the entire loop over here, avoid left turns,but eventually find ourselves at the goal as well after more steps.Let's assume there are different costs attached.The cost of motion is -1 per red cell.The cost of right turns is -2.Why would be penalize right turns?Well, as you turn right, you might have to yield for pedestrians.That might cost you some time.Let's assume an expectation that -2 accounts for the time relative to the motion of -1.What I'd like you to know is a tricky question.What is the max cost of a left turn?If we avoid left turns altogether, we'd much rather take the loop over here.I want the solution where you turn left to be distinctly more expensive, or more negative,than the solution where you turn right.When I say "max", we're dealing with negative numbers.So if you would were to look into positive numbers, what's the minimum cost of a left turn?But I want you to enter the negative number over here.[Thrun] And the answer is -15.If you look at the pure motion cost for the short route, there are 6 steps.So we wanted to turn left.You pay the penalty of -6.The longer route is 14 steps, and we add a penalty of -6 for 3 right turns.Each is -2. That gives us -20.If we penalize left turns with -15, then the total will be -21,which is smaller or higher cost, so to speak, than the alternative routethat we wish to favor.For anything larger than -15, we either have a tie or we just go the shortcut,so that's the correct number over here.It was a really nontrivial question. I'm really proud if you got this right.[Thrun] So let me give you some examples of this method in action.Here we have an actual planning technique that uses dynamic programmingand understands how far things are away.And on top of it, it also considers local rollouts to avoid local obstacles.These local rollouts are continuous trajectories.They are variant by discrete decisions,like whether to change the lane, and by various small discrete nudges around obstaclesso we can avoid obstacles.And in rolling out to a certain horizon, like up to here,and then connecting to the dynamic programming value,we can calculate in actual traffic situations what is the cost of going a certain path.Here is an attempt to turn right.You can see the vehicle approaching a stop sign.There is an entire maze of streets.The best way to go right and then into the left lane is to take the right turnand then initiate a lane shift, which is happening right now,to reach a target location that is indicated by this big orange circlethat it's crossing right now.If we increase the cost of a lane shift to a much larger value,the answer that emerges is really interesting.It doesn't turn right because of the cost of the subsequent lane shift.Instead this vehicle goes straight,takes a left turn, which happens to be much cheaper than the lane shift.It then follows this left turn, takes another left turn,and eventually takes a third left turn just to get to the left lane.And if you look very carefully, you can now reach the goal locationwithout a lane change maneuver which would have much higher cost.So here it is now in the left lane, and without lane-changing maneuverit manages to reach the goal.This illustrates that dynamic programming in the context of controlling actual carshas a big value to play.Here is the same idea applied to a passing maneuver in normal driving.You see our vehicle following another vehicle. This is actual data in preparation for the Urban Challenge.Now we placed an abandoned vehicle on the left lane,and you can see how trainers are being made that incorporate dynamic obstaclesby virtue of those rollouts and a dynamic programming functionby virtue of the background green to red to find the optimal actions.This method has really enabled us to navigate complicated situations with self-driving cars.[Thrun] The last example I want to talk about in this lectureis related to general purpose path planning where we don't have a road network.Here is an example of where this occurs.This is an example of where a blockage occurs.None of the preplanned paths are drivable by our robot,so it has to, after a certain time out here--20 seconds--find itself a path anywhere in the environment.In fact, our Urban Challenge car did just this.We don't do this today in traffic. It's a little bit dangerous.But for the Urban Challenge it was perfectly doable, and it was safe.So this car found a route that was outside any pre-given corridor.Here is an even more challenging examplewhere our robot Junior approaches a complete road blockage,but its target location is behind the road blockage.You can see that none of the paths can possibly make it thereand the only correct action is to turn around and pick a different roadto finally approach the goal location from the opposite side.You can see an attempted lane shift to the opposite lane doesn't function either,and there is time models supposed to be with all of those.Eventually, using a general purpose plannerof the type that Peter talked about in his classto find what ends up to be a really complicated multi-turn turnaroundwhere the car backs into a driveway a little bit, as you can see over here,and it is all planned completely dynamically without any preconceptionhow such a multi-point U-turn would look like.Then it goes forward, then it goes backward, and does so multiple timesuntil it finally has turned around.It's not particularly efficient or elegant, but it's very, very safe.This vehicle will eventually be able to drive in a different directionand reach the goal point behind the blockage.That was one of the tasks DARPA gave us.So you can see it do its job until it finally breaks freeand is able to navigate around this blockage onto a different street, as shown over here.[Thrun] So let's talk about robot path planning or robot motion planning,which is a rich field in itself, and I can't give you a complete surveyof all the algorithms involved.But one of the key differences to the planning algorithms we talked about beforeis that now the world is continuous.For example, we learned about A* in which we discretize the world.We might have a goal location, we might have obstacles,and then A*, a valid action sequence, might look like this.And even though this is a valid solution to the planning problem,a car can't really follow these discrete choices.There is a number of very sharp turns over here that are just irreconcilable with the motion of a car.So the fundamental problem here is A* is discrete,whereas the robotic world is continuous.So the question arises, is there a version of A* that can deal with the continuous natureand give us provably executable paths?This is a big, big question in robot motion planning.Let me just discuss it for this one exampleand show you what we've done to solve this problem in the DARPA Urban Challenge.The key to solving this with A* has to do with the state transition function.Suppose we have a cell like this and we apply a sequence of very small step simulationsusing our continuous math from before.Then a state over here might find itself right here in the corner of the next discrete state.Instead of assigning this just to the grid cell,in the algorithm called hybrid A*, it memorizes the exact x prime, y prime,and theta prime and associates it with this grid cell over herethe first time the grid cell is expanded.Then when expanding from this cell it uses a specific starting point over hereto figure out what the next cell might be.It might happen that the same cell is reached again in A*, maybe from over here,leading to a different continuous amortization of x, y, and theta,but because in A* we tend to expand cells along the shortest pathbefore we look at the longer paths, we now just cut this offand never consider the state over here again.This leads to a lack of completeness, which means there might be solutions to the navigation problemthat this algorithm doesn't capture,but it does give us correctness.So as long as our motion equations are correct, the resulting paths can be executed.Now here is a caveat.This is an approximation and is only correct to the point that these motions equations are correct that are not correct.But nevertheless, our paths that come out are nice, smooth, and curved paths,and every time we expand a grid cell we memorize explicitly the continuous values of x prime, y prime, and theta with this grid cell.[Thrun] Now here is an actual result of applying this A* algorithmfor our vehicle that sits over here.Real obstacles--these are laser scans of parked cars--and a target location over here.And while the curve isn't super smooth,you can still see it is able to find a continuous and drivable curveto the parking location over hereby this small but important modification of A*.There are a few other modifications of A* which I can't go into detail,but here you can see a typical attempt of a robot to navigate a parking lothere in simulation.You can see the tree that is being expanded in that search.And every time it gets stuck, it does a new A* search.You can see how the map is being acquired as the robot moves.In its state that's in front of the robot, it not only considers the x, y and hidden directionbut also allows the robot to go forward and backwards,and driving backwards is just a different state than going forwards.Now you can see how it backs up, finds a new path, and it is an incomplete mazeuntil it finally is able to reach the goal location through an actual opening.We made this maze really hard to test our algorithms.The nice thing is these algorithms work almost real time.It takes less than a tenth of a second to build this entire search tree,and the robot is able to navigate this parking lot really, really efficiently.This was one of the fastest motion planning algorithms that I sawin the DARPA Urban Challenge.In fact, in all of robotics it's been one of the fastest algorithms I've personally seen in my life.Here is the same algorithm applied to an actual parking example using our robot Junior.It's driving over here, it wishes to get over there,and you can see it has backed up into a parking gap over here,which is an amazing precision for a robot, and then moved forward along the line over here.Our state space is, I guess, 4-dimensional.It comprises x, y, hidden direction, and whether the car is going forward or backwards.There is a cost to changing directions, so it doesn't change direction too often.You can see it navigate to its target location.Details I am not telling you include that the trajectory that the planner generatesis subsequently smoothed using a quadratic smootherso that we get rid of the kinks,and the car drives much nicer as a result.But the workhorse here that does all the work to find the best pathis actually A* modified into hybrid A*, as I told you.And in this final video we see the car navigating a parking lot with lots of traffic cones.On the left you see the video imagery, on the right side you can see the internal mapand the path planner,and it attempts to park itself in the designated spot on the left.[Thrun] So this finishes my short lecture on robotics.Obviously the field is much, much bigger than what I just showed you,but I gave you examples of the key elements.I gave you an example of perception using particle filters.I gave you an example of planning using MDPs and also A*.These are some of the key methods we've been applying to self-driving cars.There are many other methods.Most notably, there's also reinforcement learning that has recently received a lot of attention.But I don't have time to talk about those.I hope you are able to apply these methods yourself to pretty much any robotics problemsthat you might be working on.Robotics is really a fascinating area.There's a lot of things to learn--way too much than I can offer in this single class.But what you should have noticed is there's a really strong interplaybetween artificial intelligence and the methods I showed you beforeand what we are doing, for example, to make cars drive themselves.Now, in the next class we'll talk about a topic that's equally important,which is natural language processing.So when you learned today a little bit about how to build self-driving cars,next lecture you might actually learn how to build the next Googlewhen Peter Norvig tells you all about natural language processing.So please come and see us again when the next class comes up.So, this question I want to test your knowledge aboutthe dimension of the state space of a dynamic system.In all these questions, I'm going to look at a ball, like a soccer ball.The interesting thing about a soccer ball is its orientationis an important variable, whether it's upside down and so on.So, in all of these questions we're going to explore this,and the fact that this object is rotationally invariant.Let me start with a simpler question,which is the kinematic state, which is the state without any velocitiesof the soccer ball, where it is on the ground.And I'll follow up with a question with the same kinematic stateof the ball in mid-air,the difference being between these 2 questions that on the groundit's confined to be in a 2-dimensional ground plane,whereas in mid-air, you might add another dimension or not.There's also the dynamic state on the ground and in mid-air.And for the dynamic state, I wish to ignore things like spin.I just care about velocities as a person really far away could observe them, just to make things clear.And finally, I'd like to include spin, so let me take the most complicated situationof dynamic state in mid-air considering spin.The last one is really a tricky question,so I don't mind at all if you get this wrong.But in all of those, I would like to exploit the factthat I really don't care about the absolute orientation of the soccer ball that is here,so it's invariant to its orientation, but it might still have spin.And the answer is 2, 3, 4, 6, and 9.On the ground, the static state without velocity is just X and Y.That's 2. If we add mid-air, we have height, which adds a third dimension, 3.If we add the dynamic state to the ground, which is data X and data Y over time, that's 4 in totalplus the original X and Y. Same for mid-air. Multiply 3 up to 6. And the last one is the tricky one.Clearly, a helicopter in mid-airthat looks at rotational velocities would have 12 dimensions.But again, because I don't care about the absolute coordinatesof its yaw, roll and pitch.The ball is variant.The spin variables are 3, data roll, data pitch and data yaw.If we add those to the dynamic state in mid-air,we get 9 and not 12.Once again, these are X, Y and Z: data X, data Y, data Z over time,and the velocities in the different rotational directions,which make a total of 9.This will be in dynamic programming question for a robot with 3 coordinates, X, Y and theta,even though in this diagram I just show 2.Suppose our target location is in the top right cornerfacing east as shown over here.Initially, the robot's location is in the bottom left corner facing north.Suppose our goal is worth 100.Going straight costs us -1,and we can turn on the spot, but turning on the spotcosts us -5.What would be the value of the start state?And the answer is 88.It takes 7 steps to go from start to goalif we just count the go straight steps.1, 2, 3, 4, 5, 6, 7.And we have to turn once in this spot right over here,which costs an additional -5, so we pay a total of -12.That plus 100 gives us 88.Same situation as before.We'd like to go from here to here.We have a 3-dimensional state space.The goal is worth 100.A straight motion costs -1.Turning on the spot clockwise costs -10,but turning counterclockwise, which is "C-CW," costs us 0.What is now the value of the start state?Please put your answer in here.And the answer is 93,the reason being there's 7 straight steps to the goal.You can go down here or up here.And suppose we go up here and we wanted to turn clockwise,which is the one that gets us oriented towards the goal.But that costs us -10, but we can turn 3 times counterclockwise.We first turn left and down and then right.And the total cost of this is 0 because each counterclockwise turnis worth 0, therefore, we just go straight to the goal,and we only pay the straight motion cost, which is 7 in totalbecause it's -7 for the straight penaltyplus 100 is 93.This is a particle filter question where we start with a single particle over here facing east.The particle has an X, a Y, and a heading direction,and this particle is on a checker board with black squares and white squares.Let's assume we draw 5 new particles from this particle for the motion of going right,and they end up as indicated over here.Each of these 5 new particles--1 of which falling into a2, 2 of which falling into b2,1 into c2, and 1 into b3--each of these particles will obtain an importance weight,as now that what we'll measure is on a black square.So the measurement is black.To calculate the importance weight, let me tell you that the probability of seeing blackon a black square = 0.8,whereas the probability of seeing black on a white square is only 0.1.So I want you to tell me the total importance weight that falls to a2--here we just have a single particle--into b2--please add the importance weight of both particles--c2, and b3. Please add your numbers over here.The answer is 1/19th for a2, c2, and b3, and 16/19th or 0.8421 for b2.To see, let's associate the nonnormalized probability value to each of the particles.Over here, we have 0.1. Here is 0.8, but if 2 particles, let's make this 1.6.0.1 and 0.1 again.These nonnormalized importance weights add up to 1.9,so the desired result is the division of the original particle weights by 1.9,which is the value as shown on the left.In resampling in the next motion step, let's assume the following 3 particles are usedwith the other ones are being ignored.2 of them live in b2, 1 in c2, who again moves right,and we get particles distributed as follows--2 fall into b3, 2 into b4, and 1 in c4.So using the same measurement probability as before,and now a measurement of a white square.Tell me what the cumulative importance weight for the 3 new squares,where 2 of the particles fall into b3, which happens to be a white square,2 into b4, which happens to be a black square,and 1 into c4, which happens to be a white square again.The answer is 18/31, which is approximately 0.5806.4/31 is 0.1290,and 9/31, which is half of the thing over here, 0.2903.And again, we look at the same as before.We look at the total nonnormalized measurement probabilities for our particles.In a white square, the probability of seeing white is 1 - 0.1, that is 0.9.Since we have 2 particles, the nonnormalized cumulative particle weight is 1.8.Doing the same for the black square, the probability of seeing white is 0.2,which is 1 - 0.8.We have 2 particles in the black square to get a nonnormalized total probability of 0.4, so you get a nonnormalized total importance weight of 0.4.And finally, the probability of seeing white in the white square is 1 - 0.1 is 0.9,and here we only have 1 particle, so the nonnormalized importance weight is 0.9.If you add those up, we get 3.1.So we have to divide all of those by 3.1.So 18/31 is the 1st one. 4 by 31--the 2nd, and 9 by 31--the 3rd, as indicated on the left.Our robot, Stanley, performed as follows in the DARPA Urban Challenge in 2005.He came in 1st, 2nd, 3rd, or 4th or below in the ranking.And oh, my God! Yes, we came in first!It was one of the most amazing events in my entire life.Our robot made it first across the finishing line.In this final question, I'm going to quiz you about our approximate motion model for this robot, which I restate.I'd like you to apply this exact motion model over here even though you might be suspiciousof its accuracy.Suppose a time, t = 0, coordinates are 0, 0, and 0 for all 3 variables.Delta t = 4. I'll admit the units over here.v = 10 and omega = pi/8, which is 22.5 degrees in degrees.Assuming you run one these simulations exactly every 4 time steps,I would like to know what the mobile state is after 4 of those updates, or put differently, total time of 16.So please tell me, what x will be, y, and theta.The answer surprisingly is 0, 0, 0. It just survives the way it was before.To see this, [ ] initially faces east.Next direction, it'll move forward 40, and then its heading direction changesby 4 x pi/8, which is pi(1/2), so it's going to start pointing up.It repeats the same action 3 more times, and eventually arrives at the original locationand points right again.So this is a square motion. The results are in the exact same initial state as it started out with.Now in reality, if we didn't use these equations over here, it would be on a circle, but even in a circle, it would arrive back at the original location with those parameters shown over here. So the fact that our simulation simulates a square doesn't effect the end result of this question,and I didn't even ask about the circle. I just asked about applying those equations over here, so 0, 0, 0 is the correct answer.Welcome back. We're down to our final main unit.This one is on natural language processing--that is, understand natural languages like English or German or Frenchand figuring out what to do with them.Now, this is a very interesting topic for three reasons.One is a philosophical one--we as humans have defined ourselves much in terms of our ability to speak with each other and understand each other.This ability to use language is something that we feel sets us apart from all the other animals and from the other machines.Second is in terms of applications.We really would like to be able to talk to our computers and use them for various things.Sure there are occasions where clicking with a mouse is the right thing to do,but talking is natural, and we want to be able to communicate with our machines.Then third is in terms of learning.We want out computers to be smarter,and much of human knowledge is written down in terms of paragraphs and sentences of text,and not in terms of formal databases or formal procedures written in code.It's all in text, and if we want our computers to be smart,they'd better be able to read that text and make sense of it.That's what this lesson is all about.We'll start by talking about language models.Historically, there have been two types of models that have been popularfor natural language understanding within AI.One of the types of models has to do with sequences of letters or words?These types of models tend to be probabilistic in that we're talking about the probability of a sequence,word based in that mostly what we're dealing with is the surface words themselves,and sometimes letters.But we're dealing with the actual data of what we see,Rather than some underlying extractions,and these models are primarily learned from data.Now, in contrast to that is another type of model that you might have seen before,where we're primarily dealing with trees and with abstract structures.So we say we can have a sentence, which is composed of a noun phrase and a verb phrase,and a noun phrase might be a person's name, and that might be "Sam."And the verb phrase might be a verb and we might say "Sam slept"--a very simple sentence.Now, these types of models have different properties.For one, they tend to be logical rather than probabilistic--that is whereas on this side, we're talking about the probability of a sequence of words,on this side we're talking about a set of sentences and that set defines the language,and a sentence is either in the language or not.It's a Boolean logical distinction rather than on this side a probabilistic distinction.These models are based on abstraction such as trees and categories--categories like noun phrase and verb phrase and tree structures like thisthat don't actually occur in the surface form, so the words that we can observe.An agent can observe the words "Sam" and "slept,"but an agent can't directly observe the fact that slept is a verb or that it's part of this tree structure.Traditionally, these types of approaches have been primarily hand-coded.That is, rather than learning this type of structure from data,we learn it by going out and having linguists and other experts write down these rules.Now, these distinctions are not hard to cut.You could have trees and have a probabilistic model of them.You could learn trees.We can go back and forth, but traditionally the two camps have divided up in this way.Now, we've seen probabilistic word models before.If you remember when we were doing machine learning, we talked about the bad-of-words model.What I'm showing you now is a copy of a bumper sticker that my friendOthar Hansson, who is one of the main engineers on the Google search team came up with,the bumper sticker, of course, says "Honk if you love the bag-of-words model,"but it says that in a way where the words are a bag rather than a sequence.This just kind of indicates the power of the model--that it gets the idea across while loosing all notion of sequence,and thus making the probabilistic model simpler to deal with.But we can move on from a bag-of-words model, which we can think of as a unigram--sometimes also called a naive Bayes model,because every individual word is treated as a separate factor that's unrelated or unconditionally independent of all the other words.We can move beyond those types of models to ones where we do take sequence into account.What we want then is a probabilistic model P over a word sequence,and we can write that sequence word 1, word 2, word 3, all the way up to word n,and we can use an abbreviation for that and write that the sequence of words 1 through n, using the colon.Now the next step is to say we can factor this and take these individual variables write that in terms of conditional probabilities.So, this probability is equal to the product over all i of the probability of words of igiven all the subsequent words.So that would be from word 1 up to word i - 1.The is just the definition of conventional probability--the joint probability of a set of variables can be factored out as the conditional probability of one variable given all the others,and then we can recursively do that until we've got all the variables accounted for.We can make the Markov assumptionand that's the assumption that the effect of one variable on another will be local.That is, if we're looking at the nth word, the words that are relevant to thatare the ones that have occurred recently and not the ones occurred a long time ago.What the Markov assumption means is that the probability of a word i,given the words all the was from 1 up to word i minus 1,we can assume that that's equal or approximately equal to the probabilityof the word given only the words from i minus k up to i minus 1.Instead of going all the way back to number 1, we only go back k steps.For order 1 Markov model, for an order k equals one, then this would be equal tothe probability of word i given only word i minus 1.Now, the next thing we want to do is in our mode is called the Stationarity Assumption.What that says is that the probability of each variable is going to be the same.So the probability distribution over the first word is going to be sameas the probability distribution over the nth word.Another way to look at that is if I keep saying sentences,the words that show up in my sentence depend on what the surrounding words are in the sentence, but they don't depend on whether I'm on the first sentence or the second sentence or the third sentence.Stationarity assumption we can write as the probability of a word giventhe previous word is the same for all variables.For all values of i and j, the probability of word i given the previous wordas the same as the probability of word j given the previous word.That gives us all the formalism we need to talk about these word sequence models--probabilistic word sequence models.In practice there are many tricks.One thing we talked about before, when we were doing the spam filterings and so on,is a necessity of smoothing.That is, if we're going to learn these probabilities from counts,we go out into the world, we observe some data, we figure out how often word i occurs given word i - 1 was the previous word, we're going to find out that a lot of these counts are going to be zeroor going to be some small number, and the estimates are not going to be good.And therefore we need some type of smoothing, like the Laplace smoothing that we talked about,and there are many other techniques for doing smoothing to come up good estimates.Another thing we can do is augment these models to say maybe we want to deal not just with words but with other data as well.We saw that in the spam-filtering model also.So there you might want to think about who the sender is,what the time of day is and so on,these auxiliary fields like in the header fields of the email messagesas well as the words in the message, and that's true for other applications as well.You may want to go beyond the words and consider variables that have to do with context of the words.We may also want to have other properties of words.The great thing about just dealing with an individual word like "dog"is that it's observable in the world.We see this spoken or written text, and we can figure out what it means, and we can start making counts about it and start estimating probabilities,but we also might want to know that, say, "dog" is being used as a noun,and that's not immediately observable in the world, but it is inferable. It's a hidden variable, and we may want to try to recover these hidden variables like parts of speech.We may also want to go to bigger sequences than just individual words,so rather than treat "New York City" as three separate words,we may want to a model that allows us to think of it as a single phrase.Or we may want to go smaller than that and look at a model that deals with individual letters rather than dealing with words.So these are all variations, and the type of model you choose depends on the application,but they all follow from this idea of a probabilistic model over sequences.Now, we talked about using language for learning,and this slide is demonstrating the power of language,how it has such a powerful connection to the real world that allows us to learning thingsjust by observing language use.What I've done here is I've gone to Google trends and types in two search terms--"full moon" and "ice cream" and gotten back a graph of how popular those queries are over time.We also have a graph of the news volume for those terms over time,but that's not so interesting here.What's interesting this side is the regularity in these patterns.This pattern allows me just by observing language to do amateur astronomy. What do I mean by that?Well, look at the curve for ice cream.Popular in the summer. Not so popular in the winter.What that means is if I wanted to figure out what the rotational period is of the earth around the sun,all I have to do is measure these peaks, and it would come out to 365 days--a very regular performance of language speakers using the term "ice cream" in the summerrepeatedly year after year.Now, there's a little bit of a blip here.What happened in this case?Well, it turns out that a manufacturer of cell phone operating systems decided to call the latest update to their operating system "Ice Cream Sandwich,"and so there was a lot of searching for that when it came out.But that just lasted a few days, and then things went back to normal.Similarly for the query "full moon" in blue, we see this period here,and we can measure that period to be 28 days,we we can do amateur astronomy and figure out how the moon works as well,just by observing how people in the real world use language.What can you do with language in the world besides amateur astronomy?Well, I haven't told you yet, but I want to give you a little quizand allow you to guess.And so for each of these applications here, I want you to tell me whether you think that language models, and specifically these types of word models,would be a major part of an implementation of that task.Almost all of these are great examples of applications that are used everyday,and are primarily based on word models.We've seen classification before for spam and other types of categories,language ID and so on.There's also the idea of clustering, where we don't have categories ahead of time.Yes, you can take news stories and classify them into, say, sports or weather,but you can also cluster them to say here's a cluster of stories that are all related about the latest topic that has maybe never occurred before.There's also input correction of various kinds such as spelling correction,sentiment analysis--taking reviews of products and trying to decide if they're favorable or unfavorable and rate products that way,information retrieval--web search is a problem that I've worked on and is primarily addressed with word models,question answer such as IBM's Watson did in playing the game of Jeopardy.They use a variety of techniques.Much of them are based around word models, but there are other techniques as well.Machine translation--we saw the example of Chinese menus and translating to English from examples.The examples are primarily dealt with by phrases and individual words and some augmentation to that as well.Speech recognition--as similar story.And then finally I threw in one that is no primarily a question for word models.Driving a car is autonomously is primarily a question in perception and localization.Yes, you might want to be able to talk to the car to direct it to do something,but that wouldn't be part of the autonomous part.Now, I wanted to show you how powerful n-gram models of language are.That is, if we're only looking at word sequences, what is it that we're giving up and what are we getting?So I read in the complete works of Shakespeare into a small computer program,and then built n-gram models and sampled from that model.That is, generated random sentences that come from the probability distribution defined by that model.And here are samples from the unigram model.That is sampling from words according to frequency in the corpus of Shakespeare text,but not taking into account any relationship between adjacent words.And looking at this, it doesn't make much sense.It does seem like real sentences.You can tell the vocabulary is somewhat archaic.You have words like "o'erthrown" and "thou" and "'tis" and so on,but you aren't really getting very much from this.Now we move to a bigram model,where we're sampling from the probability of a word given the previous word.Now we see a little bit of structure emerge.So you can see at the start of the sentences "I have", "hear you," "hark ye."The words seem to go together,but then as the sentences move on they ramble and don't go any definitive direction.So the sentences are locally consistent at the level of one or two words,but that consistency doesn't go very far.Now with the trigram models, we're starting to get a little bit more structure.In fact, we get complete sentences that actually make some sense--"I will never yield," and the exclamation "little pretty ones!"And we get sentences that are fairly coherent--"I would learn of noble Edward's sons"--but then break down a little bit--"what thing, avoid!"So we're getting a model that appears to be a little bit closer to actual Shakespearebut it's still incomplete.And finally this example based on 4-grams--Now we're seeing an even longer structure.Sometimes we have this generate something that makes sense like "betwixt their two estates,"and this is not something that appears in Shakespeare,but it was just generated and it made sense.Sometimes we get things that are actually quotes like "even to the frozen ridges of the alps."The model chose to duplicate something that is actually in Shakespeare.Certainly there were lots of traces it could have made."Alps" appears four or five times."Even" appears many, many times. "Frozen" appears multiple times.But it just happened to duplicate something that was a quotation from the original.And then there's lots of examples of sentences and phrases that make a lot of sense--"I know my duty," and "give me some little breath," and so on.So it looks like even though all we know is a sequence of 4 woeds,We're still capturing quite a bit of what it means to have coherent sentences but not everything.Here's a little quiz--for each of these pieces of text, which was generated from an n-gram model,I want you to try to guess if it was generated from a 1-gram model, a 2-gram, or a 3-gram.Now, I know you won't necessarily be able to get these all right. Don't worry about that.The main point is just for you to get some facility in kind of looking at these models and trying to understand them.I'll give you a hint--three of them were generate by a 1-gram model,three by a 2-gram model, and 3 of them by a 3-gram model,and one of them is an actual excerpt from the corpus of Shakespeare's work,And so leave that one blank. Don't mark it at all.Here we see the answers. You can tell that the 3-grams are more consistent, more sentence-like than the 1-gram model generated sentences,and this is an actual sentence or stage direction from the works of Shakespeare.Here's a quiz to make sure you understand how to calculate these probabilistic models.We're going to calculate the probability of the string "woe is me,"and we're going to calculate that beginning at the beginning of the sentence,which we'll mark with this dot character,given that we are starting at the beginning of the sentence.I want you to figure that out, put the probability in here,and it's going to be a small number with a lot of zeros to the right of the decimal place.So scale that by a factor of 1 billion--the probability times 10 to -9.Now, of course, I'm going to have to give you some data to make this make sense.I'm going to tell you that the probability that "woe" occurs at position igiven that the start-of-sentence marker occurs at position i minus 1.I should say what we're doing here is we're sort of artificially introducing a tokeninto our data of the start-of-sentence marker,which could be either what comes after a period or exclamation pointor at the beginning of the file. That all counts as a start-of-sentence marker.That probability is 0.002.The probability that "is" occurs at position i given that "woe" occurred at i minus 1 is 0.07,and the probability that "me" occurs at position i given that "is" occurred at i minus 1 is 0.0005.Tell me the probability of the whole string "woe is me" at the beginning of a sentencegiven that we're starting at the beginning of a sentence and put your answer in here.The answer is that we just multiply them together,and it words out to 7 parts per billion.I should note that these numbers are small, but that shouldn't bother you.So "woe is me" seems like a fairly common phrase.The reason it's small is because there are so many common phrases of three words.And so even though this one's fairly common, it works out to only a few parts per billion,because of the many other possibilities.Now let's take a step back for a second, and I'm going to talk about probabilistic letter models.Here we have a sequence of letters,and it looks like this sequence is rather infrequent in English.But what can we do with letter models that we can't do or that we can do in opposition to word models?The answer is letter models are very good in cases where we're going to be dealing with unique words that maybe we haven't seen before,but they still give you properties of the language.One very interesting task is language identification. Let's see how that would work.Let's take some example phrases--"hello, world," "guten tag, welt," "salam dunya,"and let's suppose you have the task of classifying these into the language from which they were sampled from,and we'll make this into a quiz, and we'll give you some choices--English, German, French, Spanish, and Azerbaijani--and tell me for each of these want your best guess is at the most likely language classification.That didn't seem too hard.This looks like English. This looks like German. I may not be familiar with Azerbaijan, but it doesn't look like English, German, French, or Spanish,so I'll probably choose that, and that would be the right answer.Now, how could I do that? Well, I could do it by recognizing some of the words.But it turns out I can also do it just by looking at letter sequences,the frequency of of single letters or pairs of letters or triplets of letters.In fact, you can get about 99% accuracy for language identification just looking at tables of letters.And a great thing about dealing with letter models is that the probability tables you need are much more compact.If you think about triples of words, there may be a million words in the vocabulary,so a table of triples is a million to the 3rd power.That's quite a number of entries.Whereas for letters in the alphabet, most alphabets have about 30 letters or so.So it's very easy and compact to store triples of those.Now, in doing actual language identification,it's also common to add other features, to not look only at the letter combinations.So you might add words as well.You might add a small number of words--the most common words in a language,or it may be even better to add the most discriminative words--words that show up in one language but not in another languageand count the occurrence of those words.In this table what I've done is I've taken samples of text in 3 languagesand just counted to the frequency of the letter bigrams,and then ordered them from top to bottom.And so for language A, TH was the most frequent letter bigram, TE was the second most frequent, and so onIn language B, EN was most popular, ER was the second most popular, and so on.And the same for language C.What I want you to do is just try to guess which language is which.Is language A English, German, or Azerbaijani?And do the same for B and C.If you're familiar with these languages, you probably could have guessed thatTH is the most common two-letter sequence in English.These look like German.These are a little bit unfamiliar, and it has a letter that doesn't show up in English and German,and so this one is Azerbaijani.Here I just wanted to show that even very short sequences can be identified quite easilywith language ID models based on letter frequencies.So for the 3-letter sequence T-H-E, I have trigram models for 3 different languages.And you can see that in language A there's a 1.1% chance of representing T-H-E,which is 4 times more than B and quite a bit more than C.For the 3-letter sequence D-E-R, that's 10 times more likely to be language Bthan to be language A and quite a bit more than language C.For the letter sequence R-B-A, that's 50 times more likely to be C than it is to be B and even moreso than A.What I want you to tell me is what are each of these languages?Where did this column of numbers come from?Is language A English, German or Azerbaijani, and the same for B and C?You can see that English is a language in which THE is very common.German is a language in which DER is more common.And in Azerbaijani, the sequence RBA is more common.Enough about letters. Now let's use all the tools at our disposal and tackle a new task--the task of classification into semantic classes.Say we're given a sequence of phrases and want to classify them into one of several categories.Here I've chosen just three--people, places, and drugs,and I have some examples of each.What would you use to do that?Well, you have a number of things at your disposal.One, you could memorize some common parts.So "Steve" and "Bill" are very common as that first word in a phrase which represents people."San" and "New" are common in places. "City" is common at the end of places.But not all these techniques will be unambiguous or 100% accurate.So for example, if you have a phrase where the last word is "grove" and the first word seems like part of a name,that could be a place, but it could also be a person's name.With drugs, it looks like maybe the letter-based approach is better than the word-based approach.They seem to have a much higher frequency of starting with "z" or ending in "x", for example,but you can imagine a classifier using the techniques that we've seen in machine learningthat takes all these features.What's the first word? What's the second word? What's the first letter? What's the last letter or the last two letters?Throw all those features in and build a classifier.Here's a quick quiz. Which of these would be a good algorithm or technique for doing this classification into things like people, places, and drugs.Could we use Naive Bayes, k-Nearest Neighbors, Support Vector Machines, Logistic Regression. Could we use the Unix Sort command or the Gzip command?Check all those that you think would be reasonably good algorithmsfor doing classification.The answer is that all of these are good, except for the sort command.That wouldn't be very good.It would maybe separate out the drugs that begin with "z" near the end of the list,so that would help, but it would just do probably about random for everything else.Now, you may be surprised to learn that the gzip command is actually pretty good as a classification algorithm. Let's try to understand that.Here I have 3 files containing a corpus of text in each of the languages that I want to classify into,and imagine these are much longer, so it gives you a good sample in text inEnglish, German, and Azerbaijan.Now I have a new piece of text that I want to classify against each of these possibilities.Well, I can do that using the gzip command.So I could issue this Unix command that says "concatenate together the new file with the English file, gzip them, compress them, then count the number of characters,and do the same for the German and Azerbaijani,and then figure out which one is shortest.In fact, when we do that with the files I've collected, it gives me the right answer.Now how does it do that?Well, you have to understand a little bit about how compression algorithms like gzip work.What they do is they take a file like this and they look for common subsequences,and they represent that in less than 1 byte.For example, I-S-SPACE would be represented by 3 bytes in an ASCII encoding,but in compressed encoding you could say, "Hey, I see that sequence here. I see it here again. It's going to show up many times."So maybe I can represent those 3 bytes just in terms of one,saying this is a common subsequence that I'm going to see again and again.Once we've done that for English, we come up with common subsequences in English.Then if we add in another file that has a lot of the same common sequences,like here it has I-S-SPACE again, then that's going to compress well with respect to this.It's not going to compress very well with respect to the Azerbaijan, because that won't have built up a code for I-S-SPACE.That will have built up codes for things like R-B-A rather than for I-S-SPACE.So it turns out that the ideas of compression and learning are actually very closely related,and they're related by information theory and this idea of entropy of an expressionor the information content.That wasn't discovered until fairly recently. The two fields had developed independently, but now they've come back together,and we understand how they relate.The next topic I want to address is called "Segmentation."This is the problem of given a sequence of language,figure out how to break it up into words.Now, in Chinese we don't have spaces between the words,and so in order to understand if the first word of this message corresponds to a single character or two characters or what,we have to be able to do the process of segmentation and figure out where they are.In English, we don't have that. Words have spaces between them.So we don't have the segmentation problem,but we certainly have it in speech recognition in languages like English,because this speech sounds are sometimes run together without pauses in between them,and there are places where we do have a language without segmentation.For example, in the language of URLsyou could have a URL like "choosespain.com",which is the travel site that tries to encourage you to choose Spain as your travel destination,but if you segment it wrong, you'd come up with "chooses pain,"which would not be the intended expression for that particular URL.So segmentation is an important problem. Let's talk about how to do it.Let's build a probabilistic word model of segmentation.By definition, the best segmentation, which we'll call S*, is equal to the one which maximizes the joint probability of the segmentation.So we're going to segment the text into a sequence of words--word 1 through word n--and find that segmentation into words that maximize the joint probability.By the definition of joint probability, that's the same as maximizing the product over the wordsof the probability of each word given all the previous words.Now this is going to be a little unwieldy to deal with, so we can make an approximation.We can say that the best segmentation is approximately equal to the one that maximizes,and what we could do here is we could make the Markov assumptionand say we're only going to be considering the few previous words.But I'm going to go all the way and make the naive Bayes assumptionand say we're going to treat each word independently.We just want to maximize the probability of each individual word regardless of the word that comes before or after it.Now, I know that that assumption is wrong and that the words do depend on the words to the right or the left of them,but I'm going to hope that this simplification is going to make the process of learning easierand will turn out to be good enough.Now for a quick quiz. For a given string--say we have this string with 12 characters--how many possible segmentations are there?How many ways can we break this up into words?And let's answer that not just for 12 characters, but for n characters in general.With n characters, how many ways of segmenting could there be?Could there be n-1 ways, n-1 squared, n-1 factorial, or 2^n-1?Tell me which of those you think is right.The answer is 2^n-1, and the way you can see that is here.With 12 characters, there are 11 spaces in between characters,and we can either place or not place a word segment in between each of the characters.And so 11 of them either occur or don't occur, so that's to the 11th.Now, 2^n is a lot.For example, if we have 30 characters in our string, then there'd be a billion possible segmentations to deal with.We clearly don't want to have to enumerate them all.We'd like some way of searching through them efficiently without having to consider the probability of every possible segmentation.That's one of the reasons why making this naive Bayes assumption is so helpful.It means that there's no interrelations between the various words,so we can consider them one at a time.That is, here's one thing we can say.We can say that the best segmentation is equal to the argmax over all possible segmentations of the string into a first word and the rest of the wordsof the probability of that first word times the probability of the best segmentation of the rest of the words.And notice that this is independent.The best segmentation of the rest of the words doesn't depend on the first word.And so that means we don't have to consider all interactions,and we don't need to consider all 2^n possibilities.So now we have two reasons why the naive Bayes assumption is a good thing.One is it makes this computation much more efficient,and secondly, it makes learning easier, because it's easy to come up with a unigram probability.What's the probability of an individual word from our corpus of text?It's much harder to get combinations of multiple word sequences.We're going to have to do more smoothing, more guessing what those probabilities are, because we just won't have the counts for them.So given this formula and given our input string--let's stick with the familiar one--we can start enumerating the possibilities for splitting up this string Sinto a first word and a rest part and figuring out the probabilities.So the first could be "n," could be "no," could be "now," could be "nowi," and so on,and then the rest would be "owis..." or starting with "w" or starting with "is"or starting with "s," and then what's the probability of the first.Well, that we get from our corpus by counting and then smoothing,and in our Shakespeare corpus "n" occurs infrequently"--about one in a million times--"no" occurs fairly frequently--about 0.004,"now" 0.003, and "nowi" doesn't occur at all, and so we'd use some factor based on smoothing.Then if we take the rest and multiply out this whole term,the best segmentation of the rest times the probability of the first that comes from this column,then that column will give us about 10 to -19 for the segmentation that starts with "n,"10 to -13 for the one that starts with "no," 10 t the -10 for the one that starts with "now,"and 10 to -18 for the one starts with "nowi."Again, that depends on exactly what type of smoothing you choose to do.But it turns out that this row here is at least 1,000 times better than any of the other segmentations.That is the segmentation that comes out "now is the time."So this model, simplified though it is, coming up with this naive Bayes assumption,gets this one right, and it does about 99% of the segmentations accurately.Here we have a demonstration that the implementation of this algorithm into actual codeis not that much more complicated than the mathematical formulas I just described to you.Here's the function segment, which takes a text, and it does what we just said.So it splits the text up into all possible first and rest components,and then the candidates will be the first word plus the best segmentation of the rest,and then out of all those candidates we just take the maximum according to the probability of the wordswhere the probability of the words is just the product of the probability of each individual word.So that's the naive Bayes assumption coming into this definition,and this is just the definition of how to split something up into a first and rest.And you can follow the links in the note to see the source code for thisand play with it on your own if you like.Now I want to give you an idea of how well the segmentation program performs.Here I've trained it on a corpus of 4 billion words--not just the Shakespeare corpus but a larger corpus,and then I give it some test cases to try to find the best segmentation.So I gave it the test case here. The program came up with "base rate sought to,"but the correct answer was "base rates ought to."In this case, it just seems somewhat like bad luck that that was the right answer,but both segmentations seem like good segmentations.Next was this trial.My program came up with "small and in significant,"but the correct answer was "small and insignificant."Here it seems like it really has erred that "small and insignificant"seems like a much better segmentation than the one my program came up with.What I want you to tell me is what do you think could help us do a better job of getting the right answer.Would it be helpful to gather more data?Check that box if you think that would be helpful.Would it be helpful to make a Markov assumption rather than the naive Bayes assumption?Check here.Or would it be helpful to do a better job with our smoothing algorithm? Check here.And you can check more than one.In this case, the problem really comes down to the naive Bayes assumption isa weak one, and the Markov assumption would do much better.It wouldn't really help to have more data or to do a better job of smoothing,because I already have good counts for words like "in" and "significant"as well as words like "small" and "and."They're all common enough that I have a good representation of how often they occuras a unigram as a single word.The problem is that we would like to know that the word "small" goes very wellwith the word "insignificant" but does not goes very well with the word "significant."So if we had a Markov model where the probability of "insignificant" depended on the probability of "small," then we could catch that,and we could get this segmentation correct.Now let's move on, and I want to do just one more example.Here's this input, and my program came up with "g in or mouse go"--a sequence of common words, but the correct answer was "ginormous ego."Again, what do you think could help us get the right answer this time?More data? Making the Markov assumption rather than naive Bayes assumption?Or doing a better job with smoothing. Check all the ones that you think might apply.Here is seems to be a problem of not enough data and not a very good smoothing algorithm.Now the problem was even though I had 4 billion words from which I trained by probabilistic model,I had never seen the word "ginormous"--not once in those 4 billion.Yet, I should be able to deal with it even if I haven't seen the word before.So having more data might mean that I would've seen "ginormous"and I could have some probability for it rather than just making the Laplace smoothing assumption.And having better smoothing could also help--maybe something more sophisticated than Laplace,maybe something that looks more carefully at the content of the word.So it might have a letter model to say these letters look common,ending in "ous"--that's a common ending in English--so this looks more like a word,even if I haven't seen it before, than some other combination of letters.Now let's do one more example of a probabilistic problem--this time, spelling correction.That is, given a word that is possibly misspelled, how do we come up with the best correction for that word?We're going to do the same type of analysis.We're saying we're looking for the best possible correction, C*,and that's going to be the argmax over all possible corrections c to maximizethe probability of that correction given the word.So that's the definition of what it means to have the best correction.Then we can start the analysis, and we can apply Bayes rule to saythat's going to be equal to the probability of the word given the correctiontimes the probability of the correction.Of course, in Bayes rule there's a factor on the bottom, but that cancels out,because it's equal for all possible corrections.So to choose the maximum, we just have to deal with these two probabilities.Now, it may seem like we made a backwards step.Here we had one probability to estimate.Now we've applied Bayes rule and now we have two probabilities we have to estimate,but the hope is that we can come up with data that can help us with this.And certainly, these unigram statistics--what's the probability of a correction?--those we can get from our document counts, so we look at our corpus.The probability of a correct word is from the data.We just look at those counts and apply whatever smoothing we decided is best.Now, the other part--what's the probability that somebody typed the word wwhen they meant to type to the word c--that's harder.We can't observe that directly by just looking at documents that are typed,because there we only have the words where we are.We don't have the intent and the word,but maybe we can look at lists of spelling corrections.So this is from spelling correction data.Now that kind of data is much harder to come by.It's easy to go out and collect billions of words of regular text and do those counts,but to find spelling correction data--that's harder to dounless you're, say, already running a spelling correction service.If you're a big company that happens to run that, then it's easy to collect the data.But bootstrapping it is hard.There are, however, some sites that will give you on the order of thousandsor tens of thousands of examples of misspellings, not billions or trillions.Now, here I show some data that I've gathered from sites that deal with spelling correction,and these are all examples of the correct spelling followed by misspelled wordsand maybe multiple of them.And from that we want to calculate the probability of a word given the correction.So for example, we would like to know what's the probability of P-L-U-S-Ebeing the word that's spelled when the correct word was "pulse."And we do have examples of that here. We have a single example.But it's clear that we're just not going to have enough to cover all the possible words we want to deal with and all the possible misspellings for those words.With only tens of thousands of examples,there are so many words in English that we're not going to have them all.Instead of trying to deal with word-to-word spelling errors,let's deal with letter-to-letter errors.And so let's not say that this is "pulse" misspelled as "pluse,"but rather let's say this is U-L misspelled as L-U.Here, let's say this is the E in "elegant" misspelled as an A.And we'll look at these types of edits from one word to another,a transposition between 2, a replacement, or an insertion or deletion of a single letter.We'll build up probability tables for those rather than probability tables for all the words.That's much easier to do with a smaller amount of data.Here's an example of spelling correction in action.Take the word w equals "thew,"and we want to find the correction c that maximizes the probability of w given c times the probability of c.We start searching for the possible corrections cthat are close to our target word "thew" in terms of added distance.That is, first we start with all possible c that are one letter away,replacing one letter, swapping two letters, inserting one letter, or transposing two letters.And here we have a list of a few of those possible corrections.So it could be "the" by deleting the "w.We could do no correction at all; we have to consider that as one of the possibilities.We could replace the "e" with an "a."We could add a "r."We could transpose the "w" and the "e."Then we look into our spelling correction tables,and again we reduce them from a word-based to a letter- or edit-based,and we say what's the probability of inserting a "w."Here we've conditioned the insert not just absolutely of inserting a "w" anywhere,but for insertions and deletions, we condition them on the previous letter.So what's the possibility of inserting a "w" given that the previous letter was an "e?"It turns out that's what the probability is,and then we go through the list.Here's replacing an "e" with an "a."That's one of the most common edits made in English,one of the most common spelling corrections.A 10th of a percent of all spelling errors are mistaking an "e" for an "a,"and similarly down the list.So we get this probability for the probability of w given c,and then the probability of the correction word c, that we just get by looking up in our corpus how many times we have seen this wordand applying whatever smoothing we're getting.Then we multiply them all out, and I've scaled these by a factor of 1 billion.It turns out with the model I've built that "thew" is most probably corrected to "the."And that makes sense.It's easy to imagine your finger slipping off the "e" key and going over tothe "w" since they're next to each other,and "w" is a very common word in English.But it's troubling that the second possibility,namely leaving "thew" alone and keeping it as is has such a high probability.Now, it turns out "thew" is a word.It's rather archaic. It does show up in the Shakespeare corpus.It has to do with muscle tissue,but it's a fairly uncommon word,and how high it ranks depends in large part on the probability that we assignto this edit of doing nothing at all.Here I've assigned it a probability of 0.95.That is, I've said for my probabilistic model,I've made this choice to say I think that about 95% of the words are spelled correctlyand 5% are spelled incorrectly.You have to make that choice in order to have a complete model.The probability distribution has to be spread out over all possible,and they have to sum up to one, so I've got to put it somewhere.If I had made another choice, then these two could have been swapped around.So the answer you get depends on the assumptions you make.Still, we can have spelling correcters that are highly accurate.This very simple model of just looking at unigram possibilitiesand looking at the edits achieves accuracy in the 80% range.If we go beyond that and start dealing with Markov assumptions and looking at multiple word sequences, then we can get up into the high 90%.Now, let me back up just for a minute and talk about software engineering in generalrather than talking about specific AI techniques.What I'm showing here is a small excerpt from the spelling correction codefrom a project called Htdig, which is an open-source search engine. It's a great search engine.If you ever have need of one, you might want to check it out.All the code is very straightforward and easy to deal with.It has several thousand lines of code dealing with spelling correction.Here we see a little bit of code. It has the good idea of saying one word might be misspelled for another if they sound alike,and so let's go through each word and figure out what each letter is sounding likeand see if there are other words that sound similar.So for example, here it's saying what does a "c" sound like.Well, "c" is ambiguous in English.It has this "x" sound, the "ch" sound, this "s" or "k" sound,and there's all these possibilities about how it can have one sound or another.Now imagine you're in charge of maintaining this program.In order for you to make sure that it's right you have to do several things.First, you could look at this comment and say, well, does this commentaccurately reflect the rules for English pronunciation?Here, it's talking about pronouncing a "c" as an "s" in the context of an "i," "e," or "y."What about the other vowels--"a" and "o?"Were they left out by accident or is this correct?So you'd have to do some work to check that out.Then you'd have to do more work to say if this comment correct,is the comment correctly implemented in this code here?In fact, just this sort of one page of code just dealing with a couple lettersis about the same as all the code that we use to implement the probabilistic model.But I think the most important difficulty in maintaining code like thisis that it's so specific to the English language.Imagine you're in charge of maintaining it, and you're boss or professor comes to you and says,"Great job. Now I'd like you to make this work for German and French and Azerbaijani and 50 other languages."You'd have to go through and understand the pronunciation rules in each of those languagesand edit a version of this code for each particular language.That would be quite tedious.But if you were dealing with a probabilistic modeland you were asked to work in another language,all you would have to do is go out and collect a large corpus of words in that language.Then you'd have the probability of the individuals words.And then find a corpus of spelling errors.Then you'd have the probability of the spelling edits.And so gathering that data is much faster, much easier software engineering processthan writing this code by hand.In sense, you could say that machine learning over probabilistic models is the ultimate in agile programming.No subtitles...No subtitles...No subtitles...No subtitles...No subtitles...No subtitles...No subtitles...No subtitles...No subtitles...No subtitles...No subtitles...No subtitles...No subtitles...No subtitles...No subtitles...No subtitles...The very first question, is a search question.You probably know about the Towers of Hanoi,if you don't then please go and Google them.It's a single player game, by which you try to move the tower of four slices over here,onto the right peg, over here.You can use the middle peg, but the rules are you can only move one disk at a time.And it might never happen, that a small disk sits below a larger disk.So the way to solve it is to move the disk over here,the second largest disk to the right side,the small one over,and the third largest to the center, and so on.If you know it, you know what I'm talking about. If not, just Google it.So, I would like to know, what is the size of the state space of valid disk configurations in this puzzle.Please enter this here.I'd like to know, whether the number of the disks on the left pegare an admissible heuristic, if you use A* search.And I'd like to know, what is the number of stepsthat an optimal solution will requireto move all the disks from the left peg, to the right peg.So here's a Bayes Network, with 6 variables, A, B, C, D, E, and F.And I'd like you to count parameters.If this was a binary based network,where each variable can take on two values,then, A would require one independent parameter,and B another one.And C would require four independent parameters,because there's four different ways A and B can come together in condition C.Now in this question I'd like to ask you,What happens if each node can assume three values, not just two?So A can be, A1, A2, A3.And C can be, C1, C2, C3.For each node, specify the number of independent parameters requiredto state the conditional probability of that node.And I'll tell you this is a tricky question,So for A, the correct answer is two.I won't give you the other ones.And, it's two because A can take three values,but it takes two independent parameters.The last one can be inferred from, one minus the first two.Please fill in the values for all the other variables.This is a true or false set of questions for Machine Learning.Suppose we've trained a machine learning model,and we've found really good values for our parameters in our model.And now, we're going to increase the noise,that affects our data.What should we do to accommodate the increase of noise?Shall we increase k, if we're using k nearest neighbor?True or False?Increase k if we are using the k means algorithm.True or False?Increase k if we are using Laplacian smoothing.True or False?Use fewer particles if we are using particle filters.True or False?And use more data if available.True or False?So this is a planning question.And I apologize, it's a little bit hard to read.There's a lot of text here.And I ask you to consult the pdf document to read the text.Given the resources on the left, over here,can we reach those five goals;A, B, C, D, E, on the right side?And in looking at those, there's words like 'consume',which means, the action eliminates the resource.Whereas 'use' means, you have to have it, but you retain it after using it.Now initially, you know there's a couple of books;one by Nau, about planning,one by Zweben, about scheduling,and one by Melville, about Whales.And there's also videos.Video 8 is about Planning.And Video 15 is about Scheduling.These might be our in-class videos.That's your initial state.And your goal is that you, as a student,know about planning,and you know about scheduling.So the question is, with certain resourcesthat are available in the beginning,and they differ from question to question,can you attain the state of knowing about planning and scheduling?Now, there's two ways to know about a topic.One is to study it using a book.And one is to view it using a video.In both cases, the outcome is to know about the topic over here.Now either one has a different precondition.In the 'book' case, you have to have the book,and the book has to be about the topic you care about.In which case, the action 'study' lets you understand the book and you know about the topic.So for example, if you have a book about planning,and study it, then you know about planning.In the 'view' case, you have to have a video that's about the topic,and you have to have a certain bandwidth,which happens to be 2.5.If you don't have the bandwidth 2.5, you won't be able to view the video,and you won't be able to know about the topic. That's the way the problem is set up.Now, books can be bought or borrowed.In the buying case, you consume 50 dollarsIn the borrowing case, you have to have a privilege, at the library, that's at least '1'. It might be larger but it can't be lower than '1'.And in either case after doing this, you have the book,and you can now plug this into the 'study' action,and you can read about it,and study it, and know the topic.So here are the questions.If your resource is that you have 50 dollars,and you have library privileges of '1',can you then attain the state ofknowing about planning and scheduling?Secondly, suppose your resources is no dollars,but you have library privileges of '2',can you attain the same state?Third, what about the same with library privileges of '1'?Can you get here?Fourth, what about if you have 40 dollars,and bandwidth of '3'? Can you get here?And fifth, what about if you have bandwidth of '2', and 95 dollars?Can you get here?Check all, or any, or noneof those five questions that apply.This is a question about logic.We have four different statements.Pink is True,Pink or Green is True,Pink and Green is True,and not Pink implies that Green is True.Now these statements could imply each other,and in this matrix over here,I'd like you to select each circlewhere an implication is necessarily true. It's always true. For example,if you believe that Pink impliesthat Pink or Green is True,then mark the A implies B circle, over here.If you believe the Pink is True impliesPink and Green must be True,then mark the circle A implies C, over here.And so on for the entire matrix.One hint, D looks complex,but it happens to be the same as one of the previous cases.So if you fill out the matrix for A to C first,and then copy the result over for D,it will be easier, than if you start thinking about D separately.And to find the equivalency of D,just write down the Truth Table of these different things over here,and observe D is already represented among A to C.In this question we study a particle filter.Let's just zoom in for a second.We have eight particles that land on this checkerboard.They are labeled, 'A' all the way to 'H'.And some of them are on black squares.And some of them are on white squares.Given those particles,we'll assume that the probability of measuring 'black',for any particle that falls on a black square,is 0.7.And the probability of measuring 'white',for any particle that falls on a white square,is 0.6.From that you can easily calculate the probability of measuring 'white',if a particle falls on a black square.And the probability of 'black',if the particle falls on a white square.Now I'd like to what's the normalized importance weight, after normalization,of the particle, labeled 'A',if our measurement happens to be 'white'?That's a number that you put in over here.And I'm going to ask you the same questionabout the normalized importance weight of particle 'A',if the measurement is 'black'.To calculate this,you will go through these probabilities.For each particle, you will assign the measurement probability.And then you just normalize all of those,so they add up to one. In this question, we assume that a particle, 'A', has a already normalized importanceweight of 0.2.So there might be other particles,we don't even care how many.But their importance weights add up to 0.8.We now sample 3 new particles,with replacement.What is the probability that this particle, 'A',is sampled at least once?And the way you derive this is by askingthe question, what's the probability that particle 'A' is never sampled?And then you take the compliment of this.This is a question about Alpha-Beta Pruning,in min-max search, in games.Consider the following tree,where this is the max node,and these are min nodes.We perform alpha-beta pruning.I'd like you to check all leaf nodes,of these 9 leaf nodes over hear, in this tree,that will be expanded, assumingthat we expand from the left to the right,and we expand in depth first mode, of course,as always in these game trees.These are four True or False questionsfor computer vision, and specifically,perspective projection.Consider a projective image of an object.Which of the following statements is true?If the object moves closer to the camera,the size of the projected imageof the object will increase.Is this True of False? Please just check one.If we use a camera with a longer focal length,as a result of using the longer focal length,the size of the projected image will increase.Check one.If we double the distance to the object,the projected image will be half as large, as before.Check one.And finally, the ratio of the focal lengthover the distance to the objectis the same as the projected size of the object in the camera plane.Please check True or False.A question on stereo vision.An object at range of 100 metersleads to a 2mm displacement for a stereo rig,with focal length 40mm.Now we double the baseline.What will happen to the new displacement,that used to be 2mm,what will it be now?Here's a 'Structure from Motion' type problem, that is similar to what I asked you on a homework assignment.Assume there is a world of 3 point features,that will be named; 1, 2, 3, but I won't tell you which one is which.There are 4 pinhole cameras; A, B, C, and D.And they all have a left, center, and right side.Left, center, and right side.And you should observe,that the perceived order of features in the scene,by virtue of using a pinhole,will be inverted inside the pinhole camera.So camera 'A', sees in the left position,feature '1',on the center position, feature '2',on the right position, feature '3'.I would like to know,for which of the other camera's,is it the case that feature '3'will appear in the leftmost position?So the leftmost position is 'L' over here,'L' over here, 'L' over here.Assuming the optical centers, shown over here.Please check any or all of the following;Camera B,Camera C,Camera D,or None of them.My final question is a simplified self-driving car question,that is usually solved using dynamic programming.But I have to warn you, the state space shown hereisn't the full state space.The orientation isn't really made explicit, in this state space.But suppose you have a road environment,that has a straight street over here,you can turn left, go straight, or turn right over here,and similarly you can turn left or right over here.And we assume that moving from one grid cell to the nexthas a cost of '1'.Turning left has a cost of '14'.And turning right has a cost of '1' as well.Let's assume the robot when it turns,stays in the same grid cell, but it only can turn once.After it turned, it has to actually move.So it's impossible, for example, to turn right three timesjust to avoid the cost of a left turn.I would like to know,what is the minimum total costof going from the start location, over here,to location 'A'.I realize that there is many ways to get there.I'd like to know the minimum.So what's the minimum cost to get to 'A',irrespective of what orientation you assume at 'A'? I don't really care.The same from the start location to 'B'.And from the start location to 'C'.Please enter your best guesses on the right side.