It certainly has. But are our NPCs really that much smarter now? In Mario, their AI was based on a simple loop - walk, if you hit a wall, turn around and walk some more, Repeat.
In Oblivion, each character has certain tasks that he's been given based on the time in the game - and you can talk to them, so they'll tell you different things. If you attack them and drain a certain amount of their hp, they'll attack you. But is that it?
I can take Oblivion as a good example of how AI can utterly fail:
In the Dark Brotherhood storyline, there's a quest where you have to kill all your comrades. They're all in a multi-room cave, separated by doors. So, I started the quest and did this: One guy was at the lunch table, eating an apple.
So I thought this'd be a good time to kill him off. Nobody else was around and that guy was peacefully chomping his apple, which meant that it would've taken him at least a couple of seconds to get into his combat mode to fight back - enough time to let a shitstorm of fireballs rain on his head. That's exactly what I did and it worked beautifully. He was still eating his apple as I started to firefuck his brains, dealing major damage. A couple of seconds later he lay dead on the table.
Nice, I thought! One man down - Only three more to go.
At that second, another comrade came in - apparently he was hungry, too. My first thought was: "Shit, now he's gonna see what I just did to his buddy and he's gonna kill me" (I'm talking about the fat, armored Orc) - But, thankfully, he didn't care much about the corpse and his fried brain being splattered across the table. So he went to the cupboard, took out a piece of cheese and rested his arse at the table as well, peacefully eating it while his buddies brain was still smoking.
That's the problem with open world games - as a developer, it's impossible to create events for every single situation that might happen in the game that the player might do - You already need dozens of events for each character to make them believable, it's currently impossible to keep the believability if you have a couple of hundred NPCs to deal with.
I believe we'll have AI Engines in the future - just like Physics engines like Havoc became mighty popular, having an engine that'd somehow deal with that shit and that could be implemented into various other engines while not losing a ton of performance would be great. But right now, all of the open world games are filled with puppets, mindless idiots instead of something that could fool me as human beings.
Fable 2 is another great example, cause it's a game that was just released. The population there just doesn't cut it. I'd rather center the game around one little town with 10 citizens that really could be believable rather than simulating a whole country with 500 citizens that are all stupid as shit.