Technically, Metroid Prime went from side-scrolling to first person. 3rd person implies that the camera was behind your character, which it wasn't in the previous Metroid games. They were side scrollers. Technically 3rd person, yes. But side-scrolling and 3rd person are also different genres. So, there's that.
Anyways, Metroid's change in genre worked out well for the series. You are right, not many genre changing games have worked out for the better. However, Metroid moving from side-scrolling to first person is a much, much bigger deal than just going from 2D to 3D. Going from 2D to 3D would be, say, Super Mario Bros to Super Mario 64. It felt like a natural progression, which is why a lot of people were hyped about it. Going from side-scrolling to first person, however, is a much harder endeavor, as it rarely works. Granted, it worked for Duke Nukem, who started out as side scrolling game, and later turned into a first person shooter. So that's one example of that genre switch. However, typically speaking, it was mostly unheard of, at the time, for a game to go from side-scrolling to first person, and be successful. That's why there was a lot of skepticism, and a lot of worry. Back then, that kind of genre switch just didn't happen much, if at all. And was rarely successful. That's why everybody immediately thought that it was going to fail.
However, Metroid Prime proved to be a fantastic game. And the gamble paid off. But, there was no guarantee that making that switch from side scroller to first person, was going to be a success. In fact, most people would of guessed that it was going to be a huge, huge failure. FPS's, at the time, weren't nearly as big as they are today. They were up and coming, but didn't have the mainstream popularity that they do now. In hindsight, everybody now thinks that it was a brilliant move, and say that they would of thought that it was going to be a perfect transition. However, back then, nobody thought that, except for a very, very, small few. But, in the end, it was a huge success. And yes, that surprised almost everybody back then. Granted it was only what, 10 years ago, or so. But even back then, the gaming landscape was far, far different than it is now. And yeah, most people thought for sure that it was going to fail, and fail hard. So there was wide surprise when it did great.
---
However, Metroid Prime isn't the only franchise to switch genres and be successful at it. Another franchise that switched genres successfully was...
Super Mario Bros. - They went from a side-scrolling game to an RPG series, successfully. Nobody ever thought to think of that transition. But they did it anyways, and it worked.
Others games went to similar genres. Like, the Teenage Mutant Ninja Turtles, for example. They went from the "beat'em up' genre to the "fighting game' genre. As they had several fighting games too. Not as successful, but they did it.
Warcraft went from an RTS game to an MMO game. That was a big genre switch that paid off for them. Tho, honestly, I still miss the first the RTS Warcraft games.
Even Star Fox did a genre switch. Going from a 'shooter' to a '3rd person' adventure. Tho, that didn't get nearly as much praise as the Metroid Prime games did.
Of course, even movies have switched genres between films. For instance, the most obvious one is the Aliens franchise. Alien (1) was a horror movie, while Aliens (2) was an action movie. Both were successful. The later films not so much. But yeah, genre switches can sometimes be a good thing. However, there are times when it's not.
And, back in the days when Metroid Prime was just first coming out, genre changes as big as going from a side-scrolling game to a first person shooter, were mostly considered a big "no no" by most people, as not many people thought that it would of worked. Now a days, more and more people would be willing to jump at a chance to make a first person shooter from a traditionally made side scrolling franchise. Look at Mega Man. They were working on making a first person Mega Man game. It never made it out of development, but they tried. But even with the success of Metroid Prime, most people didn't like what they saw with the first person Mega Man game. Yet, those who actually played it, said that it was going to be a great game. So that same thing that happened to Metroid was starting to be the same or similar thing that was going to happen to Mega Man. Tho now, we'll never know. But even Mega Man has changed genres too. As even Mega Man became an RPG at one point too.
Changing genres is nothing new. However, when changing to a similar genre, people tend to generally get excited. Like, if they just move from a 2D platformer to a 3D platformer, like the Mario games did, and later, the Sonic games, people tend to go along with those. But when it comes to big genre changes, like going from a side scrolling game to a first person game, people tend to be skeptical. Mostly because they fell that the game will be terrible or won't live up to their expectations. But then, sometimes it does, and that surprises them. But a lot of people tend to want to stick with what they know and love about a franchise, and don't like too much of a change. And so, when a beloved franchise gets a big change, most people tend to be against it, until they end up trying it for themselves. Sometimes their fears were right, and they hate the game. Other times, they were wrong, and they loved it. As was the case with Metroid Prime. Everybody hated the idea, and didn't think it would work (except for a very, very small few). So, when it came out, and turned out to be a fantastic game. That's when everybody turned around and said, "i knew it all along that it was going to be great, how come they didn't do this earlier". However, most of them are just lying to themselves.
But yeah, that's why most people expected Metroid Prime to fail. Because it was a huge genre change, which, at the time, hadn't been that successful for games. And, people who love a franchise, don't want to see huge changes, as changes scare them.
Take The Legend of Zelda: The Wind Waker, for example. The game was shown off originally with modern high end graphics (at the time). And then, later, they showed the more cartoony version. That version upset a lot of people, and almost everybody immediately hated the decision, and felt that The Wind Waker would flop, hard. They didn't think it would be any good, because they didn't like the changes. Changes scare committed and devoted fans of a particular franchise. Especially big franchises, like Metroid and Zelda. However, once The Wind Waker was released, everybody realised what a great game it truly was. However, much like Metroid Prime, everybody thought that that game was going to fail too.
