See, I think that the N64 was a success. And not just in the traditional sense. Did the PS1 'out perform' it, absolutely. But that doesn't mean that the N64 wasn't a success.
The N64 may have had a smaller library, however, it still had lots of beloved games on it. Also, it was the 'first' for several things. It was the first console to produce 'real' 3D worlds (unlike the PS1). It was the first console to implement and popularize the analog stick, which is now an industry standard. It was the first console to implement and popularize the rumble/force feedback feature, which is also now an industry standard.
Also, the highest selling game on the original Playstation was Gran Turismo, selling 10.85 million units. Meanwhile, Super Mario 64 sold 11.62 million. Of course, the original Playstation had more total sales. But the N64 sold more games than the Gamecube did, and the N64 had about half of the library size of the Gamecube. Yep, that's right, the Gamecube had almost twice as many games at the N64, yet, the N64 sold more game units.
Speaking of games, the N64 also had some 'groundbreaking' games as well. Super Mario 64. The Legend of Zelda: Ocarina of Time. Goldeneye 007. To name a few. These games, and the other groundbreaking games, changed the genre that they were in for the better. All because of the N64.
When comparing some N64 titles to it's next released version on the Gamecube. Super Mario 64 outsold Super Mario Sunshine. Mario Kart 64 outsold Mario Kart: Double Dash. The Legend of Zelda: Ocarina of Time outsold The Legend of Time: The Wind Waker. In fact, as a whole, the N64 sold more games than the Gamecube did, and the Gamecube had almost twice as many games on it's system. The N64 had a total 387 games released in America. While the Gamecube had 646. Yet, as of June 30, 2013, the N64 has sold over 224.97 million games, while the Gamecube sold over 208.57. And the Gamecube was also, in the end at least, a 'success'. A "small" success (tho not at first), but an overall success none-the-less.
And, if all you care about is money/sales to determine a success or not (even tho that's not the only thing that can determine whats a success or not, tho it is a big thing), then keep this in mind. The N64 may have "lost" to the Playstation, but it did "beat" the Xbox. Seriously. The N64 sold about 9 million more systems than the Xbox. Of course, it also sold more systems than the Dreamcast. In fact, the N64 sold almost as many systems as the Xbox and Dreamcast combined. Yet, people widely regard the first Xbox as being 'successful'. In fact, they don't even question it. And if that's the case, then the N64 shouldn't even be questioned about being a success either. The N64 wasn't even directly competting against the Xbox, as it was a generation older. Some may say, yeah, but the N64 was in direct competition with the Playstation, not the Xbox. Fair enough. However, the Sega Genesis was in direct competition with the Super Nintendo, and yet, both are highly regarded as successes. Nobody questions either of those. Also, the Xbox was in direct competition with the Playstation 2, and the Playstation 2 blew it out of the water. Of course, it blew every other system out of the water too. However, if you are going to argue that the N64 was a failure because the Playstation was a success, then you should argue that the Xbox was a failure because the Playstation 2 was a huge success. Especially when the N64 outperformed the Xbox. But again, nobody question's the validity of the Xbox's success. So, nobody should be questioning the validity of the N64's success. Especially when the Xbox had about 3 times as many games made for it (968 total Xbox games to 387 total N64 games), yet, the N64, overall, did better.
So, in the end, was the N64 a success? Absolutely. Without question. It may not have been a success in the way that some people would of wanted it to be, or, in a way that people remembered it being. However, it was definitely a success regardless.