The higher end phones can't even compete with the higher end phones. The smartphone market is stuck in a rat race where all the phones are barely distinguishable from each other and the more unique features get generally ignored. And even if you make the perfect phone, it's obsolete and garbage within three months. There's absolutely no money to be made.
You never specified the OS. If Nintendo makes a custom OS or even forks android, the phone will be DOA. Blackberry and Amazon have learned the cold harsh reality that not having access to the Google Play app store will kill your phone, even if it's perfectly capable of running Android apps. While being able to play Mario on a phone might be nice, it's a communications tool front and center. That means phonecalls text messages whatsapp line instagram facebook knowlegetree and a million other apps that are the One Thing that someone or another relies on and needs on their phone. There's no getting around that the Android and iOS app stores are now vital to any phone on the market. The percentage of phones that have access to either the App Store or Google Play amount to at least 95% of all phones sold.
If Nintendo uses an authorized Android OS build, they no longer have full control of their system. One of Nintendo's defining traits is that their games and systems are as glitch free as humanly possible (compared to Bethesda, for example). Using Android opens the phone up to a multitude of issues Nintendo can't control. Google can push an update that breaks a game, or a user can install an app that fucks things up. The game cannot take maximum advantage of the hardware, as the OS and other apps will be sucking the CPU dry. I don't think Nintendo can even use their own game store, so they would have to rely on Google Play and have everything linked to your Google ID.
And that's not even covering the rampant piracy that Android OS has, where people would most likely crack the games to install on their own phones. That last one is a biggie. Nintendo would be forced to sell a smartphone at near cost in order to compete, but attempts to make money by selling games would be foiled by people simply choosing to pirate the games.
The smartphone market is not a nice place. Nintendo has no business making a smartphone.
Incidentally, what did you think a so-called Nintendo phone would look like? Is it a generic slab with no tactile feedback, or does it have a whole slew of buttons and joypads that add cost and get in the way when you want to make a call? Look how chunky the 3DS and Pro Controller are. They need to be, in order to have decent travel on the physical controls. Now look at the phone in your pocket. If it's anything like mine, it's pretty fucking skinny. There would be no room for usable triggers on it, and you're dreaming if you expect a circle pad or even the c-stick. Yes, you can make the phone thicker, but then you lose pocketability big time. Are you really going to wear a phone holster like an old Crackberry addict?
The 3DS and the Wii did one thing, and they did it well. That's why they were successful. Nintendo has never been a company that made a jack of all trades product, and never will.
I'm sorry that I'm kind of hostile about this. Maybe it's because I have SDAM, but all the facts of the smartphone world and of Nintendo's actions are very clear that such an attempt would be a suicide run, and Nintendo would not even attempt it. There is simply no benefit, and it won't even make something that you'll stare at nostalgically when you rediscover it in a box in your attic.