I remember the first time I saw an Android stick PC.
It was Spring 2013. There were a few different types of stick PC’s by that time. Most of them could only be found on ebay from sellers in China. Even though I was late to the party, I knew these things would be special. Within a few days of reading that first article, I took the leap and ordered my first Android stick PC. I wanted to see what I could do with them. More importantly, I wanted other people to know about them too.
So AndroidPCReview.com was born.
Gotta start somewhere, right?
When I look back at those first few reviews, I cringe. I hate them. They didn’t really give you any new insights on the product. But, if you’ve read any of my latest reviews, you can definitely see how much I’ve improved since then.
As I tested more Android stick PC’s, then mini PC’s and finally Smart TV boxes, I found better ways to benchmark performance. Some of these I found on my own. But most came from others in the community or from reader requests. I’ve always been happy to share the credit for good ideas. So I have to send some well-deserved kudos to the community at Freaktab, Rivers at SomeCoolTech and Lon Seidman from Lon Reviews Tech.
This post doesn’t try to be the “One Test Method to Rule Them All.” It’s just my method for testing any new Android mini PC that I get my hands on. As I find new benchmarks and tests, they’ll make their way in this list. If you’ve got ideas on tests you’d like to see, leave them in the comments below. Enjoy!
Apps to test an Android Mini PC
There are a few standard benchmarks that everyone expects to see in a review, so that’s generally where I start.
AnTuTu has been the gold standard for benchmarking Android devices for several years. If you’re looking for a single number to explain your device’s performance – this is it. Lately though, manufacturers have been getting wise to how AnTuTu actually works and have tried to game the system. In the latest version, AnTuTu 5, they tried to put a stop to that.
Part of that is by focusing on a single CPU thread. Why, you ask? After all, our devices are multi-core, so why not focus on a multi-core benchmark?
Well, the trick to that is that having multiple cores actually makes it harder to get a true CPU performance number. Many applications aren’t multi-core enabled, and those that are tend to set a limit of 2-4 cores. Any more, and it’s just overkill. To give you a PC example you may be familiar with, my work PC is a 10-core Xeon workstation. However, when I try to run calculations on an Excel workbook, it’s only able to use one core with any consistency. If big names like Microsoft can’t (or won’t) optimize their software, what are the odds that that free Android app you love so much will?
3DMark Ice Storm and Sling Shot
Ice Storm is actually only one of the benchmarks in Futuremark‘s 3DMark suite. Back when I was building and overclocking PC’s, I’d run the Futuremark tests again and again in order to tweak my settings to get the most bang for my buck.
It’s not quite as easy (or as complicated, depending on your perspective) to test an Android mini PC or TV Box. There just aren’t as many settings to tweak unless you get down to the root level. Still, the 3DMark tests give you good framerate and graphics performance numbers that are very relevant if you like to game.
And who doesn’t?
In the PC world you’ve either got an Intel CPU or an AMD. There aren’t a whole lot of other options, and none that you’re going to find in a PC at your local Best Buy.
Android is a whole other story. You can get CPU’s from RockChip, AllWinner, Intel, and a ton of others. To make it more confusing, there’s no “AllWinner Inside” sticker to let you know what you’ve got under the hood.
That’s where CPU-Z comes in. This simple app gives you detailed information on processor type, manufacturer, number of cores and clock frequency, as well as memory type and chipset. You can also see real-time performance statistics. For basic information on your device, there’s no better place to start.
Ookla Speed Test
Speedtest.net is my go-to site whenever I’m having speed issues with my Internet at home. It gives simple dashboard-style results and let’s you upload those results to Facebook or Twitter if you feel like bragging. Did you know it’s also able to test how fast a device is on your personal network? Their mobile app will let you test how fast your wi-fi signal is and compare it to the other wireless devices that you own. That lets you get a pretty good idea of the wireless speed of your device.
There are a few other hardware benchmarks I’ll look at, depending on the situation. These include:
Once I look at the results from the apps, I’ll get a pretty good idea of where I need to go into a little more depth.
For example, looking at CPU-Z, I may see that a particular device is not running at it’s full advertised speed, or not using all of its cores effectively. In that case, I’ll have to do a bit more graphics testing or video testing. This lets me find out if the device is throttling its data, or if it was manufactured using sub-par components.
In another example, I would look at the Settings page to see how the memory is partitioned, or if it is at all. Some devices have restore partitions, like on your PC. So even if the device has 8 GB of storage, you really may only be able to access 4GB of it.
Boot time is a pretty simple stat to test. Grab a stopwatch and time how long the minipc takes to go from a cold boot (completely powered down) to the desktop, ready to use. I’ll usually run this test ten times and take the average of all of the results. After that, I’ll let the device cool off (if necessary) and run the same test again – this time going from a Standby state to desktop. Again, I’ll run the test ten times and take the average.
Power use isn’t really a big issue for most people, so I only include it if a device is really out of whack with other minipc’s. To test this, I’ll use a Kill-A-Watt P3 4400 to record some statistics during general usage. Lately, I’ve been testing with the Belkin WeMo Insight switch. It gets similar results faster from its mobile app.
Benchmarks are great, but it’s people that are going to be using these devices. So I look at the user experience next. This involves a few different aspects like:
- Launcher Impressions
- Look and feel
- Ease of use
- Customization options?
- Has it been updated already?
- How frequently does the manufacturer update firmware for other models?
- How do you update the firmware? OTA or paperclip?
- Video impressions
- Kodi\XBMC (with hardware performance)
- Remote impressions
- Did it include a stock remote?
- Does the stock remote provide a good experience, or do you need an air mouse?
- Using a webcam
- Gaming (this is the fun part)
There are a couple of closing thoughts I try to mention with every review.
Does the device have any limitations out of the box? Technology is great, and a lot of times manufacturers will provide updates that give new functionality later down the road. But the fact is that many users will never update the device. For better or worse, they’re stuck with the way it performs on Day 1. If it can’t do something important, users need to know.
What is the highlight of the device? Every device should have something that it does better than everyone else. Whether that’s video performance, gaming, sound or just a sweet looking user interface – this is something that really stood out in my mind.
Does the device give you good value for the money? I don’t know about you, but I can’t stand to spend money on something when I don’t have to. Are there better options for the price? Is the pricing so much better (or worse) than everyone else?
Has the device been provided to me? I think its important to let the readers know if a manufacturer has sent me a review product. I have never been paid by a manufacturer to write an article on this site, and that isn’t going to change. But, a lot of the products have been sent to me by companies just trying to get people interested in their product. That doesn’t influence my reviews, but I still think its important for you to know where the product came from.