Necromancer's notes

My home server has a long story. It all started with a Pentium 4, an old 20GB HDD and FreeBSD 6.2 … hell, I don’t even remember the exact year.
Anyway, after a few years, the hardware was finally put to rest, since it died and got resurrected thrice, I got an Intel Atom D410-based miniATX board, switched to linux, first debian, then agilia, then arch… Anyway, it used to be a nice server for personal needs, that crashed only on occasional HAD-effect, so it was… sufficient.
Now, the time has come to move on, to arm. The benefits were simple and straight:

• 10W peak power consumption
• Fully passive cooling
• eMMC for the root partition
• 4 cores!
• Always a serial terminal, starting from uboot phase, so that I don’t have to carry a monitor to the closet where it is stationed.

I picked ODROID-X2 based around Exynos4212 Prime. ODROID-U2 looked worse, since had NAND soldered onboard. eMMC looked easier to replace. And the benchmarks said eMMC was faster.

So, here go my adventures with this hardware.

First things first. Power. People from hardkernel did it the strange way. If you _instantly_ load all 4 cores, all usb/hsic stuff will reset. The very easy test in bash:

#!/bin/bash
cat /dev/urandom > /dev/null&
cat /dev/urandom > /dev/null&
cat /dev/urandom > /dev/null&
cat /dev/urandom > /dev/null&

 
Poking for a few minutes with an oscilloscope, a few minutes of thinking and voila – a workaround: I put a 4700uF cap on the power input. Yeah, yeah, an overkill, but it works. Even if you hotplug some nasty stuff that has a big inrush current. Luckily there was a test pad and ground pad nearby.
With that hack done – all these problems were instantly gone.

Second bug – power jack. The contact was loose, the cable I had was shitty, so I ditched that in favor of a JST connector.

So long for the hardware, now the software. I started by benching the storage subsystem with a simple test:

SATA HDD over usb:

necromant@fireblade:~$ sudo dd if=/dev/sda of=/dev/null
1198177+0 records in
1198176+0 records out
613466112 bytes (613 MB) copied, 31.2201 s, 19.6 MB/s

Oops, forgot to disable usb mass storage verbose debug.

necromant@fireblade:~$ sudo dd if=/dev/sda of=/dev/null 
1975674+0 records in
1975673+0 records out
1011544576 bytes (1.0 GB) copied, 46.847 s, 21.6 MB/s

SD Card:

necromant@fireblade:~$ sudo dd if=/dev/mmcblk0 of=/dev/null 
258017+0 records in
258016+0 records out
132104192 bytes (132 MB) copied, 12.2988 s, 10.7 MB/s

eMMC:

necromant@fireblade:~$ sudo dd if=/dev/mmcblk1 of=/dev/null 
718049+0 records in
718048+0 records out
367640576 bytes (368 MB) copied, 11.4995 s, 32.0 MB/s

That says it all: eMMC – rootfs, SD – webpages, since we have fast random access there, HDD for data, logs and the rest of the junk. Before doing anything, I carefully tested SD and eMMC with flashbench, and formatted according to this tutorial to improve performance.
It looks like on eMMC bl1/bl2/tz/uboot are locked into a secure piece, which is read-only via /dev/mmcblk1boot0, and /dev/mmcblk0 has only uboot env.
As a result, my eMMC partitioning looks like this:

        Device Boot      Start         End      Blocks   Id  System
/dev/mmcblk1p1            2423        4095         836+  83  Linux
/dev/mmcblk1p2            4096       32767       14336    b  W95 FAT32
/dev/mmcblk1p3           32768    15269887     7618560   83  Linux

uboot.
Well, uboot from samsung/hardkernel is a pile of junk dating back to 2010. ext2load is broken, ext4load is not there uImage doesn’t boot, only zImage and fatload. My idea to boot fresh kernels via tftp from an openwrt router also didn’t work out: Hardkernel people put ethernet on HSIC via SMSC LAN95x, not built-in exynos MII, (saved a few cents on the phy?). And uboot doesn’t have the code to run LAN95x. Shit.
Well, could be worse. My personal code shit-o-meter showed a total shittiness of something like 0.32 realteks for the samsung code.
Anyway, first partition on eMMC is for uboot env, so that I can easily access it via cmdline, next go vfat boot partition with zImage and rootfs. Simple s that. The rest is straightforward.
Ethernet is only 100mbit, and works out all of the throughput. Since I have a 100mbit network this is sufficient.

Finally, Apache Bench of Intel Atom D410 vs Exynos from outside of my network. Configs of the webserver, php and mysql are totally identical:

Atom D410:

    ab -n 1000 -c 100 http://ncrmnt.org/
    Server Software: awesome
    Server Hostname: ncrmnt.org
    Server Port: 80
    Document Path: /
    Document Length: 37804 bytes
    Concurrency Level: 100
    Time taken for tests: 33.119 seconds
    Complete requests: 1000
    Failed requests: 0
    Write errors: 0
    Total transferred: 38100000 bytes
    HTML transferred: 37804000 bytes
    Requests per second: 30.19 [#/sec] (mean)
    Time per request: 3311.930 [ms] (mean)
    Time per request: 33.119 [ms] (mean, across all concurrent requests)
    Transfer rate: 1123.42 [Kbytes/sec] received
    Connection Times (ms)
     min mean[+/-sd] median max
    Connect: 31 46 90.6 36 2016
    Processing: 202 3110 1081.8 3225 9784
    Waiting: 163 2956 885.8 3167 5907
    Total: 237 3157 1065.2 3261 9819
    Percentage of the requests served within a certain time (ms)
     50% 3261
     66% 3283
     75% 3295
     80% 3307
     90% 3423
     95% 5098
     98% 6112
     99% 6466
     100% 9819 (longest request)

Exynos:

ab -n1000 -c100 http://ncrmnt.org/
Server Software: awesome
Server Hostname: ncrmnt.org
Server Port: 80
Document Path: /
Document Length: 37804 bytes
Concurrency Level: 100
Time taken for tests: 34.914 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 38102552 bytes
HTML transferred: 37804000 bytes
Requests per second: 28.64 [#/sec] (mean)
Time per request: 3491.388 [ms] (mean)
Time per request: 34.914 [ms] (mean, across all concurrent requests)
Transfer rate: 1065.75 [Kbytes/sec] received
Connection Times (ms)
 min mean[+/-sd] median max
Connect: 32 63 192.1 35 3758
Processing: 150 2806 3082.1 1945 30606
Waiting: 78 699 1052.0 292 9206
Total: 184 2869 3098.2 2012 30655
Percentage of the requests served within a certain time (ms)
 50% 2012
 66% 3161
 75% 4218
 80% 4394
 90% 5738
 95% 7662
 98% 10636
 99% 17378
 100% 30655 (longest request)

Although, taking the time to optimize things would definitely improve things.

EDIT: optimized a few bits, tested with siege with -c 150 getting the wordpress frontpage.

Transactions: 2691 hits
Availability: 100.00 %
Elapsed time: 37.74 secs
Data transferred: 97.58 MB
Response time: 1.53 secs
Transaction rate: 71.30 trans/sec
Throughput: 2.59 MB/sec
Concurrency: 109.01
Successful transactions: 2691
Failed transactions: 0
Longest transaction: 1.79
Shortest transaction: 0.07

So far it looks like an epic win. Yes, now I’m ready for the HAD-effect!