If you’re following my blog, you might’ve read my post about Dlink’s substandard engineering, and how I’ve got debian running on the damn thing. Well, since that time I’ve made a few improvements, namely:
- Power button’s now correctly rigged as well as the rest of the buttons.
- Upstream 3.18, 3.19 kernels now works flawlessly and don’t ever freeze. I update kernels with each new kernel release
- Goodies like Marvell’s DMA engine, mv_xor, watchdog and mv_cesa are all enabled for the win and are working awesomely
- There’s now a huge guide about ‘cooking’ the damned thing into a useable state in five easy steps. And you are reading it.
- Power measurements are all there as well!
BIG FAT WARNING: I take no whatsoever responsibility if you screw up your device following these instructions. Proceed at your own risk, use your own head. This will also void warranty, btw.
Let’s first start with a small checklist of the stuff you’ll need:
- Screw driver
- TTL Serial-2-USB dongle, 2.54 pin headers and jumper wires. (Or a HC-05/HC-06 module. I prefer the latter)
- Soldering iron (DO NOT use 150W monstrosities from the 70s, be gentle!) and basic soldering skills.
- An itchy desire to void warranty and get a debian box
- A couple of hard drives that we’ll shove into our NAS
- A host PC running *SURPRISE* Debian or ubuntu
- Some spare time to waste
Step one: Take the thing apart, solder in UART, access u-boot.
Start by removing the rubber standoffs at the bottom of your NAS (don’t lose them!), remove the 4 screws. Take the thing apart – mind the wire connecting the fan, and carefully remove the PCB from it’s plastic housing. We’ll need serial access. Serial settings are: 115200 8n1. Voltage is 3.3V. Pinout:
[RXD|(gap)|3.3v|GND|TXD]
Heat up the soldering iron and solder in the unpopulated 2.54 pitch with the header. Alternatively configure and attach a HC-05/HC-06 module with a little bit of doublesided tape, some wires and your soldering ninja skills. HC-05 modules are dirt cheap and very handy and will relay all serial communications via bluetooth, so you’ll need no messing with the wires.
Once you’ve hooked serial – power on the hardware and immediately start pressing ‘1’ and hitting space. That magic dance interrupts boot and drops you to the command prompt.
Step two: Kernel and u-boot
TL;DR: Grab the sources at my github here
Since u-boot on the box does NOT support deviceTree we’ll have to append device tree to our kernel image and follow appended dtb way, e.g. CONFIG_ARM_APPENDED_DTB=y
To compile the kernel you’ll need an ARM toolchain and some commandline magic. I use the following shell script. It’s a little generic, but you get the idea.
export CROSS_COMPILE=arm-module-linux-gnueabi- uimage() { cp zImage-precious arch/arm/boot/zImage cat arch/arm/boot/dts/${1}.dtb >> arch/arm/boot/zImage ARCH=arm LOADADDR=${2} make uImage cp arch/arm/boot/uImage ./uImage-${3} cp arch/arm/boot/uImage /srv/tftp/uImage-${3} } rm -f arch/arm/boot/zImage ARCH=arm make menuconfig zImage dtbs -j8 cp arch/arm/boot/zImage ./zImage-precious uimage armada-370-dlink-dns327l 0x00008000 altmera |
There are two ways of getting our NAS to boot the kernel. First one is use tftp from another host, second one – burn it to the NAND. The second one is a bit more dangerous, since if you screw up your u-boot bootloader, you’ll have a hell of a time debricking your hardware. According to the docs, Armada 370 has some kind of debricking voodoo using serial embedded into ROM bootloader, but I haven’t tested it, so if something happens – you’re on your own. Just try not to screw up here, okay?
The TFTP way
So, loading kernel over tftp is possibly the best idea. This way you can easily update kernels, and if something goes wrong – you just replace the file on the tftp server, reboot and you’re back to normal. Setup a tftp server, copy the kernel uImage there and run the following in uboot prompt to change environment(replace with your own ip addresses).
setenv ipaddr 192.168.0.5 setenv serverip 192.168.0.1 setenv voodoo 'mw.l 0xd00184e0 0xa8a; phyWrite 0 16 2; phyWrite 0 19 77; phyWrite 0 18 5747; sleep 1; phyRead 0 19; phyRead 0 18; phyWrite 0 16 0;' setenv bootargs console=ttyS0,115200n8 root=/dev/md0 rootfstype=ext4 setenv bootcmd 'run voodoo; tftp;bootm;' saveenv
Note the ‘voodoo’ line. DLink people set the voltage levels on RGMII phy via… a script in userspace after everything including network boots up. Since it needs a crappy kernel driver that’s not in stock – we’re out of luck. So I moved it to the bootloader. See my previous post on DNS327L for the details on what it does.
From now on you can use the following to boot linux:
run bootcmd
or just wait till timeout elapses.
The NAND way
Since we’ll need tftp either way, follow instructions above and set up a working tftp server. Before we start with NAND let me remind you that NAND contains 2 copies of the kernel . One is used during normal boot, the other one is there for failsafe reasons and will be used if (for instance) you abort tftp with CTRL+C. Maybe if something bad happens or if some key combo is pressed as well. Since things may differ on your box, I document the procedure of finding out the correct NAND offsets and sizes for the kernel.
Let’s start with how the stock firmware boots. Have a look at bootcmd variable in u-boot.
The stock bootcmd looks like this on my box:
nand read.e 0xa00000 0x500000 0x400000;nand read.e 0xf00000 0xa00000 0x300000;bootm 0xa00000 0xf00000
Let’s decipher this magic. The syntax is:
nand read [physical_address_where_to_load] [offset_of_nand_in_bytes] [number_of_bytes_to_read]
We first read kernel image from 0x500000 in NAND to 0xa00000. Next goes initrd from 0xa00000 in NAND to 0xf00000 in physical memory. Next we boot those. Great!
From that above we only need to know that the first kernel image resides at address 0x500000 in NAND! That’s all we need so far.
Next we need to find out the NAND offset of the second kernel image. If we interrupt kernel boot we’ll see something like this before the kernel boots:
NAND read: device 0 offset 0x5d00800, size 0x500000
Gotcha! So the second (failsafe) kernel image resides at 0x5d00800 and should be less than 5Mibs (0x500000). There’s no way to change that, so take care!. People at Dlink must have hardcoded these into u-boot itself. Phew!
Summing up all that we’ve learned so far, we need the following to update the primary kernel image in NAND:
tftp 0xa00000 uImage-altmera
nand erase 0x500000 0x500000
nand write ${fileaddr} 0x500000 0x500000
And setup the environment like this to boot it:
setenv voodoo 'mw.l 0xd00184e0 0xa8a; phyWrite 0 16 2; phyWrite 0 19 77; phyWrite 0 18 5747; sleep 1; phyRead 0 19; phyRead 0 18; phyWrite 0 16 0;' setenv bootargs console=ttyS0,115200n8 root=/dev/md0 rootfstype=ext4 setenv bootcmd 'run voodoo; read.e 0xa00000 0x500000 0x400000; bootm 0xa00000' saveenv
If we want to replace the failsafe image, the procedure is mostly the same, just the offsets differ a little. And we have to erase on the ‘erasesize’ boundary, so the erase offset would be 0x5d00000 and not 0x5d00800.
tftp 0xa00000 uImage-altmera
nand erase 0x5d00000 0x500000
nand write ${fileaddr} 0x5d00800 0x500000
Step three: Prepare Hard Drives (RAID mirror)
We’ll have our root filesystem set up on the HDD and it in this configuration the root filesystem will be mirrored, just like the rest of it. We’ll boot using root=/dev/md0. I also ended up mirroring swap space, since it will allow true HDD hotplugging if something fails. If you don’t need mirroring the procedure will be a little different.
Since initramfs in embedded environment is usually VERY messy, I prefer to have everything I need compiled in the kernel itself and boot with no initramfs. It saves a lot of effort if something goes wrong and keeps things simple. And that’s the part where we hit the very first two pitfalls:
- To boot directly from /dev/md0 we need kernel-level raid autodetection. And that will only work if we create a raid array with version 0.90 metadata.
- Kernel does autodetection, but doesn’t scan /dev/md0 for any partitions. e.g. root=/dev/md0p1 will NOT work.
Noticing the above, we take the smallest of the two drives and partition it. I left 10GiB for root filesystem, 2GiB for swap and the rest for the actual data. You can use any tool you like, e.g. cfdisk. Remember to set partition type to 0xfd, e.g. “Linux RAID autodetect”, aligning partition sizes to 4k is also a good idea.
Once done, you can copy the partition table to the other drive using sfdisk:
sfdisk -d /dev/sda | sfdisk /dev/sdb
At this point we can start creating our arrays. If you don’t want to attach both drives to the running system you can partition only the first one and create a degraded array. We’ll add the drives to it later.
mdadm --metadata=0.90 --create /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1 mdadm --metadata=0.90 --create /dev/md1 --level=1 --raid-devices=2 /dev/sda2 /dev/sdb2 mdadm --metadata=0.90 --create /dev/md2 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3
Or, if you prefer to create a degraded array with one drive and resync later:
mdadm --metadata=0.90 --create /dev/md0 --level=1 --raid-devices=2 /dev/sda1 missing mdadm --metadata=0.90 --create /dev/md1 --level=1 --raid-devices=2 /dev/sda2 missing mdadm --metadata=0.90 --create /dev/md2 --level=1 --raid-devices=2 /dev/sda3 missing
Next, create our filesystems:
mkfs.ext4 -L root /dev/md0 mkswap -f /dev/md1 mkfs.ext4 -L data /dev/md2
Step four: Bootstrap debian
Not really much to talk about here, since bootstrapping debian systems is worth a bunch of posts on its own. You’ll need a running debian/ubuntu system on your desktop and debootstrap or multistrap.
I use the following script to debootstrap debian systems. You can use the following as a reference, but better refer to official Debian docs on this topic. They have a MUCH better writeup.
You’ll need qemu-arm-static and debootstrap at the very least.
debootstrap --foreign --variant=minbase --arch=armel wheezy debian-wheezy http://ftp.ru.debian.org/debian/ cp `which qemu-arm-static` ./debian-wheezy/usr/bin/ cp /etc/resolv.conf ./etc/resolv.conf echo "/debootstrap/debootstrap --second-stage" | chroot . |
This will give you the most basic system. Next we’ll need a proper sources.list. You can get one with the online debian sorces.list generator
And paste contents to etc/apt/sources.list of your newly created filesystem.
Next will be installing some extra packages into your chroot. My basic list is below.
apt-get install wget curl ifupdown isc-dhcp-client openssh-server net-tools nano apt-utils inetutils-ping dialog mdadm smartmontools watchdog |
You can also add ajenti web interface or anything else you’d like. Before we copy the rootfs to the hard disk drive it’s a good idea to setup root password with ‘passwd’ and enable getty on our serial port.
The latter can be achieved by adding
T0:23:respawn:/sbin/getty -L ttyS0 115200 vt100
to your etc/inittab file.
Now, finally, let’s create a proper /etc/fstab there. Here’s my fstab:
/dev/md2 /srv/ ext4 noatime,journal_async_commit 0 0 /dev/md1 swap swap defaults 0 0 tmpfs /lib/init/rw tmpfs size=16K,rw,nosuid,relatime,mode=0755 0 0 tmpfs /var/run tmpfs size=1M,rw,nosuid,mode=0755 0 0 tmpfs /var/lock tmpfs size=512K,rw,noexec,nosuid,nodev,mode=1777 0 0 tmpfs /var/tmp tmpfs size=512K,rw,noexec,nosuid,nodev,mode=1777 0 0
And it’s time to put this filesystem on our drive! I do that with tar (something like that, copying it from my head, might need slight tweaks to work)
cd rootfs sudo tar cpf - . --exclude=sys --exclude=dev --exclude=proc| sudo tar vxpf - -C /media/nas_hdd_mount_point |
After that, create all the directories we’ve skipped: /proc /sys /dev /mnt and /srv on the target drive. And you’re done! Unmount the filesystems, detach the drive and put it into your NAS.
Step five: Boot up & set up your system
If everything went okay, your NAS should boot and see the login prompt. Login as root, setup the system as you see fit. It’s just a common linux box now that you can use in whatever way you like. Something like a 5-buck VPS, just with a little bit more storage and processing power.
Step five: Bells and whistles
I made up a cron script, that checks smart params periodically and turns on red light, should any of the drives detect bad blocks. Just drop it into your /etc/cron.hourly:
#!/bin/bash # A more or less hacky way to get drives in the manner they are PHYSICALLY wired # in the system. We can't use /dev/sdX names, because they may swap in respect to # which initializes first. Numbering starts with '1' get_drive() { DRIVE=`find /sys/|grep ata${1}|grep events_async|cut -d"/" -f 12` echo "/dev/$DRIVE" } check_drive() { DISK=`get_drive $1` REALLOC=`smartctl -A $DISK|grep Reallocated_Sector_Ct | awk '{print $10}'` if [ "$REALLOC" != "0" ]; then echo default-on > /sys/class/leds/$2/trigger echo "Yikes! Drive $DISK has $REALLOC reallocs!" else echo none > /sys/class/leds/$2/trigger fi } check_drive 1 dns327l:amber:sata-l check_drive 2 dns327l:amber:sata-r |
Next we need some fan control. So we’ll need to cross-compile dns320l-daemon and put it onto our filesystem. Here’s my /etc/dns320l-daemon.ini:
[Serial] Port = /dev/ttyS1 NumberOfRetries = 3 [Daemon] ServerPort = 57367 ServerAddr = 127.0.0.1 SyncTimeOnStartup = 0 SyncTimeOnShutdown = 0 [GPIO] SysfsGpioDir = /sys/class/gpio PollTime = 100 [Fan] PollTime = 10 TempLow = 25 TempHigh = 34 Hysteresis = 2
It also needs a proper initscript. I’ve written one for you:
#! /bin/sh ### BEGIN INIT INFO # Provides: dns320ldaemon # Required-Start: $remote_fs $syslog # Required-Stop: $remote_fs $syslog # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: DNS320L daemon # Description: Fan and RTC daemon # ### END INIT INFO # PATH should only include /usr/* if it runs after the mountnfs.sh script PATH=/sbin:/usr/sbin:/bin:/usr/bin DESC="NAS Fan and RTC daemon" NAME=dns320l-daemon DAEMON=/usr/sbin/$NAME DAEMON_ARGS="" PIDFILE=/var/run/$NAME.pid SCRIPTNAME=/etc/init.d/$NAME # Exit if the package is not installed [ -x "$DAEMON" ] || exit 0 # Read configuration variable file if it is present [ -r /etc/default/$NAME ] && . /etc/default/$NAME # Load the VERBOSE setting and other rcS variables . /lib/init/vars.sh # Define LSB log_* functions. # Depend on lsb-base (>= 3.2-14) to ensure that this file is present # and status_of_proc is working. . /lib/lsb/init-functions dns_cmd() { { echo "${*}"; sleep 1; } | telnet localhost 57367 } # # Function that starts the daemon/service # do_start() { # Return # 0 if daemon has been started # 1 if daemon was already running # 2 if daemon could not be started start-stop-daemon --start --quiet --exec $DAEMON --test > /dev/null \ || return 1 start-stop-daemon --start --quiet --exec $DAEMON -- \ $DAEMON_ARGS \ || return 2 dns_cmd hctosys # Add code here, if necessary, that waits for the process to be ready # to handle requests from services started subsequently which depend # on this one. As a last resort, sleep for some time. } # # Function that stops the daemon/service # do_stop() { dns_cmd systohc killall -9 dns320l-daemon return 0 } # # Function that sends a SIGHUP to the daemon/service # do_reload() { # # If the daemon can reload its configuration without # restarting (for example, when it is sent a SIGHUP), # then implement that here. # start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME return 0 } case "$1" in start) [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME" do_start case "$?" in 0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;; 2) [ "$VERBOSE" != no ] && log_end_msg 1 ;; esac ;; stop) [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME" do_stop case "$?" in 0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;; 2) [ "$VERBOSE" != no ] && log_end_msg 1 ;; esac ;; status) status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $? ;; #reload|force-reload) # # If do_reload() is not implemented then leave this commented out # and leave 'force-reload' as an alias for 'restart'. # #log_daemon_msg "Reloading $DESC" "$NAME" #do_reload #log_end_msg $? #;; restart|force-reload) # # If the "reload" option is implemented then remove the # 'force-reload' alias # log_daemon_msg "Restarting $DESC" "$NAME" do_stop case "$?" in 0|1) do_start case "$?" in 0) log_end_msg 0 ;; 1) log_end_msg 1 ;; # Old process is still running *) log_end_msg 1 ;; # Failed to start esac ;; *) # Failed to stop log_end_msg 1 ;; esac ;; *) #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2 echo "Usage: $SCRIPTNAME {start|stop|status|restart|force-reload}" >&2 exit 3 ;; esac : |
You may also want to setup watchdog and wd_keepalive to make use of the hardware /dev/watchdog the system has. Btw, the stock firmware does NOT use it. What a shame!
Bonus #1
I keep mysqld running on this box with decent performance. Enough to keep this very blog running. Here’s my my.cnf. I spent a few weeks carefully tuning that, gathering statistics. I preferred myisam to innodb, since myisam has less memory requirements.
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /srv/mysql.altmera/
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
skip-name-resolve
default-storage-engine=myisam
key_buffer_size = 32M
max_allowed_packet = 4M
table_open_cache = 512
sort_buffer_size = 128K
read_buffer_size = 256K
read_rnd_buffer_size = 256K
net_buffer_length = 2K
thread_stack = 64K
query_cache_type = 1
query_cache_size = 32M
query_cache_limit = 8K
thread_stack = 256K
thread_cache_size = 4
tmp_table_size = 16M
max_heap_table_size = 16M
# For low memory, InnoDB should not be used so keep skip-innodb uncommented unless required
skip-innodb
# Uncomment the following if you are using InnoDB tables
#innodb_data_home_dir = /var/lib/mysql/
#innodb_data_file_path = ibdata1:10M:autoextend
#innodb_log_group_home_dir = /var/lib/mysql/
#innodb_log_arch_dir = /var/lib/mysql/
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 16M
#innodb_additional_mem_pool_size = 2M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 5M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates
[isamchk]
key_buffer = 8M
sort_buffer_size = 8M
[myisamchk]
key_buffer = 8M
sort_buffer_size = 8M
[mysqlhotcopy]
interactive-timeout
Here’s mysqltuner.pl output:
root@altmera:~# perl mysqltuner.pl >> MySQLTuner 1.4.0 - Major Hayden <[email protected]> >> Bug reports, feature requests, and downloads at http://mysqltuner.com/ >> Run with '--help' for additional options and output filtering [OK] Logged in using credentials from debian maintenance account. [OK] Currently running supported MySQL version 5.5.40-0+wheezy1 [OK] Operating on 32-bit architecture with less than 2GB RAM -------- Storage Engine Statistics ------------------------------------------- [--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED -InnoDB +MRG_MYISAM [--] Data in MyISAM tables: 73M (Tables: 269) [--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17) [--] Data in MEMORY tables: 0B (Tables: 1) [!!] Total fragmented tables: 6 -------- Security Recommendations ------------------------------------------- [OK] All database users have passwords assigned -------- Performance Metrics ------------------------------------------------- [--] Up for: 10d 11h 1m 29s (2M q [2.402 qps], 36K conn, TX: 5B, RX: 508M) [--] Reads / Writes: 75% / 25% [--] Total buffers: 80.0M global + 1.0M per thread (151 max threads) [OK] Maximum possible memory usage: 231.0M (46% of installed RAM) [OK] Slow queries: 0% (0/2M) [OK] Highest usage of available connections: 26% (40/151) [OK] Key buffer size / total MyISAM indexes: 32.0M/11.9M [OK] Key buffer hit rate: 99.9% (7M cached / 4K reads) [OK] Query cache efficiency: 67.0% (1M cached / 1M selects) [OK] Query cache prunes per day: 0 [OK] Sorts requiring temporary tables: 0% (0 temp sorts / 37K sorts) [!!] Temporary tables created on disk: 48% (64K on disk / 131K total) [OK] Thread cache hit rate: 99% (244 created / 36K connections) [OK] Table cache hit rate: 25% (368 open / 1K opened) [OK] Open file limit used: 54% (642/1K) [OK] Table locks acquired immediately: 99% (827K immediate / 827K locks) -------- Recommendations ----------------------------------------------------- General recommendations: Run OPTIMIZE TABLE to defragment tables for better performance Enable the slow query log to troubleshoot bad queries When making adjustments, make tmp_table_size/max_heap_table_size equal Reduce your SELECT DISTINCT queries without LIMIT clauses Variables to adjust: tmp_table_size (> 16M) max_heap_table_size (> 16M)
Bonus #2
Power measurements.
DNS-327L is a very power efficient box. With a 12V power supply and 2 HDD drives(1TiB WD Black + 1TiB WD Green) it’s drawing 1.15 amps when idle (13.8 Watts) (maximum fan speed). 1.28 Amps with 100% CPU load (dd if=/dev/urandom of=/dev/null), 1.4Amps with some disk load (dd if=/dev/md0 of=/dev/null). 1.38Amps when both of the above tests are running concurrently. That gives a maximum of 16.8 Watts of power consumption under load which is definitely a huge win!
Consumption of the hard disk drives makes up most of these, so they may differ.