Some black magic: bash, cgi and file uploads

Before you declare me an insane old man, I have to say, that I needed it for an embedded board with only 64MiBs of RAM. And (mostly due to the specifics of the tasks), I ended up using bash hooked to lighttpd via cgi. Bringing on heavy artillery (php or python), really complicated the task, since I mostly needed the outputs of different shell utils.
Anyway, I quickly got to the part where I needed to handle POST requests to upload binary files. First, I googled for a solution, and one of them eventually did what I needed. But alas, it had a drawback.

If you have a look at the source from the link

if [ "$REQUEST_METHOD" = "POST" ]; then
  cat >$TMPOUT
  # Get the line count
  LINES=$(wc -l $TMPOUT | cut -d ' ' -f 1)
  # Remove the first four lines
  tail -$((LINES - 4)) $TMPOUT >$TMPOUT.1
  # Remove the last line
  head -$((LINES - 5)) $TMPOUT.1 >$TMPOUT
  # Copy everything but the new last line to a temporary file
  head -$((LINES - 6)) $TMPOUT >$TMPOUT.1
  # Copy the new last line but remove trailing \r\n
  tail -1 $TMPOUT | perl -p -i -e 's/\r\n$//' >>$TMPOUT.1

You will see that a temporary file is used. And that pretty much sucks, since my flash media on that board was horribly slow, I didn’t want to waste erase cycles, nor I had enough RAM to handle all of that in tmpfs. Ideas? Here goes my implementation of that… thing.

read boundary
read disposition
read ctype
read junk
#Holy fuck, this sucks!
#Due to \n\r line breaks we have 2 extra bytes per line read, 
#6 + 2 newlines == 10 junk bytes
echo  "Content-Type: text/html"
echo ""
echo "Upload complete, $SIZE bytes stored<br>"
echo "Written to $dir, filename $2 ($dir/$file)<br>"
dd ibs=1 obs=512 count=$SIZE of=$dir/$file

That’s it! No temporary files, no need to buffer the whole file in memory, like PHP does. Just one plain dd and some utter voodoo.
No POST variables can be handled, so instead I pump two variables via GET, separated by a ‘:’.
In this raw way it’s pretty much a hell of a security hole, but since the thing I’m working on will unlikely ever be exposed to the internets, and online only when there’s a user nearby – why not?

3 thoughts on “Some black magic: bash, cgi and file uploads”

  1. Hi, thanks for the tips!

    Here’s how I do it, with a POST form upload (slight variation of your script) :

    echo "Content-type: text/html"
    echo ""
    echo ""
    cat header.html
    debugmsg "HTTPS=$HTTPS"

    if [[ "$REQUEST_METHOD" == "POST" ]]; then

    echo "Uploading file, please wait ..."

    # single file upload
    # we must strip the first 4 lines (http headers) of received data, and the last 2 at the end
    read hline1 </dev/stdin && debugmsg "$hline1"
    read hline2 </dev/stdin && debugmsg "$hline2"
    read hline3 </dev/stdin && debugmsg "$hline3"
    read hline4 </dev/stdin && debugmsg "$hline4"

    # calculate remaining extra bytes to strip at the end of data (boundary string + extra cr/lf)
    size=$(( ${#hline1} + 5 ))

    # read remaining data from stdin, except the last $size bytes (http headers)
    head -c -$size "$outfile"
    if [[ $? -eq 0 ]];then
    echo "File has been uploaded successfully"
    echo "Error : Unable to write file!"
    echo "File is too big for uploading, maximum allowed file size is $MAX_FILE_UPLOAD_SIZE bytes"
    echo "Error: Invalid method"

  2. For info, I noticed that some http clients do not work well with this method : the native web browser on Android 2.3.6 on my smartphone, for exemple (badly formatted http headers, although I’m not sure if they are really breaking the rules, I would need to read the RFC’s!).

    But Firefox Mobile and Opera on the same platform work OK 🙂

Leave a Reply