Simple Log Shipper To AWS S3

Background

I had a need to ship uncompressed application logs to an S3 bucket. Logs were originally CSV files created by a specific application, and not standard .log type files that were converted to JSON. You could use this shipping method for just about any file with minor modification. This guide assumes you already have the AWS CLI setup in your environment. Script is set in cron to start at system boot, and runs in the background monitoring the shipping directory for incoming files. I had to observe the following in the process:

• Not to touch or process (ship) anything that hadn’t completed writing yet!
• Detect when an incoming json file completed writing and only then, act upon it.
• Compress the json file before uploading it to S3
• After compression, upload the file.

#!/bin/bash

# this script takes incoming json files, compresses them, then uploads to S3.  After uploading, deletes the last file uploaded.

# Uncompressed json files needing to be shipped to AWS S3 landed here:
in="/var/log/shipping/out/"
# variables to hold counts
jcount=`ls -lhtr "${in}"*.json | wc -l`
zcount=`ls -lhtr "${in}"*.gz | wc -l`

# Compress before we ship (because storage costs bucks)
compress() {
        while [ $jcount -gt 1 ] 
        do
                gzip ${pfile}
                rm -f "${pfile}"
                jcount=`ls -lhtr "${in}"*.json | wc -l`
                pfile=`ls -lhtr "${in}"*.json | tail -"${jcount}" | head -1 | awk '{ print $9 }'`
        done
}

# Ship only compressed files, first in first out.
ship() {
        while [ $zcount -gt 1 ]
        do
                aws s3 cp ${zfile} s3://my-log-bucket
                rm -f "${zfile}"
                zcount=`ls -lhtr "${in}"*.gz | wc -l`
                zfile=`ls -lhtr "${in}"*.gz | tail -"${zcount}" | head -1 | awk '{ print $9 }'`
        done
}

# Main loop that keeps checking for incoming files to process and ship.
inotifywait -m "${in}" -e close_write | while read file
do
        jcount=`ls -lhtr "${in}"*.json | wc -l`
        compress;
        zcount=`ls -lhtr "${in}"*.gz | wc -l`
        ship;
done;

exit 0;