Posted: Tue Jun 26, 2018 22:20 Post subject: R7000 nginx build for read-only /opt
Hi folks, I figured someone else might find this useful.
I have an R7000 and am planning on using it for a reverse proxy, among other things, to host various virtual machines I use for web development in my house. I want to run /opt as read-only, so that whatever is being run from there doesn't wear out the flash memory.
Lighttpd is included in Kong's build(s), but unfortunately it does not support SSL passthrough like Nginx does. The problem with making the switch from Lighttpd is that Nginx as-compiled by the maintainer for the Entware package points to /opt for the log, pid, and lock files by default.
This means that before Nginx will write a log anywhere else, it checks for its log folder under /opt to be writable. If you wish to run packages on /opt mounted read-only to avoid wearing out the internal flash, as I do, then that won't work. Nginx never releases that default logfile location. As long as Nginx is running, you can't remount /opt as read-only.
If only Nginx were trying to put its default log under the ramdisk on /tmp, this would be fixed and Nginx would be totally usable with /opt mounted on the internal flash as read-only. You could then, say... redirect Nginx logs to a remote syslog on another machine, which makes way more sense than writing logs to a USB stick (and also eliminates the obvious physical security issue with putting SSL keys on a USB stick, too). Nginx supports remote syslogs by default, so that would be a trivial (just config, no re-compile) change to make.
So this re-build of the Nginx binary does just that: points Nginx logs, lock, and pid to existing folders on the /tmp ramdisk.
How to use, assuming you want opt on the internal flash:
Enable jffs2 as /opt from Administration -> Management
Install Nginx from optware like you normally would (opkg install nginx)
Replace the binary in /opt/sbin with the one linked here (rm -f /opt/sbin/nginx && scp user@192.168.1.xx:nginx /opt/sbin/nginx)
Modify your config file in /opt/etc/nginx to point the logs somewhere else, or disable the logs entirely, or whatever you want to do with them. BIG FAT WARNING: if you do not do this, the logs will fill up all your RAM and (probably) crash your router, you must turn them off or send them somewhere else.
Run nginx once so it can chown all of its directories on /opt to itself (/opt/etc/init.d/S80nginx start && /opt/etc/init.d/S80nginx stop)
In your Administration -> Commands -> Startup, add commands to mount and read-only the internal flash at boot: (mount --bind /jffs/opt /opt && mount -o remount,ro /opt)
Reboot the router
You could of course modify the above to remount your USB as read-only as well, if you prefer.
Here's the Makefile diff if you wish to compile yourself...
As if this writing the current Entware version of Nginx is 1.12.2, compiled against the 2.6 armv7 kernel. Far into the future, you probably want to recompile this yourself using the diff, by following the Entware instructions
In my case I don't really care about the cgi directories since I'm just using it as a proxy. Yours works because of the pointing of those initial read+write files to /var
Joined: 08 May 2018 Posts: 14208 Location: Texas, USA
Posted: Wed Jun 27, 2018 15:56 Post subject:
RNCTX wrote:
Looks like nothing affecting 1.12.2+ (current version on Entware repo).
As I noticed, I guess I read that wrong, and/or they were intentionally ambiguous. My fond memories of nginx is it telling me, "No, you wait, you no need internet access, you been here for hour, you go!" Of course, that was about 10 years ago. Maybe things have improved since then, or maybe that was just the router's configuration that was crap...
I'm not sure of the history myself. Last time I worked with web servers (early 2000s) there was Apache way up here, and everyone else down there. With the Apache foundation splintering into lots of different projects, I suppose the door was left open for someone to come along and compete on "just a web server."
In this case it's the most lightweight option for an SSL reverse proxy that I know of. Built-in support for remote syslog is a big plus for putting it on a router, too.
Joined: 08 May 2018 Posts: 14208 Location: Texas, USA
Posted: Thu Jun 28, 2018 14:05 Post subject:
RNCTX wrote:
I'm not sure of the history myself. Last time I worked with web servers (early 2000s) there was Apache way up here, and everyone else down there. With the Apache foundation splintering into lots of different projects, I suppose the door was left open for someone to come along and compete on "just a web server."
In this case it's the most lightweight option for an SSL reverse proxy that I know of. Built-in support for remote syslog is a big plus for putting it on a router, too.
Definitely a good thing, lightweight. I think the router market definitely benefitted with the development of nginx and others... but it's yet another thing I need to learn eventually. Taking such a long hiatus has proven to be entertaining jumping back into it. Really itching to dig out all my antiques and tinker, but one thing at a time.
As mentioned in the OP it's still going to touch the lock, pid, and logs under var, but after it's running it will send over the logs to the syslog on the webserver behind the router running on 192.168.1.80. By default the syslog entries will be prepended with hostname (ddwrt) and optionally the tag included in each entry above. If the webserver is running ubuntu it has a network-capable syslog by default. You simply need to open up the listener in /etc/rsyslog.conf
The above makes them look like this when some script-kiddie is scanning your machine ...
tl;dr: no writing, to ram or otherwise, other than the initial startup log entry, the lock file, and the pid file.
Since the connection will be SSL end-to-end the same keys must exist on both the router and the webserver(s).
After installing socat and ca-certificates from Entware's repo and acme.sh into /opt/bin, here's a shell script that you can run once per month to renew keys...
Lets Encrypt supports wildcard certificates but only with dns verification. acme.sh has lots of pre-written scripts for various providers, as you can see in mine my domains are on AWS, so I'm using the dns_aws script. The dns provider script goes into whatever folder you specify as your acme home (in the script I explicitly point it to /opt/root/.acme) The sleep for 180 seconds is unfortunately unavoidable, acme.sh has a two minute sleep built in when it needs to put a verification key on the domain, so waiting for 160 seconds gives it plenty of time before starting any subsequent keys.
You will of course have to set up an API key for acme to use and specify it in the dns provider script. In AWS it's just a simple IAM role with permission to list and modify records in the zone. There are examples on the acme.sh github page for lots of providers.
On each of your servers you'll want a cron job that runs about five minutes after each renewal cron job runs to get the new certs over from the router by sftp or scp or some similar. If five minutes isn't real-time enough, the most obvious solution that comes to mind is having the router renewal script FTP the keys to each of your webserver(s) and using pure-ftpd on each webserver. Pure-FTPD has the option to fire shell scripts as triggers based on upload actions, so you could get really close to real-time that way.
Obviously this can't be completely write-free unless I want my ssl keys sitting on the ram disk waiting for the power to go out . But only writing to the jffs once a month for a few bytes to renew SSL certs should be okay.
todo: Error handling on the ssl_renew script, and it really should mail me every time it runs to make sure that the cert renewal succeeded, lest my keys expire without me remembering to check them. msmtp is in the Entware repo too, that should work using a Gmail account I would think. Since acme.sh saves a log it would be simple to cat the log > body-of-email in another script that runs on the end of the ssl_renew script.
