Transmission Crashing, PIA, portforwarding, killswitch,cron

Post new topic   Reply to topic    DD-WRT Forum Index -> Advanced Networking
Author Message
portsup
DD-WRT User


Joined: 20 Oct 2018
Posts: 210

PostPosted: Sun Aug 04, 2019 14:39    Post subject: Transmission Crashing, PIA, portforwarding, killswitch,cron Reply with quote
OK so I have had for a long time a good setup for Transmission using PIA so openvpn with portforwarding and a killswitch. Except I have had some other issues which I resolved because of a fragmented SMR drive aka nightmare drive. Remaining issues were transmission crashing or just going a bit funny. So I have come to solve it with cron scripts to monitor things and restart periodically. I also came to rework much of the way I was setting up transmission for the VPN with killswitch etc. I think the code is now much better.

I will now give the code,setup and explanation.

I setup jffs in nvram. I had an issue with mount times and openvpn starting when using another drive. You could restart openvpn to avoid the need for nvram.

You obviously need transmission from entware installed, but I think you also need busybox to run scripts as "#!/bin/sh", which is important.

This is my first script portforward.sh


Code:
#!/bin/sh
cp -a  /jffs/openvpncl/* /tmp/openvpncl/
cp -a /opt/transmission/config/settingscopyfrom.json /opt/transmission/config/settings.json


I load it in the openvpn config with "up /jffs/portforward.sh". This script overwrites some things. At /jffs/openvpncl/ I have previously saved the route-up.sh and route-down.sh scripts from /tmp/openvpncl/. I have altered them so when copied back they have extra code I want run. The transmission overwrite is to restore the settings file from one I backed up. I have another problem where the settings file gets altered by the various remotes I use to control transmission.... this really messed me around before I worked it out. Anyway this solves that.

Next script route-up.sh

Code:

#!/bin/sh
iptables -D POSTROUTING -t nat -o tun1 -j MASQUERADE
iptables -I POSTROUTING -t nat -o tun1 -j MASQUERADE
iptables -D INPUT -i tun1 -j ACCEPT
iptables -D FORWARD -i tun1 -j ACCEPT
iptables -D FORWARD -o tun1 -j ACCEPT
iptables -I INPUT -i tun1 -j ACCEPT
iptables -I FORWARD -i tun1 -j ACCEPT
iptables -I FORWARD -o tun1 -j ACCEPT
for IP in `cat /tmp/openvpncl/policy_ips` ; do
    ip rule add from $IP table 10
done
ip addr add 192.168.168.168 dev tun1
ip route add default via $route_vpn_gateway table 10
ip route flush cache
echo $ifconfig_remote >>/tmp/gateway.txt
echo $route_vpn_gateway >>/tmp/gateway.txt
echo $ifconfig_local >>/tmp/gateway.txt
stopservice dnsmasq -f
startservice dnsmasq -f
cat /tmp/resolv.dnsmasq > /tmp/resolv.dnsmasq_isp
env | grep 'dhcp-option DNS' | awk '{ print "nameserver " $3 }' > /tmp/resolv.dnsmasq
cat /tmp/resolv.dnsmasq_isp >> /tmp/resolv.dnsmasq
sleep 2
touch /tmp/resolv.dnsmasq
/jffs/portforward1.sh


2 bits I added to this. One is "ip addr add 192.168.168.168 dev tun1" which gives the vpn device tun1 that IP in addition to others it has. This sets up some of the routing for that IP to work over the VPN. I also added 192.168.168.168 to the pbr, which sets up the rest of the routing, you could add it here with "ip rule add from 192.168.168.168 table 10",. I bind transmission to that ip. That ip has no other routing than through the vpn, so it can not leak, vpn up down or not running. You could use another ip just make sure it is not in the usual range you use otherwise it could leak, unless you where to use other methods, iptables or preserving the pbr ip rules etc .

The other code I add is "/jffs/portforward1.sh" to run the next script which is

Code:
#!/bin/sh
/jffs/portforward2.sh &
exit 0


Why am I using a script to jump to the next script? Well openvpn has some restrictions on the way scripts run from it and some how this gets around it. This is the next script

Code:
#!/bin/sh
nice -n 1 /opt/bin/transmission-daemon -g /opt/transmission/config
sleep 15
port=$(curl --interface tun1 "http://209.222.18.222:2000/?client_id=##YOUR SHA CALC HERE##" 2>/dev/null | awk -F ':' '{ print $2 }'| awk -F '}' '{ print $1 }')
echo $port>/tmp/port.txt
/opt/bin/transmission-remote 192.168.3.1 -p "$port"
iptables -t nat -I PREROUTING -p tcp --dport $port -j DNAT --to 192.168.168.168


"nice -n 1 /opt/bin/transmission-daemon -g /opt/transmission/config" this runs transmission with my config stored at that location. I am actually running it on an additional non smr hard drive (mounted as /opt/) to also save it's parts there and then copy to the SMR drive to avoid "messing it up"... Nice -n 1 reduces the process priority so transmission isn't so much of a hog on the rest of things. AKA makes it play "nice" lol linux and names... Smile

"port=$(curl --interface tun1 "http://209.222.18.222:2000/?client_id=##insertyour sha256sum from calc here##" 2>/dev/null | awk -F ':' '{ print $2 }'| awk -F '}' '{ print $1 }')" is actually ONE LINE ( forum wraps it) and you will need to enter your sha256sum calc number to make it work. Here https://emn178.github.io/online-tools/sha256.html the first google result for a "sha256sum calculator", just hit some random caracters into the top box for your number in the bottom one. This code gets the port from PIA and saves it as $port. The "sleep 15" above it is import to make this work! You can take a look at https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=313661 for more info. There I posted my original code which worked fine. The next poster obviously didn't read it and went on for another 7 pages....

"echo $port>/tmp/port.txt" is just for debug and unnecessary.

"/opt/bin/transmission-remote 192.168.3.1 -p "$port"" this sends the port to transmission while it is running, hence you need to start it before.

"iptables -t nat -I PREROUTING -p tcp --dport $port -j DNAT --to 192.168.168.168" this forwards the port to that ip I am using for transmission.

Other code is the route-down.sh script
Code:

#!/bin/sh
iptables -D INPUT -i tun1 -j ACCEPT
iptables -D POSTROUTING -t nat -o tun1 -j MASQUERADE
ip route flush table 10
killall transmission-daemon


I only add "killall transmission-daemon" which is sort of a extra or double kill switch but isn't necessary for that as it won't leak without it. It does prevent a duplicate process when the route comes back up and my script launches transmission again. You could setup the launch of transmission outside the vpn process and so not need this. Or do an if statement to check if it is already running. It may be better to get the port entered into the settings file for my next bit. I am not sure if transmission writes to the settings file when crashing? or when it writes to the settings file, maybe only on shutdown. It does lock it or something when running.


So here is the cronjobs

Code:

*/5 * * * * root /jffs/transmissionstayup.sh
0 */5 * * * root /jffs/transmissionrestart.sh


this runs the first script transmissionstayup.sh every 5 minutes. Which is this

Code:
#!/bin/sh


vpn=$(ip link show |grep -v grep |grep -c tun1)
trans=$(ps |grep -v grep |grep -c transmission-daemon)

if [ "$vpn" -eq "1" ] && [ "$trans" -eq "0" ]
then
nice -n 1 /opt/bin/transmission-daemon -g /opt/transmission/config
fi

if [ "$vpn" -eq "0" ] && [ "$trans" -eq "1" ]
then
killall transmission-daemon
sleep 30
openvpn --config /tmp/openvpncl/openvpn.conf --route-up /tmp/openvpncl/route-up.sh --route-pre-down /tmp/openvpncl/route-down.sh --daemon
fi

if [ "$vpn" -eq "0" ] && [ "$trans" -eq "0" ]
then
openvpn --config /tmp/openvpncl/openvpn.conf --route-up /tmp/openvpncl/route-up.sh --route-pre-down /tmp/openvpncl/route-down.sh --daemon
fi

if [ "$trans" -gt "1" ]
then
killall transmission-daemon
sleep 30
nice -n 1 /opt/bin/transmission-daemon -g /opt/transmission/config
fi



This restarts transmission if it crashes, kills duplicates of transmission, restarts openvpn if it crashes.

The second script transmissionrestart.sh runs every 5 hours and is this



Code:
#!/bin/sh


vpn=$(ip link show |grep -v grep |grep -c tun1)
trans=$(ps |grep -v grep |grep -c transmission-daemon)

if [ "$vpn" -eq "1" ] && [ "$trans" -eq "1" ]
then
killall transmission-daemon
sleep 30
nice -n 1 /opt/bin/transmission-daemon -g /opt/transmission/config
fi


Doing this just keeps transmission from going strange. Time from restart is obviously something you would have to play at. Every hour seems silly but 5 is reasonable.

Extra info( may help)

This is my transmission config, with port changed and password for some anonymity here.

Code:
{
    "alt-speed-down": 50,
    "alt-speed-enabled": false,
    "alt-speed-time-begin": 540,
    "alt-speed-time-day": 127,
    "alt-speed-time-enabled": false,
    "alt-speed-time-end": 1020,
    "alt-speed-up": 50,
    "bind-address-ipv4": "192.168.168.168",
    "bind-address-ipv6": "::",
    "blocklist-enabled": false,
    "blocklist-url": "http://www.example.com/blocklist",
    "cache-size-mb": 12,
    "dht-enabled": true,
    "download-dir": "/tmp/mnt/sda1/torrents",
    "download-queue-enabled": true,
    "download-queue-size": 20,
    "encryption": 0,
    "idle-seeding-limit": 30,
    "idle-seeding-limit-enabled": false,
    "incomplete-dir": "/opt/transmission/parts",
    "incomplete-dir-enabled": true,
    "lpd-enabled": false,
    "message-level": 2,
    "peer-congestion-algorithm": "",
    "peer-id-ttl-hours": 6,
    "peer-limit-global": 1020,
    "peer-limit-per-torrent": 200,
    "peer-port": 34987,
    "peer-port-random-high": 65535,
    "peer-port-random-low": 49152,
    "peer-port-random-on-start": false,
    "peer-socket-tos": "default",
    "pex-enabled": true,
    "port-forwarding-enabled": true,
    "preallocation": 2,
    "prefetch-enabled": false,
    "queue-stalled-enabled": true,
    "queue-stalled-minutes": 30,
    "ratio-limit": 2,
    "ratio-limit-enabled": false,
    "rename-partial-files": true,
    "rpc-authentication-required": false,
    "rpc-bind-address": "0.0.0.0",
    "rpc-enabled": true,
    "rpc-host-whitelist": "",
    "rpc-host-whitelist-enabled": true,
    "rpc-password": "deleted",
    "rpc-port": 9091,
    "rpc-url": "/transmission/",
    "rpc-username": "",
    "rpc-whitelist": "127.0.0.1,192.168.3.*",
    "rpc-whitelist-enabled": true,
    "scrape-paused-torrents-enabled": true,
    "script-torrent-added-enabled": false,
    "script-torrent-added-filename": "",
    "script-torrent-done-enabled": false,
    "script-torrent-done-filename": "",
    "seed-queue-enabled": false,
    "seed-queue-size": 10,
    "speed-limit-down": 100,
    "speed-limit-down-enabled": false,
    "speed-limit-up": 100,
    "speed-limit-up-enabled": false,
    "start-added-torrents": true,
    "trash-original-torrent-files": false,
    "umask": 18,
    "upload-slots-per-torrent": 100,
    "utp-enabled": true
}


These settings are working for me with a r7000. Using parts storage (incomplete-dir) on another drive protects the the smr drive. I use preallocate on that drive to reduce fragmentation but it is not really needed. Preallocate is meant to reduce messing up a SMR drive if you write to it, but I don't see that at all working as torrents get written piece by piece all over the place and even though the space is reserved it will still damage the data next to each piece as it is written and the drive goes on long long i/o waits fixing it's self. SMR is fine but you can only use them for write large and seldom, read when ever.

Disabling encryption speeds things up a bit. But an interesting thing I stumbled on was setting the peer-limit-global closer to 1024 restricts the amount of files transmission can keep open and so reduces drive i/o, and so improving drive access for other things like ftp. I have it at 1020 but 1024 before, both seem to work well, transmission, ftp are all more responsive. Side effect is I have occasionally crashed openvpn or the tunnel with that setting so I will have to see how that goes with my new cron scripts. I found a post here https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=311060 about monitoring the tunnel with a ping through it, I may use that if I have more issues with it.

Also using EXT4 on the drives with ddwrt significantly improves performance over NTFS. NTFS is like really bad and the linux drivers still mess things up on it after so many years.

Ext2fsd https://sourceforge.net/projects/ext2fsd/ is a great driver to read write to ext drives on windows. It even includes the linux command mke2fs.exe to format drives to any ext. JUST BE CAREFUL as I learnt the hard way with one drive wiped that using windows device safely remove corrupts ext4 partitions. Would be nice if that bug was "fixed"....

That's all for now, hope I have helped someone.
Sponsor
portsup
DD-WRT User


Joined: 20 Oct 2018
Posts: 210

PostPosted: Tue Sep 29, 2020 7:50    Post subject: Reply with quote
Wow Pia has been very unreliable with their old API for getting a port. I have made new scripting for the newer "Next Gen" API. Rolling Eyes

I made portforward2.sh push to another script just so I could test it as it was complicated.

Code:
#!/bin/sh
nice -n 1 /opt/bin/transmission-daemon -g /opt/transmission/config
sleep 15
/jffs/portforward3new.sh


Code:
#!/bin/sh



###### PIA Variables ######
curl_max_time=15
curl_retry=5
curl_retry_delay=15
user='YOURPIAUSERNAMEHERE'
pass='YOURPIAPASSWORDHERE'



tok=$(curl --insecure --silent --show-error --request POST --max-time $curl_max_time --header "Content-Type: application/json" --data "{\"username\":\"$user\",\"password\":\"$pass\"}" "https://www.privateinternetaccess.com/api/client/v2/token" | /opt/bin/jq -r '.token')
echo $tok   
echo $tok>/tmp/token.txt
vpn_ip=$(traceroute -m 1 209.222.18.222 | tail -n 1 | awk '{print $2}')
pf_host="$vpn_ip"
pf_getsig=$(curl --insecure --get --silent --show-error --retry $curl_retry --retry-delay $curl_retry_delay --max-time $curl_max_time --data-urlencode "token=$tok" $verify "https://$pf_host:19999/getSignature")
pf_payload=$(echo $pf_getsig | /opt/bin/jq -r .payload)
pf_getsignature=$(echo $pf_getsig | /opt/bin/jq -r .signature)
pf_bind=$(curl --insecure --get --silent --show-error --retry $curl_retry --retry-delay $curl_retry_delay --max-time $curl_max_time --data-urlencode "payload=$pf_payload" --data-urlencode "signature=$pf_getsignature" $verify "https://$pf_host:19999/bindPort")
pf_port=$(echo $pf_payload | /opt/bin/base64 -d | /opt/bin/jq -r .port)
pf_token_expiry_raw=$(echo $pf_payload | /opt/bin/base64 -d | /opt/bin/jq -r .expires_at)

echo $pf_payload   
echo $pf_payload>/tmp/pf_payload.txt
echo $pf_getsignature   
echo $pf_getsignature>/tmp/pf_getsignature.txt

echo $pf_port   
echo $pf_port>/tmp/port.txt
iptables -t nat -I PREROUTING -p tcp --dport $pf_port -j DNAT --to 192.168.168.168
/opt/bin/transmission-remote 192.168.3.1 -p "$pf_port"


This would need to be run I think at least once a week. I am running it at least once a day. You will get a new port when you run it.

I also added the PIA dns servers to my routeup.sh

Code:
#!/bin/sh
iptables -D POSTROUTING -t nat -o tun1 -j MASQUERADE
iptables -I POSTROUTING -t nat -o tun1 -j MASQUERADE
iptables -D INPUT -i tun1 -j ACCEPT
iptables -D FORWARD -i tun1 -j ACCEPT
iptables -D FORWARD -o tun1 -j ACCEPT
iptables -I INPUT -i tun1 -j ACCEPT
iptables -I FORWARD -i tun1 -j ACCEPT
iptables -I FORWARD -o tun1 -j ACCEPT
for IP in `cat /tmp/openvpncl/policy_ips` ; do
    ip rule add from $IP table 10
done
ip addr add 192.168.168.168 dev tun1
#ip rule add from $ifconfig_local table 10
ip rule add to 209.222.18.222 table 10
ip rule add to 209.222.18.218 table 10
ip rule add to 1.1.1.1 table 10
ip route add default via $route_vpn_gateway table 10
ip route flush cache
echo $ifconfig_remote >>/tmp/vpnremote.txt
echo $route_vpn_gateway >>/tmp/vpngateway.txt
echo $ifconfig_local >>/tmp/vpnip.txt
#sed -i 's/.*bind-address-ipv4.*/    "bind-address-ipv4": "'$ifconfig_local'",/' /opt/transmission/config/settings.json
stopservice dnsmasq -f
startservice dnsmasq -f
cat /tmp/resolv.dnsmasq > /tmp/resolv.dnsmasq_isp
env | grep 'dhcp-option DNS' | awk '{ print "nameserver " $3 }' > /tmp/resolv.dnsmasq
cat /tmp/resolv.dnsmasq_isp >> /tmp/resolv.dnsmasq
sleep 2
touch /tmp/resolv.dnsmasq
/jffs/portforward1.sh


specifically the bit

Code:
ip rule add to 209.222.18.222 table 10
ip rule add to 209.222.18.218 table 10
ip rule add to 1.1.1.1 table 10


in addition to this I put in the gui section services/services/dnsmasq/Additional DNSMasq Options/


Code:
no-resolv
server=209.222.18.222
server=209.222.18.218
server=1.1.1.1


These were to solve dns issues the router had. It also means that those dns servers go over VPN so when you run


Code:
vpn_ip=$(traceroute -m 1 209.222.18.222 | tail -n 1 | awk '{print $2}')


you'll get your vpn internal server ip address. There is many ways you could get that.

In addition to the main script of portforward3new.sh

you will need this portforwardrebind.sh

Code:
#!/bin/sh



###### PIA Variables ######
curl_max_time=15
curl_retry=5
curl_retry_delay=15

   
pf_payload=$(cat /tmp/pf_payload.txt)
echo $pf_payload
pf_getsignature=$(cat /tmp/pf_getsignature.txt)
echo $pf_getsignature


vpn_ip=$(traceroute -m 1 209.222.18.222 | tail -n 1 | awk '{print $2}')
pf_host="$vpn_ip"
pf_bind=$(curl --insecure --get --silent --show-error --retry $curl_retry --retry-delay $curl_retry_delay --max-time $curl_max_time --data-urlencode "payload=$pf_payload" --data-urlencode "signature=$pf_getsignature" $verify "https://$pf_host:19999/bindPort")
 
echo done rebinding port


This needs to be run by a cronjob every 15minutes or you will lose that port.

Code:
*/15 * * * * root /jffs/portforwardrebind.sh


the script relies on info stored in text files from portforward3new.sh . I am not certain how it all works but I think if you ask for the info again you get a new port. So if you ran portforward3new.sh every 15mins you would also get a new port every 15mins. Not great for keeping torrents going well.

Other things I needed to make it all work were entware packages jq and coreutils-base64, for the bins jq and base64.

so

Code:
opkg install coreutils-base64

Code:
opkg install jq


The scripts can likely be improved much, but it works well.
egc
DD-WRT Guru


Joined: 18 Mar 2014
Posts: 12837
Location: Netherlands

PostPosted: Tue Sep 29, 2020 9:31    Post subject: Reply with quote
Nice job, PIA portforwarding is going to change, like you are describing Sad

https://github.com/fm407/PIA-NextGen-PortForwarding/blob/master/pia-nextgen-pf.sh

_________________
Routers:Netgear R7000, R6400v1, R6400v2, EA6900 (XvortexCFE), E2000, E1200v1, WRT54GS v1.
Install guide R6400v2, R6700v3,XR300:https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=316399
Install guide R7800/XR500: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=320614
Forum Guide Lines (important read):https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=324087
eibgrad
DD-WRT Guru


Joined: 18 Sep 2010
Posts: 9157

PostPosted: Tue Sep 29, 2020 10:08    Post subject: Reply with quote
Good grief, are you telling me that PIA is changing their crappy port forwarding API once again, after all we did to support it w/ my own scripting many months ago?!

https://pastebin.com/P9nmpyxh

_________________
ddwrt-ovpn-split-basic.sh (UPDATED!) * ddwrt-ovpn-split-advanced.sh (UPDATED!) * ddwrt-ovpn-client-killswitch.sh * ddwrt-ovpn-client-watchdog.sh * ddwrt-ovpn-remote-access.sh * ddwrt-ovpn-client-backup.sh * ddwrt-mount-usb-drives.sh * ddwrt-blacklist-domains.sh * ddwrt-wol-port-forward.sh * ddwrt-dns-monitor.sh (NEW!)
egc
DD-WRT Guru


Joined: 18 Mar 2014
Posts: 12837
Location: Netherlands

PostPosted: Tue Sep 29, 2020 10:28    Post subject: Reply with quote
Oh yes believe it or not a lot of things are going to change they are now switching over to their next gen servers.

Indeed that was a lot of "fun" getting PIA port forwarding getting to work but all down the drain, I heard it is not even functioning any more

Their new recent setup guide from end of august is horribly wrong.
I contacted their support pointing them to my updated setup guide, their answer "Thank you for your interest" but did not changed anything (I did not look the last week)

My subscription is going to end in October and as they are still not supporting WireGuard on the router (they might in the future, but are saying that for over a year Sad ) I think I will end it.

https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=326414

_________________
Routers:Netgear R7000, R6400v1, R6400v2, EA6900 (XvortexCFE), E2000, E1200v1, WRT54GS v1.
Install guide R6400v2, R6700v3,XR300:https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=316399
Install guide R7800/XR500: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=320614
Forum Guide Lines (important read):https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=324087
portsup
DD-WRT User


Joined: 20 Oct 2018
Posts: 210

PostPosted: Tue Sep 29, 2020 12:12    Post subject: Reply with quote
Yes the old port forwarding broke many months ago, then they only fixed it for a few servers, now it seems dead for good.

What is interesting is the portforwarding for next gen that I got working is labeled v2. There is another slightly different but far more complicated version labeled v3. Having to make a bind request ever 15min is well interesting. I guess they don't want to run out of ports.

It's great you guys put so much effort into things, but the previous port forwarding was quite easily covered with a few lines. You can see some of the very complicated scripts made for this new version and on trying to make them work they didn't and had no errors. On stripping them down and reading and understanding them, then also came errors and I managed to make it work.

I think much effort on scripts would be better spent working on the ddwrt code?
bushant
DD-WRT Guru


Joined: 18 Nov 2015
Posts: 2029

PostPosted: Tue Sep 29, 2020 12:22    Post subject: Reply with quote
egc wrote:
they are still not supporting WireGuard on the router (they might in the future,
but are saying that for over a year


I talked to them Saturday, their answer was:
"Keep checking the website".

My subscription will parbly go the way of the Dodo bird.
portsup
DD-WRT User


Joined: 20 Oct 2018
Posts: 210

PostPosted: Tue Sep 29, 2020 22:28    Post subject: Reply with quote
Let me help you check the website ...

https://www.privateinternetaccess.com/helpdesk/news/posts/introducing-next-generation-port-forwarding

The scripts have wireguard stuff in them in addition to ovpn things.
Display posts from previous:    Page 1 of 1
Post new topic   Reply to topic    DD-WRT Forum Index -> Advanced Networking All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum