Setup Automatic TV Downloads using SickRage and SABnzbd

The goal here is to setup an automated system that check for new episodes of TV shows, download the episodes automatically, copy them to a central media PC, and rename them as needed.

Disclaimer:  This info is for educational purposes only.   Don’t pirate stuff, kids.

Newsgroups can be used to distribute large files.  Tracking these items is handled using NZB files.   Think of NZB as the newsgroup equivalent of a torrent tracker.   With the NZB file of the content you want, you can download the media to your PC and watch it just like you would with a torrent.

So here’s what we are after, in a nutshell:

  • Automatically Find the nzb for content you want.
  • Use the nzb to download the item from a news server
  • Manage the nzb and downloaded content so that files are renamed and copied to the correct folders automatically.
  • Watch for new episodes and download them when available.

 

You will need an account on a news server.  If your ISP still offers NNTP news servers then great.  Otherwise you’ll need a 3rd party news source.   Giganews, Easynews, Usenet, are a few examples.   Most of these have a trial periods to try before you buy.  Easynews has an NNTP only, unlimited account for $10 per month.  For this setup, NNTP is all we need.

Next, we will need an account on a host that serves as a NZB indexer.  There are a few good ones out there so try to find a private one or find one outside the US (if you have hardware and bandwidth to spare you could even run your own if you really wanted).  I went with oznzb.  Whichever one you choose, this will be is the source of the NZB you would use to grab media from the news server.

Now that we have services ready to go, lets setup our server.   I am using an Ubuntu 10.0.4 LTS server for this example, but I’ve tested and used this on 12.04 as well.

Install sabnzbdplus using APT. SAB is the app that will manage the downloading of NZBs and their content. Think of it as a ‘uTorrent’ for news. I will also add the mobile theme in case you want to manage the app from a mobile phone. The ubuntu repo has an older version so we need to add a custom repo for the latest code:

sudo add-apt-repository ppa:jcfp/ppa
sudo apt-get update
sudo apt-get install sabnzbdplus sabnzbdplus-theme-mobile

Create a user called nbzd. The service can run as this user with no extra privileges.

sudo adduser nzbd

Edit the file /etc/default/sabnzbdplus and edit the file to include the nzbd user, allow hosts on network to connect using 8080.

# [required] user or uid of account to run the program as:
USER=nzbd
HOST=0.0.0.0
PORT=8080

Now you can start the nzbd as a service.

$ sudo /etc/init.d/sabnzbd start

Now you can access the config page and wizard at
http://server-ip:8080/sabnzbd
It will run you through a wizard to get your configuration information.

sabnzbdplus

1) Select your language
2) Select your theme
3) Select access options and id/pw.
4) Enter your Usenet provider info from whatever service you subscribed to from above.
5) If you signed up for oznzb, then enter the info here. the API key is found on the oznzb account page.

Now take some time to go through the Configs and setup your download location folders, temp folders, etc.

Now lets install SickRage. This is the app that will automatically grab nzb files for content we want to download.
SickRage is written in python. Every ubuntu server should already have python installed and ready to go. SickRage is a fork off the the very popular SickBeard. I prefer SickRage because it is updated more often, has a native email notification, and handles torrent files better then other options.

Start with getting some pre-reqs:

 
sudo apt-get install git-core python python-cheetah

Use git to get the latest code:

cd /opt/
git clone https://github.com/SiCKRAGETV/SickRage.git sickrage

Change owners

 chown -R nzbd:root /opt/sickrage

In the sickrage folder look for directory autoProcessTV/. Rename the sample file

cd /opt/sickrage
cp -a autoProcessTV/autoProcessTV.cfg.sample autoProcessTV/autoProcessTV.cfg

Edit the file and change it to support SSL if needed:

 
[SickBeard]
host=localhost
port=8081
username=
password=
web_root=
ssl=1

Now we need to tell SABnzbd where to find the file.
Since I have SABnzbd doing different types of post processing, I create a folder and fill that folder with softlinks to the actual scripts.

 
mkdir /opt/sabprocessingscripts
chown nzbd:root /opt/sabprocessingscripts
cd /opt/sabprocessingscripts
ln -s /opt/sickrage/autoProcessTV/sabToSickBeard.py sabToSickBeard.py

From the SABnzbd web pages go to FOLDERS and find the field called Post-Processing Scripts folder. Enter the path to the file we just edited:

 /opt/sabprocessingscripts

Now go to the CATEGORIES link and enter a new line Enter
Category: TV
Priority: Default
Processing: Default
Script: sabToSickbeard.py
Folder: tv
Group: tv
And ADD it when done.

You can start it from the cli using

/opt/sickrage/SickBeard.py

I wanted it to start as a service so it would launch at startup as well.

To autostart SickRage as a service, we need to copy and edit 2 files.

sudo cp /opt/sickrage/runscripts/init.ubuntu /etc/init.d/sickrage
sudo chmod +x /etc/init.d/sickrage

Edit the file /etc/init.d/sickrage and make sure you have the following:

# Source SickBeard configuration
if [ -f /etc/default/sickrage ]; then
    . /etc/default/sickrage
else
    echo "/etc/default/sickrage not found using default settings.";
fi

# Source init functions
. /lib/lsb/init-functions

# Script name
NAME=sickrage

# App name
DESC=SickRage

Next, edit /etc/default/sickrage and add the following:

SR_HOME=/opt/sickrage/
SR_DATA=/opt/sickrage/
SR_USER=nzbd

Give the file execute permissions:

 
chmod +x /etc/default/sickrage

Let the service start on bootup

 update-rc.d sickrage defaults

Start the service

 $ /etc/init.d/sickrage start

Note: if you get any permission errors, run this:

 
chown -R nzbd:root /opt/sickrage

You should be able to open up the sickrage webpages now at http://server-ip:8081. You’ll want to go into CONFIG – SEARCH PROVIDERS and enter in the info for the NZB service you signed up for above or create a custom entry on that page if it is not listed.

Next we need to get SickRage working with SABnzbd. In the SickRage interface go to CONFIG -SEARCH SETTINGS. Check the box for NZB Search and select Sabnzbd+ from the list. Enter in the details for our SABnzbd install including the SAB API key which you can get from the SABnzbd web page at CONFIG -GENERAL -API. Test and Save changes when done.

Next go into SickRage web pages CONFIG – NAMING

  • Pick a Name pattern that you like from the dropdown.

Under Metadata

  • Use the Pulldown to select the Type depending upon what devices you have available

And with that, you should be complete. Go into SickRage, add in TV directory where you keep or want to keep you media files, and start adding shows.

Android Phone Email Provisioning Errors with Exchange Active Sync

This problem came for me recently. Several employees where trying to setup Android phone email clients to check mail on an Exchange 2010 mail server. They would constantly get the following message: “Remote Security Administration, The Server requires that you allow it to remotely control some security features of your phone.”

Android Email Client Active Sync Error

There was a similar issue in Android with email account setups traced to some bug in the way Android handles Active Sync Policies.    This time around, we confirmed this to be the case with all versions of android from 2.2 through ICS using the stock client.

The Fix / Workaround:

To get around this issue, we removed the default Active sync policy on the accounts in question.   Not a great fix since you don’t get the ability to force pins and the like, but since the policy wasn’t applied correctly anyway, it was an easy choice.

Step 1 – In Exchange MMC, Create a blank A.S. policy.   MMC -> Organization Config -> Client Access -> Active Sync Policy Tab.     Create the new policy here and call it “Deleteme”     Highlight the policy and make it the default.

Step 2 – Since the GUI forces you to select a default policy, open up the Power shell.   Run the following command:

 Set-ActiveSyncMailboxPolicy “Deleteme” -IsDefaultPolicy $False

This forces the Deleteme policy to not be the default. The GUI should now show no default policy at all.

Step 3 – Assign the Deleteme policy to the mailbox for the user. MailBox Properties -> Mailbox Features -> Exchange ActiveSync (Click properties). Browse and select the Deleteme policy.

Step 4 – Go back to MMC -> Organization Config -> Client Access -> Active Sync Policy Tab and delete the Deleteme policy.

Now check the assigned Active sync policy on the mailbox and it should be blank.  

Have the user test the email account setup.    After I did this, all the mailboxes could be provisioned correctly and would no longer get that popup error.

I also found out that after any Service Pack or Rollup pack was applied to the Exchange Server, a Default Active Sync Policy was recreated.    I had to run through the same procedure again after the update to SP2 and to Rollup 4.

 

 

Python Script to reset Home DVR

My Home DVR system seems to be built on a Busybox Linux install with the DVR running from that OS.   The issue with my DVR is that the consumer/home dvr devices for under $500 seem to have issues with remote viewing.   I have a system that works well enough when you are using the client on the same LAN.  However, when trying to view remote video, the DVR viewing software would shutdown.  Sometimes the system will just lock up for no reason and stop all recording.    Error in the code maybe?  Buffer issue, possibly?   Whatever the reason, If I used a smartphone to peek in on my cameras or perhaps the wind blew the wrong way, I run the risk of locking up the DVR app and thus stop recording video.   The Linux OS is still up and running, it’s just the Recording and playback app that crashed.

After several tech calls to the company and getting the latest firmware for the unit, as well as trying the same process from various Android and iOS apps, the issue was still there. What I found is that my unit as well as most cheap comsumer units is built off the same busybox clone.

So I could remove the system and get another that would work with the remote viewers.    Not an attractive option.  Considering I might end up with the same flaky code.  I also can not afford more expensive units.

Since I have a home Linux server running various functions I including a small Check_mk/Nagios install for monitoring, I did the following:

The viewing app listens on 18004 by default for remote requests.   In nagios, I added the following check to the DVR system’s cfg file in nagios/conf.d.

define service{
use                     generic-service
host_name               <my dvr host name>
service_description     TCP_18004
check_command           check_tcp!18004
event_handler           restart-dvr
}

This will watch port 18004.  If the app locks up for any reason,  18004 stops listening.

Next I added the following item to nagios/conf.d/commands.cfg.

define command{
command_name    restart-dvr
command_line    /opt/scripts/resetdvr.py
}

In the event 18004 is not reachable (indicating a failure) nagios calls the command “restart-dvr”.   This command simply executes the python script I wrote at /opt/scripts/resetdvr.py

#!/usr/bin/python
import sys
import telnetlib
HOST = "<dvr IP address>"
user = "<root username>"
password = "<your password"
tn = telnetlib.Telnet(HOST)
tn.read_until("login: ")
tn.write(user + "n")
tn.read_until("Password: ")
tn.write(password + "n")
tn.read_until("~ $")
tn.write("rebootn")
tn.write("exitn")
tn.read_all()

To help lock it up a bit:

chown root:root /opt/scripts/resetdvr.py
chmod 700 /opt/scripts/resetdvr.py

When the failure is detected, nagios runs the python script which simply issues a reboot.  The DVR comes back in in about 45 seconds or so and its back online.

Have a better suggestion?  Leave me a comment.

Setup DHCP and DNS for Dynamic DNS updates on Ubuntu Server

After a setup of Bind and DHCP, this section will let the DNS zone dynamically update when new DHCP clients get an IP lease.

Reference the previous posts at
Install DHCP Services on Ubuntu 10.04
Installing DNS Services on Ubuntu Server

1)      Backup the Zone files  

cp /etc/bind/zones/* /var/lib/bind/

2)      Change the ownership of the copied files

chown -R bind:bind /var/lib/bind/ 

3)      Create a shared secret key between DHCP and DNS so no one else can update the DNS zones. 

dnssec-keygen -r /dev/urandom -a HMAC-MD5 -b 128 -n USER DHCP_UPDATER

  You should see something like “Kdhcp_updater.+221+012129”

4)      Cat the kdhcp private file and locate the “key=”. It will look something like this “012345+ABCD12345670ABCD==”

5)      Copy the key so that it can be pasted into the config later.

6)      Add the key to the bind config and specify the zones that can be updated.

7)      Edit /etc/bind/named.conf.local and add the following

#Secret key for Updated from DHCP
key DHCP_UPDATER {
algorithm HMAC-MD5.SIG-ALG.REG.INT;
# Paste in the generated key here.   Should be in quotes
       secret "PASTE KEY HERE";
}; 

zone "mydomain.local" {
type master;
# Writable File is in /var/lib instead of etc/bind/zones
        file "/var/lib/bind/mydomain.local.db";

#Allow Updates from any service that has secret key in DHCP UPDATER
   allow-update {key DHCP_UPDATER; };
};

zone "1.168.192.in-addr.arpa" {
type master;
notify no;

# Writable File is in /var/lib instead of etc/bind/zones
     file "/var/lib/bind/rev-192.168.1.db";

#Allow Updates from any service that has secret key in DHCP UPDATER
      allow-update {key DHCP_UPDATER; };
};

8)      Edit the /etc/dhcp3/dhcpd.conf IF you are on 12.04, Edit the /etc/dhcp/dhcpd.conf

9)      Add/Edit the following items to the file

# Change update style to interim
ddns-update-style  interim; 
 ignore client-updates;  #overwrite client configured FQDNs 
 ddns-domainname "mydomain.local."; 
 ddns-rev-domainname "in-addr.arpa."; 

 key DHCP_UPDATER { 
         algorithm HMAC-MD5.SIG-ALG.REG.INT; 

         #Paste in the generated key here.   Should be in quotes 
         secret "<paste key here>"; 
 }; 

 zone mydomain.local. { 
         primary 127.0.0.1; 
         key DHCP_UPDATER; 
 } 

 zone 1.168.192.in-addr.arpa. { 

         primary 127.0.0.1; 
         key DHCP_UPDATER; 
 }

10)   Tighten permissions on Config files.   The config files have the secret key which we don’t want people reading so we remove READ rights from the files.
For 10.04

chmod o-r /etc/bind/named.conf.local 
chmod o-r /etc/dhcp3/dhcpd.conf

For 12.04

chmod o-r /etc/bind/named.conf.local 
chmod o-r /etc/dhcp/dhcpd.conf

11)   Restart Services and test setup.

  1. Restart networking on any host
  2. Check DNS using “host <pc name>
  3. Check reverse using host <ip address>

12)   Cleanup Files

  1. Remove key file “rm Kdhcp_updater.*
  2. Remove old zone files “rm –R /etc/bind/zones

Some “important” pointers

Database files being rewritten by bind
The dns database files are now being rewritten by the bind service. Always stop the bind service before making any changes to the database files, otherwise they might be overwritten by bind.

Examples of how to stop and start the bind service:

sudo /etc/init.d/bind9 stop
sudo /etc/init.d/bind9 start

Install DHCP Services on Ubuntu

The goal here is simply to install DHCP Services so that the Ubuntu host will allocate IP addresses to clients on the network.   Using this method allows more control over DHCP options handed out to the clients over most consumer based routers.  I’ve done this on both Ubuntu 10.04 LTS and 12.04 LTS with one minor difference.   In 10.04, conf files are in /etc/dhcp3.  In 12.04, conf files are in /etc/dhcp.     With that in mind, here are the steps.

Get the DHCP server from repository.

apt-get install dhcp3-server

   (Don’t worry if it says startup fails, its not configured yet)

For 10.04, edit /etc/dhcp3/dhcpd.conf

For 12.04, edit /etc/dhcp/dhcpd.conf

Add the following sections for the subnet

#option definitions common to all supported networks...
#replace mydomain.local your internal dns namespace
#replace server1.mydomain.local with the domain name of the dns server
option domain-name "mydomain.local";
option domain-name-servers server1.mydomain.local;

#If this DHCP server is the official DHCP server for the local
# network, the authoritative directive should be uncommented.
authoritative;

# This is a very basic subnet declaration.
subnet 192.168.1.0 netmask 255.255.255.0 {
range 192.168.1.100 192.168.10.150;
option routers gateway.mydomain.local;
}

# also add in the following for each ip reservations:
# Instead of assigning an IP address here, I referred
# to the dns entry. This way, if I need to change the IP
# I only need to edit the DNS entry.

host  PCNAME;{
hardware ethernet MAC ADDRESS GOES HERE;
fixed-address PCNAME.mydomain.local;
}

Start the DHCP service on 10.04 with

 /etc/init.d/dhcp3-server start 

Start the DHCP service on 12.04 with

 start isc-dhcp-server

Add user dhcpd as owner and assign permissions
For 10.04

Chown dhcpd:root /etc/dhcp3/dhcpd.conf
Chmod 660 /etc/dhcp3/dhcpd.conf

For 12.04

Chown dhcpd:root /etc/dhcp/dhcpd.conf
Chmod 660 /etc/dhcp/dhcpd.conf

Install Bind9 for DNS Services on Ubuntu

This article gives you a step by step on installing bind9 to Ubuntu Server for dns resolution for an internal ‘local’ domain. This was done on Ubuntu 10.04 LTS 64 bit and also on 12.04 LTS 64 bit.   The zone ‘mydomain.local’ can be renamed to anything really.   I like to keep internal domains as ‘local’ on split dns so that there is a clear distinction between mydomain.com and mydomain.local addresses.  Internal subnet space used here is 192.168.1.0/255.255.255.0

Install Bind9 for DNS services

1) Run  apt-get install bind9

2) Create a zones folder “mkdir /etc/bind/zones

3) Edit /etc/bind/named.conf.local

zone "mydomain.local" {
type master;
file “/etc/bind/zones/mydomain.local.db";
};

zone "1.168.192.in-addr.arpa" {
type master;
notify no;
file “/etc/bind/zones/rev-192.168.1.db";
};

4) Create the Zone files for mydomain.local. Replace ‘server1’ with your dns server name and adjust IP addresses as needed.

vim /etc/bind/zones/mydomain.local.db

 $ORIGIN .
 $TTL 604800     ; 1 week
 mydomain.local             IN SOA  server1.mydomain.local. admin.mydomain.local. (
 2011032909 ; serial
 604800     ; refresh (1 week)
 86400      ; retry (1 day)
 2419200    ; expire (4 weeks)
 604800     ; minimum (1 week)
 )

NS      server1.mydomain.local.
 $ORIGIN mydomain.local.
 server1                 A       192.168.1.10
 server2                 A       192.168.1.11
 gateway                A       192.168.1.1

5) Create the reverse Zone file vim /etc/bind/zones/rev-192.168.1.db

$ORIGIN .
 $TTL 604800     ; 1 week
 1.168.192.in-addr.arpa IN SOA  server1.mydomain.local. admin.mydomain.local. (
 2012080301 ; serial
 604800     ; refresh (1 week)
 86400      ; retry (1 day)
 2419200    ; expire (4 weeks)
 604800     ; minimum (1 week)
 )

NS      server1.
 $ORIGIN 1.168.192.in-addr.arpa.
 1                       PTR     gateway.mydomain.local.
 15                     PTR     server1.mydomain.local.
 245                   PTR     server2.mydomain.local.

6) Lastly don’t forget to add the forwarders by editing /etc/bind/named.conf.options

forwarders {
 208.67.222.222;
 208.67.220.220;
 };

How to Install VMWare Tools v4 on an Ubuntu Host

This should work with Ubuntu 12.04 LTS and 11.10 server.

Unlike window VM within the vSphere window, adding VMTools requires just a little work.   This install requires the ‘build-tools’ packages be installed if not already present.

“apt-get install linux-headers-server build-essential”

1) On the host to be updated, select the “Install/Upgrade VMware Tools” menu selection from the menu.  This will make the VMWare tools ISO available to the host.

2) Make sure you update the server.

sudo apt-get update
sudo apt-get upgrade

3) Create a mount point

sudo mkdir -p /media/cdrom

4) Mount the VMWare Tools ISO

sudo mount /dev/cdrom /media/cdrom

5) CD into the new mount

cd /media/cdrom

6) Copy the VM*.tar file to a temporary directory.

sudo cp VM*.tar.gz /temp

7) CD to the temp folder

cd /temp

8) Unmount the ISO, we are done with it.

sudo umount /media/cdrom

9) Extract the TAR file

sudo tar xzvf VM*.tar.gz

10) CD into the new vmware tools folder

cd vmware-tools-distrib

11) If you are on Ubuntu 11.10, you should create a /usr/lib64 directory to get around an error in the install script.

sudo mkdir /usr/lib64

12) Run the install script.  Note that the -d flag will ‘auto-answer’ all questions with the default.   Omit the -d if you want to answer each manually.

sudo ./vmware-install.pl -d

13) Reboot to complete.

sudo shutdown -r now