Getting started with Amazon AWS and Boto3

If you are starting from scratch with using Python and Boto3 to script and automate Amazon AWS environments, then this should help get you going.

To begin, you’ll need a few items:
1) Download and install the latest Amazon AwsCli. The is the Command Line client for AWS.
2) You will need credentials for an Amazon AWS environment of course. There are lot’s of articles on how to setup AWS CLI with an AWS account.
3) Download and install the latest Python Release from python.org. For these examples I’m using Python 3.5.2.
4) You will need to ‘pip install boto3’ in a Python environment. You can do this from command line. I recommend creating a Virtual Environment for AWS-Boto3 to keep packages separate.
5) If you’ve never used a Python IDE before, try the free Pycharms Community Edition. I use it and it really helps speed up your coding.

At this point, you should have a working AWS CLI, Python Intepreter, and have pip installed the boto3 library.

It’s a good idea to keep the boto3 documentation handy in another browser tab. Because of how boto3 works, there is no premade library and, when working in pycharms, no autocomplete. Having the docs available to reference the objects and methods saves a lot of time.

In my experience with Boto3, there resources and there are clients.

import boto3
ec2 = boto3.resource('ec2')
ec2client = ec2.client('ec2')

I use the resource to get information or take action on a specific item. I think of it as being at a ‘higher’ level than the client. When using the boto3 resource, you usually have to provide an id of the item. For example:

import boto3
ec2 = boto3.resource('ec2')
vpc = ec2.Vpc('vpc-12345678')

There is usually no searching or enumerating items at the resource level. Now that the class vpc is defined, you can look at the attributes, references, or collections. You will also have a series of actions you can perform on the item.

# Attributes
print(vpc.cidr_block)
print(vpc.state)
# Actions
vpc.attach_internet_gateway(InternetGatewayId="igw-123456")
vpc.delete()
# Collections
vpclist = vpc.instances.all()
for instance in vpclist:
print(instance)

When using a client instead of a resource, you get a level of control over AWS items that is very close to the AWS CLI interface. With the client, you get more detail, can search, and can get very granular in your filters and tasks. Almost every resource has a client available.
For example:

import boto3
# Ec2
ec2 = boto3.resource('ec2')
ec2client = ec2.client('ec2')
# S3
s3 = boto3.resource('s3')
s3client = s3.client('s3')

One of the most useful benefits of using a client is that you can describe the AWS items in that resource, you can filter or iterate for specific items, and manipulate or take actions on those items.
In the VPC example above, when defining the VPC resource, we needed to know the ID of the vpc. With a client, you can list all the items and find the one you want.

import boto3
ec2 = boto3.resource('ec2')
ec2client = ec2.client('ec2')
response = ec2client.describe_vpcs()
print(response)

What you get back is a dictionary with the important item called “Vpcs”. This dictionary item is the output containing all the VPCs. (We could expect this from the Boto3 docs).
So let’s list out all the VPCs from the response

import boto3
ec2 = boto3.resource('ec2')
ec2client = ec2.client('ec2')
response = ec2client.describe_vpcs()
print(response)
for vpc in response["Vpcs"]:
print(vpc)

Now you can see how it starts to break down into the info you want. Here you can see a list of dictionaries from the Vpc.
Next, lets get to the individual items of each VPC.

import boto3
ec2 = boto3.resource('ec2')
ec2client = ec2.client('ec2')
response = ec2client.describe_vpcs()
for vpc in response["Vpcs"]:
print("VpcId = " + vpc["VpcId"] + " uses Cidr of " + vpc["CidrBlock"])

That’s the very basics of it. Post any questions in the comments below. And I’ll be adding more posts on Boto3 in the coming days and weeks.

Chaining Operations and Operators in Linux

This is my list of my most used chaining operators in Ubuntu.

1 .   Semi-colon operator ;


The semi-colon allows you to chain multiple commands so that they run in-order.

# apt-get update ; apt-get upgrade ; echo 'upgrades are done'

 

2 . Single Ampersand &


The single ampersand will execute a command in the background and can chain commands to run in the background. You use the command followed by a space and the ampersand per command.
To run a single command:

# ping www.slsmk.com &

To run more than 1 command in the background:

# ping www.slsmk.com & cp ~/* . & apt-get update &

 

3 . Double Ampersand AND Operator &&


The && symbol, also called the AND Operator, links and executes commands in order only if the previous command is successful.
Technically a command is successful if it completes with exit status 0.
For example, I want to create a directory and file, but I only want to create the file is the directory is created correctly.

# mkdir ~/test && touch ~/test/tempfile1

 

4. Single PIPE |


The PIPE operator is used when you want the output of one command to be the input of a following command.
For example, this will list installed packaged then search for lines with ‘java’.

# dpkg -l | grep java

 

5. OR Operator ||


The OR operatorm || is similar to the AND operator, only here it will execute the following command only if the previous command failed. A command fails if it exits with status code 1.

# mkdir ~/test || echo 'The command failed'

 

6. NOT Operator !


The NOT operator ! is used in a command to identify those items that should be exempt from the command.
For example, imagine a directory with various filetypes and you wanted to remove all files except the PDFs.

# rm -r !(*.pdf)  

 

7. AND OR Operator && ||


This combination of the AND && and the OR || operators delivers what is basically an if-else statement based on the exit status code of the 1st command. BASH Shell has other command to get an if-else result, but this is using just the Operators.
For Example:

# mkdir ~/testdir && echo 'Directory Created' || echo 'Directory Creation Failed'

 

8. Precedence ()


When using && and ||, the exit codes determine whether or not to execute the following commands. Also, it is important to understand that the && and || only evaluate the 2 commands preceding and following the operators. So when using multiple operators, setting groups and precedence comes in handy when you want to ensure groups of commands complete or fail in a certain way.
In this example, Both commands in the 1st set () must exit with 0 in order for the next () set to execute.

# (ls *.pdf > pdf-files.list && cp *.pdf ~\) && (ls *.tar > tar-tiles.list && cp *.tar ~\) || echo 'Needs attention' 

 

9. IF THEN ELSE


Precedence can become very unreadable very quickly. I prefer using BASH’s IF THEN ELSE commands. These work just like any programming languange… IF something is true, THEN execute this command, otherwise (ELSE) run this command. Note that in BASH the if is concluded with ‘fi’.
There are pages of options for IF THEN ELSE which you should explore.
For example, this is a very basic example:

# if ls *.pdf ; then echo 'There are PDFs here' ; else echo 'There are no PDFs here' ; fi 

How to Install OwnCloud to Ubuntu 14.04 LTS

This quick how-to steps you through a simple installation of OwnCloud to a Ubuntu 14.04 server.

First, you need some prereqs:

 sudo apt-get install php5-gd mysql-server

To begin, you need to add the repository for ownCloud for ubuntu 14.04.

wget -nv https://download.owncloud.org/download/repositories/stable/xUbuntu_14.04/Release.key -O Release.key
apt-key add - < Release.key

Next, update the lists and install the package.

sh -c "echo 'deb http://download.owncloud.org/download/repositories/stable/xUbuntu_14.04/ /' >> /etc/apt/sources.list.d/owncloud.list"
apt-get update
apt-get install owncloud

Once the package is installed, access the ownCloud interface at http://SERVERNAME/owncloud

The first time you launch it, it will prompt you to create an admin id and password. Optionally, you can pick the Data folder location and choose MySQL vs SQLite.

owncloud setup 1404

 

After you create the admin user and sign into the OwnCloud Interface, if you are installing this for home use, you will probably want to enable some basic plug-ins.   If you plan on syncing calendars and contacts, then you will need at least those 2 add-ons.

Click on the Files entry on the upper left for the pull down menu shown below.   Click on the + to add new apps.

owncloud setup 1404

owncloud setup 1404

 

Select ‘Productivity’ from the left hand menu and Enable the Calendar and Contacts Applications.

owncloud setup 1404

owncloud setup 1404

 

Once completed, you will be taken to the web interface. Here you can add users and adjust settings as needed. I create the users here and that will pretty much complete the basic install.

ownCloud Admin menu

Last thing to do is load a desktop client available from ownCloud’s web page http://owncloud.org/sync-clients/

Next Steps that you should consider:
1) Enforce https connectivity to the owncloud, this is done through the admin menu selection.
2) Turn on Antivirus. Enabling this app in the Admin->apps menu sets up ClamAV to scan all uploaded content.
3) Make sure you are backing up the Owncloud Data Directory and the MySQL database.

 

There are additional options here for LDAP authentication, email alerts, and much more beyond this basic setup. Explore the add-on applications as well.

How to Install Mylar for use with SABnzbd on Ubuntu Linux

In other posts, I have installed SabNZBD, Sickbeard, and Couchpotato for auto downloading of TV and Movies. In this case, I wanted to use the same process to download Comic Book Files, CBRs, automatically based on created lists, have SABnzbd download them, and have them moved to a permanent folder for viewing access.

Mylar is a great solution. It’s an open-source project maintained by evilhero and works very well and is still under active development.

Before you start, make sure you have a working SABnzbd install on the Host.

Let’s get the pre-reqs out of the way also:

sudo apt-get install git-core

Download the latest copy of Mylar. I like using the /opt directory for full contained applications. Mylar is still in active development so I opt to have the latest development branch downloaded via git.

cd /opt
git clone https://github.com/evilhero/mylar.git -b development

To have Mylar start automatically, copy out the /opt/mylar/mylar.init.d file to /etc/init.d, make it executable, and add it to auto start.

sudo cp /opt/mylar/init-scripts/ubuntu.init.d /etc/init.d/mylar
sudo chmod +x /etc/init.d/mylar
sudo update-rc.d mylar defaults

Create the user account for mylar to use and optionally add it to a user group.

## Create the system only mylar user
sudo adduser --system --no-create-home mylar

## If you use my tutorials, add mylar to the nabd group so it can access the common Downloads folder.   Otherwise, you don't need this. 
sudo usermod -g nzbd mylar

Change the owner of the /opt/mylar directory to the newly created ‘mylar’ user

chown -R mylar:root /opt/mylar

Now create/edit the /etc/default/mylar file and add the following items:

# path to app
APP_PATH=/opt/mylar
# user
RUN_AS=mylar

Start the service

/etc/init.d/mylar start

Mylar should now be available from that host on port 8090 using http://SERVERNAME:8090 The web interface should load and provide you with additional config options.

Now I want to get mylar to work with SABnzbd.

Bring up mylar web interface and click on Settings in the upper right and then click on the Download Settings tab. Fill in the information for your SABnzbd host. Note that in the API field, I used the NZB API from SABnzbd. Mylar doesn’t need full control over SAB, it only needs to add NZBs to SAB. Fill in the fields with the values for your system. If you have been using my examples, the image below should match the values you would use.

Configure Mylar

Last thing to do in order to get the basics up and running is to add a search provider. Here I use a custom NZB indexer. Click on upper right, for settings, select the search providers tab, check the box for Use Newznab, and fill in your indexer’s details. Be sure to save your changes.

Mylar Newznab Config

Now lets get the Post-processing scripts ready. Locate the scripts in /opt/mylar/sabnzbd.   Copy the 1 file  to the directory you have configured in SAB to hold the processing scripts, I prefer to use a softlink instead of copying the file for easier upgrades.

ln -s /opt/mylar/post-processing/sabnzbd/ComicRN.py /opt/sabprocessingscripts/ComicRN.py

In SAB, you will want to add a new category for the mylar downloads. I choose the category ‘comics’ but you can call it whatever you want, just so long as the mylar download category matches exactly.
Mylar Sab Config

With that, you should have a basic, working Mylar installed.

Troubleshooting Permissions
If you encounter problems with the creation, copying, renaming of any files, check your mylar account permissions. In this example, I added mylar to a group called nzbd. When Mylar creates new sub-directories for content, the Linux default is to create the directory with permissions of 0755 meaning that the group nzbd does not have write permissions. You can solve this issue in many ways.

Make sure that you have both Mylar and SABNzbd configured to create directories with permissions of 777 and 0777 respectively.

Mylar Permissions

Mylar Permissions

SABNzbd Permissions

To make life easy, if you already have an nzbd user for SABnzbd to use, you can just run mylar under the nzbd account instead of creating a mylar account. This way mylar and SABNzbd are both running with the same user permissions.   If you do this, you’ll want to use ‘chown -R nzbd:nzbd /opt/mylar’ and change the RUN_AS user from mylar to nzbd.

For the security conscious, you can implement ACLs in linux to handle the permissions on directories. This requires the installation and configuration of an ACL utility on your system.

How to Install Universal Media Server UMS on Ubuntu in Headless mode

Updated for Ubuntu 14.04
10-19-2015

Universal Media Server is a fork off the very useful PS3 Media Server. And although the PS3MS was a great solution, it did have some shortcomings, especially with certain file formats or file containers. I tried UMS and loved it. It is easy to install and, at least for now, streams and transcodes every media file I have to support playback on any device including the PS3 and the sony SMP-N200 I use on other TVs.

So, working with any Ubuntu 14.04 server, here is my step by step to get UMS installed and working.

First you must have Java 7 JRE installed on the server. OpenJava will not work.

apt-get install software-properties-common
apt-get update
apt-get install openjdk-7-jre openjdk-7-headless

With Java installed, we now need to add some other pre-reqs:

apt-get install mediainfo dcraw vlc-nox mplayer mencoder

I’m going to use the /opt directory for the install. Then we download the latest UMS package from sourceforge. You can check the UMS webpage to find the latest version. As I write this, the latest is 5.2.3. After the download is complete, unpack the file with tar. I create a softlink using /opt/ums so that when we need to upgrade, we can just point the softlink to the new directory while not touching the config files that we will be using in /etc/ later on.

cd /opt
wget http://sourceforge.net/projects/unimediaserver/files/Official%20Releases/Linux/UMS-5.2.3-Java7.tgz
tar -xvzf UMS-5.2.3-Java7.tgz
ln -s /opt/ums-5.2.3 ums
rm UMS-5.2.3-Java7.tgz

Next we need to create the init.d script to auto start the app when the server boots, as well as have better control over the service.
We will create /etc/init.d/ums.

nano /etc/init.d/ums

Copy the following into the new file:

#!/bin/bash
#
### BEGIN INIT INFO
# Provides:          ums
# Required-Start:    $local_fs $remote_fs $network
# Required-Stop:     $local_fs $remote_fs $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Starts UMS program.
# Description:       Java Upnp Media Server dedicated to PS3
### END INIT INFO

#set -x

# Author: Papa Issa DIAKHATE <paissad@gmail.com>
#
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DESC="Universal Media Server"
NAME=ums
UMS_PROFILE=/etc/UMS.conf
DAEMON=/opt/ums/UMS.sh
DAEMON_OPTS="console"
SCRIPTNAME=/etc/init.d/ums
UMS_START=1 # Wether to start or not UMS ver at boot time.
DODTIME=30  # Time to wait for the server to die, in seconds.
            # If this value is set too low you might not
            # let the program to die gracefully and 'restart' will not work

test -x $DAEMON || exit 1

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions

# Include ums defaults if available
if [ -f "/etc/default/$NAME" ] ; then
        . /etc/default/$NAME
fi

# May we run the init.d script ?
[ $UMS_START = 1 ] || exit 1

#--------------------------------------------------------------------------
# Some color codes
txtred=$'\e[0;31m' # Red
txtylw=$'\e[0;33m' # Yellow
txtrst=$'\e[0m'    # Text Reset
#--------------------------------------------------------------------------
warnout(){
    echo >&2 -e ""$txtylw"Warning:$txtrst $1"
}
#--------------------------------------------------------------------------
running(){
    pid=`pgrep -f 'java .*ums.jar.*'`
}
#--------------------------------------------------------------------------
do_start(){
    running && { warnout "$NAME is already running !"; exit 0; }
    echo "Starting $DESC : $NAME"
    UMS_PROFILE="$UMS_PROFILE" start-stop-daemon --start --quiet --background --oknodo \
        --exec $DAEMON -- $DAEMON_OPTS

}
#--------------------------------------------------------------------------
do_stop(){
    running || { warnout "$NAME is NOT running !"; exit 0; }   
    local countdown="$DODTIME"
    echo -e "Stopping $DESC : $NAME \c "
    kill -9 $pid
    while running; do
        if (($countdown >= 0)); then
            sleep 1; echo -n .;
            ((--countdown))
        else
            break;
        fi
    done
    echo
    # If still running, then try to send SIGINT signal
    running && { \
        echo >&2 "Using kill -s SIGINT instead"; \
        echo >&2 "If you see this message again, then you should increase the value of DODTIME in '$0'."; \
        kill -2 $pid; \
    }

    if [ -e "/usr/share/ums/debug.log" ]; then
    count=9
    while [ $count -ge 1 ]
    do
    if [ -e "/usr/share/ums/debug.log.$count" ]; then
        plus=$((count+1))
        mv "/usr/share/ums/debug.log.$count" "/usr/share/ums/debug.log.$plus"
    fi
        count=$((count-1))
    done
    if [ -e "/usr/share/ums/debug.log" ]; then
        mv "/usr/share/ums/debug.log" "/usr/share/ums/debug.log.1"
    fi
    fi

    return 0
}
#--------------------------------------------------------------------------
do_force-stop(){
    running || { warnout "$NAME is NOT running !"; exit 0; }   
    echo "Stopping $DESC : $NAME"
    kill -9 $pid
    if [ -e "/usr/share/ums/debug.log" ]; then
    count=9
    while [ $count -ge 1 ]
    do
    if [ -e "/usr/share/ums/debug.log.$count" ]; then
        plus=$((count+1))
   mv "/usr/share/ums/debug.log.$count" "/usr/share/ums/debug.log.$plus"
    fi
       count=$((count-1))
    done
    if [ -e "/usr/share/ums/debug.log" ]; then
   mv "/usr/share/ums/debug.log" "/usr/share/ums/debug.log.1"
    fi
    fi
}
#--------------------------------------------------------------------------
do_status(){
    echo -n " * $NAME is "
    ( running || { echo "NOT running "; exit 0; } )
    ( running && { echo "running (PID -> $(echo $pid))"; exit 0; } )
}
#--------------------------------------------------------------------------
case "$1" in

    start|stop|force-stop|status)
        do_${1}
        ;;
    restart|reload)
        do_stop
        do_start
        ;;
    force-restart|force-reload)
        do_force-stop
        do_start
        ;;
    *)
        echo "Usage: $SCRIPTNAME {start|stop|force-stop|restart|force-restart|reload|force-reload|status}"
        exit 1
        ;;
esac

Now add execute permissions to the script and add the UMS script to update-rc.d

chmod +x /etc/init.d/ums
update-rc.d ums defaults

A sample conf file that you could use is at /opt/ums/UMS.conf and could be copied into /etc/UMS.conf and edited to fit your needs. It has all the configurable options and is probably more than most will need. You should also copy in the WEB.conf file as well to handle web streams if you use that functionality. (Thanks Wolfgang Hochweller)

cp /opt/ums/UMS.conf /etc/
cp /opt/ums/WEB.conf /etc/

Configuration is done to the /etc/UMS.conf file. At the very least you will want to add the location of the media to share.

folders=/mnt/media/tv, /mnt/media/movies, /mnt/media/music 

Pay attention to the following items, especially for those hosts with multiple NICs.

network_interface=
hostname=
port=
folders=

Now start UMS:yty

service ums start

That’s it, it should be running and advertising itself as UPNP/DLNA on the local network.

Install Couchpotato on Ubuntu Server

Couchpotato is an automatic NZBD downloader. Very similar to SickBeard in that it will watch newsgroups for NZBD files, download the NZBD, and hand the file off to SABNzbd, other download utility, or save it off to a directory so that the system can download the media.

In this example, I have a Linux server running SABNzbd. If you don’t have this setup, have a look here: http://www.slsmk.com/setup-automatic-tv-downloads-using-sickbeard-and-sabnzbd/
The goal here is to have couchpotato pull watch for and pull the nzbd, hand it off to SABnzbd, and let the server download the file.

I like to keep these “all in one” apps in /opt/.

 cd /opt

If you don’t already have Git installed, you will need that:

 apt-get install git-core

Grab the source from git.

 git clone https://github.com/RuudBurger/CouchPotatoServer.git

Once the GIT clone is complete, you should have a /opt/CouchPotatoServer directory.

Since I don’t want to run couchpotato as root, I am going to change the user it runs under. I already had a user called ‘nzbd’ from the previous SABnzbd/Sickbeard installations, so I’m going to use that since it would already have permissions to write out the nzbd files to the correct directories for SAB.

Here lets change the permissions on the new CouchPotato Directory.

 chown -R nzbd:root CouchPotatoServer/

Next, copy the init script out to /etc/init.d/ and make it executable.  Copy the Defaults file into /etc/default

cp /opt/CouchPotatoServer/init/ubuntu /etc/init.d/couchpotato
chmod +x /etc/init.d/couchpotato
cp /opt/CouchPotatoServer/init/ubuntu.default /etc/default/couchpotato

Edit the /etc/default/couchpotato file and edit the path and user as needed.   If you are using my tutorials, I have a common account called nzbd for all Download applications that use SABnzbd.

nano /etc/default/couchpotato

# path to app
APP_PATH=/opt/CouchPotatoServer/           
# user
RUN_AS=nzbd   

And finally, add it to start automatically.

# update-rc.d couchpotato defaults

With the Install completed, open up a browser and connect to the linux host on port 5050 to access couchpotato’s config page.

Welcome
Since this is a new install, we don’t need to import a data.db. If you have one, Enter the location of the data.db here.

General
Here we add an id/pw to the couchpotato website. I always add one to all my sites. You can also change the default port if you wanted. I left it at the default of 5050.

Downloaders
Here you can specify the nzbd download app you are using. In this example, we use SABnzbd. So I check the box next to SAB and enter the information.
1) SAB host, I’m running on localhost:8080
2) SAB api key. This is found in the SAB webpage -> config -> general. Copy and paste the API key from the API key field.
3) Add a category. I used ‘movies’.

Providers
This is where you add in the information on your news provider. Whatever service you have, check the box and fill in to info required. The API keys for these services can be found on that service’s website under your account logon. Copy and paste the API key if needed.

Renaming
Couchpotato will move and rename files for you after a complete download. Fill in the fields as needed. In my example, I am moving items:

FROM    /home/nzbd/Downloads/complete/
TO      /mnt/media/movies

Leave everything else as default.

Add movies to your wanted list
I love this little feature. Add the +Couchpotato link to your browser and as you browse IMDB or other sites, you can add the movie directly to your couchpotato want-list.

That’s it. Open up your browser, point it to your host on 5050, and start adding items.

If you enter a search item and nothing pops up, restart couchpotato on the server to kick it into gear.

Block the Win 10 style Data Gathering from Windows 7/8/Server2008

Microsoft Windows 10 has some pretty intrusive data gathering built into the operating system.   It collects anonymous and not-so-anonymous data from users of the Operating System.   Today, MS released 4 Optional Updates to bring the Win 10 Style Telemetry and Data Gathering to previous versions of Windows.

The 4 updates are:

KB3075249
KB3077715
KB3078667
KB3080149

MS Fanboys will claim there’s nothing personal that MS takes… but I say, do you really trust them?
People will claim ios and android already do this… but I say, why give them anything about your self at all if you can avoid it?
Many say we already post about ourselves online in social media… For those of us who don’t, I say don’t apply these updates .

To block them, you simply hide the updates in Windows Update.
Got to the Control Panel -> Windows Updates or START -> Search -> Windows Update

Select Optional Updates
Block win 7 Data gathering 1

Right Click each of the 4 updates and select ‘Hide Updates’
Block win 7 Data gathering 2

Hit OK and that’s it. Gone until MS tries to sneak it in again down the road.

Another option is to use this OpenSource Program to disable the tracking:
https://github.com/10se1ucgo/DisableWinTracking

Lastly, If you already installed the KBs, you can uninstall them directly from the “Programs and Features” in Control Panel.

StrongSwan Ipsec VPN for Remote Users with Certificate Based Authentication

This is a working strongswan ipsec config that can be used for a roadwarrior setup for remote users utilizing certificate based authentication instead of id/pw.

This is a pure IPSEC with ESP setup, not L2tp.
This is not 2 factor, it is cert only.

To get started:

sudo apt-get install strongswan

You need is a CA that is capable of registering AltNames in a cert. OpenSSL can do this easily. I used this guide to setup the basic openssl CA. http://www.freebsdmadeeasy.com/tutorials/freebsd/create-a-ca-with-openssl.php

Once the CA is ready and you have generated your ca cert and ca private key, you next need to create a cert for the ipsec host and a cert for the end user.

For the Server:
Since I need the Alt Names in the certs, make a copy of /etc/ssl/openssl.cnf to be used for the Server.

cp /etc/ssl/openssl.cnf /etc/ssl/openssl-for-server.cnf
# Extension copying option: use with caution. copy_extensions = copy [ v3_req ] # Extensions to add to a certificate request basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName=@alt_names [alt_names] DNS.1 = ipsecvpn.mydomain.com DNS.2 = another_name.mydomain.com

Now, using Openssl, create the request for the server, fill in the details of the req as needed, then sign the request.

openssl req -new -nodes -out ipsechost-req.pem -keyout private/ipsechost-key.pem -config /etc/ssl/openssl-for-server.cnf
openssl ca -config /etc/ssl/openssl-for-server.cnf -out ipsechost-cert.pem -in ipsechost-req.pem

Copy the certs to the correct locations for strongswan to use.

cp cacert.pem /etc/ipsec.d/cacerts
cp ipsechost-cert.pem /etc/ipsec.d/certs
cp private/ipsechost-key.pem /etc/ipsec.d/private/

Stongswan is configured using the /etc/ipsec.conf and /etc/ipsec.secrets files.
This is a very simple config that will work for providing access to remote users:
Edit /etc/ipsec.conf

 
# /etc/ipsec.conf - strongSwan IPsec configuration file
config setup

conn %default
        ikelifetime=60m
        keylife=20m
        rekeymargin=3m
        keyingtries=1

conn common
        left=IP_OF_IPSEC_HOST          # Ip of the host
        leftcert=ipsechost-cert.pem    # the cert we just created and copied
        leftid=@ipsecvpn.mydomain.com  # the Alt name in the Cert we just created
        leftsubnet=172.16.31.0/24      # The internal subnet the remote user wants to access
        right=%any                     # Connections can come from anywhere
        rightsourceip=192.168.1.0/24   # Use this pool of IPs to assign to these inbound connections
        auto=add

conn ikev2
        keyexchange=ikev2 
        also=common

Edit the /etc/ipsec.secrets file

: RSA ipsechost-key.pem

Restart/Reload IPsec.

 
ipsec restart

I like to watch logs just to be sure there are no errors:

 
tail -f /var/log/syslog /var/log/auth.log

Next we create a client cert. We need another copy of the openssl config file for user requests since the Alt Name changes from DNS to Email.

cp /etc/ssl/openssl-for-server.cnf /etc/ssl/openssl-for-users.cnf
[ v3_req ] # Extensions to add to a certificate request basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName=email:copy #[alt_names] #DNS.1 = ipsecvpn.mydomain.com #DNS.2 = another_name.mydomain.com

Create the request, fill in the details as needed for the user especially the email address, and Sign the request. The email address specified in the request prompts will be used in the cert for the Alt name and in the config for the user’s side of the tunnel.

openssl req -new -nodes -out user1-req.pem -keyout private/user1-key.pem -config /etc/ssl/openssl.cnf 
openssl ca -config /etc/ssl/openssl-for-users.cnf -out user1-cert.pem -in user1-req.pem 

You need to copy the user1-cert.pem, user1-key.pem, and the cacert.pem to the user’s machine. We will need these file to complete the VPN.

On the User’s Side:

sudo apt-get install strongswan

Copy the files into the proper directories
user1-cert.pem to /etc/ipsec.d/certs
user1-key.pem to /etc/ipsec.d/private
cacert.pem to /etc/ipsec.d/cacerts

Edit the client side ipsec.conf. This is a working config:

conn %default
     ikelifetime=60m
     keylife=20m
     rekeymargin=3m
     keyingtries=1
     keyexchange=ikev2

conn roadwarrior
     leftsourceip=%config                # This will take an IP from the ip pool on server
     leftcert=ipsecuser1-cert.pem        # The user cert we copied in
     leftid=ipsecuser1@openpeak.org      # This is the email included in the Alt Name in the user cert
     leftfirewall=yes
     right=cert-ipsec-vpn.openpeak.org   # The location of the host, FQDN or IP 
     rightid=cert-ipsec-vpn.openpeak.org # the Altname used by the ipsec host
     rightsubnet=172.16.31.0/24          # the subnet on the servers side you want to access. 
     auto=start

Edit the ipsec.secrets file

: RSA ipsecuser1-key.pem

On the client, issue an “ipsec restart” and it should attempt to build the tunnel with that you are done.

Troubleshooting:
Use “ipsec statusall” to get details on the tunnels. From the server, a healthy tunnel looks like this:

Security Associations (1 up, 0 connecting):
       ikev2[11]: ESTABLISHED 3 minutes ago, HOST_IP[vpnhost.mydomain.com]...REMOTE_IP[ipsecuser1@mydomain.com]
       ikev2[11]: IKEv2 SPIs: 49c4512b56436e5b_i 6276554588ce1803_r*, public key reauthentication in 50 minutes
       ikev2[11]: IKE proposal: AES_CBC_128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_2048
       ikev2{9}:  INSTALLED, TUNNEL, ESP in UDP SPIs: cd1e015f_i cd3cb1c1_o
       ikev2{9}:  AES_CBC_128/HMAC_SHA1_96, 0 bytes_i, 0 bytes_o, rekeying in 12 minutes
       ikev2{9}:   HOST_INTERNAL_SUBNET=== POOL_IP_ASSIGNED_TO_REMOTEUSER

Use “ipsec listall” for details on the host’s certs and configs. Here we want to be sure Alt Names are good, and the CA and certs are loaded correctly.

List of X.509 End Entity Certificates:
  altNames:  ipsecvpn.mydomain.com
  subject:  "C=US, ST=NY, O=OpenPeak, OU=IT, CN=ipsecvpn.mydomain.com, E=ipsechost@mydomain.com"
  issuer:   "C=US, ST=NY, L=NY, O=mydomain, OU=IT, CN=ipsecserver-ca, E=ca@mydomain.com"
  serial:    10:03
  validity:  not before Mar 18 20:44:25 2015, ok
             not after  Nov 24 20:44:25 2028, ok 
  pubkey:    RSA 2048 bits, has private key
  keyid:     10:15:77:ae:2e:a4:e8:3f:cc:1f:6d:a9:d9:80:bd:9f:41:fb:63:e5
  subjkey:   3f:0c:bf:01:2f:c7:16:be:d4:83:5c:76:81:56:a9:f1:3a:84:b4:5f
  authkey:   b7:61:d7:32:19:65:c3:10:1a:43:23:27:bc:46:29:e5:ff:df:03:1c

List of X.509 CA Certificates:

  subject:  "C=US, ST=NY, L=NY, O=mydomain, OU=IT, CN=ipsecserver-ca, E=ca@mydomain.com"
  issuer:   "C=US, ST=NY, L=NY, O=mydomain, OU=IT, CN=ipsecserver-ca, E=ca@mydomain.com"
  serial:    db:e9:16:e0:44:a3:57:83
  validity:  not before Mar 18 15:49:45 2015, ok
             not after  Mar 15 15:49:45 2025, ok 
  pubkey:    RSA 2048 bits
  keyid:     18:47:07:92:b8:3d:a0:bb:88:bf:27:2b:4d:0b:a7:79:8b:c1:1b:ba
  subjkey:   b7:61:d7:32:19:65:c3:10:1a:43:23:27:bc:46:29:e5:ff:df:03:1c
  authkey:   b7:61:d7:32:19:65:c3:10:1a:43:23:27:bc:46:29:e5:ff:df:03:1c

Note that if you want to enable 2 factor with this, change the openssl request for the Clients to omit the -nodes option. This will prompt you for a password during the certificate creation that must be entered every time the client wants to connect.

Windows 2008 R2 Server Windows update unknown error

I’m putting this out there for anyone else. For the last 3 months I had a windows 2008 R2 server that would not apply windows updates. “An unknown error has occurred” is all I would get.

This was fixed today. The cause seemed to be an Disk Filter applied to the Local System Disk. I had never heard of this before today.

The command FLTMC lists out the Filter names on the disk. This is from a working server. I don’t know if yours will match exactly.

C:Usersadministrator.server>fltmc

Filter Name                     Num Instances    Altitude    Frame
------------------------------  -------------  ------------  -----
VirtFile                                0       429999.280700    0
msnfsflt                                0       364000         0
luafv                                   1       135000         0

Now when my server was in an error condition, this listing had an additional entry with the highest ‘Altitude’ value.

C:Usersadministrator.server>fltmc

Filter Name                     Num Instances    Altitude    Frame
------------------------------  -------------  ------------  -----
CpsFsJnl                                0       429999.999999    0
VirtFile                                0       429999.280700    0
msnfsflt                                0       364000         0
luafv                                   1       135000         0

This filter was the cause of my issue. This was a remnant of a Symantec CPS (continuous protection server) installation that was supposedly uninstalled years ago. It apparently left this filter installed and active on the server. It must have been dormant there for years until a windows update or something caused the error condition.

The filer is an system file called cpsfsjnl.sys and a quick search found it buried in the Program Files Directory. I deleted the file (after making a backup just in case). I also exported then deleted the following registry entries:

[HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesCpsFsJnl]
[HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesCpsFsJnlEnum]

I rebooted the server, checked FLTMC to be sure the CpsFsJnl was no longer listed, and then ran the Windows updates.

So, bottom line, in my case, Symantec CPS left a Virtual Disk Filter on the server that caused the error condition. Removing it fixed Windows updates.

Exchange 2007 Outlook Web Access Error The Password does not meet the minimum security requirements

In Microsoft Exchange OWA (outlook web access) a user will attempt to change the account password and encounter an error “The password supplied does not meet the minimum security requirements”. This error happens even though the GPO Policy has password complexity disabled.

OWA error message

The cause of the error is the minimum password age in one or more of the applied GPOs on the OWA host.

Change the Policy for Minimum password age to 0. Then either run GPUPDATE /FORCE or just reboot the host.

GPO for Min Password Age