Install Splunk on Ubuntu

Splunk is a popular enterprise level tool for log collection, analysis and management. While you can obtain an enterprise license, most functions are available in the free community edition.

Setup is very easy:

  1. Download and move the .tar.gz file to the appropriate server (i386 vs. amd64)
  2. sudo dpkg -i splunk*.deb
  3. Start the server:

    sudo /opt/splunk/bin/splunk start

    The first time you run after installation or update you will have to accept terms.

  4. Access the admin screen:

    http://HOSTNAME:8000

    login (admin/changeme)
    change password

  5. Go to Settings/Forwarding * Receiving
    – add new (port 9997)
  6. Open firewall port (if enabled):

    sudo ufw allow 8000
  7. Now to start as a service…

    sudo /opt/splunk/bin/splunk enable boot-start

Competitors:

REFERENCES

DomainKeys Identified Mail (DKIM) Email

DomainKeys (originally from Yahoo!) and Cisco, and later as an industry collaboration, is a means for and organization to claim responsibility for sending a message, in a way that can be validated by a recipient. As a result, emails are “signed” by the outgoing SMTP server and can be verified against a DNS record. Depending upon the receiver, unsigned emails MAY be treated or marked as SPAM as they could be forgeries.

The below instructions assume Ubuntu (Debian) and Postfix, but could likely be modified for other platforms.

  • Install OpenDKIM:
    sudo apt-get install opendkim opendkim-tools
  • Setup initial configuration:

    sudo vi /etc/opendkim.conf

    ADD:
    Domain example.com
    KeyFile /etc/postfix/dkim.key
    Selector dkim
    SOCKET inet:[email protected]
    PidFile /var/run/opendkim/opendkim.pid
    #Canonicalization relaxed/relaxed
    ExternalIgnoreList file:/etc/opendkim/TrustedHostList
    InternalHosts file:/etc/opendkim/TrustedHostList
    LogWhy yes
  • Add trusted hosts… (and folder path, if needed)

    sudo mkdir /etc/opendkim


    sudo vi /etc/opendkim/TrustedHostList

    ADD:

    # External Hosts that OpenDKIM will Trust (add any appropriate values)
    localhost
    127.0.0.1
    10.1.10.1
  • sudo vi /etc/default/opendkim
    ADD:

    SOCKET="inet:[email protected]"
  • sudo vi /etc/postfix/main.cf
    ADD:

    # DKIM
    # --------------------------------------
    milter_default_action = accept
    milter_protocol = 2
    smtpd_milters = inet:127.0.0.1:8891
    non_smtpd_milters = inet:127.0.0.1:8891
  • Take a look around the following file, you may need it later:
    sudo vi /etc/postfix/master.cf
  • Generate your keys:

    opendkim-genkey -t -s dkim -d example.com

    NOTE: this creates dkim.private & dkim.txt, you “might” want to make backups of them 🙂

  • Change permissions on the file:

    sudo chown opendkim:opendkim dkim.private
  • Copy to the postfix folder:

    sudo cp dkim.private /etc/postfix/dkim.key
  • NOTE: I initially had a problem with dkim refusing connections, this MIGHT be needed.

    sudo adduser postfix opendkim
  • Start things back up together:

    sudo service opendkim start
    sudo service postfix restart
  • sudo vi dkim.txt
    (copy contents, remove t=y; as it indicates test mode)
    dkim._domainkey IN TXT ( "v=DKIM1; k=rsa; p=xxxxxxxxx" ) ;

  • Add DNS for DomainKey:

    _domainkey.example.com TXT o=~
  • Add DNS for DKIM:

    dkim._domainkey.example.com TXT v=DKIM1; k=rsa; p=xxxxxxxxx
  • NOTE: you will likely need to wait a few hours for your DNS entries to propagate.

  • Simple verification…. send an email to this address and you will receive a response with your SPF, DKIM and DMARC compliance status:
    mailto:[email protected]

REFERENCES:

Install New Relic Server Monitor on Ubuntu

I’ve found New Relic to be a great free addition to my suite of tools for server monitoring and alerting as I shifted to a DevOps support environment.

Installation is very fast an simple once you’ve created a free accound. Paid options are available and allow for more features.

You will need to record/save YOUR_LICENSE_KEY from your account for step 5 below.

  1. sudo sh -c 'echo deb http://apt.newrelic.com/debian/ newrelic non-free > /etc/apt/sources.list.d/newrelic.list'
  2. wget -O- https://download.newrelic.com/548C16BF.gpg | sudo apt-key add -
  3. sudo apt-get update
  4. sudo apt-get install newrelic-sysmond
  5. sudo nrsysmond-config --set license_key=YOUR_LICENSE_KEY
  6. sudo /etc/init.d/newrelic-sysmond start

You are done! Within a few minutes you should start seeing data on your consoles at the New Relic website.

REFERENCES:

Install Google mod_pagespeed for Apache HTTP

Website network performance is usually a very complicated process. Over the years, I’ve outlined many development techniques that can be used toward this goal. I’d heard about mod_pagespeed for some time, but never had the opportunity to experiment with it until recently. My first impression is that it is a VERY EASY means to gain performance improvements without reworking your existing website to implement techniques for establishing far-future expires, cache-busting, minification and static file merging.

Out of the box, most common techniques are automatically applied to your assets and a local server cache is created to utilize them.

Default installation is trivial:

  1. Download the latest version for your server architecture:

    wget https://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-stable_current_amd64.deb

    OR

    wget https://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-stable_current_i386.deb

  2. sudo dpkg -i mod-pagespeed-*.deb

  3. sudo apt-get -f install
    (if required)

  4. sudo service apache2 restart

NOTE: Using tools like Firebug will enable you to see an HTTP Header indicating the version being used for your requests.

If you need to modify configuration from the default:

sudo vi /etc/apache2/mods-available/pagespeed.conf

For VirtualDomains, you can selectively enable and disable PageSpeed and many of it’s settings in your appropriate configuration files with:

<IfModule pagespeed_module>
ModPagespeed on
</IfModule>

NOTE: Appending ?ModPagespeed=off to your URL will disable functions for that request.

REFERENCES:

Java User-Agent detector and caching

It’s often important for a server side application to understand the client platform. There are two common methods used for this.

  1. On the client itself, “capabilities” can be tested.
  2. Unfortunately, the server cannot easily test these, and as such must usually rely upon the HTTP Header information, notably “User-Agent”.

Example User-agent might typically look like this for a common desktop browser, developers can usually determine the platform without a lot of work.

"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3)"

Determining robots and mobile platforms, unfortunately is a lot more difficult due to the variations. Libraries as those described below simplify this work as standard Java Objects expose the attributes that are commonly expected.

With Maven, the dependencies are all resolved with the following POM addition:

<dependency>
<groupid>net.sf.uadetector</groupid>
<artifactid>uadetector-resources</artifactid>
<version>2014.10</version>
</dependency>


/* Get an UserAgentStringParser and analyze the requesting client */
final UserAgentStringParser parser = UADetectorServiceFactory.getResourceModuleParser();
final ReadableUserAgent agent = parser.parse(request.getHeader("User-Agent"));

out.append("You're a '");
out.append(agent.getName());
out.append("' on '");
out.append(agent.getOperatingSystem().getName());
out.append("'.");

As indicated on the website documentation, running this query for each request uses valuable server resources, it’s best to cache the responses to minimize the impact!

http://uadetector.sourceforge.net/usage.html#improve_performance

NOTE: the website caching example is hard to copy-paste, here’s a cleaner copy.


/*
* COPYRIGHT.
*/
package com.example.cache;

import java.util.concurrent.TimeUnit;
import net.sf.uadetector.ReadableUserAgent;
import net.sf.uadetector.UserAgentStringParser;
import net.sf.uadetector.service.UADetectorServiceFactory;
import com.google.common.cache.Cache;
import com.google.common.cache.CacheBuilder;

/**
* Caching User Agent parser
* @see http://uadetector.sourceforge.net/usage.html#improve_performance
* @author Scott Fredrickson [skotfred]
* @since 2015jan28
* @version 2015jan28
*/
public final class CachedUserAgentStringParser implements UserAgentStringParser {

private final UserAgentStringParser parser = UADetectorServiceFactory.getCachingAndUpdatingParser();
private static final int CACHE_MAX_SIZE = 100;
private static final int CACHE_MAX_HOURS = 2;
/**
* Limited to 100 elements for 2 hours!
*/
private final Cache<String , ReadableUserAgent> cache = CacheBuilder.newBuilder().maximumSize(CACHE_MAX_SIZE).expireAfterWrite(CACHE_MAX_HOURS, TimeUnit.HOURS).build();

/**
* @return {@code String}
*/
@Override
public String getDataVersion() {
return parser.getDataVersion();
}
/**
* @param userAgentString {@code String}
* @return {@link ReadableUserAgent}
*/
@Override
public ReadableUserAgent parse(final String userAgentString) {
ReadableUserAgent result = cache.getIfPresent(userAgentString);
if (result == null) {
result = parser.parse(userAgentString);
cache.put(userAgentString, result);
}
return result;
}
/**
*
*/
@Override
public void shutdown() {
parser.shutdown();
}
}

REFERENCES:

HTTP Strict Transport Security (HSTS)

The HTTP Strict Transport Security feature lets a web site inform the browser that it should never load the site using HTTP, and should automatically convert all attempts to access the site using HTTP to HTTPS requests instead.

Example Use case:
If a web site accepts a connection through HTTP and redirects to HTTPS, the user in this case may initially talk to the non-encrypted version of the site before being redirected, if, for example, the user types http://www.foo.com/ or even just foo.com.

Problem:
This opens up the potential for a man-in-the-middle attack, where the redirect could be exploited to direct a user to a malicious site instead of the secure version of the original page.

Risk:
For HTTP sites on the same domain it is not recommended to add a HSTS header but to do a permanent redirect (301 status code) to the HTTPS site.

Bonus:
Google is always “tweaking” their search algorithms, and, at least at present time, gives greater weight to secure websites.


# Optionally load the headers module:
LoadModule headers_module modules/mod_headers.so

<VirtualHost *:443>
# Guarantee HTTPS for 1 Year including Sub Domains
Header always set Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"
</VirtualHost>

Then you might (optionally, but recommended) force ALL HTTP users to HTTPS:

# Redirect HTTP connections to HTTPS
<VirtualHost *:80>
ServerAlias *
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTPS} off
#RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [redirect=301]
</IfModule>
</VirtualHost>

That’s it…

REFERENCES:

Poodle.. or rather, what’s all the fuss with SSLv3

The “Poodle” attack on websites and browsers was all over the media a few weeks ago, following in the shadow of Heartbleed.

Here’s what most users need to know… This is an vulnerability that exists in secure internet communication because…

  1. While most newer systems rely on TLS security, they still support older protocols (SSLv3 in particular for this issue)
  2. As secure communications generally attempt to find a “common” method, they will often “drop down” to older supported versions (even if they are now often considered insecure!)
  3. Most browser and server software (unless recently patched) will allow for this “drop down” in security.
  4. Most software provides a mechanism to disable this by the user or in configuration.
  5. Upgrading your software will usually remove these “problematic” vulnerabilities.

Simply put… for a consumer, it’s best to upgrade to a newer browser or find the appropriate configuration to disable SSLv3 if you are unable to upgrade. Server administrators generally should update their sofware on a regular basis for security items such as this one!

NOTE: Many CDN’s such as CloudFlare are proactive and block this vulnerability.

Technical details on the Poodle vulnerability (if you’re into that sort of thing!):

Instructions here are for Apache HTTPd 2.2.23 and newer, other servers will require a similar change:


  1. sudo vi /etc/apache2/mods-enabled/ssl.conf
  2. Change the following line from:
    SSLProtocol All -SSLv2
    to:
    SSLProtocol All -SSLv2 -SSLv3
  3. sudo service apache2 reload
  4. sudo service apache2 restart

Can be tested at the following websites:

REFERENCES:

“msapplication-config” and browserconfig.xml

Windows-8/MSIE-11 introduced Tiles, as such server administrators may have started seeing HTTP 404 errors in their server logs as it attempts to look for a “browserconfig.xml” file at the root of a website domain. If you are inclined to use this file, you should definitely look into the documentation for how to best make use of it. Others may just wish to prevent the error from making “noise” in their log files.

To remove the error, add the following to your pages; alternately you COULD define the URL of your file as the ‘content’ attribute:

<meta name="msapplication-config" content="none" />

You can alternately place an empty /browserconfig.xml on your web server for each domain.

An common example of how to use this file is below:

<?xml version="1.0" encoding="utf-8"?>
<browserconfig>
<msapplication>
<tile>
<square70x70logo src="/mstile-70x70.png"/>
<square150x150logo src="/mstile-150x150.png"/>
<wide310x150logo src="/mstile-310x150.png"/>
<square310x310logo src="/mstile-310x310.png"/>
<TileColor>#8bc53f</TileColor>
<TileImage src="/mstile-150x150.png" />
</tile>
</msapplication>
</browserconfig>

REFERENCES:

Remove Landscape on Ubuntu

The Canonical/Ubuntu Landscape service has been around for as long as I can remember using Ubuntu. A free trial period is enabled (re-enabled?) when a new installation occurs, that allows for a server administrator to see performance metrics and uptime information for any hardware that is running the client. After the trial ends, it is still a quick means of visually observing some key statistics in the terminal MOTD at login. I’d also noticed that it was still doing DNS lookups to “landscape.canonical.com” on a regular basis, and while I did not look for it, I assume that some information was still being collected and reported upon.

As there are MANY other ways to get server performance information, I decided that it was time to be rid of landscape itself.

Removal is easy, as only one line is required… I chose to “purge” all references, though you can “remove” if you feel inclined to leave any configuration for possible later re-installation.


sudo apt-get purge landscape-client landscape-client-ui landscape-client-ui-install landscape-common

REFERENCES:

Install Subversion Server on Ubuntu

Subversion is a commonly used central version control system for software development. There are currently still a large number of organizations that rely upon it, many have since moved on to Git.

  1. sudo apt-get install apache2 apache2-utils
  2. sudo apt-get install subversion subversion-tools libapache2-svn
  3. sudo mkdir /home/svn
  4. svnadmin create /home/svn/test
  5. Create a group for subversion users:
    sudo groupadd subversion
  6. sudo adduser USERNAME
  7. Add a user to the group:
    sudo useradd -G USERNAME subversion
  8. sudo chown -R www-data:subversion /home/svn/test
  9. sudo chmod -R g+rws /home/svn/test
  10. sudo a2enmod dav_svn
  11. To create/clobber a new file for the first user:
    sudo htpasswd -c /etc/apache2/.htpasswd YOURUSER
  12. To add additional users:
    sudo htpasswd /etc/apache2/.htpasswd YOURUSER
    (repeat for new users without the -c as that creates/clobbers the file)
  13. sudo vi /etc/apache2/sites-available/000-default.conf
    Then add to the bottom:
    (NOTE1: the LimitExcept can be enabled to allow anonymous access):
    (NOTE2: the LimitXMLRequestBody can be uncomment to allow large commits)

    <Location /svn>
    DAV svn
    SVNParentPath /home/svn
    AuthType Basic
    AuthName "Subversion Repository"
    # AuthUserFile /etc/svn-auth
    AuthUserFile /etc/apache2/.htpasswd
    #LimitXMLRequestBody 0
    #<LimitExcept GET PROPFIND OPTIONS REPORT>
    Require valid-user
    #</LimitExcept>
    </Location>
  14. sudo service apache2 reload
  15. sudo service apache2 restart

    NOTE: At this point you should be able to browse and do a remote checkout of the code from another machine….

    http://YOUR-IP-OR-HOSTNAME/svn
    and
    svn co http://YOUR-IP-OR-HOSTNAME/svn/test --username YOURUSER --password YOURPASS

  16. sudo vi /etc/init/svnserve.conf
    Add the following:

    # svnserve - Subversion server
    description "Subversion server"
    start on (local-filesystems and net-device-up IFACE=lo and started udev-finish)
    stop on runlevel [06]
    chdir /home/svn
    respawn
    respawn limit 2 3600
    exec /usr/bin/svnserve --foreground --daemon --config-file /home/svn/repos/conf/svnserve.conf --root /home/svn/repos/
  17. Then execute:
    sudo initctl start svnserve
  18. Back on the client side…
    Create a new folder inside your user folder:
    cd ~/test
  19. Check out the project into this folder:
    svn checkout http://YOUR-IP-OR-HOSTNAME/svn/test
  20. Let us just add a new HTML index file to the folder:
    vi index.html
  21. Add it to version control:
    svn add index.html
    Commit the new file:
    svn commit -m "commit message"
    Update:
    svn up
  22. That should cover most cases for you…

REFERENCES: