Install Splunk on Ubuntu

Splunk is a popular enterprise level tool for log collection, analysis and management. While you can obtain an enterprise license, most functions are available in the free community edition.

Setup is very easy:

  1. Download and move the .tar.gz file to the appropriate server (i386 vs. amd64)
  2. sudo dpkg -i splunk*.deb
  3. Start the server:

    sudo /opt/splunk/bin/splunk start

    The first time you run after installation or update you will have to accept terms.

  4. Access the admin screen:

    http://HOSTNAME:8000

    login (admin/changeme)
    change password

  5. Go to Settings/Forwarding * Receiving
    – add new (port 9997)
  6. Open firewall port (if enabled):

    sudo ufw allow 8000
  7. Now to start as a service…

    sudo /opt/splunk/bin/splunk enable boot-start

Competitors:

REFERENCES

Apache Commons-Email java implementation steps

Many java developers are familiar with the venerable javax.mail.* packages and make use of them in their applications.

While it works well, it can often be cumbersome to work with and difficult to implement new features. Apache Commons-Email, now at version 1.4 (May 2015), provides a simpler interface to send emails with HTML format and attachments.

NOTE: The below examples assume that you are using an SMTP server that verifies the sender. You may need to modify the examples for your specific configuration. Additionally, I’ve left out the try/catch blocks for “Exceptions” that you will have to add.

Using javax.mail.* to send an text formatted message:

final String body = "Example email body";
final String emailFrom = "From User ";
final String emailTo = "To User
";
/* NOTE: 'session' and 'conn' are outside of the scope of this example but generally contain host and authentication information */
javax.mail.Session session = getSession(conn);
final javax.mail.Message message = new javax.mail.internet.MimeMessage(session);
message.setFrom(new javax.mail.internet.InternetAddress(emailFrom));
message.setRecipients(javax.mail.Message.RecipientType.TO, javax.mail.internet.InternetAddress.parse(emailTo));
message.setSubject(subj);
message.setText(body);
javax.mail.Transport.send(message);

Using commons-email for HTML email.

final String body = "Example email body";
final String emailFromAddr = "[email protected]";
final String emailFromName = "User From";
final String emailToAddr = "[email protected]";
final String emailToName = "User To";
final String username = "myusername";
final String password = "mypassword";

final org.apache.commons.mail.HtmlEmail email = new org.apache.commons.mail.HtmlEmail();
email.setHostName("localhost");
email.setSmtpPort(25);
email.setAuthentication(username, password);
email.setAuthenticator(new org.apache.commons.mail.DefaultAuthenticator(username, password));
//email.setSSLOnConnect(true);
final String charset = "UTF-8";
email.setCharset(charset);
email.setFrom(emailFromAddr,emailFromName);
email.setSubject(subj);

email.addTo(emailToAddr, emailToName);
//email.setDebug(true);

// set the alternative message
email.setTextMsg("Your email client does not support HTML messages.");

// set the html message
final StringBuilder sb = new StringBuilder();
sb.append("");
sb.append(body);
sb.append("");
email.setHtmlMsg(sb.toString());
email.send();

Using commons-email with an inline attachment:

final org.apache.commons.mail.HtmlEmail email = new org.apache.commons.mail.HtmlEmail();
/* (insert code from example above above) */
String cid = null;
try{
final URL url = new URL("http://www.example.com/logo.gif");
final String img = email.embed(url, "Logo");
cid = "\"\"";
}
catch(final MalformedURLException ex){
// eat it!
}
// set the html message
final StringBuilder sb = new StringBuilder();
sb.append("");
if(cid!=null){ sb.append(cid); }
sb.append(body);
sb.append("");
email.setHtmlMsg(sb.toString());
email.send();

Code changes to use the library should not take very long as Commons-Email builds on top of javax.mail.*. In most cases, For Maven projects, you can remove the javax.mail references and simply add the new commons-email one to your pom.xml:


<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-email</artifactId>
<version>1.4</version>
</dependency>

REFERENCES:

Windows 7+ “GodMode”

To give you complete control over all the configurable options in Windows 7+ at a single press of a button just simply create a new folder anywhere and rename it to this:


GodMode.{ED7BA470-8E54-465E-825C-99712043E01C}

The folder you create will now have 270 items that are to do with configurable options in Windows 7. Interestingly it also works for Windows 8.1 and 10 as well.

NOTE: Vista partially supported this feature, but was prone to crashing until the folder was removed.

REFERENCES:

Hackintosh – running Apple OS/X inside VirtualBox (on Ubuntu or Windows)

I’ve done a LOT of web and software development in my career, one of the biggest problems I’v e had is doing proper testing on various platforms. VirtualBox can be run on most platforms, and allows for you to virtualize various operating systems. Apple’s OS X can be problematic to install, here’s some functioning instructions and settings that I’ve successfully used for my virtual Hackintosh test environment.

WARNING: As this uses an image of the software, it is advisable to also have a valid licenced copy as to use this without a licence would be “stealing”.

NOTE: this is a version of Snow Leopard, unfortunately you will NOT be able to update or add “fixes” to this test copy as it will usually break. I recommend making a copy of the VM if you wish to try!

My VirtualBox settings:

Mac OS X
Mac OS X Snow Leopard (64-bit)
1024MB
1 core
Disable EFI
6MB+ video

Steps:

  1. English, continue
  2. Utilities… disk Utilities… Select drive, Erase, name, erase… close
  3. continue
  4. Select disk… continue
  5. Customize (see instructions for: updates, kernels & bootloaders)
  6. NOTE: i only did 10.6.1 (an earlier 10.6.2 failed)

  7. Install
  8. Resolution fix – (I could not save file, but here are the instructions…

    Finder > OSX Drive (On desktop) > Extras > com.apple.Boot.plist (open with TextEdit – use the field to find)


<key>Graphics Mode</key>
<string>1280×1024x32</string>

REFERENCES:

Cache Busting URL’s with querystring params?

There is a LOT of noise and discussion about this online, most often attributed to the following source,
http://www.stevesouders.com/blog/2008/08/23/revving-filenames-dont-use-querystring/

While it was relevant back when this was written in 2008, but the main reason cited was proxy server behavior, particularly Squid Proxy. FWIW, Squid had this capability before, but it was not the default setting. They changed their default policy with caching dynamic URLs with their 2.7 release, stable since 31 May 2008.

http://wiki.squid-cache.org/ConfigExamples/DynamicContent

Outline: The obsolete default configuration of squid prevents the caching of dynamic content (pages with ? in the URI), like so:


hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
cache deny QUERY

NOTE: That policy setting was created at a time when dynamic pages rarely contained proper Cache-Controls, that has now changed. From the release of Squid 2.7 and 3.1 the squid developers are advocating a change to this caching policy. These changes will also work in 3.0 and 2.6 releases despite not being officially changed for their squid.conf.default.

The changed policy is to remove the QUERY ACL and paired cache line. To be replaced by the refresh_patterns below:


refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320

Squid Release history, for reference.

Version First STABLE release Date Latest Release Latest Release Date

3.5   Jan 2015
3.4	09 Dec 2013	3.4.12 	 18 Feb 2015 
3.3	09 Feb 2013	3.3.13 	 28 Aug 2014 
3.2	14 Aug 2012	3.2.13 	 13 Jul 2013 
3.1	29 Mar 2010	3.1.23 	 09 Jan 2013 
3.0	13 Dec 2007	STABLE26 28 Aug 2011 
2.7	31 May 2008	STABLE9 16 Mar 2010 
2.6		01 Jul 2006 	STABLE23 17 Sep 2009 
2.5		25 Sep 2002 	STABLE14 20 May 2006 
2.4		20 Mar 2001 	STABLE7 02 Jul 2002 

NOTE: if you would like to verify how widespread QueryString versioning is, please look at the source of microsoft.com or just about any WordPress powered website.

REFERENCES:

Domain-based Message Authentication, Reporting & Conformance (DMARC) Email

DMARC was published in 2012 to build upon the SPF and DKIM email conventions for authorizing senders. It allows specification of policies and provides for reporting of actions performed under those policies.

Assistant:
http://www.kitterman.com/dmarc/assistant.html

DNS Entry Resembles:

_dmarc.example.com TXT v=DMARC1; p=none; rua=mailto:[email protected]; ruf=mailto:[email protected]; fo=0; adkim=r; aspf=r; pct=100; rf=afrf; ri=86400; sp=none

Simple verification…. send an email to this address and you will receive a response with your SPF, DKIM and DMARC compliance status:
mailto:[email protected]

REFERENCES:

DomainKeys Identified Mail (DKIM) Email

DomainKeys (originally from Yahoo!) and Cisco, and later as an industry collaboration, is a means for and organization to claim responsibility for sending a message, in a way that can be validated by a recipient. As a result, emails are “signed” by the outgoing SMTP server and can be verified against a DNS record. Depending upon the receiver, unsigned emails MAY be treated or marked as SPAM as they could be forgeries.

The below instructions assume Ubuntu (Debian) and Postfix, but could likely be modified for other platforms.

  • Install OpenDKIM:
    sudo apt-get install opendkim opendkim-tools
  • Setup initial configuration:

    sudo vi /etc/opendkim.conf

    ADD:
    Domain example.com
    KeyFile /etc/postfix/dkim.key
    Selector dkim
    SOCKET inet:[email protected]
    PidFile /var/run/opendkim/opendkim.pid
    #Canonicalization relaxed/relaxed
    ExternalIgnoreList file:/etc/opendkim/TrustedHostList
    InternalHosts file:/etc/opendkim/TrustedHostList
    LogWhy yes
  • Add trusted hosts… (and folder path, if needed)

    sudo mkdir /etc/opendkim


    sudo vi /etc/opendkim/TrustedHostList

    ADD:

    # External Hosts that OpenDKIM will Trust (add any appropriate values)
    localhost
    127.0.0.1
    10.1.10.1
  • sudo vi /etc/default/opendkim
    ADD:

    SOCKET="inet:[email protected]"
  • sudo vi /etc/postfix/main.cf
    ADD:

    # DKIM
    # --------------------------------------
    milter_default_action = accept
    milter_protocol = 2
    smtpd_milters = inet:127.0.0.1:8891
    non_smtpd_milters = inet:127.0.0.1:8891
  • Take a look around the following file, you may need it later:
    sudo vi /etc/postfix/master.cf
  • Generate your keys:

    opendkim-genkey -t -s dkim -d example.com

    NOTE: this creates dkim.private & dkim.txt, you “might” want to make backups of them 🙂

  • Change permissions on the file:

    sudo chown opendkim:opendkim dkim.private
  • Copy to the postfix folder:

    sudo cp dkim.private /etc/postfix/dkim.key
  • NOTE: I initially had a problem with dkim refusing connections, this MIGHT be needed.

    sudo adduser postfix opendkim
  • Start things back up together:

    sudo service opendkim start
    sudo service postfix restart
  • sudo vi dkim.txt
    (copy contents, remove t=y; as it indicates test mode)
    dkim._domainkey IN TXT ( "v=DKIM1; k=rsa; p=xxxxxxxxx" ) ;

  • Add DNS for DomainKey:

    _domainkey.example.com TXT o=~
  • Add DNS for DKIM:

    dkim._domainkey.example.com TXT v=DKIM1; k=rsa; p=xxxxxxxxx
  • NOTE: you will likely need to wait a few hours for your DNS entries to propagate.

  • Simple verification…. send an email to this address and you will receive a response with your SPF, DKIM and DMARC compliance status:
    mailto:[email protected]

REFERENCES:

Sender Policy Framework (SPF) Email

This is a simple mechanism, using DNS to certify that email from your domain comes from authorized servers. This is accomplished by adding a DNS record to identify the servers from which you send legitimate email. Emails sent from other servers MAY then be assumed as forged (SPAM) and blocked by the receiving email server.

NOTE: This can be easily spoofed, as such it should be a portion of your email security strategy, look into DKIM and DMARC too!

One thing that I initially did not understand… if you are supporting IPv6 and IPv4, you should merge your records onto a single DNS TXT entry:


example.com TXT v=spf1 a mx ip4:xxx.xxx.xxx.xxx ip6:xxxx:x:xxx:xxxx:xxx:xxxx:xxxx:xxx -all

REFERENCES:

Install New Relic Server Monitor on Ubuntu

I’ve found New Relic to be a great free addition to my suite of tools for server monitoring and alerting as I shifted to a DevOps support environment.

Installation is very fast an simple once you’ve created a free accound. Paid options are available and allow for more features.

You will need to record/save YOUR_LICENSE_KEY from your account for step 5 below.

  1. sudo sh -c 'echo deb http://apt.newrelic.com/debian/ newrelic non-free > /etc/apt/sources.list.d/newrelic.list'
  2. wget -O- https://download.newrelic.com/548C16BF.gpg | sudo apt-key add -
  3. sudo apt-get update
  4. sudo apt-get install newrelic-sysmond
  5. sudo nrsysmond-config --set license_key=YOUR_LICENSE_KEY
  6. sudo /etc/init.d/newrelic-sysmond start

You are done! Within a few minutes you should start seeing data on your consoles at the New Relic website.

REFERENCES:

Install Google mod_pagespeed for Apache HTTP

Website network performance is usually a very complicated process. Over the years, I’ve outlined many development techniques that can be used toward this goal. I’d heard about mod_pagespeed for some time, but never had the opportunity to experiment with it until recently. My first impression is that it is a VERY EASY means to gain performance improvements without reworking your existing website to implement techniques for establishing far-future expires, cache-busting, minification and static file merging.

Out of the box, most common techniques are automatically applied to your assets and a local server cache is created to utilize them.

Default installation is trivial:

  1. Download the latest version for your server architecture:

    wget https://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-stable_current_amd64.deb

    OR

    wget https://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-stable_current_i386.deb

  2. sudo dpkg -i mod-pagespeed-*.deb

  3. sudo apt-get -f install
    (if required)

  4. sudo service apache2 restart

NOTE: Using tools like Firebug will enable you to see an HTTP Header indicating the version being used for your requests.

If you need to modify configuration from the default:

sudo vi /etc/apache2/mods-available/pagespeed.conf

For VirtualDomains, you can selectively enable and disable PageSpeed and many of it’s settings in your appropriate configuration files with:

<IfModule pagespeed_module>
ModPagespeed on
</IfModule>

NOTE: Appending ?ModPagespeed=off to your URL will disable functions for that request.

REFERENCES: