Squid3 Proxy on Ubuntu

Using a personal proxy server can be helpful for a variety of reasons, such as:

  • Performance – network speed and bandwidth
  • Security – filtering and monitoring
  • Debugging – to trace activity

Here are some simple steps to get you started,  obviously you will need to further “harden” security to make it production ready!


sudo apt-get install squid3


cd /etc/squid3/
sudo mv squid.conf squid.orig
sudo vi squid.conf

NOTE: the following configuration works, but will likely need to be adapted for your specific usage.


http_port 3128
visible_hostname proxy.EXAMPLE.com
auth_param digest program /usr/lib/squid3/digest_file_auth -c /etc/squid3/passwords
#auth_param digest program /usr/lib/squid3/digest_pw_auth -c /etc/squid3/passwords
auth_param digest realm proxy
auth_param basic credentialsttl 4 hours
acl authenticated proxy_auth REQUIRED
acl localnet src 10.0.0.0/8 # RFC 1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC 1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC 1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
#acl SSL_ports port 443
#http_access deny to_localhost
#http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow localhost
http_access allow authenticated
via on
forwarded_for transparent

Create the users and passwords:

sudo apt-get install apache2-utils (required for htdigest)
sudo htdigest -c /etc/squid3/passwords proxy user1
sudo htdigest /etc/squid3/passwords proxy user2

Open up firewall port (if enabled):

sudo ufw allow 3128

Restart the server and tail the logs:

sudo service squid3 restart
sudo tail -f /var/log/squid3/access.log

OTHER FILE LOCATIONS:

/var/spool/squid3
/etc/squid3

MONITORING with Splunk…

sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/squid3/access.log -index main -sourcetype Squid3
sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/squid3/cache.log -index main -sourcetype Squid3

REFERENCES:

IPv6 and IPv4 for Apache Tomcat

If you’ve recently upgraded your network from IPv4 to IPv6, you might find that some software no longer works as it had before. Apache Tomcat is one that I recently stumbled upon, as it seems to prefer the IPv6 connection and stops listening on IPv4 with the default configuration.

The solution is simple, you just have to tell the server to listen on all incoming IP addresses. This worked for me with versions 7.x and 8.x, and I suspect that older and newer versions would be similar.

  1. sudo vi /opt/tomcat/conf/server.xml
  2. To each <Server> entry add:
    address="0.0.0.0"
  3. Restart Tomcat

REFERENCES:

Brotli Compression

If you look at HTTP Headers as often as I do, you’ve likely noticed something different in Firefox 44 and Chrome 49. In addition to the usual ‘gzip’, ‘deflate’ and ‘sdhc’ , a new value ‘br’ has started to appear for HTTPS connections.

Request:

Accept-Encoding:br

Response:

Content-Encoding:br

Compared to gzip, Brotli claims to have significantly better (26% smaller) compression density woth comparable decompression speed.

The smaller compressed size allows for better space utilization and faster page loads. We hope that this format will be supported by major browsers in the near future, as the smaller compressed size would give additional benefits to mobile users, such as lower data transfer fees and reduced battery use.

Advantages:

  • Brotli outperforms gzip for typical web assets (e.g. css, html, js) by 17–25 %.
  • Brotli -11 density compared to gzip -9:
  • html (multi-language corpus): 25 % savings
  • js (alexa top 10k): 17 % savings
  • minified js (alexa top 10k): 17 % savings
  • css (alexa top 10k): 20 % savings


NOTE: Brotli is not currently supported Apache HTTPd server (as of 2016feb10), but will likely be added in an upcoming release.

http://mail-archives.apache.org/mod_mbox/httpd-users/201601.mbox/%[email protected]%3E

Until there is native support, you can pre-compress files by following instructions here…
https://lyncd.com/2015/11/brotli-support-apache/

REFERENCES:

Upgrade Splunk server

Initially this seemed a bit problematic for me. Each time the browser client is started it (by default) checks for a new server release and prompts the user to upgrade. The installation automatically identifies the currently installed version and takes the necessary steps to migrate configuration.

Steps are similar to initial installation.

  1. Download the appropriate build for your server (i386 vs. amd64)
  2. transfer to the server via sftp or other secure means
  3. sudo /opt/splunk/bin/splunk stop
  4. sudo dpkg -i splunk*
  5. sudo /opt/splunk/bin/splunk start
  6. … accept terms… Y
  7. MIGRATE “y”
  8. http://HOSTNAME:8000
  9. sudo /opt/splunk/bin/splunk enable boot-start

Install Splunk on Ubuntu

Splunk is a popular enterprise level tool for log collection, analysis and management. While you can obtain an enterprise license, most functions are available in the free community edition.

Setup is very easy:

  1. Download and move the .tar.gz file to the appropriate server (i386 vs. amd64)
  2. sudo dpkg -i splunk*.deb
  3. Start the server:

    sudo /opt/splunk/bin/splunk start

    The first time you run after installation or update you will have to accept terms.

  4. Access the admin screen:

    http://HOSTNAME:8000

    login (admin/changeme)
    change password

  5. Go to Settings/Forwarding * Receiving
    – add new (port 9997)
  6. Open firewall port (if enabled):

    sudo ufw allow 8000
  7. Now to start as a service…

    sudo /opt/splunk/bin/splunk enable boot-start

Competitors:

REFERENCES

DomainKeys Identified Mail (DKIM) Email

DomainKeys (originally from Yahoo!) and Cisco, and later as an industry collaboration, is a means for and organization to claim responsibility for sending a message, in a way that can be validated by a recipient. As a result, emails are “signed” by the outgoing SMTP server and can be verified against a DNS record. Depending upon the receiver, unsigned emails MAY be treated or marked as SPAM as they could be forgeries.

The below instructions assume Ubuntu (Debian) and Postfix, but could likely be modified for other platforms.

  • Install OpenDKIM:
    sudo apt-get install opendkim opendkim-tools
  • Setup initial configuration:

    sudo vi /etc/opendkim.conf

    ADD:
    Domain example.com
    KeyFile /etc/postfix/dkim.key
    Selector dkim
    SOCKET inet:[email protected]
    PidFile /var/run/opendkim/opendkim.pid
    #Canonicalization relaxed/relaxed
    ExternalIgnoreList file:/etc/opendkim/TrustedHostList
    InternalHosts file:/etc/opendkim/TrustedHostList
    LogWhy yes
  • Add trusted hosts… (and folder path, if needed)

    sudo mkdir /etc/opendkim


    sudo vi /etc/opendkim/TrustedHostList

    ADD:

    # External Hosts that OpenDKIM will Trust (add any appropriate values)
    localhost
    127.0.0.1
    10.1.10.1
  • sudo vi /etc/default/opendkim
    ADD:

    SOCKET="inet:[email protected]"
  • sudo vi /etc/postfix/main.cf
    ADD:

    # DKIM
    # --------------------------------------
    milter_default_action = accept
    milter_protocol = 2
    smtpd_milters = inet:127.0.0.1:8891
    non_smtpd_milters = inet:127.0.0.1:8891
  • Take a look around the following file, you may need it later:
    sudo vi /etc/postfix/master.cf
  • Generate your keys:

    opendkim-genkey -t -s dkim -d example.com

    NOTE: this creates dkim.private & dkim.txt, you “might” want to make backups of them 🙂

  • Change permissions on the file:

    sudo chown opendkim:opendkim dkim.private
  • Copy to the postfix folder:

    sudo cp dkim.private /etc/postfix/dkim.key
  • NOTE: I initially had a problem with dkim refusing connections, this MIGHT be needed.

    sudo adduser postfix opendkim
  • Start things back up together:

    sudo service opendkim start
    sudo service postfix restart
  • sudo vi dkim.txt
    (copy contents, remove t=y; as it indicates test mode)
    dkim._domainkey IN TXT ( "v=DKIM1; k=rsa; p=xxxxxxxxx" ) ;

  • Add DNS for DomainKey:

    _domainkey.example.com TXT o=~
  • Add DNS for DKIM:

    dkim._domainkey.example.com TXT v=DKIM1; k=rsa; p=xxxxxxxxx
  • NOTE: you will likely need to wait a few hours for your DNS entries to propagate.

  • Simple verification…. send an email to this address and you will receive a response with your SPF, DKIM and DMARC compliance status:
    mailto:[email protected]

REFERENCES:

Install New Relic Server Monitor on Ubuntu

I’ve found New Relic to be a great free addition to my suite of tools for server monitoring and alerting as I shifted to a DevOps support environment.

Installation is very fast an simple once you’ve created a free accound. Paid options are available and allow for more features.

You will need to record/save YOUR_LICENSE_KEY from your account for step 5 below.

  1. sudo sh -c 'echo deb http://apt.newrelic.com/debian/ newrelic non-free > /etc/apt/sources.list.d/newrelic.list'
  2. wget -O- https://download.newrelic.com/548C16BF.gpg | sudo apt-key add -
  3. sudo apt-get update
  4. sudo apt-get install newrelic-sysmond
  5. sudo nrsysmond-config --set license_key=YOUR_LICENSE_KEY
  6. sudo /etc/init.d/newrelic-sysmond start

You are done! Within a few minutes you should start seeing data on your consoles at the New Relic website.

REFERENCES:

Install Google mod_pagespeed for Apache HTTP

Website network performance is usually a very complicated process. Over the years, I’ve outlined many development techniques that can be used toward this goal. I’d heard about mod_pagespeed for some time, but never had the opportunity to experiment with it until recently. My first impression is that it is a VERY EASY means to gain performance improvements without reworking your existing website to implement techniques for establishing far-future expires, cache-busting, minification and static file merging.

Out of the box, most common techniques are automatically applied to your assets and a local server cache is created to utilize them.

Default installation is trivial:

  1. Download the latest version for your server architecture:

    wget https://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-stable_current_amd64.deb

    OR

    wget https://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-stable_current_i386.deb

  2. sudo dpkg -i mod-pagespeed-*.deb

  3. sudo apt-get -f install
    (if required)

  4. sudo service apache2 restart

NOTE: Using tools like Firebug will enable you to see an HTTP Header indicating the version being used for your requests.

If you need to modify configuration from the default:

sudo vi /etc/apache2/mods-available/pagespeed.conf

For VirtualDomains, you can selectively enable and disable PageSpeed and many of it’s settings in your appropriate configuration files with:

<IfModule pagespeed_module>
ModPagespeed on
</IfModule>

NOTE: Appending ?ModPagespeed=off to your URL will disable functions for that request.

REFERENCES:

Java User-Agent detector and caching

It’s often important for a server side application to understand the client platform. There are two common methods used for this.

  1. On the client itself, “capabilities” can be tested.
  2. Unfortunately, the server cannot easily test these, and as such must usually rely upon the HTTP Header information, notably “User-Agent”.

Example User-agent might typically look like this for a common desktop browser, developers can usually determine the platform without a lot of work.

"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3)"

Determining robots and mobile platforms, unfortunately is a lot more difficult due to the variations. Libraries as those described below simplify this work as standard Java Objects expose the attributes that are commonly expected.

With Maven, the dependencies are all resolved with the following POM addition:

<dependency>
<groupid>net.sf.uadetector</groupid>
<artifactid>uadetector-resources</artifactid>
<version>2014.10</version>
</dependency>


/* Get an UserAgentStringParser and analyze the requesting client */
final UserAgentStringParser parser = UADetectorServiceFactory.getResourceModuleParser();
final ReadableUserAgent agent = parser.parse(request.getHeader("User-Agent"));

out.append("You're a '");
out.append(agent.getName());
out.append("' on '");
out.append(agent.getOperatingSystem().getName());
out.append("'.");

As indicated on the website documentation, running this query for each request uses valuable server resources, it’s best to cache the responses to minimize the impact!

http://uadetector.sourceforge.net/usage.html#improve_performance

NOTE: the website caching example is hard to copy-paste, here’s a cleaner copy.


/*
* COPYRIGHT.
*/
package com.example.cache;

import java.util.concurrent.TimeUnit;
import net.sf.uadetector.ReadableUserAgent;
import net.sf.uadetector.UserAgentStringParser;
import net.sf.uadetector.service.UADetectorServiceFactory;
import com.google.common.cache.Cache;
import com.google.common.cache.CacheBuilder;

/**
* Caching User Agent parser
* @see http://uadetector.sourceforge.net/usage.html#improve_performance
* @author Scott Fredrickson [skotfred]
* @since 2015jan28
* @version 2015jan28
*/
public final class CachedUserAgentStringParser implements UserAgentStringParser {

private final UserAgentStringParser parser = UADetectorServiceFactory.getCachingAndUpdatingParser();
private static final int CACHE_MAX_SIZE = 100;
private static final int CACHE_MAX_HOURS = 2;
/**
* Limited to 100 elements for 2 hours!
*/
private final Cache<String , ReadableUserAgent> cache = CacheBuilder.newBuilder().maximumSize(CACHE_MAX_SIZE).expireAfterWrite(CACHE_MAX_HOURS, TimeUnit.HOURS).build();

/**
* @return {@code String}
*/
@Override
public String getDataVersion() {
return parser.getDataVersion();
}
/**
* @param userAgentString {@code String}
* @return {@link ReadableUserAgent}
*/
@Override
public ReadableUserAgent parse(final String userAgentString) {
ReadableUserAgent result = cache.getIfPresent(userAgentString);
if (result == null) {
result = parser.parse(userAgentString);
cache.put(userAgentString, result);
}
return result;
}
/**
*
*/
@Override
public void shutdown() {
parser.shutdown();
}
}

REFERENCES:

HTTP Strict Transport Security (HSTS)

The HTTP Strict Transport Security feature lets a web site inform the browser that it should never load the site using HTTP, and should automatically convert all attempts to access the site using HTTP to HTTPS requests instead.

Example Use case:
If a web site accepts a connection through HTTP and redirects to HTTPS, the user in this case may initially talk to the non-encrypted version of the site before being redirected, if, for example, the user types http://www.foo.com/ or even just foo.com.

Problem:
This opens up the potential for a man-in-the-middle attack, where the redirect could be exploited to direct a user to a malicious site instead of the secure version of the original page.

Risk:
For HTTP sites on the same domain it is not recommended to add a HSTS header but to do a permanent redirect (301 status code) to the HTTPS site.

Bonus:
Google is always “tweaking” their search algorithms, and, at least at present time, gives greater weight to secure websites.


# Optionally load the headers module:
LoadModule headers_module modules/mod_headers.so

<VirtualHost *:443>
# Guarantee HTTPS for 1 Year including Sub Domains
Header always set Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"
</VirtualHost>

Then you might (optionally, but recommended) force ALL HTTP users to HTTPS:

# Redirect HTTP connections to HTTPS
<VirtualHost *:80>
ServerAlias *
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTPS} off
#RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [redirect=301]
</IfModule>
</VirtualHost>

That’s it…

REFERENCES: