Squid3 Proxy on Ubuntu

Using a personal proxy server can be helpful for a variety of reasons, such as:

  • Performance – network speed and bandwidth
  • Security – filtering and monitoring
  • Debugging – to trace activity

Here are some simple steps to get you started,  obviously you will need to further “harden” security to make it production ready!


sudo apt-get install squid3


cd /etc/squid3/
sudo mv squid.conf squid.orig
sudo vi squid.conf

NOTE: the following configuration works, but will likely need to be adapted for your specific usage.


http_port 3128
visible_hostname proxy.EXAMPLE.com
auth_param digest program /usr/lib/squid3/digest_file_auth -c /etc/squid3/passwords
#auth_param digest program /usr/lib/squid3/digest_pw_auth -c /etc/squid3/passwords
auth_param digest realm proxy
auth_param basic credentialsttl 4 hours
acl authenticated proxy_auth REQUIRED
acl localnet src 10.0.0.0/8 # RFC 1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC 1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC 1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
#acl SSL_ports port 443
#http_access deny to_localhost
#http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow localhost
http_access allow authenticated
via on
forwarded_for transparent

Create the users and passwords:

sudo apt-get install apache2-utils (required for htdigest)
sudo htdigest -c /etc/squid3/passwords proxy user1
sudo htdigest /etc/squid3/passwords proxy user2

Open up firewall port (if enabled):

sudo ufw allow 3128

Restart the server and tail the logs:

sudo service squid3 restart
sudo tail -f /var/log/squid3/access.log

OTHER FILE LOCATIONS:

/var/spool/squid3
/etc/squid3

MONITORING with Splunk…

sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/squid3/access.log -index main -sourcetype Squid3
sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/squid3/cache.log -index main -sourcetype Squid3

REFERENCES:

HTML5 Link Prefetching

Link prefetching is used to identify a resource that might be required by the next navigation, and that the user agent SHOULD fetch, such that the user agent can deliver a faster response once the resource is requested in the future.


<link rel="prefetch" href="http://www.example.com/images/sprite.png" />

<link rel="prefetch" href="/images/sprite.png" />

Supported in:

  • MSIE 11+/Edge
  • Firefox 3.5+ (for HTTPS)
  • Chrome
  • Opera

REFERENCES:

SonarLint for Eclipse IDE

I’ve always been fan of tools for automation of development and testing. I’ve used SonarQube for a long time, and even connect it to my IDE (usually Eclipse), so that I can act on any warnings for code as I’m working on it.

SonarLint takes that to a new level, as it gives notifications before the code is even commited for SonarQube to analyze.

While the instructions here are for Eclipse, SonarLint is also available for IntelliJ IDEA, VisualStudio, and as a command line tool for download from the website.

Eclipse Update Site:
https://www.sonarlint.org/eclipse/

REFERENCES:

HTML5 preconnect

In addition to dns-prefetch, you can take browser performance one step further by actually creating a new connection to a resource.

By initiating an early connection, which includes the DNS lookup, TCP handshake, and optional TLS negotiation, you allow the user agent to mask the high latency costs of establishing a connection.

Supported in:

  • Firefox 39+ (Firefox 41 for crossorigin)
  • Chrome 46+
  • Opera


<link rel="preconnect" href="//example.com" />
<link rel="preconnect" href="//cdn.example.com" crossorigin />

REFERENCES:

HTML5 DNS prefetch

I often get into some fringe areas of micro-optimizations of website performance, DNS prefetching is another one of those topics.

To understand how this can help, you must first understand the underlying concepts that are used within the communications used to build your web page.

The first of these is a “DNS Lookup”, where the domain name (www.example.com) is converted into a numerical address, the IP address of the server that contains the file(s).

In many websites, content is included from other domains for performance or security purposes.

When the domain names are known in advance, this approach can save time on the connection as the lookup can fetched in advance, before it is required on the page to retrieve assets.

This can be particularly useful for users with slow connections, such as those on mobile browsers.


<link rel="dns-prefetch" href="//www.example.com" />

Supported in:

  • MSIE9+ (MSIE10+ as dns-prefetch)/Edge
  • Firefox
  • Chrome
  • Safari
  • Opera

REFERENCES:

IPv6 and IPv4 for Apache Tomcat

If you’ve recently upgraded your network from IPv4 to IPv6, you might find that some software no longer works as it had before. Apache Tomcat is one that I recently stumbled upon, as it seems to prefer the IPv6 connection and stops listening on IPv4 with the default configuration.

The solution is simple, you just have to tell the server to listen on all incoming IP addresses. This worked for me with versions 7.x and 8.x, and I suspect that older and newer versions would be similar.

  1. sudo vi /opt/tomcat/conf/server.xml
  2. To each <Server> entry add:
    address="0.0.0.0"
  3. Restart Tomcat

REFERENCES:

IPv6 DNS configuration

As an IT professional, I’ve long been aware of the impending IPv4 exhaustion. To the layperson, this can easily be compared to phone numbers… there are now so many devices connected to the Internet that the size of the number used to identify and reach each of them uniquely is impossible.

IPv6 is a newer addressing system that supports a drastically increased number of addresses/numbers for use. Unfortunately, like Digital TV (in the US), adoption and migration of users and websites is slow.

To do your part as a user, you can change the settings in your gateway/router/modem to allow for IPv6 DNS lookups as most providers already support IPv6 traffic.

You can test your connection here:
http://www.test-ipv6.com/

Here are a few common values, I’ve also provided the Comcast/Xfinity values for reference:

OpenDNS IPv4:

  • 208.67.222.222 (resolver1.opendns.com)
  • 208.67.220.220 (resolver2.opendns.com)
  • 208.67.222.220 (resolver3.opendns.com)
  • 208.67.220.222 (resolver4.opendns.com)

OpenDNS IPv6:

  • 2620:0:ccc::2
  • 2620:0:ccd::2

Google IPv4:

  • 8.8.8.8
  • 8.8.4.4

Google IPv6:

  • 2001:4860:4860::8888
  • 2001:4860:4860::8844

Comcast IPv4:

  • 75.75.75.75
  • 76.76.76.76

Comcast IPv6:

  • 2001:558:feed::1
  • 2001:558:feed::2

REFERENCES:

Redirect within a javascript file

There often comes a time when you are working on a large project and find a need to refactor javascript resources. Unfortunately, if those assets are accessed by 3rd parties or other code you cannot easily update, you might find yourself stuck.

If you have access to the Tier1 (HTTP server such as ApacheHTTPd) you can often do this within the httpd.conf, or an .htaccess file update. If not, you can always do a simple function within the old javascript file itself, such as the one below.

Put this in the old javascript file location, it is in a closure to prevent the variables from “leaking” into the global namespace.


/* MOVED */
(function(){
"use strict";
var u='/js/newfile.js';
var t=document.createElement('script');t.type='text/javascript';t.src=u;
var s=document.getElementsByTagName('script')[0];s.parentNode.insertBefore(t,s);
})();

Brotli Compression

If you look at HTTP Headers as often as I do, you’ve likely noticed something different in Firefox 44 and Chrome 49. In addition to the usual ‘gzip’, ‘deflate’ and ‘sdhc’ , a new value ‘br’ has started to appear for HTTPS connections.

Request:

Accept-Encoding:br

Response:

Content-Encoding:br

Compared to gzip, Brotli claims to have significantly better (26% smaller) compression density woth comparable decompression speed.

The smaller compressed size allows for better space utilization and faster page loads. We hope that this format will be supported by major browsers in the near future, as the smaller compressed size would give additional benefits to mobile users, such as lower data transfer fees and reduced battery use.

Advantages:

  • Brotli outperforms gzip for typical web assets (e.g. css, html, js) by 17–25 %.
  • Brotli -11 density compared to gzip -9:
  • html (multi-language corpus): 25 % savings
  • js (alexa top 10k): 17 % savings
  • minified js (alexa top 10k): 17 % savings
  • css (alexa top 10k): 20 % savings


NOTE: Brotli is not currently supported Apache HTTPd server (as of 2016feb10), but will likely be added in an upcoming release.

http://mail-archives.apache.org/mod_mbox/httpd-users/201601.mbox/%[email protected]%3E

Until there is native support, you can pre-compress files by following instructions here…
https://lyncd.com/2015/11/brotli-support-apache/

REFERENCES:

Java Dependency Vulnerability scanning with Maven victims-enforcer

One of the OWASP guidelines for secure applications is to not use components with known vulnerabilities. Unfortunately it can be a very difficult and time consuming task to keep up with these manually, automation can save you countless hours!

See https://www.owasp.org/index.php/Top_10_2013-A9-Using_Components_with_Known_Vulnerabilities.

NOTE: victims-enforcer can be used in conjunction with the OWASP dependency scanner. I have only found it to be problematic in ‘tycho’ builds.


<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>1.4.1</version>
<dependencies>
<dependency>
<groupId>com.redhat.victims</groupId>
<artifactId>enforce-victims-rule</artifactId>
<version>1.3.4</version>
<type>jar</type>
</dependency>
</dependencies>
<executions>
<execution>
<id>enforce-victims-rule</id>
<goals>
<goal>enforce</goal>
</goals>
<configuration>
<rules>
<rule implementation="com.redhat.victims.VictimsRule">
<!--
Check the project's dependencies against the database using
name and version. The default mode for this is 'warning'.

Valid options are:

disabled: Rule is still run but only INFO level messages and no errors.
warning : Rule will spit out a warning message but doesn't result in a failure.
fatal : Rule will spit out an error message and fail the build.
-->
<metadata>warning</metadata>

<!--
Check the project's dependencies against the database using
the SHA-512 checksum of the artifact. The default is fatal.

Valid options are:

disabled: Rule is still run but only INFO level messages and no errors.
warning : Rule will spit out a warning message but doesn't result in a failure.
fatal : Rule will spit out an error message and fail the build.
-->
<fingerprint>fatal</fingerprint>

<!--
Disables the synchronization mechanism. By default the rule will
attempt to update the database for each build.

Valid options are:

auto : Automatically update the database entries on each build.
daily : Update the database entries once per day.
weekly: Update the database entries once per week.
offline : Disable the synchronization mechanism.
-->
<updates>daily</updates><!-- was: auto -->

</rule>
</rules>
</configuration>
</execution>
</executions>
</plugin>

Vulnerability database is sourced from: https://victi.ms with backing from RedHat.

REFERENCES: