Sonatype Nexus2 Repository Manager OSS

To allow for repeatable, faster builds in a continuous build environment, it’s often a good idea to use a central repository to cache common assets and prevent the need to download assets from the internet for each build. Using Nexus allows for those transfers to occur over your local network for previously downloaded assets.

You can download the WAR from:
http://www.sonatype.org/downloads/nexus-latest.war

And install on your Java application server, such as Apache Tomcat, via normal means.

If you are using Maven, you’ll need to make appropriate changes in (/.m2/settings.xml) to direct your builds to use Nexus.

Jenkins and other build automation tools will require similar changes.

REFERENCES:

Squid3 Proxy on Ubuntu

Using a personal proxy server can be helpful for a variety of reasons, such as:

  • Performance – network speed and bandwidth
  • Security – filtering and monitoring
  • Debugging – to trace activity

Here are some simple steps to get you started,  obviously you will need to further “harden” security to make it production ready!


sudo apt-get install squid3


cd /etc/squid3/
sudo mv squid.conf squid.orig
sudo vi squid.conf

NOTE: the following configuration works, but will likely need to be adapted for your specific usage.


http_port 3128
visible_hostname proxy.EXAMPLE.com
auth_param digest program /usr/lib/squid3/digest_file_auth -c /etc/squid3/passwords
#auth_param digest program /usr/lib/squid3/digest_pw_auth -c /etc/squid3/passwords
auth_param digest realm proxy
auth_param basic credentialsttl 4 hours
acl authenticated proxy_auth REQUIRED
acl localnet src 10.0.0.0/8 # RFC 1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC 1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC 1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
#acl SSL_ports port 443
#http_access deny to_localhost
#http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow localhost
http_access allow authenticated
via on
forwarded_for transparent

Create the users and passwords:

sudo apt-get install apache2-utils (required for htdigest)
sudo htdigest -c /etc/squid3/passwords proxy user1
sudo htdigest /etc/squid3/passwords proxy user2

Open up firewall port (if enabled):

sudo ufw allow 3128

Restart the server and tail the logs:

sudo service squid3 restart
sudo tail -f /var/log/squid3/access.log

OTHER FILE LOCATIONS:

/var/spool/squid3
/etc/squid3

MONITORING with Splunk…

sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/squid3/access.log -index main -sourcetype Squid3
sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/squid3/cache.log -index main -sourcetype Squid3

REFERENCES:

Install Splunk Universal Forwarder on Ubuntu

After a while it can get tedious to access and review server logs via the command line. There are several tools available that can provide the same information in a graphical manner. Recently I’ve migrated to Splunk as there are both Enterprise and Free versions available.

  1. Of course, you’ll need a Splunk server installed first, as the forwarder is really just another (lighter) instance that will forward the log information to a central location.
  2. Download the system appropriate installer from:
    http://www.splunk.com/download/universalforwarder
  3. Check to see if you are running 32 or 64 bit OS.uname -aIf you see i686 you are 32 bit, if x86_64 you are 64 bit!
  4. Download, you’ll likely need a different version:sudo dpkg -i splunkforwarder-6.1.3-220630-linux-2.6-intel.deb
    or
    sudo dpkg -i splunkforwarder-6.1.3-220630-linux-2.6-amd64.deb
  5. Enable auto-start on reboot:cd /opt/splunkforwarder/bin/

sudo ./splunk enable boot-start

    1. Start the server:sudo service splunk start
    2. Set the password:

      The default ‘admin‘ password is ‘changeme‘ so we need to change it immediately to do anything else, or we will see errors in future steps.

      sudo /opt/splunkforwarder/bin/splunk edit user admin -password YOUR_NEW_PASSWORD -auth admin:changeme

    3. Set the server:sudo /opt/splunkforwarder/bin/splunk add forward-server YOUR_SERVER_ADDRESS:9997

      NOTE: if you get prompted for a splunk username/password you likely skipped the above step. Remember – the forwarder is a new ‘light’ installation of the server and as such has it’s own users!

    4. Enable some monitors on the box:Some common services and log locations to get you started…
      Apache2 HTTPd
      sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/apache2 -index main -sourcetype Apache2
      Tomcat7
      sudo /opt/splunkforwarder/bin/splunk add monitor /opt/tomcat7/logs -index main -sourcetype Tomcat7
      MySQL
      sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/mysql -index main -sourcetype MySQL
      Postfix (SMTP)
      sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/mail.log -index main -sourcetype Postfix
      Squid3 (Proxy)
      sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/squid/access.log -index main -sourcetype Squid3
      sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/squid/cache.log -index main -sourcetype Squid3

      SonarQube
      sudo /opt/splunkforwarder/bin/splunk add monitor /opt/sonar/logs -index main -sourcetype Sonar
      PM2
      sudo /opt/splunkforwarder/bin/splunk add monitor /home/{user}/.pm2/logs -index main -sourcetype PM2
      NPM
      sudo /opt/splunkforwarder/bin/splunk add monitor /home/scott/.npm/_logs -index main -sourcetype NPM
  1. (OPTIONAL) Verify configuration by opening file at the following:sudo su
    vi /opt/splunkforwarder/etc/apps/search/local/inputs.conf
    exit
  2. You now should be able to log into your server and see new data flowing from the forwarder.

    NOTE: this requires you to enable ‘receiving’ of data on the port specified above, usually 9997.

REFERENCES:

Private SSH SOCKS Tunnel

The use of SSH private tunnels as a “poor man’s VPN” is pretty common. When doing so, you can easily tunnel (map) traffic destined to specific ports thru the tunnel to their intended destination.

Common Ports:

  • CVS:  L2401
  • SVN: L3690
  • git: L9418 tcp
  • mysql: L3306
  • rdp: L3389 tcp
  • socks4/socks5: D1080 (no host required)

REFERENCES:

 

Web Proxy Autodiscovery Protocol (WPAD)

If you take a close look at your logs you may occasionally see requests for a file named wpad.dat. This file is related to automatic proxy configuration in many browsers.

To provide this capability to your users and website,

  1. DNS:

    Default behavior is to traverse the domain in reverse, looking for one with a file named /wpad.dat

    Example (using my domain for example):
    wpad.www.giantgeek.com
    wpad.giantgeek.com
    wpad.com

  2. Then in httpd.conf, set the MIME type:
    AddType application/x-ns-proxy-autoconfig .pac
  3. Also in httpd.conf, add a redirect to the actual file you wish to use.
    Redirect permanent /wpad.dat http://www.giantgeek.com/proxy.pac
  4. In the new file, add the following default contents, modify if you use a proxy:

    /* 'proxy.pac' - This is the main function called by any browser
    NOTE: there is NO proxy!
    */
    function FindProxyForURL(url, host)
    {
    return “DIRECT”;
    } // End function FindProxyForUrl

REFERENCES:

Configuring Apache webserver for browser caching of web content…

This is a HUGE topic, I’ve outlined some simple steps below as well as my initial configuration for you to start with…

NOTE: this is for simple ‘static’ content such as images, additional work is required for dynamic (program generated) content, such as that generated in PHP.

1. In ‘httpd.conf’ make sure the following line is uncommented.

LoadModule expires_module modules/mod_expires.so

2.  In ‘httpd.conf’ add the following:

ExpiresActive On
### Expire images 1 day from when they’re accessed
ExpiresByType application/java-archive “access plus 1 day”
ExpiresByType image/gif “access plus 1 day”
ExpiresByType image/png “access plus 1 day”
ExpiresByType image/jpg “access plus 1 day”
ExpiresByType image/jpeg “access plus 1 day”
ExpiresByType image/x-icon “access plus 1 day”
ExpiresByType text/css “access plus 1 day”
ExpiresByType text/javascript “access plus 1 day”
ExpiresByType text/xml “access plus 1 day”
ExpiresByType application/xml “access plus 1 day”
ExpiresByType text/plain “access plus 1 month”
 

3. (Optional) Set default expiry of content in ‘httpd.conf’:

### Expire everything else 1 day from when it’s last modified
ExpiresDefault “modified plus 1 day”

NOTE: These we’re my original settings, you may want to add attitional MIME type and expiry configurations particular to your web content.

REFERENCES:

Proxy Auto-config

There comes a need for many organizations (or individuals) to establish proxy servers on their network. This is usually done for reasons of security or network topology. While the use of proxy servers simpifies some aspects of networking, it comes at the cost of maintaining the browser configuration of every network device (usually browsers). Netscape provided a mechanism to automate much of this problem by allowing the browser to retrieve the proxy configuration from a centrally managed server.

The proxy autoconfig file is written in JavaScript, it should be a separate file that has the proper filename extension and MIME type when provided from a webserver.

The file must define the function:

function FindProxyForURL(url, host)
{
...
}

1. FILENAME EXTENSION:
.pac

2. MIME TYPE:
application/x-ns-proxy-autoconfig

3. REFERENCES:

4. ApacheHTTP config.

Add the following to the httpd.conf file:

Redirect permanent /wpad.dat {yourdomain}/proxy.pac
AddType application/x-ns-proxy-autoconfig .pac

5. EXAMPLE:

/* 'proxy.pac' - This is the main function called by any browser */
function FindProxyForURL(url, host)
{

if (isPlainHostName(host) || // No Proxy for Non FQDN names
shExpMatch(host, “*.localnet”) || // No Proxy for internal network
shExpMatch(host, “127.0.0.1”) || // No Proxy for LocalHost
shExpMatch(host, “localhost”) || // No Proxy for LocalHost
shExpMatch(host, “mailhost”) || // No Proxy for MailHost
dnsDomainIs(host, “giantgeek.com”) || // No Proxy
return “DIRECT”;

else {
return “PROXY proxy.giantgeek.com:8080; PROXY proxy.giantgeek.com:8090; PROXY proxy2.giantgeek.com:8080”;

} //End else

} // End function FindProxyForUrl

NOTE: Also see my ‘WPAD’ blog entry.