Squid3 Proxy on Ubuntu

Using a personal proxy server can be helpful for a variety of reasons, such as:

  • Performance – network speed and bandwidth
  • Security – filtering and monitoring
  • Debugging – to trace activity

Here are some simple steps to get you started,  obviously you will need to further “harden” security to make it production ready!


sudo apt-get install squid3


cd /etc/squid3/
sudo mv squid.conf squid.orig
sudo vi squid.conf

NOTE: the following configuration works, but will likely need to be adapted for your specific usage.


http_port 3128
visible_hostname proxy.EXAMPLE.com
auth_param digest program /usr/lib/squid3/digest_file_auth -c /etc/squid3/passwords
#auth_param digest program /usr/lib/squid3/digest_pw_auth -c /etc/squid3/passwords
auth_param digest realm proxy
auth_param basic credentialsttl 4 hours
acl authenticated proxy_auth REQUIRED
acl localnet src 10.0.0.0/8 # RFC 1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC 1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC 1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
#acl SSL_ports port 443
#http_access deny to_localhost
#http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow localhost
http_access allow authenticated
via on
forwarded_for transparent

Create the users and passwords:

sudo apt-get install apache2-utils (required for htdigest)
sudo htdigest -c /etc/squid3/passwords proxy user1
sudo htdigest /etc/squid3/passwords proxy user2

Open up firewall port (if enabled):

sudo ufw allow 3128

Restart the server and tail the logs:

sudo service squid3 restart
sudo tail -f /var/log/squid3/access.log

OTHER FILE LOCATIONS:

/var/spool/squid3
/etc/squid3

MONITORING with Splunk…

sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/squid3/access.log -index main -sourcetype Squid3
sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/squid3/cache.log -index main -sourcetype Squid3

REFERENCES:

Install Splunk on Ubuntu

Splunk is a popular enterprise level tool for log collection, analysis and management. While you can obtain an enterprise license, most functions are available in the free community edition.

Setup is very easy:

  1. Download and move the .tar.gz file to the appropriate server (i386 vs. amd64)
  2. sudo dpkg -i splunk*.deb
  3. Start the server:

    sudo /opt/splunk/bin/splunk start

    The first time you run after installation or update you will have to accept terms.

  4. Access the admin screen:

    http://HOSTNAME:8000

    login (admin/changeme)
    change password

  5. Go to Settings/Forwarding * Receiving
    – add new (port 9997)
  6. Open firewall port (if enabled):

    sudo ufw allow 8000
  7. Now to start as a service…

    sudo /opt/splunk/bin/splunk enable boot-start

Competitors:

REFERENCES

Install Subversion Server on Ubuntu

Subversion is a commonly used central version control system for software development. There are currently still a large number of organizations that rely upon it, many have since moved on to Git.

  1. sudo apt-get install apache2 apache2-utils
  2. sudo apt-get install subversion subversion-tools libapache2-svn
  3. sudo mkdir /home/svn
  4. svnadmin create /home/svn/test
  5. Create a group for subversion users:
    sudo groupadd subversion
  6. sudo adduser USERNAME
  7. Add a user to the group:
    sudo useradd -G USERNAME subversion
  8. sudo chown -R www-data:subversion /home/svn/test
  9. sudo chmod -R g+rws /home/svn/test
  10. sudo a2enmod dav_svn
  11. To create/clobber a new file for the first user:
    sudo htpasswd -c /etc/apache2/.htpasswd YOURUSER
  12. To add additional users:
    sudo htpasswd /etc/apache2/.htpasswd YOURUSER
    (repeat for new users without the -c as that creates/clobbers the file)
  13. sudo vi /etc/apache2/sites-available/000-default.conf
    Then add to the bottom:
    (NOTE1: the LimitExcept can be enabled to allow anonymous access):
    (NOTE2: the LimitXMLRequestBody can be uncomment to allow large commits)

    <Location /svn>
    DAV svn
    SVNParentPath /home/svn
    AuthType Basic
    AuthName "Subversion Repository"
    # AuthUserFile /etc/svn-auth
    AuthUserFile /etc/apache2/.htpasswd
    #LimitXMLRequestBody 0
    #<LimitExcept GET PROPFIND OPTIONS REPORT>
    Require valid-user
    #</LimitExcept>
    </Location>
  14. sudo service apache2 reload
  15. sudo service apache2 restart

    NOTE: At this point you should be able to browse and do a remote checkout of the code from another machine….

    http://YOUR-IP-OR-HOSTNAME/svn
    and
    svn co http://YOUR-IP-OR-HOSTNAME/svn/test --username YOURUSER --password YOURPASS

  16. sudo vi /etc/init/svnserve.conf
    Add the following:

    # svnserve - Subversion server
    description "Subversion server"
    start on (local-filesystems and net-device-up IFACE=lo and started udev-finish)
    stop on runlevel [06]
    chdir /home/svn
    respawn
    respawn limit 2 3600
    exec /usr/bin/svnserve --foreground --daemon --config-file /home/svn/repos/conf/svnserve.conf --root /home/svn/repos/
  17. Then execute:
    sudo initctl start svnserve
  18. Back on the client side…
    Create a new folder inside your user folder:
    cd ~/test
  19. Check out the project into this folder:
    svn checkout http://YOUR-IP-OR-HOSTNAME/svn/test
  20. Let us just add a new HTML index file to the folder:
    vi index.html
  21. Add it to version control:
    svn add index.html
    Commit the new file:
    svn commit -m "commit message"
    Update:
    svn up
  22. That should cover most cases for you…

REFERENCES:

Install Splunk Universal Forwarder on Ubuntu

After a while it can get tedious to access and review server logs via the command line. There are several tools available that can provide the same information in a graphical manner. Recently I’ve migrated to Splunk as there are both Enterprise and Free versions available.

  1. Of course, you’ll need a Splunk server installed first, as the forwarder is really just another (lighter) instance that will forward the log information to a central location.
  2. Download the system appropriate installer from:
    http://www.splunk.com/download/universalforwarder
  3. Check to see if you are running 32 or 64 bit OS.uname -aIf you see i686 you are 32 bit, if x86_64 you are 64 bit!
  4. Download, you’ll likely need a different version:sudo dpkg -i splunkforwarder-6.1.3-220630-linux-2.6-intel.deb
    or
    sudo dpkg -i splunkforwarder-6.1.3-220630-linux-2.6-amd64.deb
  5. Enable auto-start on reboot:cd /opt/splunkforwarder/bin/

sudo ./splunk enable boot-start

    1. Start the server:sudo service splunk start
    2. Set the password:

      The default ‘admin‘ password is ‘changeme‘ so we need to change it immediately to do anything else, or we will see errors in future steps.

      sudo /opt/splunkforwarder/bin/splunk edit user admin -password YOUR_NEW_PASSWORD -auth admin:changeme

    3. Set the server:sudo /opt/splunkforwarder/bin/splunk add forward-server YOUR_SERVER_ADDRESS:9997

      NOTE: if you get prompted for a splunk username/password you likely skipped the above step. Remember – the forwarder is a new ‘light’ installation of the server and as such has it’s own users!

    4. Enable some monitors on the box:Some common services and log locations to get you started…
      Apache2 HTTPd
      sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/apache2 -index main -sourcetype Apache2
      Tomcat7
      sudo /opt/splunkforwarder/bin/splunk add monitor /opt/tomcat7/logs -index main -sourcetype Tomcat7
      MySQL
      sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/mysql -index main -sourcetype MySQL
      Postfix (SMTP)
      sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/mail.log -index main -sourcetype Postfix
      Squid3 (Proxy)
      sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/squid/access.log -index main -sourcetype Squid3
      sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/squid/cache.log -index main -sourcetype Squid3

      SonarQube
      sudo /opt/splunkforwarder/bin/splunk add monitor /opt/sonar/logs -index main -sourcetype Sonar
      PM2
      sudo /opt/splunkforwarder/bin/splunk add monitor /home/{user}/.pm2/logs -index main -sourcetype PM2
      NPM
      sudo /opt/splunkforwarder/bin/splunk add monitor /home/scott/.npm/_logs -index main -sourcetype NPM
  1. (OPTIONAL) Verify configuration by opening file at the following:sudo su
    vi /opt/splunkforwarder/etc/apps/search/local/inputs.conf
    exit
  2. You now should be able to log into your server and see new data flowing from the forwarder.

    NOTE: this requires you to enable ‘receiving’ of data on the port specified above, usually 9997.

REFERENCES:

Install SonarQube for Continous Inspection

Installation of Sonar requires but a few simple steps, though they can be rather obscure to many developers.

Connect to MySQL:

  1. CREATE DATABASE sonar CHARACTER SET utf8 COLLATE utf8_general_ci;
  2. grant all privileges on sonar.* to 'sonar@localhost' identified by 'sonar';
  3. flush privileges;

Easiest method, if you are on Ubuntu:

  1. sudo vi /etc/apt/sources.list
  2. Then add the following line:
    deb http://downloads.sourceforge.net/project/sonar-pkg/deb binary/
  3. sudo apt-get update
  4. sudo apt-get install sonar
  5. You will then have to stop/start as above to edit the configuration, generally to change database config:
    sudo vi /opt/sonar/conf/sonar.properties

Manual installation (and upgrade?) if you prefer to do things the hard way 🙂

  1. Download and unzip the release:
    wget http://dist.sonar.codehaus.org/sonar-3.6.1.zip
    unzip sonar-3.6.1.zip
    sudo mv sonar-3.6.1 /opt/sonar
    sudo rm -r /opt/sonar (to remove old link)
    sudo ln -s /opt/sonar-3.6.1/ /opt/sonar
  2. cd /opt/sonar/conf
  3. vi sonar.properties
    (uncomment lines for MySQL, comment out H2)
  4. sudo cp /opt/sonar/bin/linux-x86-64/sonar.sh /etc/init.d/sonar
  5. sudo vi /etc/init.d/sonar
    add the following 2 lines:
    SONAR_HOME=/opt/sonar
    PLATFORM=linux-x86-64

    change:
    WRAPPER_CMD="${SONAR_HOME}/bin/${PLATFORM}/wrapper"
    WRAPPER_CONF="${SONAR_HOME}/conf/wrapper.conf"
    PIDDIR="/var/run"

  6. sudo update-rc.d -f sonar remove
  7. sudo chmod 755 /etc/init.d/sonar
  8. sudo update-rc.d sonar defaults
  9. /etc/init.d/sonar start

REFERENCES:

Upgrades can be a little odd, if you see the maintenance page at http://localhost:9000/maintenance, go to http://localhost:9000/setup and do the required steps.

REFERENCES:

MySQL (Windows) service startup error 1067

I’ve installed and managed dozens of MySQL installations for several years, occasionally it seems that an install just doesn’t run like it has in the past.

Recently I had a problem where the service would not start (Error 1067) on Windows Server 2003 (R2)… which is running under VMWare. After checking the obvious places and turning up nothing I started down the list of potential solutions exposed by Google search.

The ultimate solution it seems is that the ‘my.ini’ file needed to include the specific path information required by the service…. without it the service would not start.

Here’s my current file (c:\windows\my.ini) for reference:

[WinMySQLAdmin]
Server=C:/mysql40/bin/mysqld-nt.exe
[mysqld]
basedir=c:/mysql40
datadir=c:/mysql40/data

For the really observant readers of this entry… you will notice that this is for MySQL 4.0 (which is no longer officially supported). It’s mostly used as it is widely compatible across various host systems that are sometimes problematic with newer releases.

Cheers.

The Oath of Enlistment

Let’s review this for just one moment… all ‘sworn’ government officials take a similar oath when they assume office. This is the wording used for enlisted members of the U.S. military.

I,­­­______________, do solemnly swear (or affirm) that I will support and defend the Constitution of the United States against all enemies, foreign and domestic; that I will bear true faith and allegiance to the same; and that I will obey the orders of the President of the United States and the orders of the officers appointed over me, according to regulations and the Uniform Code of Military Justice. So help me God.

Without delving into the details, I personally feel that many of our elected officials chose to ignore their oaths, instincts, and the people they represent. Let’s not forget these facts come election time as voting along “traditional” party lines doesn’t ever seem to fix matters, it’s time for a change!

Disturbed, especially by H1B and illegal immigration.