Archive

Posts Tagged ‘http’

Install Google mod_pagespeed for Apache HTTP

March 14th, 2015 No comments

Website network performance is usually a very complicated process. Over the years, I’ve outlined many development techniques that can be used toward this goal. I’d heard about mod_pagespeed for some time, but never had the opportunity to experiment with it until recently. My first impression is that it is a VERY EASY means to gain performance improvements without reworking your existing website to implement techniques for establishing far-future expires, cache-busting, minification and static file merging.

Out of the box, most common techniques are automatically applied to your assets and a local server cache is created to utilize them.

Default installation is trivial:

  1. Download the latest version for your server architecture:

    wget https://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-stable_current_amd64.deb

    OR

    wget https://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-stable_current_i386.deb

  2. sudo dpkg -i mod-pagespeed-*.deb

  3. sudo apt-get -f install
    (if required)

  4. sudo service apache2 restart

NOTE: Using tools like Firebug will enable you to see an HTTP Header indicating the version being used for your requests.

If you need to modify configuration from the default:

sudo vi /etc/apache2/mods-available/pagespeed.conf

For VirtualDomains, you can selectively enable and disable PageSpeed and many of it’s settings in your appropriate configuration files with:

<IfModule pagespeed_module>
ModPagespeed on
</IfModule>

NOTE: Appending ?ModPagespeed=off to your URL will disable functions for that request.

REFERENCES:

Java User-Agent detector and caching

February 21st, 2015 No comments

It’s often important for a server side application to understand the client platform. There are two common methods used for this.

  1. On the client itself, “capabilities” can be tested.
  2. Unfortunately, the server cannot easily test these, and as such must usually rely upon the HTTP Header information, notably “User-Agent”.

Example User-agent might typically look like this for a common desktop browser, developers can usually determine the platform without a lot of work.

"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3)"

Determining robots and mobile platforms, unfortunately is a lot more difficult due to the variations. Libraries as those described below simplify this work as standard Java Objects expose the attributes that are commonly expected.

With Maven, the dependencies are all resolved with the following POM addition:

<dependency>
<groupid>net.sf.uadetector</groupid>
<artifactid>uadetector-resources</artifactid>
<version>2014.10</version>
</dependency>


/* Get an UserAgentStringParser and analyze the requesting client */
final UserAgentStringParser parser = UADetectorServiceFactory.getResourceModuleParser();
final ReadableUserAgent agent = parser.parse(request.getHeader("User-Agent"));

out.append("You're a '");
out.append(agent.getName());
out.append("' on '");
out.append(agent.getOperatingSystem().getName());
out.append("'.");

As indicated on the website documentation, running this query for each request uses valuable server resources, it’s best to cache the responses to minimize the impact!

http://uadetector.sourceforge.net/usage.html#improve_performance

NOTE: the website caching example is hard to copy-paste, here’s a cleaner copy.


/*
* COPYRIGHT.
*/
package com.example.cache;

import java.util.concurrent.TimeUnit;
import net.sf.uadetector.ReadableUserAgent;
import net.sf.uadetector.UserAgentStringParser;
import net.sf.uadetector.service.UADetectorServiceFactory;
import com.google.common.cache.Cache;
import com.google.common.cache.CacheBuilder;

/**
* Caching User Agent parser
* @see http://uadetector.sourceforge.net/usage.html#improve_performance
* @author Scott Fredrickson [skotfred]
* @since 2015jan28
* @version 2015jan28
*/
public final class CachedUserAgentStringParser implements UserAgentStringParser {

private final UserAgentStringParser parser = UADetectorServiceFactory.getCachingAndUpdatingParser();
private static final int CACHE_MAX_SIZE = 100;
private static final int CACHE_MAX_HOURS = 2;
/**
* Limited to 100 elements for 2 hours!
*/
private final Cache<String , ReadableUserAgent> cache = CacheBuilder.newBuilder().maximumSize(CACHE_MAX_SIZE).expireAfterWrite(CACHE_MAX_HOURS, TimeUnit.HOURS).build();

/**
* @return {@code String}
*/
@Override
public String getDataVersion() {
return parser.getDataVersion();
}
/**
* @param userAgentString {@code String}
* @return {@link ReadableUserAgent}
*/
@Override
public ReadableUserAgent parse(final String userAgentString) {
ReadableUserAgent result = cache.getIfPresent(userAgentString);
if (result == null) {
result = parser.parse(userAgentString);
cache.put(userAgentString, result);
}
return result;
}
/**
*
*/
@Override
public void shutdown() {
parser.shutdown();
}
}

REFERENCES:

HTTP Strict Transport Security (HSTS)

January 24th, 2015 No comments

The HTTP Strict Transport Security feature lets a web site inform the browser that it should never load the site using HTTP, and should automatically convert all attempts to access the site using HTTP to HTTPS requests instead.

Example Use case:
If a web site accepts a connection through HTTP and redirects to HTTPS, the user in this case may initially talk to the non-encrypted version of the site before being redirected, if, for example, the user types http://www.foo.com/ or even just foo.com.

Problem:
This opens up the potential for a man-in-the-middle attack, where the redirect could be exploited to direct a user to a malicious site instead of the secure version of the original page.

Risk:
For HTTP sites on the same domain it is not recommended to add a HSTS header but to do a permanent redirect (301 status code) to the HTTPS site.

Bonus:
Google is always “tweaking” their search algorithms, and, at least at present time, gives greater weight to secure websites.


# Optionally load the headers module:
LoadModule headers_module modules/mod_headers.so

<VirtualHost *:443>
# Guarantee HTTPS for 1 Year including Sub Domains
Header always set Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"
</VirtualHost>

Then you might (optionally, but recommended) force ALL HTTP users to HTTPS:

# Redirect HTTP connections to HTTPS
<VirtualHost *:80>
ServerAlias *
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTPS} off
#RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [redirect=301]
</IfModule>
</VirtualHost>

That’s it…

REFERENCES:

Selenium Firefox modifyheaders

January 4th, 2015 No comments

A few of my tests require access to modify the HTTP Request headers. Unfortunately, Selenium hides access to them to allow for portability, and to better emulate what “users” generally can change. To work around this a Firefox extension can be used and configured at runtime for this purpose.

NOTE: for Maven, you need to place a copy of the .xpi file referenced into the /src/test/resources folder for Selenium to locate it.

In the example below, I’m setting the HTTP Header for “DNT” to “1”.

public FirefoxDriver createFirefoxDriver() throws URISyntaxException, IOException {
// Specify the install location (if not default)
System.setProperty("webdriver.firefox.bin","C:\\path\\to\\Firefox.exe");
// Prevent Console log "noise" from the Selenium Firefox plugin
System.setProperty("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.SimpleLog");
System.setProperty("org.apache.commons.logging.simplelog.log.httpclient.wire", "OFF");
System.setProperty("org.apache.commons.logging.simplelog.log.org.apache.commons.httpclient", "OFF");

final FirefoxProfile profile = new FirefoxProfile();
final URL url = this.getClass().getResource("/modify_headers-0.7.1.1-fx.xpi");
final File modifyHeaders = modifyHeaders = new File(url.toURI());

profile.setEnableNativeEvents(false);
profile.addExtension(modifyHeaders);

profile.setPreference("modifyheaders.headers.count", 1);
profile.setPreference("modifyheaders.headers.action0", "Add");
profile.setPreference("modifyheaders.headers.name0", "DNT");
profile.setPreference("modifyheaders.headers.value0", "1");
profile.setPreference("modifyheaders.headers.enabled0", true);
profile.setPreference("modifyheaders.config.active", true);
profile.setPreference("modifyheaders.config.alwaysOn", true);

final DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setBrowserName("firefox");
capabilities.setPlatform(org.openqa.selenium.Platform.ANY);
capabilities.setCapability(FirefoxDriver.PROFILE, profile);
return new FirefoxDriver(capabilities);
}

Load Testing web application with Selenium and TestNG

December 27th, 2014 No comments

I’ve used Selenium for while to do verification tests of web applications, recently I discovered a very simple way to use it with TestNG and Maven to do some performance testing. TestNG allows for the use of annotations to allow multi-threading and iterations.

pom.xml:

<dependencies>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.8.7</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>2.44.0</version>
<scope>test</scope>
</dependency>
<dependencies>

And as for a simple test to get started with… scripting of steps is available online or could be in a future blog post.

/*
* COPYRIGHT. none
*/
package com.example.selenium;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebDriverException;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testng.Assert;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
/**
* Simple test example for Selenium
*/
public class SeleniumTest {

private static final Logger LOGGER = LoggerFactory.getLogger(SeleniumTest.class);
/**
* TODO Un-comment or change if needed to set your local path!
*/
@BeforeClass
public void oneTimeSetUp() {
System.out.println("-------------------------------------- init ----------------------------------------");
//System.setProperty("webdriver.firefox.bin","C:\\path\\to\\firefox.exe");
}
/**
* NOTE: uses TestNG - behaves differently than JUnit
*/
@Test(invocationCount = 1, threadPoolSize = 5)
public void testLoadApp() {

final String fn = "testLoadApp";
final String baseUrl = "http://www.giantgeek.com/index.php";
LOGGER.debug("[START] Thread Id: {} is started!", Thread.currentThread().getId());

WebDriver driver = null;
final long start = System.currentTimeMillis();
try{
driver = (WebDriver)new FirefoxDriver();
driver.get(baseUrl);

final String actual = driver.getTitle();
LOGGER.debug("Page Title is {}", actual);
final String expected = "GIANTGEEK.COM";
Assert.assertEquals(actual,expected);
//perform whatever actions, like login, submit form or navigation

}catch(final WebDriverException ex){
LOGGER.warn(fn+":WebDriverException:{}",ex);
}catch(final Exception ex){
LOGGER.warn(fn+":Exception:{}",ex);
}
finally {
final long elapsed = System.currentTimeMillis() - start;
LOGGER.debug("[END] Thread Id: {}, elapsed={}", Thread.currentThread().getId(),elapsed);
if(driver != null){
driver.quit();
}
}
}
}

WARNING: Selenium Tests MAY fail if the browser used for testing is updated in the Operating System. Updating the pom.xml to a newer release usually helps!

REFERENCES:

“msapplication-config” and browserconfig.xml

November 15th, 2014 No comments

Windows-8/MSIE-11 introduced Tiles, as such server administrators may have started seeing HTTP 404 errors in their server logs as it attempts to look for a “browserconfig.xml” file at the root of a website domain. If you are inclined to use this file, you should definitely look into the documentation for how to best make use of it. Others may just wish to prevent the error from making “noise” in their log files.

To remove the error, add the following to your pages; alternately you COULD define the URL of your file as the ‘content’ attribute:

<meta name="msapplication-config" content="none" />

You can alternately place an empty /browserconfig.xml on your web server for each domain.

An common example of how to use this file is below:

<?xml version="1.0" encoding="utf-8"?>
<browserconfig>
<msapplication>
<tile>
<square70x70logo src="/mstile-70x70.png"/>
<square150x150logo src="/mstile-150x150.png"/>
<wide310x150logo src="/mstile-310x150.png"/>
<square310x310logo src="/mstile-310x310.png"/>
<TileColor>#8bc53f</TileColor>
<TileImage src="/mstile-150x150.png" />
</tile>
</msapplication>
</browserconfig>

REFERENCES:

Install Subversion Server on Ubuntu

October 18th, 2014 No comments

Subversion is a commonly used central version control system for software development. There are currently still a large number of organizations that rely upon it, many have since moved on to Git.

  1. sudo apt-get install apache2 apache2-utils
  2. sudo apt-get install subversion subversion-tools libapache2-svn
  3. sudo mkdir /home/svn
  4. svnadmin create /home/svn/test
  5. Create a group for subversion users:
    sudo groupadd subversion
  6. sudo adduser USERNAME
  7. Add a user to the group:
    sudo useradd -G USERNAME subversion
  8. sudo chown -R www-data:subversion /home/svn/test
  9. sudo chmod -R g+rws /home/svn/test
  10. sudo a2enmod dav_svn
  11. To create/clobber a new file for the first user:
    sudo htpasswd -c /etc/apache2/.htpasswd YOURUSER
  12. To add additional users:
    sudo htpasswd /etc/apache2/.htpasswd YOURUSER
    (repeat for new users without the -c as that creates/clobbers the file)
  13. sudo vi /etc/apache2/sites-available/000-default.conf
    Then add to the bottom:
    (NOTE1: the LimitExcept can be enabled to allow anonymous access):
    (NOTE2: the LimitXMLRequestBody can be uncomment to allow large commits)

    <Location /svn>
    DAV svn
    SVNParentPath /home/svn
    AuthType Basic
    AuthName "Subversion Repository"
    # AuthUserFile /etc/svn-auth
    AuthUserFile /etc/apache2/.htpasswd
    #LimitXMLRequestBody 0
    #<LimitExcept GET PROPFIND OPTIONS REPORT>
    Require valid-user
    #</LimitExcept>
    </Location>
  14. sudo service apache2 reload
  15. sudo service apache2 restart

    NOTE: At this point you should be able to browse and do a remote checkout of the code from another machine….

    http://YOUR-IP-OR-HOSTNAME/svn
    and
    svn co http://YOUR-IP-OR-HOSTNAME/svn/test --username YOURUSER --password YOURPASS

  16. sudo vi /etc/init/svnserve.conf
    Add the following:

    # svnserve - Subversion server
    description "Subversion server"
    start on (local-filesystems and net-device-up IFACE=lo and started udev-finish)
    stop on runlevel [06]
    chdir /home/svn
    respawn
    respawn limit 2 3600
    exec /usr/bin/svnserve --foreground --daemon --config-file /home/svn/repos/conf/svnserve.conf --root /home/svn/repos/
  17. Then execute:
    sudo initctl start svnserve
  18. Back on the client side…
    Create a new folder inside your user folder:
    cd ~/test
  19. Check out the project into this folder:
    svn checkout http://YOUR-IP-OR-HOSTNAME/svn/test
  20. Let us just add a new HTML index file to the folder:
    vi index.html
  21. Add it to version control:
    svn add index.html
    Commit the new file:
    svn commit -m "commit message"
    Update:
    svn up
  22. That should cover most cases for you…

REFERENCES:

Security through obscurity – hiding your server version information

August 31st, 2014 No comments

I’ve recently spent a lot of time reviewing the OWASP documentation, and (like many corporations) realized that I’d neglected to keep up with this configuration item.

By sharing the exact version of each piece of server software you are using, “hackers” are able to quickly identify unpatched systems and their known vulnerabilities.

To make their work harder, there are a few simple steps that the server admin can take to remove this information from the HTTP Headers and error pages.

Apache HTTPd:

  1. sudo vi /etc/apache2/conf-enabled/security.conf
  2. Add:

    ServerTokens ProductOnly
    ServerSignature Off
  3. If using virtual hosts, add the following to each one:
    ServerSignature Off
  4. sudo service apache2 restart

Apache Tomcat:

  1. vi /opt/tomcat7/conf/server.xml
  2. Find the <Connector > entry and add:
    server="Apache"
  3. cd /opt/tomcat7/lib
  4. mkdir -p org/apache/catalina/util
  5. vi /opt/tomcat7/lib/org/apache/catalina/util/ServerInfo.properties
    server.info=Apache Tomcat
  6. sudo service tomcat7 restart

PHP “X-Powered-By: PHP/5.x.x-1ubuntuX.X”

  1. sudo vi /etc/php5/apache2/php.ini
    expose_php = Off
  2. sudo service apache2 restart

REFERENCES:

Preventing Blackberry browser from messing up your UI

August 10th, 2014 No comments

I’ve previously given steps to prevent phone numbers (and other elements) from being automatically reformatted by Skype Toolbar and IOS Safari, there is still a small segment of the user population that uses Blackberry devices that can similarly benefit from a little code.

The following stops auto detection and formatting of phone and email addresses on devices with the BlackBerry Browser.

HTML:

<meta http-equiv="x-rim-auto-match" content="none" />

WML:

<meta name="x-rim-auto-match" http-equiv="x-rim-auto-match" forua="true" content="none" />

REFERENCES:

Comcast Business Class gateway forwarding port 22 for SSH

March 13th, 2014 No comments

For as long as I’ve had Comcast, and other providers for that matter, I’ve been able to configure my internet gateway/router to allow port 22 (SSH) access to an internal machine. It came as a surprise to me earlier this week that I was blocked when I tried to use their web admin console to change the internal forwarding to a newer machine. As usual, Technical Support was less that helpful and said that it was not possible to do so, and never should have been as Comcast uses that port to administer the gateway. To make matters more disturbing, I was told that I could not have similar SSH access to the gateway, and that replacing their hardware, while permitted, would prevent my use of a static IP.

Back to the solution, as I know that I had only setup this forwarding about a year ago, and it was working only minutes before I tried to change it, I knew that the configuration was possible if I could figure out how it was being blocked. The message in the web console was a javascript alert(); and gave me a starting point. I opened up Firefox and used Firebug to look for the message. Here are a few interesting findings from:

http://HOSTNAME/user/feat-firewall-port-forward-edit.asp

var RemoteManagementPortsCgiBase = “8080,8080,1\|8181,8181,1\|2323,2323,1\|22,22,1\|”;

msg += “Public Port Range conflict with Remote Management Ports.\n”;

if (msg.length > 1)
{
alert(msg);
return false;
}
return true;
}

If you even a little bit of javascript (or simple computer programming for that matter), the solution is clear…. if the ‘msg’ value is empty you will not see the alert or be prevented from making the change you desire.

Lesson to be learned by the Comcast developers (or most likely = subcontractors), always validate submitted form data in your application code, NEVER rely upon javascript alone to verify user entered data!

I also find it interesting that they are also preventing 8080, 8081 and 2323… perhaps that’s their other back doors in these gateways for their access. The same approach should work for those ports if you need it!