Lynx is a command line web browser which I often use for checking the behaviour of redirection headers and content types in a web page’s http headers. This first post shows how to get the headers and the second post how to set the user agent.
Why use Lynx?
There are browser extensions for getting http headers from a page and the browser can often show this sort of information about a page as well, but I find it easy to use Lynx; I have a lot of control over setting things like the user-agent string, and can run it from the command line on remote hosts as well as locally.
Using Lynx to get the HTTP headers
This first example shows how I use Lynx to get the HTTP headers:
lynx -head -dump http://www.example.com/
-head tells Lynx to do a "head" request and -dump to "dump" the output to standard output, i.e. the command line. The above command will show something like this:
HTTP/1.1 200 OK Date: Thu, 16 Aug 2012 02:13:45 GMT Server: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 PHP/5.3.6 X-Powered-By: PHP/5.3.6 Set-Cookie: featured=2222; path=/ Vary: Accept-Encoding Connection: close Content-Type: text/html; charset=iso-8859-1
If the page tells the browser it should be redirected to another page, it will look something like below. Note the "301 Moved Permanently" header, and the "Location" header showing where to redirect to:
HTTP/1.1 301 Moved Permanently Date: Thu, 16 Aug 2012 02:25:03 GMT Server: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 PHP/5.3.6 X-Powered-By: PHP/5.3.6 Location: http://www.example.com/ Vary: Accept-Encoding Content-Type: text/html
Use -mime-header to get the headers for a GET request
Note that the above example sends a "HEAD" request and not a regular "GET" request. If this is a problem you can use the -mime_header flag instead like so:
lynx -mime_header -dump http://www.example.com/
The downside with this approach is it will also dump all the HTML from the page as well. I haven’t found any information in the Lynx man page which indicates if this can be suppressed or not; if it can please let me know in the comments.
Some example output from the second example is as follows:
HTTP/1.1 200 OK Date: Thu, 16 Aug 2012 02:18:41 GMT Server: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 PHP/5.3.6 X-Powered-By: PHP/5.3.6 Set-Cookie: featured=3641; path=/ Vary: Accept-Encoding Connection: close Content-Type: text/html; charset=iso-8859-1 <!DOCTYPE html> <html> ... more html ... </html>
Use -auth=login:password if password protected
If the site being tested is password protected you will get an error message like this:
HTTP: Access authorization required.
Use the -auth=id:pw parameter.
Looking up www.example.com
Making HTTP connection to www.example.com
Sending HTTP request.
HTTP request sent; waiting for response.
Alert!: Access without authorization denied -- retrying
lynx: Can't access startfile http://www.example.com/
Add the auth flag like so, replacing "login" with the actual login name and "password" with the actual password:
lynx -auth=login:password -head -dump http://www.example.com/
Setting the user-agent
I’ve been setting up a mobile version of a website with an m dot domain, and there’s server-sided redirection for mobile devices from the www version to the m version. I used Lynx to test the redirection was working as expected. This is covered in the "Setting the user agent in Lynx" follow-up post.