The Apache server power commander part 3
By Dirk Brockhausen
In the two preceding parts of this tutorial we explained the basics of Rules and Conditions.
We will now follow up with two examples to illustrate their use for somewhat more complex applications.
The first example deals with dynamicall generated pages while the second example will cover calling up ".txt" files.
For our first example, let's assume that you want to sell several items of merchandise on your web site.
Your clients are guided to various detailed product descriptions via a script:
http://www.yoursite.com/cgi-bin/shop.cgi?product1
http://www.yoursite.com/cgi-bin/shop.cgi?product2
http://www.yoursite.com/cgi-bin/shop.cgi?product3
These URLs are included as links on your site.
If you want to submit these dynamic pages to the search engines, you are confronted with the problem that most of them will not accept URLs containing
the "?" character.
However, it would be perfectly possible to submit an URL of the following format:
http://www.yoursite.com/cgi-bin/shop.cgi/product1
Here, the "?" character has been replaced by "/".
Yet more pleasing to the eye would be a URL of this type:
http://www.yoursite.com/shop/product1
To the search engine, this appears to be just another acceptable hyperlink, with "shop" presenting a directory containing files "product1", "product2", etc.
If a visitor clicks this link on a search engine's results page, this URL must be reconverted to make sure that "shop.cgi?product1" will actually be called.
To this effect we will make use of mod_rewrite with the following entries:
RewriteEngine on
Options +FollowSymlinks
RewriteBase /
RewriteRule ^(.*)shop/(.*)$ $1cgi-bin/shop.cgi?$2
The variables $1 and $2 constitute so-called
"backreferences". These are related to text groups.
Everything called in the clicked URL which is located before "shop" plus everything following "shop/" is defined by and stored in the two variables $1 and $2
Up to this point our given examples made use of rules such as this one:
RewriteRule ^.htaccess*$ - [F]
However, we did not yet achieve a true rewrite in the sense that one URL would be switched to another.
For the entry in our current example:
RewriteRule ^(.*)shop/(.*)$ $1cgi-bin/shop.cgi?$2
this general syntax applies:
RewriteRule currentURL rewrittenURL
As you can see, this command executes a real rewrite.
In addition to installing the ".htaccess" file, all links in your normal HTML pages which follow the format "cgi-bin/shop.cgi?product" must be changed to: "shop/product" (without the quotes).
When a spider visits a normal HTML page of this kind it will also follow or crawl the product links because there is no question mark contained in the link anymore to prevent it from doing so.
So employing this method you can convert dynamically generated product descriptions into seemingly static web pages and feed them to the search engines.
In our second example we will discuss how to redirect calls for ".txt" files to a program script.
Many webspace providers running Apache will feature system log files only in common format. What this means is that these logs will not store visitor Referrers and UserAgents.
However, in relation to "robots.txt" calls it is preferable to have access to this information in order to learn more about visiting spiders than merely their IPa.
To effect this, the entries in ".htaccess" should be as follows:
RewriteEngine on
Options +FollowSymlinks
RewriteBase /
RewriteRule ^\robots.txt$ /text.cgi?%{REQUEST_URI}
Now, when "robots.txt" is called, the applied Rule will redirect your visitor to the program script "text.cgi".
Furthermore, a variable is conveyed to the script which will be processed by the program.
"REQUEST_URI" defines the name of the file you expect to be called. In out example this is "robots.txt".
The script will now read the contents of "robots.txt" and will forward them to the web browser or the search engine spider.
Finally, the visitor hit is archived in the log file. To this effect, the script will pull the environmental variables "$ENV{'HTTP_USER_AGENT'}" etc. This will provide the required information.
Here is the source code for the cgi script mentioned above:
<BEGIN SOURCE CODE>
#!/usr/bin/perl
# If required, adjust line above to point to Perl 5.
######################################################
# (c) Copyright 2000 by fantomaster.com #
# All rights reserved. #
######################################################
$stats_dir = "stats";
$log_file = "stats.log";
$remote_host = "$ENV{'REMOTE_HOST'}";
$remote_addr = "$ENV{'REMOTE_ADDR'}";
$user_agent = "$ENV{'HTTP_USER_AGENT'}";
$referer = "$ENV{'HTTP_REFERER'}";
$document_name = "$ENV{'QUERY_STRING'}";
open (FILE, "robots.txt");
@TEXT = <FILE>;
close (FILE);
&get_date;
&log_hits
("$date $remote_host $remote_addr $user_agent $referer $document_name\n");
print "Content-type: text/plain\n\n";
print @TEXT;
exit;
sub get_date {
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime();
$mon++;
$sec = sprintf ("%02d", $sec);
$min = sprintf ("%02d", $min);
$hour = sprintf ("%02d", $hour);
$mday = sprintf ("%02d", $mday);
$mon = sprintf ("%02d", $mon);
$year = scalar localtime;
$year =~ s/.*?(\d{4})/$1/;
$date="$year-$mon-$mday, $hour:$min:$sec";
}
sub log_hits {
open (HITS, ">>$stats_dir/$log_file");
print HITS @_;
close (HITS);
}
<END SOURCE CODE>
To install the script, upload it to your web site's main or DocumentRoot directory by ftp and change file permissions to 755.
Next, create the directory "stats".
A more detailed description on how to install a script can he found in our online manuals, e.g. here:
< http://www.fantomaster.com/fantomasSuite/logFrog/lfhelp.txt >
If your server's configuration does not permit execution of Perl or CGI scripts in the main directory (DocumentRoot), you may wish to try the following RewriteRule instead:
RewriteRule ^\robots.txt$ /cgi-bin/text.cgi?%{REQUEST_URI}
Note, however, that in this case you will have to modify the paths accordingly in the program script!
Finally, here's the solution to our quiz from the previous issue of fantomNews:
RewriteCond %{REMOTE_ADDR} ^216\.32\.64
RewriteRule ^.*$ - [F]
Quiz question:
If we don't write "^216\.32\.64\." for a regular
expression in the configuration above, but
"^216\.32\.64" instead, will we get the identical
effect, i.e. will this exclude the same IPs?
The regular expression ^216\.32\.64 will apply e.g. to the following strings:
216.32.64
216.32.640
216.32.641
216.32.64a
216.32.64abc
216.32.64.12
216.32.642.12
Hence, "4" may be followed by any character string.
However, IP addresses can only have the maximal value
255.255.255.255 - which implies that e.g. 216.32.642.12 is not a valid IP. The only valid IP in the list above is 216.32.64.12!
Although the two regular expressions "^216\.32\.64\." and "^216\.32\.64" allow for different strings, due to the technical limitation of IP addresses to 0-255 this range of IPs will remain excluded.
Continue with this tutorial >>>
| DISCLAIMER: The content provided in this article is not warranted or guaranteed by Developer Shed, Inc. The content provided is intended for entertainment and/or educational purposes in order to introduce to the reader key ideas, concepts, and/or product reviews. As such it is incumbent upon the reader to employ real-world tactics for security and implementation of best practices. We are not liable for any negative consequences that may result from implementing any information covered in our articles or tutorials. If this is a hardware review, it is not recommended to open and/or modify your hardware. |
More Web Development Articles
More By Developer Shed
 developerWorks - FREE Tools!
|
<a href="http://zeus.developershed.com/shonuff.php?blackbird=3853&zoneid=442&source=&dest=http%3A%2F%2Fwww.ibm.com%2Fdeveloperworks%2Fspaces%2Fjazz%3FS_TACT%3D105AGY31%26S_CMP%3DDEVSHED&ismap="><img src="http://images.devshed.com/corp/img/news/jazz01.gif" alt="developerWorks Jazz space" align="left"></a>You've heard the buzz about Jazz... want to know more about it from a developer's perspective? Check out the Jazz space on developerWorks. This space is an up-to-date resource for developers, including technical information about Jazz and products built on Jazz, like Rational Team Concert Express. The Jazz space includes content from a wide variety of sources, including links, feeds, and comments from experts.
FREE! Go There Now!
|
|
|
|
David Barnes, Lead Evangelist for IBM Emerging Internet Technologies will discuss aspects of Web 2.0 that bring value to corporations, academia, and government. He'll also discuss IBM's vision around Web 2.0, including the importance of remixability and consumability. The discussion will culminate with examples of various IBM Software Group solutions you can use to get ahead of the Web 2.0 adoption curve.
FREE! Go There Now!
|
|
|
|
As businesses grow increasingly dependent upon Web applications to provide services to customers, employees and partners, these complex applications become more difficult to secure. Although traditional security solutions protect Internet infrastructure layers, they do not guard against HTTP and HTML attacks. Many organizations that conduct security testing still deploy applications that allow attackers to manipulate their logic and wreak havoc on their business. To mitigate this risk, development and delivery teams must address Web application security throughout the lifecycle, addressing the many layers detailed in this paper.
FREE! Go There Now!
|
|
|
|
Visit IBM developerWorks to download the latest trial version of IBM Data Studio V1.1 at no cost. IBM Data Studio is a comprehensive data management solution that helps you effectively design, develop, deploy and manage your data, databases, and database applications throughout the data management life cycle utilizing a consistent and integrated user interface. Unlike other client-side data management solutions that focus on only one aspect of the application lifecycle or database administration, Data Studio complements the Rational Software Delivery platform, providing unparalleled flexibility for a heterogeneous data server environment across platforms.
FREE! Go There Now!
|
|
|
|
Join this webcast to see how IBM Data Studio Developer and pureQuery can take the pain out of Java data access. uApplications developed using both Java and SQL have become a common requirement. Database connectivity using Java Database Connectivity (JDBC) to create an application is a multi-step tedious process, and tooling that covers both SQL and Java has been unavailable, until now. IBM Data Studio introduces the pureQuery platform: a high-performance, Java data access platform focused on simplifying the tasks of developing, managing, and optimizing database applications and services.
FREE! Go There Now!
|
|
|
|
This tutorial shows new users of IBM WebSphere Business Monitor Version 6.0.2 how to perform the "Hello World" equivalent for monitoring business process applications. It is intended to help you get familiar with the capabilities of the product.
FREE! Go There Now!
|
|
|
|
Learn from the best! Find out how developers use Rational ClearCase to be more flexible, innovative and deliver higher quality code in the Rational ClearCase Power Users eKit. This complimentary eKit provides a collection of materials, like articles, whitepapers, and demos that can help you become a power user of Rational ClearCase.
FREE! Go There Now!
|
|
|
|
Discover how Rational tools and best practices for testing can make your job easier. The new Rational Testing eKits provide you with valuable resources – including demos, webcasts, tutorials, and articles – that help you address your specific testing needs across the software lifecycle. Five new eKits are available covering the topics of Requirements and Test Management, Functional Testing, Performance Testing, Code Quality and Embedded Systems, and SOA and Web Services Testing.
FREE! Go There Now!
|
|
|
|
Try the latest version of IBM Rational Manual Tester V7.0.1 by downloading a free trial from IBM developerWorks. This manual test authoring and execution tool promotes test step reuse to reduce the impact of software change on testers and business analysts and addresses the needs of teams performing at least a portion of their testing manually.
FREE! Go There Now!
|
|
|
|
IBM Lotus Notes 8 provides a wide range of developers the ability to provide customized, integrated user interfaces via composite applications and via custom sidebar and toolbar plug-ins. This webcast provides you with tips and techniques to use with out-of-the-box capabilities of Lotus Notes 8, and survey how you can share useful components within your own company and within a larger community.
FREE! Go There Now!
|
|
|
|
All FREE IBM® developerWorks Tools!
|
