CGIProxy 2.1beta15

HTTP/FTP Proxy in a CGI Script

Home > Web Tools > CGIProxy > Beta version

Donate


News | Intro | Installation | SSL Support | Usage | Limits | Disclaimer | Options
Latest Stable Download | Latest Beta Download | FAQ | Changelog | Online Demo (login "free/speech")

October 26, 2006-- Released version 2.1beta15, which contains yet more bug fixes and workarounds, i.e. works with more sites. This may be the last release for a while.

New features in the 2.1 series include:

I don't normally release beta versions, but debugging has already delayed this a long time and I don't want to delay it further. Thus, the standard: This is beta software, and should not be used in a production environment! That said, only the new features (i.e. JavaScript support) will have new bugs; old features should work as well as or better than in the latest stable release.

Here's a more complete changelog, including previous versions.


What it is, what it is

This CGI script acts as an HTTP or FTP proxy. Through it, you can retrieve any resource that is accessible from the server it runs on. This is useful when your own access is limited, but you can reach a server that in turn can reach others that you can't. In addition, the user is kept as anonymous as possible from any servers. Common uses include: anonymous proxies similar to The Anonymizer, other personal uses, VPN-like functionality, and others. It's very simple to install, and very configurable.

When an HTML resource is retrieved, it's modified so that all links in it point back through the same proxy, including images, form submissions, and everything else. Once you're using the proxy, you can browse normally and (almost) forget it's there.

Configurable options include text-only support (to save bandwidth), selective cookie and script removal, simple ad filtering, access restriction by server, custom encoding of target URLs and cookies, and more-- there are more than 50 options so far. It can run under mod_perl unchanged. It requires Perl 5.6.1 or later.

Download CGIProxy 2.1beta15, or just view the source code.

Of course, you must read the Legal Disclaimer before using this software.

Here's a demo (username "free", password "speech"), that lets you browse in the domains jmarshall.com, slashdot.org, network-tools.com, and io.com. (Sorry, the other all-access demo was being misused for anonymous attacks, so I had to take it down.)

The original seed for this was a program I wrote for Rich Morin's article in the June 1996 issue of Unix Review.

Here are older versions, but don't use them anymore.


To Install

To run this, your server must support Non-Parsed Header (NPH) CGI scripts. Most servers do, but not all. (Starting in version 1.3.2, there may be a way to run this script without NPH support; see the $NOT_RUNNING_AS_NPH option below and read the warnings about it in the source code.)

Quick answer: Put nph-proxy.cgi on a Web server and call it.

Longer answer:

  1. Unpack the distribution.
  2. Set any desired options in nph-proxy.cgi by editing the file. See all the options below; the defaults are probably fine if you don't feel like messing with it. If you have special server or network issues, like an SSL server on a non-standard port or an HTTP or SSL proxy that you must use, then see the options category "OPTIONS RELATED TO YOUR SERVER/NETWORK ENVIRONMENT". If you don't know Perl, this can get you started.
  3. Install the script like any other CGI script (set permissions and path to the Perl interpreter). Be sure it's installed as an NPH script. In Apache and related servers, do this by starting the filename with "nph-". If you're unfamiliar with installing CGI scripts, this FAQ entry may help.

If you prefer, Zoltan Milosevic has made an online automatic installer for CGIProxy. Give it your server and account information, and it places the script for you.

If you're installing on a Windows machine, Bennett Haselton of Peacefire has made a downloadable automatic installer for CGIProxy. It includes the packages CGIProxy relies on, like Perl and a Web server.

To add SSL support (which lets you access secure servers), install the two packages as described in the SSL Support section below. Once these packages are installed, CGIProxy will automatically detect them and support SSL. (If the packages aren't present, then CGIProxy will still work fine for everything else except access to secure servers.) Note that if you add SSL support, it is strongly recommended that you run CGIProxy itself on a secure server. Otherwise, secure data will be compromised on the link between the browser and CGIProxy!

To add gzip support (compression), install the Perl module Compress::Zlib if it's not already on your system. Installation is similar to that of the Net::SSLeay module, described in the SSL Support section below.

If heavy use of this proxy puts a load on your server, see "NOTES ON PERFORMANCE" in the source code. By far, the single biggest gain can be made by running this under mod_perl, if you're not doing so already.


SSL Support

To access secure (SSL) servers through CGIProxy, you need to add SSL support to it by installing these two additional packages on the server:

  1. OpenSSL, a freely-available library of SSL and cryptography tools
  2. Net::SSLeay, a Perl module to interface with OpenSSL

OpenSSL is already installed on many servers. You can usually tell which version you have (if any) by entering "openssl version" at a Unix prompt. The Net::SSLeay module is not as common, but you can check whether it's installed and which version you have with:

perl -MNet::SSLeay -e 'print "$Net::SSLeay::VERSION\n"'

Either you get a version number, or it fails if Net::SSLeay isn't installed.

Installing these packages is "beyond the scope of this document" ;), but usually they both install easily with no problems. If you don't have root access on your server, you may need to change the default installation directory, maybe by manually editing the PREFIX setting in Makefile. Once these packages are correctly installed where nph-proxy.cgi can find them, the script will automatically detect them and support SSL; no changes to nph-proxy.cgi are needed. If you have to install Net::SSLeay somewhere that's not on the standard Perl module path (i.e. @INC), then add a "use lib" command to nph-proxy.cgi to tell the script where to find Net::SSLeay, e.g. "use lib 'path/to/your/modules'".

If you're installing on Windows, I'm told that you can install Net::SSLeay from the PPM repository at uwinnipeg.ca by entering these three commands in the PPM shell:

rep add uwinnipeg http://theoryx5.uwinnipeg.ca/cgi-bin/ppmserver?urn:/PPMServer58
search Net_SSLeay.pm
install Net_SSLeay.pm

Note that these two packages are completely unaffiliated with CGIProxy, and may have their own terms of use.

If you need to use an SSL proxy e.g. to get through a firewall, then be sure to set $SSL_PROXY and $SSL_PROXY_AUTH as needed.

IMPORTANT NOTE: It is HIGHLY RECOMMENDED that if you install SSL support for CGIProxy, then CGIProxy itself should be running on a secure server (i.e. accessed with a URL starting with "https://")! Otherwise, you open a serious security hole: any secure data sent to or from a target server will be transmitted insecurely between CGIProxy and the browser, undermining the whole purpose of secure servers.

If you're wondering what happened to the old two-script form of SSL support, here's an answer.


To Use

Call the script directly to start a browsing session. Once you've gotten a page through the proxy, everything it links to will automatically go through the proxy. You can bookmark pages you browse to, and your bookmarks will go through the proxy as they did the first time.


Limits and Bugs


Legal Disclaimer

Censorship is a controversial subject, and some governments and companies have rules about what information you should have access to. If you use my software to bypass rules that have been imposed on you, you assume all legal risks and responsibilities involved. I'm providing the software as a demonstration and teaching tool, and for when legitimate access is needed to non-accessible servers. I won't encourage you to break any rules, because I would get in trouble if I did. I can't prevent you from using this software in illegitimate ways, but I believe the value of it as a teaching tool is far too great to let a few miscreants ruin it for everybody.


Configurable Options in CGIProxy

Here's a list of all the configuration options in CGIProxy, sorted into rough categories. The default settings are in square [] brackets, and should work fine for almost all situations. If you have special server or network considerations, see the options in the category "OPTIONS RELATED TO YOUR SERVER/NETWORK ENVIRONMENT". For more information on any option, see the comments in the source code where it is set, in the user configuration section.

MAIN CONFIGURATION OPTIONS:
$TEXT_ONLY [0] Allow only text resources through the proxy, to save bandwidth.
$REMOVE_COOKIES [0] Ban all cookies to or from all servers. To allow and ban cookies by specific servers, see @ALLOWED_COOKIE_SERVERS and @BANNED_COOKIE_SERVERS.
$REMOVE_SCRIPTS [1] Prevent any script content from any server from reaching the browser. This includes script statements within HTML pages, external script files, etc. To allow and ban script content by specific servers, see @ALLOWED_SCRIPT_SERVERS and @BANNED_SCRIPT_SERVERS. Anonymity is unreliable if you don't either remove scripts, or browse with scripts turned off in your browser.
$FILTER_ADS [0] Remove ads from pages, based on the patterns in @BANNED_IMAGE_URL_PATTERNS. Also ban ad-related cookies by setting $NO_COOKIE_WITH_IMAGE.
$HIDE_REFERER [1] Don't tell servers which link you followed to get to their page. (Yes, it's misspelled on purpose.)
$INSERT_ENTRY_FORM [1] At the top of every page, include a small form that lets you enter a new URL, change your options, or manage your cookies.
$ALLOW_USER_CONFIG [1] Let users set their own $REMOVE_COOKIES, $REMOVE_SCRIPTS, $FILTER_ADS, $HIDE_REFERER, and $INSERT_ENTRY_FORM, via checkboxes on the entry form.
sub proxy_encode {}, proxy_decode {} (Requires minor programming.) You can customize the encoding of destination URLs by modifying these routines. The default is a simple unobscured URL, but sample obscuring code is included in the comments. Note: If you're not removing scripts, then you also need to change _proxy_jslib_proxy_encode() and _proxy_jslib_proxy_decode()-- see the comments.
sub cookie_encode {}, cookie_decode {} (Requires minor programming.) You can customize the encoding of cookies sent to the user's machine by modifying these routines. The default is a simple unobscured cookie, but sample obscuring code is included in the comments. Note: If you're not removing scripts, then you also need to change _proxy_jslib_cookie_encode() and _proxy_jslib_cookie_decode()-- see the comments.
@ALLOWED_SERVERS,
@BANNED_SERVERS [empty]
Allow or ban specific servers from being accessed through the proxy, based on their hostname. Each array is a list of patterns (regular expressions) to match, not just single servers.
@BANNED_NETWORKS
[('127', '192.168', '172', '10', '169.254', '244.0.0')]
Ban specific IP addresses or networks from being accessed through the proxy. Recommended for security when this script is run on a firewall.
@ALLOWED_COOKIE_SERVERS,
@BANNED_COOKIE_SERVERS [empty]
Allow or ban cookies from specific servers. Each array is a list of patterns (regular expressions) to match, not just single servers.
@ALLOWED_SCRIPT_SERVERS,
@BANNED_SCRIPT_SERVERS [empty]
Allow or ban script content from specific servers. Each array is a list of patterns (regular expressions) to match, not just single servers.
@BANNED_IMAGE_URL_PATTERNS
[sample list in source code]
If $FILTER_ADS is set, then ban images that match any pattern in this list.
$RETURN_EMPTY_GIF [0] If an image is banned, then replace it with a 1x1 transparent GIF to show blank space instead of a broken image icon.
$NO_COOKIE_WITH_IMAGE [1] Ban all cookies that come with images or other non-text resources. Those are usually just Web bugs, to track you for marketing purposes.
$QUIETLY_EXIT_PROXY_SESSION [0] (NOT for use with anonymous browsing!!!) For VPN-like installations, let the user browse directly from proxied pages to unproxied pages, with no intermediate warning screens. See the comments for more info.
$PROXIFY_SCRIPTS [1] Proxify all supported script content. Currently, only JavaScript is supported.
$ENCODE_URL_INPUT [0] When submitting a URL through either the start form or the top form, encode it first by using proxy_encode().
$USER_IP_ADDRESS_TEST [''] This lets you call an external test to authorize the user. See comments for more details.
$DESTINATION_SERVER_TEST [''] This lets you call an external test to determine if the destination server is allowed (as opposed to using @ALLOWED_SERVERS and @BANNED_SERVERS). See comments for more details.
OPTIONS RELATED TO YOUR SERVER/NETWORK ENVIRONMENT:
To enable access to secure servers: Install the separate packages OpenSSL and Net::SSLeay. If Net::SSLeay is not in the standard Perl module path, then add a command like "use lib 'path/to/your/modules'" to the script.
To enable compressed (gzip'd) content: Install the Compress::Zlib Perl module. If Compress::Zlib is not in the standard Perl module path, then add a command like "use lib 'path/to/your/modules'" to the script.
$RUNNING_ON_SSL_SERVER [''] Set this if the script is running on an SSL server (i.e. accessed with an "https://" URL). Or, the default value of '' means to guess based on the server port, which almost always works: the script assumes SSL if and only if the server port is 443.
$NOT_RUNNING_AS_NPH [0] Set this if the script is not running as an NPH script (not recommended; see comments for possible dangers).
$HTTP_PROXY,
$SSL_PROXY,
$NO_PROXY [none]
If this script has to use an HTTP proxy (like a firewall), then set $HTTP_PROXY to that proxy's host (and port if needed). Set $SSL_PROXY similarly when using an SSL proxy. $NO_PROXY is a comma-separated list of servers or domains that should be accessed directly, i.e. NOT through the proxies in $HTTP_PROXY and $SSL_PROXY. Also see $USE_PASSIVE_FTP_MODE below when using a firewall.
$PROXY_AUTH,
$SSL_PROXY_AUTH [none]
If either or both of the proxies in $HTTP_PROXY and $SSL_PROXY require authentication, then set these two variables respectively to the required credentials.
@PROXY_GROUP [empty] This is an experimental feature which may help with load balancing, or may have other creative uses. Cookies won't work if you use this. See the comments for further info.
INSERTING A STANDARD HEADER INTO EACH PAGE:
$INSERT_HTML [none] Insert your own block of HTML into the top of every page.
$INSERT_FILE [none] Insert the contents of the named file into the top of every page. Can't be used with $INSERT_HTML.
$ANONYMIZE_INSERTION [0] If $INSERT_HTML or $INSERT_FILE is used, then anonymize that HTML along with the rest of the page.
$FORM_AFTER_INSERTION [0] If $INSERT_HTML or $INSERT_FILE is used, and $INSERT_ENTRY_FORM is set, then put the URL entry form after the inserted HTML instead of before it.
$INSERTION_FRAME_HEIGHT
[80 or 50, depending on $ALLOW_USER_CONFIG]
On pages with frames, make the top frame containing any insertions this many pixels high.
MINOR OR SELDOM-USED OPTIONS:
$SESSION_COOKIES_ONLY [0] Force all cookies to expire when the current browser closes.
$MINIMIZE_CACHING [0] Try to prevent the user's browser from caching, i.e. from storing anything locally. Better privacy, but consumes more bandwidth and seems slower.
$USER_AGENT [none] Tell servers you're using this browser instead of what you're really using.
@TRANSMIT_HTML_IN_PARTS_URLS [empty] Transmit each part of certain HTML pages back to the user as they are received, rather than wait for the whole page. This is set to a list of patterns that match URLs for which you want this treatment.
$USE_PASSIVE_FTP_MODE [1] When doing FTP transfers, use "passive mode" instead of "non-passive mode". Passive mode tends to work better when this script runs behind a firewall, but that varies by network.
$SHOW_FTP_WELCOME [1] When showing FTP directories, always display the FTP welcome message, instead of never displaying it.
$PROXIFY_COMMENTS [0] Proxify the inside of HTML comments as if it's not inside comments.
$USE_POST_ON_START [1] Use POST instead of GET when submitting the URL entry form.
$REMOVE_TITLES [0] Remove titles from HTML pages.
$NO_BROWSE_THROUGH_SELF [0] Prevent the script from calling itself.
$NO_LINK_TO_START [0] Don't link to the start page from error pages.
$MAX_REQUEST_SIZE
[4194304 = 4 Meg]
(Obscure.) The largest request that can be handled in certain rare situations involving password-protected sites.
$ALLOW_UNPROXIFIED_SCRIPTS [0] Allow scripts of unsupported type to pass through the proxy.
$COOKIE_PATH_FOLLOWS_SPEC [0] When handling cookies with no path, treat it according to the cookie spec. If not set, behave as browsers (erroneously) do, i.e. set the path to "/".
$RESPECT_THREE_DOT_RULE [0] Restrict cookie domains as they should be, based on how many dots they have. If not set, behave as browsers (erroneously) do, i.e. loosen restrictions on cookie domains with two dots.


Up to Web Tools page


© 1998-2006 James Marshall
(comments welcome; for questions, please scan the FAQ first)

Last Modified: October 26, 2006 http://www.jmarshall.com/tools/cgiproxy/