Remote Access Authorization for Commercial Databases Requiring IP validation and Username/Password Validation -- Lynx URL Rewriter in use at TCU  Kerry Bouchard

3/25/1996, last revised 4/20/2000

NOTE: The Lynx-based solution described here is no longer in production use at TCU. We are now using EZproxy. For a discussion of the TCU EZproxy implementation and other authentication solutions, see the Remote Authorization and Authentication Presentation given at the March 2000 DRA Users' Conference.

Explanation:
Libraries have encouraged commercial database providers to write license agreements that allow library patrons to access databases regardless of the patron's physical location (access is based on "who you are" rather than "where you are"). When these services are remotely mounted and accessed through the telnet protocol, scripting languages (such as DCL) used in combination with C-Kermit provided a simple way to write logon scripts that validated users as being patrons of the library (based on patron id number or some other unique identifier), and then logged the user on to the remote database without ever revealing the library's institutional username and password.

Increasingly, remote access to these same databases has been shifted from telnet to the World Wide Web. The WWW HTTP protocol provides a mechanism to accomplish the first part of the above sequence (validating users by patron id), but not the second part (getting the patrons "logged on" to a remote server without knowing the institutional username and password). This problem is due to the "stateless" nature of the HTTP protocol, and the way the HTTP protocol handles access authorization. (For details, see the draft HTTP 1.1 specification, Chapter 11 --"Access Authentication".) In the case of servers that validate by IP address, rather than name and password, there is still a problem, since many library patrons may be accessing the library menus from a third-party Internet Service Provider. The remote server will see the third party ISP's IP address, not the library's.

The collection of CGI scripts and the executable file below provide a mechanism to work around these problems on library systems running the CERN WWW server under OpenVMS. I have also included some VMS and DRA-specific tools for building password files to accomplish the first task (validating by patron id before users can access the CGI scripts that in turn access the remote database server).

Requirements:

• A server running the CERN HTTP server under OpenVMS. The CERN server is free software; the World Wide Web Consortium currently lists the VMS version as available at: http://delonline.cern.ch/delphi$www/public/vms/distribution.html. (It is possible that some/all of the code below might work with the "Region 6 Threaded HTTP Server", but I have not done any testing with it.)
• An up-to-date version of Lynx, a character-mode WWW browser for UNIX and OpenVMS. This is also free software. The TCU library server is currently using Lynx 2.7. This is distributed as source code (the "LYNX2- 7.ZIP" link on the page referenced above). You will need the OpenVMS DECC C compiler to compile it. Also see my note on fixing a Multinet compile- time bug in the Lynx 2.7 distribution.
• A file of patron id numbers (or some other unique identifier) that you can incorporate into password protection files for the CERN server. (Documentation on one way to do this is provided below).
• A working knowledge of the VMS DCL scripting language, and system-level access to the CERN server HTBIN directory and rule file. At minimum, you will need to modify the username and password symbols in the sample CGI scripts below; to write scripts for database servers not provided in the samples, you may need to modify other statements.
•  

If you have not done so already, you will need to set up Access Authorization password files on your own system to restrict access to the CGI scripts documented below. Otherwise you will be granting the whole world access to the remote databases and violating your license agreements. The example below documents the approach currently used on the Mary Couts Burnett Library system at Texas Christian University.

The CERN server follows UNIX rules for password files: every Username must be unique, but all users can have the same password. The password fields in the file must be encrypted. I have not found any way to force a Web browser to prompt only for password; they always prompt for Username and Password when a server sends an authentication challenge.

Therefore I have used a somewhat awkward mechanism where the user must answer the "Username:" prompt with their ID number (which is unique), and then use a text string common to all patrons as their "Password." (This is explained to the user with a disclaimer screen that comes up before the username and password validation prompt.) The reason for using a generic string as the password, (rather than say, individual last names) is that the HTADM.EXE file that CERN supplies for maintaining server password files is only intended to be run interactively; it is far too resource-intensive to run from a batch job that builds a file containing thousands of ids (assuming you want to update the file on a nightly/weekly basis). By using a generic text string ("tcu") as the "password" for all library patrons, I was able to use the HTADM.EXE program once, to see how it encrypts "tcu", and then include this encrypted string in a GEMbase (Report Writer) job that builds an updated set of password files every night. In these files, every user has a unique "username" (their id number), and "tcu" as their password.

• Step 0: [Preliminary, only do this once before writing the DCL and Report Writer code that performs the steps below]. Decide on a generic "password" text string, and then use the CERN HTADM.EXE program to build a one-line password file with any string as [what HTADM calls] the "username" and the string you chose as [what HTADM calls] the "password." (The password string will always be encrypted the same way, regardless of what username string you choose). Then type this file to the screen to see what 13-letter encryption string the HTADM program generated for your password string.
• Step 1: Build a file or files of patron id numbers.
• Step 2: Use the file(s) generated above and output a new file in which each line consists of: PatronID+":"+Encrytion_String+":". If you grant access to different databases to different classes of patrons (e.g., all patrons get "Core First Search", but only graduate students and faculty get "Extended First Search") you will need to build more than one file.
• Step 3: Convert the file(s) generated in Step 2 to STREAM_LF format so that the CERN server (which expects UNIX file structures) can read it.
• Step 4: [Do once for each CGI script]. Add a rule to the CERN "HTTP80.CONF" file that associates a password file with each logon CGI script you are using. (See Rules in the Configuration File in the CERN HTTP server documentation.)

The following sample programs are available for download. Except for STREAM_LF.FDL, all contain site-specific code that must be edited:

• LIB$BUILD_PASSWORD_FILES.COM -- DCL program to execute steps 1 through 3. This program is called from a nightly job that runs at 3am every morning.
• TCU Nightly Job -- This is a page that links to TCU's nightly batch job files. Most older sites already have a job in place that does the equivalent, so you probably don't need to look at this. It is extremely site-specific.
• PATRON_EXTRACT.IDML -- This GEMbase job builds a patron id file that is used by several other batch jobs (and DCL menu password checks) on our system. Since this job was in place for years before WWW became a factor, the GEMbase jobs below extract their data from the .DAT file produced by this job, rather than by going out to the DRA patron datafiles a second time.
• ALLB_EXTRACT.IDML, and ALLB_REPORT.IDML -- GEMbase code to extract all TCU-affiliated patron id numbers from our DRA Patron database, (executes Step 2).
• FACGRAD_EXTRACT.IDML, and FACGRAD_REPORT.IDML -- GEMbase code to extract faculty-graduate student IDs from the file generated by ALLB_EXTRACT (executes Step 2).
• STREAM_LF.FDL -- FDL file used to execute Step 3 for the two password files generated above.
• HTTP80.CONF -- current CERN server configuration file in use at TCU; shows examples of mapping the password files to the CGI scripts.

Many thanks to Joachim Martin of Harvard University Library for responding to my January 1996 posting to Web4Lib and suggesting using Lynx with the -post_data parameter as a way of automating First Search logons.

The basic technique used takes advantage of the Lynx "-auth", "-source", and "-post_data" command line parameters. In the case of servers that validate by Username and Password, the "-auth" parameter tells Lynx to pass the username and password to the remote server. Since Lynx is invoked from within a CGI script, these parameters are hidden from the user. In the case of servers that validate by IP address, the fact that Lynx is running on the library's server means that it is coming from the IP address that the remote server expects to see (so the "-auth" parameter is not needed). The "-source" parameter tells Lynx to dump the HTML (or other data) it finds at the specified URL to the SYS$OUTPUT device, which can be redirected to a disk file with DCL commands. For servers, such as OCLC's, that ask the user to input a Username and Password into a form, the "-post_data" parameter tells Lynx to "fill in the form" with the username and password automatically; i.e., is an alternative to the "-auth" parameter.

Using Lynx with these parameters, it is possible to write a CGI script that invokes Lynx to access a remote URL, dump the output to a file, and then send this file to the user's browser. However, simply sending the raw file to the user's browser is not enough, for two reasons:

• Most pages contain many "relative" URL's, for example: "/www/stuff..htm", as opposed to: "http://www.myserver.edu/www/stuff.htm". Since the user's browser started out with a URL pointing to a CGI script on the library's server, it doesn't know how to resolve the relative URL's for the remote server.
• For most servers (OCLC is an exception), even absolute URL's will not work, because the Access Authentication dialog took place between Lynx and the remote server, not the user's browser and the remote server. So the next link the user clicks on will challenge them for a Username and Password (or else tell them that they're coming from the wrong IP address), and we are back to the original problem.

The "LYNX_PROXY.EXE" program (available below) addresses these two problems by taking the Lynx "source" file as input and sending output to the user's browser in which: 1) all relative URL's are converted to absolute URL's, and 2) all URL's pointing to files that require Access Authentication have the library server's CGI script prepended to the URL, e.g. "http://www.remote.com/protected.htm" becomes: "http://www.library.edu/htbin/remote.pp?http://www.remote.com/protected.htm".   So access to protected files continues to be redirected through the CGI script.

This is not a very good way to solve the problem - the good way would be to extend the HTTP Protocol "Access Authentication" methods to include commands that would allow a CGI script on a library server to tell a user's browser what authentication data to use for a remote database server. Until/unless the standards group that works on revisions to HTTP implements such extensions, the programs documented below may fill a need.

The proxy scripts are nearly identical, with the exception of several symbol definitions at the beginning of each script, some of which contain site-specific passwords, which must be edited. (In some scripts, such as EI_VILLAGE.PP, there are other modifications to accommodate the quirks of a particular server.) The "TCU$UTIL" logical in each script must be replaced by logical pointing to the directory where Lynx is installed on your system. You will also need to either define a system-level "WWW$SCRATCH" logical pointing to a directory that can be used for temp files, or replace this logical in the scripts. (All of the passwords contained in the files are dummy passwords.)

LYNX_PROXY.EXE (AXP executable) or LYNX_PROXY.EXE (VAX executable) -- this program is called by all of the CGI scripts listed below. The current version of this program was last modified in 6/10/98.
VALIDATE.PP -- this CGI script provides a front-end to all the scripts below.  Since the majority of databases validate by IP address only, VALIDATE.PP can check for a local IP address and send the user directly to the remote database if they are coming from a local IP address.

-###-