Description of the changes for release 1.8d of LISTSERV(R) ---------------------------------------------------------- Copyright 1997-1999 L-Soft international, Inc. 1 March 1999 ************************************************************************* ********************** LISTSERV maintainer's notes ********************** ************************************************************************* The release notes for version 1.8d of LISTSERV(R) have been split into two documents: list owner's notes and LISTSERV maintainer's notes. The present LISTSERV maintainer's notes describe changes that are specific to server or host machine configuration, or too technical to be included in the list owner's notes. LISTSERV maintainers should also read the owners' notes. ************** * Highlights * ************** - Year 2000 compliance for L-Soft products - New developer's guide available - New web-based list/server administration interface - [non-VM] DBMS and mail-merge support - DISTRIBUTE validation changes - [non-VM] TCPGUI support (lcmdx) - Synonymous hostname registration via ALIASES NAMES - [VMS] Failover support ******************************************** * Year 2000 compliance for L-Soft products * ******************************************** Beginning with version 1.8d, LISTSERV is Year 2000 Compliant under all supported architectures except for VM, contingent on operating system support for Year 2000. LISTSERV (and LMail) for VM have the code for Year 2000 compliance but have not yet (as of 1 March 1999) been certified Y2K compliant. Beginning with version 1.1b, LSMTP is Year 2000 Compliant under all supported architectures. ****************************************** * New developer's guide and mailing list * ****************************************** A new guide targeted primarily toward third-party developers has been split off from the existing Site Manager's Operations Manual and will available at release time from http://www.lsoft.com/manuals/index.html along with the other updated LISTSERV manuals. L-Soft has also established a mailing list for third-party developers: LISTSERV-DEVELOPERS@PEACH.EASE.LSOFT.COM . ************************************************************* * [non-VM] New WWW list and server administration interface * ************************************************************* In addition to the web archive interface enhancements (see the list owner's notes for details), version 1.8d introduces a fully-featured list and server administration interface. List owners and site managers can perform almost any list-related maintenance task via the password- protected list owner interface, and in addition, site managers can create lists and manage the overall "look and feel" of the web interface. ********************************************************* * [non-VM] New WWW interface templates and style-sheets * ********************************************************* LISTSERV 1.8d's new WWW interface pages are highly-configurable. Most of the new pages are found in DEFAULT.WWWTPL, and can be configured on a site-wide basis by placing edited copies of the template forms in the file SITE.WWWTPL in LISTSERV's home or MAIN directory. List owners can individualize the look-and-feel of most elements of their web pages by placing edited template forms in listname.WWWTPL. Individual templates can be edited via the new list and server administration interface described above, making the entire template editing experience seamless. Finally, LISTSERV provides an extensible style-sheet feature that makes it possible to harmonize the look-and-feel of your entire site with a few mouse clicks. **************************************** * [non-VM] DBMS and mail-merge support * **************************************** Two of the most significant enhancements in version 1.8d of LISTSERV are DBMS and mail-merge support. Following is a brief description of these new features. Specific installation instructions and samples are available in the new Developer's Guide for LISTSERV, available at http://www.lsoft.com/manuals . While the DBMS and mail-merge functions were designed to work together, they can also be used independently from each other. That is, you may find the DBMS support useful even if you have no need for mail-merge functionality, and likewise you can use the mail-merge functions without a DBMS back-end. Both functions require LISTSERV Classic or LISTSERV HPO, and are unavailable in LISTSERV Lite. DBMS support ------------ LISTSERV's DBMS support allows you to: - Direct LISTSERV to store subscriber information in a DBMS, on a list by list basis. That is, you may have a mix of traditional LISTSERV lists and DBMS lists. Furthermore, you can adjust the layout of your DBMS lists to match existing or current applications. You can map each list to a private table if this is what makes sense for you, or you can put all the lists in the same table, place related lists in one table, etc. You can add as many columns as you want to store additional information about subscribers. - Use the DBMS as a back-end for mail-merge jobs. LISTSERV can execute arbitrary SELECT statements to extract recipients from your DBMS, and make related information (name, country, account number, etc.) available for mail-merge operations. DBMS support is available through Microsoft's ODBC interface on Windows NT, and Oracle's OCI 8 interface on Windows NT, OpenVMS Alpha, Digital Unix, AIX and Solaris (SPARC only). Why require Oracle 8? --------------------- Oracle has introduced major changes to the OCI API in version 8. While many of the concepts remain similar, all the function calls have been renamed and most have a different calling sequence. Supporting both OCI 7 and OCI 8 would require more than a handful of #ifdef statements we would need to develop and support a separate interface for OCI 7. In addition, we would need to license Oracle 7 on all supported systems, which would significantly increase our development costs. While we recognise that there is a large Oracle 7 installed base, a year or so from now most of them will have upgraded to Oracle 8, thus the cost for developing an OCI 7 interface and purchasing six Oracle 7 licenses would have to be amortised over a period of about a year, leading to much higher licensing costs for the OCI 7 version. Furthermore, it is possible to use LISTSERV's OCI interface with an Oracle 7 server: the only component which must be at version 8 is the client, i.e. SQL*Net. As Oracle-based LISTSERV installations typically run on a separate, dedicated system, this simply means that you need to purchase a version 8 client license for the system in question. It is not necessary to upgrade the server to Oracle 8. Oracle 7 is also supported through the ODBC interface. Mail-merge ---------- LISTSERV's mail-merge support allows you to send individually customised messages to large numbers of recipients with very high throughput. The mail-merge functions support: - Simple substitutions, such as "Dear &firstname;". - Conditional blocks, such as a birthday greeting sent when the message happens to coincide with the recipient's birthday, or a warning when the balance of the account is negative. - Special facilities to send promotional banners to a randomly generated subset of the recipients. For instance, you can indicate that a first banner should be sent to a random subset of 200 recipients, while another banner is sent to a randomly selected (but distinct) series of 500 recipients, and others receive a third banner, or no banner at all. - Easy support for "few of many" topic subscription, such as a service offering news about movie actors (many registered actors, while most people will only want news about a handful of them). - Full integration with the DBMS interface, allowing recipients to be selected through arbitrary SELECT statements, while every column that can be converted to a character string is made available as a mail-merge field. - A simple bounce processing and collection system LISTSERV processes and decodes all bounces, and writes the failing addresses to a plain-text file. You can group related mailings in the same bounce file or use a separate file for each mailing, whichever makes the most sense in your context. As each message is sent in "probe" format, even non-compliant bounces will be processed accurately. Mail-merge support requires version 1.1b of LSMTP. LSMTP is currently available for Windows NT and OpenVMS Alpha, and will shortly be available for Digital Unix, AIX and Solaris (SPARC only) the same systems for which the DBMS interface is available. Note that LSMTP and LISTSERV need not reside on the same hardware, that is, it is possible to run the DBMS-enabled version of LISTSERV on (say) AIX, with LSMTP 1.1b running on Windows NT. Why require LSMTP? ------------------ The largest obstacle to the successful deployment of a full-scale mail-merge operation is by far performance. The feedback we received from the customers who participated in the design and early testing of this system was unanimous a mail-merge system would have no value to them unless it could be scaled to at least hundreds of thousands of recipients, if not millions. Unsurprisingly, while customers were willing to pay a moderate surcharge in hardware costs for the mail-merge functionality, they were unable to budget order of magnitude increases. Their requirement, and our design goal, was the ability to run a "normal" 6 figure mail-merge workload on a single high-end PC system. Performance requirements varied from as little as 10,000 hourly deliveries to a demanding 250,000/hour. Out of the box, a normal sendmail configuration running on a high-end PC system will not even reach the 10,000/hour mark. A mail-merge job leads to a large number of spool files (one recipient = one message), and a large spool is sendmail's bane. If you were to construct an artificial workload to demonstrate sendmail's technical flaws, it would be very close to a typical mail-merge workload with the usual 20% of bad addresses. To get an idea of the maximum mail-merge throughput from a system not specifically designed for this purpose, we conducted "brute force" (one recipient = one message) performance tests using LSMTP 1.1a on a 1-CPU, 200MHz Pentium Pro system. Using live traffic from our outsourcing customers and jobs in the 200-300,000 recipient range, we were able to sustain 150,000 hourly deliveries. While well within the target range of 10-250,000/hour, the resulting I/O rate was close to the reasonable practical limit of (then) current disk technology. With a (then) hypothetical 400MHz Pentium Pro CPU, we expected to achieve 175-200,000/hour rather than 300,000/hour. Clearly, the "brute force" approach could only be a short-term solution, as CPU performance improves much faster than I/O throughput. With the "one message = many recipients" approach we adopted in LSMTP 1.1b, we were able to achieve same-LAN throughput rates on the order of 1,000,000/hour on a 200MHz Pentium Pro. While LAN figures are meaningless as an estimate of actual performance with live addresses, in the present case they clearly indicate that I/O is no longer a bottleneck. We chose not to give customers the option of using sendmail, because the programming for a "brute force" approach is vastly different from LSMTP 1.1b's. Development and support costs for the mail-merge functions would have about doubled, whereas the only value of the "brute force" implementation would be in demonstrating that LSMTP is required for good performance. To this end, we have a number of internal (unsupported) utilities which can be used to generate "brute force" mailings, and which we would be pleased to provide to any customers wishing to evaluate the performance of sendmail for mail-merge operations. ******************************************** * [non-VM] New "no-list" bounce processing * ******************************************** LISTSERV version 1.8d introduces a new "no-list" bounce processing feature. This feature is specifically aimed at large mail-merge jobs sent out via DISTRIBUTE (as described in the preceding section). By including the following keyword: FROM=owner-nolist-logname@hostname in the DISTRIBUTE command (where "logname" is whatever you specify), LISTSERV 1.8d silently processes non-delivery notices and uses the new "change-log" feature to record the failing addresses in a file called NOLIST-logname.CHANGELOG. In the past it was necessary to create a so-called "back-end" dummy list in order to make use of LISTSERV's auto-delete features for this kind of DISTRIBUTE job, but with LISTSERV 1.8d this requirement has been removed (and in fact you should not make a back-end list called "NOLIST-logname", or this won't work). LISTSERV records bounces in the changelog like this: 19980704100221 BOUNCE jack@xyz.com 19980704100223 BOUNCE Joe.Doe@abc.def.com so it is easy to go back after the distribution and find the bad addresses. As with the DBMS and mail-merge features, full information is available in the new Developer's Guide for LISTSERV. ********************************* * DISTRIBUTE validation changes * ********************************* To control potential abuse of LISTSERV's DISTRIBUTE command, starting with version 1.8d, the ability to send DISTRIBUTE jobs is limited to LISTSERV Maintainers by default, and requires a personal LISTSERV password for authentication. "Trusted" users may be allowed to send DISTRIBUTE jobs by including them in the new DIST_ALLOWED_USERS site configuration variable. It is possible (if so desired) to revert to the old (1.8c and earlier) behavior by coding DIST_SECURITY=0 in the site configuration. *********************************** * [non-VM] TCPGUI support (lcmdx) * *********************************** A new "TCPGUI" interface is available under LISTSERV 1.8d. In conjunction with an enhanced LCMD interface called LCMDX, it is now possible to speak directly to LISTSERV via a TCP/IP connection. The LCMDX source code is available and can be incorporated into "home-grown" applications to send LISTSERV commands directly from such applications. For more information please see the new Developer's Guide for LISTSERV or the Developer's Corner web site mentioned below. ****************************************************** * Synonymous hostname registration via ALIASES NAMES * ****************************************************** LISTSERV has supported hostname aliases since BITNET added support for this function in 1990. You could define that BITNET node (say) VTVM1 was the same as VPIVM1 and VPIVM2 (old names) and was also known as VTVM1.CC.VT.EDU. Since this was designed into the first major rewrite of LISTSERV, it is very efficient and there is almost no performance penalty. It also works globally, i.e., once the equivalence is defined, it works for all lists and all users. However, the Internet did not have a similar mechanism for registering aliases, so this function was only useful to BITNET sites, although the underlying code would also have worked with Internet aliases if there had been a way to register them. LISTSERV 1.8d introduces support for a centralized list (called ALIASES NAMES) of synonymous Internet hostnames. The main purpose is to address problems with ISPs where the "From:" line is rewritten from (say) "joe@isp.net", which is what Joe wanted, to "joe@alpha.isp.net", "joe@beta.isp.net", "joe@gamma.isp.net" and so forth, where "alpha", "beta" and so on are known machine names (with the understanding that they may add machines from time to time) and "joe" is the same in all cases. In this way it is possible for LISTSERV to match joe@alpha.isp.net with his actual subscribed address of joe@isp.net or any of the other cluster hosts in his domain. This can also handle a situation where an ISP changed name and for instance "joe@oldname.net" is rewritten to "joe@newname.net". However this does not handle "joe@isp.net" -> "J.Doe@isp.net" and the like. For the time being requests by ISPs to be added to the master list should be sent to support@lsoft.com . Note that while it is possible to add entries to your local copy for your local host, you should NOT do this as they will not be propagated through the network and will simply be overwritten by the next update. ************************** * [VMS] Failover support * ************************** (VMS only) Version 1.8d introduces automatic failover support within VMS clusters. The failover facility allows you to build a fully redundant LISTSERV configuration based on an OpenVMS cluster with shared mass storage (CI, SCSI cluster, etc). Failover can take place among any number of suitably configured VAX or AXP nodes. Following a failover, the service will automatically migrate back to the designated primary when it is returned to operation. A priority scheme allows you to establish a hierarchy of failover nodes from most to least preferred (nodes with the same priority are treated as peers). For more information, please request the failover support white paper from the support staff.. ******************************************* * [non-VM] New default LISTSERV home page * ******************************************* Version 1.8d introduces a new (customizable) top-level home page for LISTSERV, which is reached by invoking the 'wa' script without any parameters. This home page contains default links to all aspects of the LISTSERV web interface as well as to L-Soft's CataList service and documentation home pages. ************************************************ * New FILTER_ALLOW site configuration variable * ************************************************ This new site configuration variable determines what, if any, exemptions are made to the standard set of addresses normally bounced by LISTSERV as part of its loop-checking heuristic. For instances, if you want to allow POSTMASTER@SOMEHOST.NET to post to lists like a normal user, you can add that value to this variable. Or if you should decide that any postmaster account anywhere should be allowed to post to lists, you can add the value POSTMASTER@* (but please note that L-Soft does not recommend this). FILTER_ALLOWS works in conjunction with the enhancements (mentioned above) to the Filter= list header keyword. The syntax of the variable is a blank-delimited list of addresses or wildcards, e.g., FILTER_ALLOW='*@GOODNODE.COM POSTMASTER@SOMEHOST.NET' ****************************** * New POST_FILTER exit point * ****************************** A new POST_FILTER exit point has been added to allow filter processing via a list exit. The exit point is called as follows: Name: POST_FILTER Parameters: One; e-mail address of the poster. Return code: 0=Accept, 1=Reject Description: This exit point is called for messages posted to a list. It can be used to compare the e-mail address of the poster to a list of users who should (or should not) be allowed to post. It can further be used to scan the body of the e-mail message, ie, to accept or reject the message based on its content. The exit places the message to be checked in the file $D/listserv.cmsut1 (where $D is the value assigned with .SD D in system.cfg or CORPORAT SYSVARS, D= in go.sys, or D\ in system_config.dat, depending on your OS platform; in any case, by default this is LISTSERV's TMP directory or minidisk). **************************************************************************** * New BOUNCES_TO configuration setting for certain administrative messages * **************************************************************************** Starting with LISTSERV 1.8d it is possible to divert some of the server-level error traffic (that is to say, error traffic not specific to a given list, or errors that should never have been sent to LISTSERV to begin with) to a specified place other than the default (the non-quiet LISTSERV maintainers). This is done by adding the BOUNCES_TO= variable to your site configuration file and restarting LISTSERV. The following specific types of traffic are routed to the BOUNCES_TO= address: spam alerts, spoofing alerts, quota errors (for sites running with the SCOPE=ISP option), and DISTRIBUTE error messages. BOUNCES_TO= can point to one or more users (use the same syntax as for POSTMASTER=) or to a mailing list set up for the purpose. *************************************************************** * Miscellaneous new and enhanced site configuration variables * *************************************************************** In addition to the site configuration variables discussed above, the following are either enhanced or completely new in version 1.8d. IGNORE_EXTERNAL_PRIME (new) is a Boolean value which determines whether or not LISTSERV will ignore the PRIME setting on incoming DISTRIBUTE jobs. This parameter can be used to expedite the delivery of DISTRIBUTE jobs which have an explicit PRIME setting. The point of this is simply to keep the spool clear of DISTRIBUTE jobs that might have a very narrow window (e.g., jobs set for weekend delivery only). Most sites will not need to use this setting, and it is disabled by default. SEARCH_DISABLED (new) is a string variable which, if set to anything but the null string, disables the (new) SEARCH command and returns the text in the string to anyone attempting an archive search. Similar in function to the (Boolean) DATABASE variable for VM, but allows you to customize the string returned to the invoker. This variable is provided primarily for servers such as the IBM P/390 which may not have sufficient resources to offer database searching functionality. This variable should not be used unless there is a good reason to disable the SEARCH command altogether. SORT_RECIPIENTS (changed) has been changed from a Boolean to an integer value. The values 0 and 1 perform as before, i.e., 0= don't sort, 1= sort. Under 1.8d and following, if this variable is set to the value 2, LISTSERV will not sort the recipient list if the number of recipients is less than or equal to your MAXBSMTP setting. Conversely, for jobs where the number of recipients is greater than MAXBSMTP, LISTSERV will sort the recipient list (keeping XYZ.COM together and thereby conserving resources). This enhancement is useful when you are using LSMTP as your outgoing MTA, since LSMTP doesn't care whether or not the recipient list is sorted. The default for all platforms except Windows NT is 0; the default for Windows NT is 2. WEB_BROWSER_CONFIRM (changed) now defaults to 0 (i.e., the feature is disabled by default). Additionally a bug in the original release version of LISTSERV 1.8c that prevented the feature from being disabled was fixed early in the 1.8d development process. WWW_AUTHINFO_DISABLE (new) is a Boolean value which can be set to 1 to disable the IP address verification function of the web archive interface ('wa'), or 0 to re-enable the function. The default is 1, i.e., disable IP address verification. The default setting allows you to bypass certain proxy problems that prevent the 'wa' interface from working properly. There is a certain minimum security exposure involved in that someone with a good memory can watch you using 'wa', memorize the URL, and then type it into another browser and use your login ticket. The ticket will still expire but until then the "thief" has full reign on your list. If the minimum security exposure noted above is unacceptable, this variable can be set to 0 (i.e., enable the authorization). *********************************** * Miscellaneous changes and fixes * *********************************** - In version 1.8c, LISTSERV maintainers were required to use the CREATEPW when updating the "where" parameter of the Notebook= keyword. Under 1.8d LISTSERV maintainers can use their personal LISTSERV passwords instead. - The 'listview' utility (Windows, unix) has a new -e option which prints just the list of email addresses of the list's subscribers (without the "real name" field or the subscriber option string). - There is a new REP option for the PW command (i.e., PW REP newpassword) which acts like PW CHANGE if a password already exists for the user, and acts like PW ADD otherwise. This was added primarily to streamline the web interface but works via mail as well. - If List-ID= is set for a list with web archives, LISTSERV will create HTML pages for both the long and short list names. - If a message without a Subject: line is sent to a list, LISTSERV will now generate a Subject: line with the subject tag for users who have the SUBJECThdr option set. - If Auto-Delete= is set to Max(0) and/or Delay(0), no daily error monitoring report is generated for the list. - A bug in 1.8c under unix and Windows NT that prevented GET with F=MIME/APPL from working properly for binary files has been fixed. - Moderation problems on mixed EBCDIC/ASCII peer chains have been fixed. - The date format yyyymmdd is now accepted as input for database functions. ------------------------------------------------------------------------- LISTSERV is a registered trademark licensed to L-Soft international, Inc. L-SOFT and LMail are trademarks of L-Soft international. LSMTP is a trademark of L-Soft international, Inc. EASE and CataList are service marks of L-Soft international, Inc. UNIX is a registered trademark of X/Open Company Limited. VMS and OpenVMS are trademarks of Digital Equipment Corporation. Microsoft is a registered trademark of, and Windows, Windows NT and Windows 95 are trademarks of Microsoft Corporation. All other trademarks, both marked and not marked, are the property of their respective owners. -------------------------------------------------------------------------