L-Soft international, Inc.

Developers Guide
for
LISTSERV®, version 1.8d

22 March 2001
Revision 2

The reference number of this document is 0103-MD-02.

Information in this document is subject to change without notice. Companies, names and data used in examples herein are fictitious unless otherwise noted. L-Soft international, Inc. does not endorse or approve the use of any of the product names or trademarks appearing in this document.

Permission is granted to copy this document, at no charge and in its entirety, provided that the copies are not used for commercial advantage, that the source is cited and that the present copyright notice is included in all copies, so that the recipients of such copies are equally bound to abide by the present conditions. Prior written permission is required for any commercial use of this document, in whole or in part, and for any partial reproduction of the contents of this document exceeding 50 lines of up to 80 characters, or equivalent. The title page, table of contents and index, if any, are not considered to be part of the document for the purposes of this copyright notice, and can be freely removed if present.

The purpose of this copyright is to protect your right to make free copies of this manual for your friends and colleagues, to prevent publishers from using it for commercial advantage, and to prevent ill-meaning people from altering the meaning of the document by changing or removing a few paragraphs.

LISTSERV is a registered trademark licensed to L-Soft international, Inc.
L-SOFT and LMail are trademarks of L-Soft international.
LSMTP is a trademark of L-Soft international, Inc.
EASE and CataLIst are service marks of L-Soft international, Inc.
UNIX is a registered trademark of X/Open Company Limited.
AIX and IBM are registered trademarks of International Business Machines Corporation.
Alpha AXP, Ultrix and OpenVMS are trademarks of Digital Equipment Corporation.
OSF/1 is a registered trademark of Open Software Foundation, Inc.
Microsoft is a registered trademark and Windows, Windows NT and Windows 95 are trademarks of Microsoft Corporation.
HP is a registered trademark of Hewlett-Packard Company.
Sun is a registered trademark of Sun Microsystems, Inc.
IRIX is a trademark of Silicon Graphics, Inc.
PMDF is a registered trademark of Innosoft International.
Pentium and Pentium Pro are registered trademarks of Intel Corporation.
All other trademarks, both marked and not marked, are the property of their respective owners.

All of L-Soft's manuals for LISTSERV are available in ascii-text format via LISTSERV and in popular word-processing formats via ftp.lsoft.com. They are also available on the World Wide Web at the following URL:

URL: http://www.lsoft.com/manuals/index.html

L-Soft invites comment on its manuals. Please feel free to send your comments via e-mail to MANUALS@LSOFT.COM, and mention which manual you are commenting on.

"Hot fix" revisions to this and other L-Soft manuals are posted as they are made to the master document, on the announcement-only mailing list:

LSOFT-DOC-UPDATES@PEACH.EASE.LSOFT.COM
Reference Number 9908-MD-02

Revision History

1. LISTSERV's Archive Search Functions

DOCUMENTATION NOTE

The information in this chapter applies to the "SEARCH" command introduced in LISTSERV 1.8c. It does not contain information pertaining to the old VM "DATABASE" command syntax, nor to LISTSERV DBMS support introduced in version 1.8d.

1.1. Preface

This chapter is an introduction to the LISTSERV archive search functions first implemented in L-Soft's LISTSERV 1.8c and later. It is intended to be a reference document for general users with little or no knowledge of database systems. It does not contain any technical information that general users would have to worry about.

Note: For LISTSERV on VM only, this chapter can be used with current versions (1.8c or later) of LISTSERV. For 1.8b or earlier VM servers, the "INFO DATABase" command will retrieve the manual for the old command syntax. The old database documentation is also available at L-Soft's ftp site in the file

ftp://ftp.lsoft.com/documents/old-listdb.memo This chapter will discuss the syntax and operational characteristics of the LISTSERV database subsystem. It is assumed that the reader is familiar with his or her e-mail client and familiar with sending commands to a LISTSERV server. If you just need a "quick start", read the next two sections for basic instructions. If you want a detailed tutorial on how to use the SEARCH command itself, you might want to skim the next two sections and then start reading with the section entitled "The SEARCH Command". 1.2. A basic database session Let's say that you are looking for messages in the EASE-HOME mailing list that pertain to the list header keyword "Digest=". To search for the term "Digest=" in the EASE-HOME list on HOME.EASE.LSOFT.COM, create a new mail message addressed to LISTSERV@HOME.EASE.LSOFT.COM and in the body (not the subject) of the message, simply type: Search 'Digest=' in EASE-HOME LISTSERV might respond to you with the following: +--------------------------------------------------------------------+ | > Search 'Digest=' in EASE-HOME | | -> 10 matches. | | | | Item # Date Time Recs Subject | | ------ ---- ---- ---- ------- | | 000058 96/01/26 14:44 41 What happened | | 000059 96/01/26 18:14 38 Re: What happened | | 000066 96/02/02 22:51 31 Digest Problem | | 000074 96/02/03 15:01 75 Re: Digest Problem | | 000075 96/02/03 18:52 49 Re: Digest Problem | | 000076 96/02/03 16:27 52 Re: Digest Problem | | 000112 96/02/13 23:37 29 not receiving mail | | 000126 96/02/25 20:20 63 error/bounce msg posted to list How? | | 000172 96/03/13 09:11 12 Digest Mailing Time | | 000483 96/06/22 17:36 34 Header Info | | | | To order a copy of these postings, send the following command: | | | | GETPOST EASE-HOME 58-59 66 74-76 112 126 172 483 | | | | >>> Item #58 (26 Jan 1996 14:44) - What happened | | I never touched the Limits= command or the notebook= All I did | | was try and add: Digest= Yes,Daily | | ^^^^^^^ | | I have tryed this several times with the same reply message: | | | | >>> Item #59 (26 Jan 1996 18:14) - Re: What happened | | > I never touched the Limits= command or the notebook= All I did | | >was try and add: Digest= Yes,Daily | | ^^^^^^^ | +--------------------------------------------------------------------+ Figure 1.1. Sample SEARCH output (further output deleted) Note that LISTSERV includes excerpts from the indexed postings showing the context of the search term(s). We've deleted all but the first 2 in the example above to save space. You would then use the GETPOST command to order the specific posts you wanted to read. For instance, we want to read posts numbered 66, 74 through 76, and 126. You would make another new message (or reply to the response from LISTSERV without quoting the text) and type in the body: GETPOST 66 74-76 126 LISTSERV would then respond with the desired postings. For the non-VM servers, GETPOST is analogous to the old database command "PRINT". There is no corresponding command for the old database command INDEX, since the response to a SEARCH command includes the index of matching postings. 1.3. Narrowing the search It is possible to add further parameters to your search in order to narrow it. You can limit a search by date with a "since. . . " predicate. Likewise, you can limit by sender and/or by the subject line with a "where . . ." predicate. For instance: Search 'Digest=' in LSTOWN-L since 94/01/01 Search 'Digest=' in LSTOWN-L where sender contains 'Thomas' Search * in LSTOWN-L where sender is ERIC@SEARN Search * in LSTOWN-L since 94/01/01 where subject contains 'Digest' are all valid search commands that will (depending on how well you've crafted your predicate) dramatically reduce the number of entries returned to you. 1.4. The SEARCH command This chapter will introduce the formal syntax of the SEARCH command. (Note: The minimum abbreviation of this command is "S".) The syntax of this command is a bit complex, and will be introduced step by step. 1.4.1. Basic search functions The two most important things you have to indicate when you search list archives are: The name of the list whose archives you want to search. What you want to search the individual documents for. The name of the list to be searched is specified after the words or phrases to be sought and is prefixed with an IN keyword. For example, we might do this: Search Rosemary in MOVIES This would select all the entries from list "MOVIES" containing the string "ROSEMARY". Now if you just wanted to see the list of all the movies you can see, you could have used an asterisk as search argument to select all the entries in the list: Search * in MOVIES Note that the mailing list name doesn't have to be uppercased. If you want to "narrow" your previous search, i.e. perform additional tests on the documents that have been previously selected, you must omit the IN keyword. In that case, the search will be applied to the previous "hits" and will create a new "hit list". But in most cases, we will want to search for something longer than one word, for example part of a "key" sentence. Search Hardware problem with a 4381 in IBMFORUM Another problem is that we might not remember the exact original sentence. This is not very important, since LISTSERV will search each word individually: in the above example, any entry that contained the words "hardware", "problem", "with", "a" and "4381" would have matched the search, even if the words appeared in a different order. But what if the original document had "4381-13" in it, instead of "4381"? This is again no problem, as LISTSERV does not require the word to be surrounded by blanks to find a match. Case is also ignored when performing the search operation. That is, "problem" would have found a match on "problems"... and "with" would have found a match on "without" or "withstand"! This may sound like inconsistent behaviour, but you should keep in mind that it is always possible to "narrow down" a search operation. However, once a document has been excluded from the list of "hits", it is very difficult to bring it back. Now what if I want to search for an exact string? For example, I am interested in the string "in C". It is very likely that just any document in the database will contain both a "in" and the letter C. But what I am interested in is things which have been written, or programmed, or implemented, "in C". In that case, it is possible to force LISTSERV to group words together by quoting them, as in: Search 'in C' in UTILITY This method can also be used to insert extra blanks between or before words: leading and trailing blanks are normally removed automatically, but they are preserved inside quoted strings. Please note that quotes must be doubled when specified inside quoted strings, as in: Search 'Rosemary''s baby' in MOVIES The search for 'in C' resulted in over fifty hits, because a match was erroneously found against "in clear", "in core", etc. However, I do not want to search for 'in C ' because there might be hits with "in C." or "in C," in the database and I don't want to miss them. If the search respected the capital C, it would no longer find all those irrelevant hits. To do this, you must enclose your search string in double-quotes instead of single quotes, for example: Search "in C" in UTILITY Note that single quotes should not be doubled inside double-quoted strings, and vice-versa. Only quotes of the same type as the string should be doubled. It is important to understand the difference between the two types of quoting. If you request a search for 'TEXT', you will find a match on "TEXT", "Text", "text" or even "teXt". This is the same behaviour as unquoted text. However, if you request a search for "TEXT", it will only find a match on "TEXT", not on "text" or "Text". Quoting is also the only way to search for a reserved keyword like "IN": if you tried "Search in in UTILITY", LISTSERV would report that database "IN" does not exist and would reject the command. This is because the keyword IN indicates the end of your search arguments. If you quote it, however, it will not be recognized and will be searched as you wanted it done. Similarly, if you want to search for an asterisk, you will have to quote it since Search * indicates that all entries should be selected. Now the problem is that there may be sentences starting with a capital I, e.g. "In C, it would be coded this way:". How can I catch these sentences? Actually, you have been using "complex search expressions" from the beginning without even being aware of it. When you specified a search on Hardware problem with a 4381 , you had in fact been asking LISTSERV for: "Hardware NEAR problem NEAR with NEAR a NEAR 4381". The "NEAR" is implicit, but it may be overriden. [Note that this is a change from the way the "locate" clause was implemented in 1.8b and earlier. Earlier versions used a default of "AND" instead of "NEAR" between discrete search terms. The difference is that Search JOE SMITH in XYZ-L looks for JOE and SMITH close to each other rather than simply looking for instances of both terms in the entire document. You can still use AND explicitly. Note that 'a NEAR b NEAR c' is defined as '(a NEAR b) AND (b NEAR c)', so the NEAR operator is not fully commutative.] You may even use parenthesis if needed: Search ("in C" or "In C") and program in UTILITY The "NEAR" can still be implied, as in: +--------------------------------------------------------------------+ | Search wooden chair (blue or green) in CHAIRS | | Search (wooden chair) or (plastic chair) in CHAIRS | | Search plastic chair (blue or green but not streaked) in CHAIRS | | | | The following commands are strictly equivalent: | | Search (wooden chair) or (plastic chair not blue) in CHAIRS | | Search chair (wooden or (plastic not blue)) in CHAIRS | | Search chair (wooden or (plastic but not blue)) in CHAIRS | | Search chair NEAR (wooden OR (plastic AND NOT blue)) in CHAIRS | +--------------------------------------------------------------------+ Figure 1.2. Sample SEARCH commands using complex document search arguments 1.4.2. Date specifications Since each document has been assigned a "date/time" field, it is possible to select documents based on this date field. This is accomplished by appending "date search rules" to the search expression, as in: +--------------------------------------------------------------------+ | Search problem (serious or severe) in BBOARD since july | | Search problem in BBOARD since oct 85 | | Search symptom in BBOARD since 12/28 | | Search error report from 12 january to august in BBOARD | | Search user complaint until 18 sept in BBOARD | | Search data check since today 11:53 in EREP | +--------------------------------------------------------------------+ Figure 1.3. Sample SEARCH commands using date search arguments The default values for omitted arguments are always chosen so as to exclude as few entries as possible. For example, "July" would mean "1 July 00:00:00" in a SINCE specification, and "31 July 23:59:59" in an UNTIL clause. The only exception is the year field, which always defaults to the current year. 1.4.3. Keyword search specifications The last thing you may wish to search is the "keywords" list. For example, you might want to select those plastic chairs which cost less than 50 dollars. It is assumed that the price will vary often (maybe almost daily), and that it is therefore kept externally from the document describing the chair. Thus, you would have a "Price" keyword which you could search in the following way: Search plastic chair in CHAIRS where price < 50 You may of course use complex expressions (with parenthesis) in the WHERE clause. There are new comparison operators available for this clause, like IS, CONTAINS, all the usual arithmetical comparison operators, and some more. However, the AND operation is no longer implied, but it can still be specified explicitly of course: Search plastic chair in CHAIR where price < 50 and avail > 4 The problem now is that, as the search commands become more and more complex, they will no longer fit in a single line. To solve this problem, we begin the command with the string "// " (two front-slashes and a space) and follow it with the SEARCH command and the search specifications. Any database command ending in a comma indicates that more is to follow on the next line. This process can be repeated several times if desired. +--------------------------------------------------------------------+ | // Search chair (wooden or (blue or green but not streaked)) , | | in CHAIRS , | | where price < 50 & avail > 4 | | | | // Search chair (wooden or (blue or green but not streaked)) , | | in CHAIRS where price < 50 & avail > 4 | | | | // Search chair (wooden or ( , | | blue or green but not streaked) , | | ) , | | in CHAIRS where price < 50 & avail > 4 | +--------------------------------------------------------------------+ Figure 1.4. Sample SEARCH commands with continuation lines: All these commands are strictly identical, although the first one is obviously more legible. The only "trick" about this continuation line business is that you should always keep quoted strings on a single line. The process of identifying continuation lines and concatenating them afterwards may cause unwanted blanks to be inserted in the command line, which is no problem outside a quoted string since blanks are ignored, but might cause erroneous results in a quoted string. If you want to search for several possible values in a given keyword, you do not have to repeat the keyword name and operator: +--------------------------------------------------------------------+ | // Search * in BBOARD where , | | subject contains (PC or (Personal and computer)) | | | | is strictly equivalent to: | | | | // Search * in BBOARD where , | | subject contains PC or , | | (subject contains Personal and subject contains computer) | +--------------------------------------------------------------------+ Figure 1.5. Sample use of "factorization" However, it should be noted that this "factorization" is performed according to the rules of logic, which may not necessarily match those of English grammar. This removes any possible ambiguity as to the meaning of these clauses. Let's consider the following example: machine does not contain (IBM and DEC) This clause will get translated into: machine does not contain IBM and machine does not contain DEC In English you would probably say "machine contains neither IBM nor DEC" . This is how LISTSERV will understand it. However, if you read the clause aloud, you will probably not pronounce the parenthesis and will end up saying "machine does not contain IBM and DEC", in other words, "machine does not contain both IBM and DEC" , which is a totally different thing (and would most probably be true all the time). The "English meaning" could be obtained with the following clause: not (machine contains (IBM and DEC)) In the former case, the negative "does not contain" operator is inserted inside the parenthesis. In the latter, only "contains" is moved, and the negation remains outside. +--------------------------------------------------------------------+ | // Search gateway problem , | | in BBOARD , | | since sept 86 , | | where sender contains (john or paul but not mick) , | | and subject does not contain lost | | | | -> 5 matches. | | | | Item # Date Time Recs Subject | | ------ ---- ---- ---- ------- | | 000012 87/10/18 13:09 12 The gateway has stopped working | | 000017 87/08/24 09:18 9 Glory glory alleluja! Again!!! | | 000018 87/10/18 13:09 8 You know what? It WORKS!!! | | 000024 87/10/18 13:09 7 Guess what happened today? | | 000205 87/10/04 16:59 9 Who's going to babysit it today? | | | | | | You might now wish to narrow your search down to exclude postings | | whose subject contains "work". For instance, | | | | // Search gateway problem , | | in BBOARD , | | since sept 86 , | | where sender contains (john or paul but not mick) , | | and subject does not contain (lost or work) | | | | -> 3 matches. | | | | Item # Date Time Recs Subject | | ------ ---- ---- ---- ------- | | 000017 87/08/24 09:18 9 Glory glory alleluja! Again!!! | | 000024 87/10/18 13:09 7 Guess what happened today? | | 000205 87/10/04 16:59 9 Who's going to babysit it today? | +--------------------------------------------------------------------+ Figure 1.6. Sample SEARCH commands with keyword search clauses 1.4.4. Phonetic search There may be cases where you are looking for a certain value of a keyword, the exact spelling of which you cannot remember. In these cases, it may be useful to try a phonetic search. A phonetic search will yield a match for anything that "sounds like" your search string, as dictated by a predefined algorithm which is of course not perfect. It may give a hit for something which does not actually sound like your search string, or, more rarely, omit a keyword which did sound like what you entered. The main reasons for this are that the algorithm must be fast to execute on the machine and therefore not too sophisticated, and that the way a given word is pronounced depends on the idiom in which the word was written. For example, the phonetical transcription of the name "Landau" will be different in French, English, German and Russian. Thus, it is impossible to decide whether a word sounds like another if the language in which the words are pronounced is not known (and of course LISTSERV does not have, a priori, any way to know it). Phonetic searches are performed through the use of the SOUNDS LIKE and DOES NOT SOUND LIKE operators, which are syntactically similar to CONTAINS and DOES NOT CONTAIN. That is, you could do something like: Search * in PHONEBOOK where NAME sounds like WOLF There is a little trick with the SOUNDS LIKE operator that you should be aware of. If your search string (WOLF in our above example) is a single word, it will be compared individually to all the words in the reference string (i.e. the data from the database), and will be considered a hit if it "sounds like" any of the words in the reference string. Thus, the search word "Ekohl" sounds like the reference string "Ecole Normale Superieure" because it matches the first word. If the search string contains more than one word, the search and reference strings will be compared phonetically as a whole (and "Ekohl Dzentrahll" will therefore not match "Ecole Normale Superieure"). Note that any search string containing more than a single word must be quoted, as explained in the previous sections of this chapter. +--------------------------------------------------------------------+ | > Search * in BITEARN where site sounds like (COHRNEAL and | | LAPORRADRY) | | | | -> 3 matches. | | | | Ref# Conn Nodeid Site name | | ---- ---- ------ --------- | | 0292 87/03 CRNLASSP Cornell University Cornell Laboratory of Atomic| | 0301 87/03 CRNLION Cornell University Cornell Laboratory of Plasma| | 0307 87/06 CRNLNUC Cornell University Laboratory of Nuclear Studes| | | | > Search * in BITEARN where SITE sounds like HOPTIKK | | | | -> 2 matches. | | | | Ref# Conn Nodeid Site name | | ---- ---- ------ --------- | | 0751 87/09 FRIHAP31 Assistance Publique - Hopitaux de Paris | | 2120 87/04 UOROPT University of Rochester The Institute of Optics | | | | > Search * in BITEARN where SITE sounds like SCHIKAGO | | | | -> 1 match. | | | | Ref# Conn Nodeid Site name | | ---- ---- ------ --------- | | 0140 86/03 BMLSCK11 Studiecentrum voor Kernenergie (SCK/CEN), Mol, | +--------------------------------------------------------------------+ Figure 1.7. Sample SEARCH commands involving phonetic match In the figure above, the first command shows an example of accurate phonetic match, where the result is exactly what the user expected. In the second example, the user found what he was looking for ("Optics"), but an additional unwanted entry was selected. This is by far the most common case. The last command is a typical example of phonetic clash, where the algorithm did not translate the search string into phonetics as the user expected it, with the result that the desired name ("Chicago") was not found and that completely irrelevant entries were presented instead. The phonetic matching algorithm used by LISTSERV is a slightly modified version of SOUNDEX -- a well-known algorithm that provides reasonably accurate matches at a very low CPU cost. Although it gives best results with the English language, for which it was originally designed, it is not too strongly tied to it and can still be used with other languages. It is of course absolutely impossible to write an program that would work for all the languages in the world, or even for the most widely used ones, since their interpretation of the most common combinations of letters are completely incompatible. 1.4.5. What to do about "100 matches (more available)." LISTSERV limits the number of matching records in a given response to no more than 100. This is done primarily to stop hackers from tying up the server with SEARCH requests on lists with thousands of archived postings, but it also keeps the size of the response down to a manageable level. For instance, sending a "SEARCH *" command for an old, very large list could result in a response measuring in megabytes if not for the 100-record limitation. There are a couple of different approaches to a solution: (Preferred) Narrow your search parameters as explained in 1.3 and following. Specify the first record for your search so that LISTSERV knows to start in a certain place. For instance, if you sent a search command like search * in lstown-l where sender contains nathan@lsoft.com that resulted in more than 100 "hits", and the last four hits were something like > +--------------------------------------------------------------------+ | 007696 95/08/23 16:02 13 Re: How to send 'urgent' messages to | | digest users? | | 007698 95/08/23 18:10 52 Re: Blocking expletives | | 007699 95/08/23 20:00 41 Re: How to send 'urgent' messages to | | digest users? | | 007716 95/08/25 10:34 33 Re: ignore subsequent lines to | | listserv? | +--------------------------------------------------------------------+ you could send a followup search command using the following syntax: search * in lstown-l.7716- where sender contains nathan@lsoft.com to get the next 100 hits starting with message number 7716. Note that it is important to include the trailing hyphen after the starting message number, as otherwise you will get back a response containing only a reference to the message number you specified. 1.4.6. Specifying the last n posts as a range Starting with LISTSERV 1.8d it is possible to specify that your search criteria be applied to only the last n posts in the archive. For instance, say you are only interested in checking the last 50 postings of LSTOWN-L to see if nathan@lsoft.com posted. You would send search * in lstown-l.last50- where sender contains nathan@lsoft.com Again you would have to use the hyphen after the number. The number n can be any integer, but as with any other search, if LISTSERV finds more than 100 hits only the first 100 will be returned to you. Thus it would be possible for n to be 1000, or even 10,000, but you will still have to narrow your search if you want more hits. 1.4.7. Exact syntax description This section describes the exact syntax of the "SEARCH" command in technical terms. You can skip it if you are not interested in learning about the details of this command. General syntax +--------+-----------------------------------------------------------+ | | | | Search | search-rules <optional-rules> | | | | | | Optional rules are: | | | | | | date-rules | | | keyword-rules | +--------+-----------------------------------------------------------+ The optional "date-rules" and "keyword-rules" arguments may appear in any order. Date rules specification You may optionally restrict the search to only those entries that lay within a given interval of time. This is accomplished by specifying one of the following date rules: SINCE date-spec <time-spec> FROM date-spec1 <time-spec1> TO date-spec2 <time-spec2> UNTIL date-spec <time-spec> The format of a "date-spec" is quite complex because of the number of different ways date/time specifications are usually expressed: TODAY yy dd mm <dd><->monthname<-><yy> mm/yy mm-yy yy/mm/dd yy-mm-dd yyyymmdd (yyyymmdd is accepted by LISTSERV version 1.8d and following.) Month names can be abbreviated to any length. If there is an ambiguity, the first month in chronological order is retained. For example, "J" would mean "January", "JU" would be "June" and "JUL" would unambiguously select "July". The format of a "time-spec" is simply <hh:mm<:ss>>. +--------------------------------------------------------------------+ | FROM 14 july TO oct 97 | | SINCE 96 | | UNTIL 23-JUN-97 | | SINCE today 11:30 | +--------------------------------------------------------------------+ Figure 1.8. Sample date clauses NOTE: Case is irrelevant in date specifications. The keywords (SINCE, UNTIL, etc) have been capitalized only for better legibility, and can be entered in lower case if desired. Keyword rules specification You may request the actual document search to take place only for those entries which match a set of "keyword comparison" rules. The syntax is the following: WHERE kwd-expression WITH kwd-expression "kwd-expression" is, generally speaking, an mathematical expression of keyword/value comparisons, possibly bound by logical operators. Comparison operators have a higher precedence than logical operators, that is, "A>10 AND B=20" is interpreted as "(A>10) AND (B=20)". The available comparison operators are listed below. All the operators appearing on a given line are synonyms. +--------------------------------------------------------------------+ | = IS | | ^= <> IS NOT | | > | | < | | >= | | <= | | SOUNDS LIKE | | DOES NOT SOUND LIKE | | CONTAINS | | DOES NOT CONTAIN | +--------------------------------------------------------------------+ Figure 1.9. Comparison operators for WHERE clauses All these operators are self-explanatory, except the last two which allow you to search the keyword value for a given "substring". That is, "Sender contains jeff" would be true if the value of the "Sender" keyword was "Jeff Smith" or "Jeffrey Donaldson". The case is ignored during the comparison unless the search operand is double-quoted. If no valid comparison operator is specified between two arguments, "IS" (identity) is assumed. The available logical operators are: +--------------------------------------------------------------------+ | ^ NOT | | & AND BUT | | | / OR | +--------------------------------------------------------------------+ Figure 1.10. Logical (boolean) operators Please note that the logical operators AND and OR have equal precedence and are evaluated left-to-right. Finally, keywords and operators can be "factorized" when the same comparison is to be applied to a given keyword and a series of comparands. For example, you might enter: Search * where sender contains ('CS Dept' and (Jack or Phil)) This is internally expanded to: // SEARCH * WHERE sender CONTAINS 'CS Dept' AND , (sender CONTAINS Jack OR sender CONTAINS Phil) Please note that the expression must always be enclosed in parenthesis, even if it is a simple one: Search * where sender contains (Joe or Morris) This stems from the fact that comparison operators have a higher priority than logical (boolean) ones. +--------------------------------------------------------------------+ | WHERE Sender is "Arthur Dent" , | | and Subject does not contain tea | | | | WITH Refcode 8467272 and Location Roubaix | | | | WITH (QTY > 100 | PRICE > 1000) & MAT = COPPER | | | | Where Sender is (Atiaran@Land or Elena@Land) , | | and Subject contains ('Be true' but not Ur-Lord) | +--------------------------------------------------------------------+ Figure 1.11. Sample WHERE clauses Search rules specification Finally, you must specify what is to be searched inside the document. If you do not want anything to be sought at all (e.g. if you are only selecting known items from the database), you can specify an asterisk as a placeholder to waive the search. Otherwise you must specify a mathematical expression where arguments are search strings, possibly bound by logical operators (see Figure 10 for a comprehensive list). The default operator is AND, so that a search for "INTERPRET STEM PROBLEM" will select all entries where "INTERPRET", "STEM" and "PROBLEM" can be found (not necessarily in the same line). +--------------------------------------------------------------------+ | Search * | | | | Search 'I/O' Error | | | | // Search Interpret (performance or tips , | | but not (bug or question)) | +--------------------------------------------------------------------+ Figure 1.12. Sample document-search clauses Reserved words and quoting When to quote strings Keyword names and search arguments need not be quoted, unless: They are formed of more than one word (search arguments only). They contain leading or trailing blanks (search arguments only). Their name matches one of the "reserved keywords" of the LISTSERV database system, and appears in a context where it can be mistaken for such. The "reserved keywords" are: FROM, IN, SINCE, TO, UNTIL, WHERE, WITH. They contain a parenthesis, logical operator or comparison operator symbol. More generally, you should quote any string that contains one of the following characters: ( ) < > = | & ^ / Any non-quoted word will be stripped of leading and trailing blanks and converted to uppercase before the search. Single-quoted strings Strings quoted in single-quotes (') are converted to upper case and cause case to be ignored during the search. That is, they behave in the same manner as un-quoted strings as far as the search algorithm is concerned. As a rule of thumb, any string can be single-quoted if desired, even if it does not have to. Single quotes must be doubled inside single-quoted strings, but double quotes should not: Search '"T''amo, ripetilo, si caro accento' in OPERA Double-quoted strings Strings quoted in double-quotes (") are not converted to upper case. They result in a case-sensitive search, which means that you should never double-quote a string unless you want case to be respected during the search. Double quotes must be doubled inside double-quoted strings, but single quotes should not: Search """T'amo, ripetilo, si caro accento" in OPERA 2. LISTSERV "Commands-Job" Feature and CJLI Interpreter 2.1. Introduction The "Commands-Job" feature of LISTSERV was designed in an attempt to allow for powerful inter-LISTSERV (and more generally, program-to-LISTSERV) command transmission with message redirection and multi-line arguments capability, while still allowing inexpert users to send commands to LISTSERV for execution in a very simple, intuitive" way. The implementation of Commands-Jobs has therefore been split into two different layers: the 'core' command job language interpreter, with its exacting, powerful but stern control cards syntax, and the 'outer' interface to the user which provides the required default control cards whenever they have been omitted, translating an "intuitive" series of commands to execute into an actual commands-job that can be processed by the 'core' interpreter. Since this documentation is primarily intended for postmasters and LISTSERV applications programmers, it will be oriented towards a description of the 'core' interpreter. The work of the 'outer interface' will only be mentioned for better understanding. The 'core' interpreter will be referred to as 'Commands Job Language Interpreter' (CJLI) in the following discussion. Warning: if you are familiar with MVS and JCL, you will probably notice some similarity between command job control cards and JCL. This similarity is purposeful and was intended to make CJLI easier to understand for JCL adepts and to make MVS users more comfortable with CJLI (MVS users who do not have any mailing system such as UCLA mail and have difficulties sending/receiving messages are often forced to use (basic) CJLI control cards to send commands to LISTSERV). However, there are a number of differences between CJLI and JCL and you should not assume that a given JCL card has the same meaning and syntax as its CJLI counterpart. Some JCL features which were deemed to be unnecessary (eg dataset concatenation with a blank DD card) have not been implemented, and new features have been implemented whenever required. 2.2. Overview -- the JOB entity As soon as the 'outer' interface detects that a file contains commands (as opposed to a mail or non-mail file intended for redistribution to a list), it passes it to the CJLI for execution. This physical file will contain one or more logical 'JOB's with interspersed comment lines. Each 'JOB' will contain zero or more commands, start with a "// JOB" card and end with a "// EOJ" card. The 'outer' interface will provide these cards if omitted, but this will be detailed later on. Anything before the first "// JOB" card, after the last "// EOJ" or between "// EOJ" and "// JOB" cards is ignored. Notably, mail headers and "Acknowledge-To:" fields (at the bottom of the mail file) would be ignored. A physical job file containing two jobs might look like this: (any number of header lines, ignored) //jobname1 JOB options . . . //jobname1 EOJ (any number of lines, ignored) //jobname2 JOB options . . . //jobname2 EOJ (any number of lines before EOF, ignored) Each job in the physical file is a completely independent entity which contains commands and the 'dataset' definitions required to execute the commands properly. Jobs are executed in sequential order; if a job does not execute successfully, the other jobs in the physical file are still executed, unless the job has been terminated by a 'global' error (eg error reading the physical file), in which case the CJLI terminates after transferring the file to the postmaster. Within a given job, commands are executed in sequential order regardless of the result of the previous commands. Each job generates a separate 'output', which is sent to its recipients (see below) before the next sequential job is executed. This 'output' consists in a series of messages which are (unless specified otherwise -- see below) sent back as a single mail file. There are basically three kind of cards in a job stream: A. Control cards, which start with "//" in column 1 and are interpreted by the CJLI. Control cards further subdivide into three categories: Pre-execution control cards, of the form: "//label kwd args". "label" and "args" can be omitted but there must still be a blank between the "//" string and the keyword name, ie "// kwd". If CJLI does not recognize the keyword, it strips off the leading "//" and considers the card as a command-card (see below). Comments, of the form: "//* any_comment_text". These cards are ignored by CJLI. Note the blank between the asterisk and the first character of the comment text. Execution-time control cards, of the following format: "//*kwd args". Note that the keyword is concatenated to the "//*" string to differentiate this from a comment. If the keyword is not recognized, the card is treated as a comment control card and ignored. It is otherwise processed by CJLI at execution time, as if it were a normal command card, and in sequence order with the other command cards. These cards are actually commands which are only available from within a command job because they would be irrelevant outside of a job context. B. Command cards, which contain the text of the actual commands to be executed. They are ignored by CJLI which passes them to the main LISTSERV command interpreter for execution. If the card starts with an asterisk, it is treated as a comment and ignored. C. Data cards, which are assembled into a dataset under control of a "// DD" control card. They are removed from the job stream by CJLI and are kept in a separate pool for later reference at command execution time. 2.3. Control Cards -- general syntax rules All control cards, except comment control cards, follow the same syntactic rules: Control card starts with the string "//" in column 1. Control cards can span any number of physical records: continuation cards can be defined by placing a comma at the end of the first physical line, and having the continuation card start with the string "// " (note the blank) in column 1. This process can be repeated any number of times. No blank is inserted between the end of the first card and the beginning of the continuation card; however, anything before the comma is kept. Examples: //card1 DD "Some very long text which, // requires a continuation card" --> "Some very long text whichrequires a continuation card" //card1 DD "Some very long text which , // requires a continuation card" --> "Some very long text which requires a continuation card" Since this approach makes it impossible to "cut" a line which ends in a large string of blanks, an alternate method was designed for blanks-sensitive cutting. If the continuation card starts with the string "//+ " in column 1 instead of just "// ", the continuation card is not stripped of leading blanks and data from columns 5-80 is appended to the first card. Example: //card1 DD "Some very long text which, //+ requires a continuation card" --> "Some very long text which requires a continuation card" Control cards can contain a label of any length starting in column 3. This label is translated to uppercase. If the label is omitted, there must be a blank in column 3. The label can contain any character, except blank and the slash sign ("/"). The label is followed by at least one blank. The next word in the card is the "card name", which is translated to uppercase. Arguments can be specified after the "card name", and must be separated from it by at least one blank. Arguments are separated by commas, and there must not be any blank before or after the comma. There are two categories of arguments: a. Positional arguments, which must appear in the correct order (which will usually depend on action being performed). b. Keywords, of the form "name=data". They can appear in any order and can be freely intermixed with positional keywords. They do not affect the sequence order of positional keywords. Keyword names are translated to uppercase, and can contain any character except blank, comma, double-quote or equal sign. For example, //JOB1 JOB XDZ,ECHO=NO,FRECP11,PW=EMERALD //JOB1 JOB PW=EMERALD,XDZ,FRECP11,ECHO=NO //JOB1 JOB Echo=NO,pW=EMERALD,XDZ,FRECP11 are three different wordings of the same arguments string. There are furthermore two different forms of "data" for arguments: a. Quoted data: in that case the data is enclosed in double quotes and can contain one or more blank delimited words as well as leading or trailing blanks. Quoted data is case-sensitive, and can contain any character except a double-quote. b. Non-quoted data: in that case the data consists of a single word (i.e., blanks cannot appear in the data), which is translated to uppercase. Non-quoted data cannot contain blanks, commas or double-quotes. In the above example, specifying 'Echo=No' is functionally identical to 'ECHO=NO', while 'Echo="No"' would leave the argument in mixed case. Quoted data is used for list of recipients, full-names, etc. Comments can be included at the end of the card, and must be separated from the arguments by at least one blank. If you did not specify any argument string, and still want to place comments at the end of the control card, you must specify a null argument string before the comments by putting a ", " before the comment text, e.g.: //JOB1 JOB , These are comments //JOB1 JOB XDZ,FRECP11 These are comments too 2.4. The JOB control card The JOB control card indicates the start of a new job and allows you to define several "execution options" for the job. The format of the JOB card is: //jobname JOB options 'jobname' is the name you want to assign to the job. If you leave it blank, CJLI will default it to be your 'userid'. The following options are available: Echo=YES|NO The default value (if the keyword is omitted) is YES and indicates that each command must be echoed to the job output before execution. The command is then prefixed with a "> ", and preceded by a blank line on the job output. Reply-to=Sender|None|"u@n1 u@n2..." The default value is "Sender" and indicates that the output of the job is to be sent to the sender of the job. "None" indicates that no output should be generated, and is used whenever the sender is a server which is not programmed to parse the output of a LISTSERV job. Also it makes sure that no loop can ever occur due to an error in a job. "u@n1 u@n2..." can be used whenever replies are to be sent to a different person/list of persons. These persons will first receive a separate message telling them that they are receiving the output of another person's job. Reply-via=Mail|Message|MSG The default value, "Mail", indicates that the job output is to be sent to its recipients in the form of a mail file. "Message" and "MSG" both indicate that interactive messages are desired. Note that an attempt to send interactive messages to a node which cannot handle them will result in LISTSERV sending a piece of mail containing the text of the various messages. Stat=YES|NO (Starting with 1.8d) Determines whether or not server statistics ("Summary of resource utilization") for a given JOB are output in the command response to the invoker. The default is YES. A server-wide default (also starting with 1.8d) may be set with the JOB_STAT_DEFAULT site configuration variable. PW=password Is the default password to use on commands where an explicit "PW=" keyword has not been specified. It makes it easier to write jobs performing several maintenance commands on the same distribution list, for example. All other keywords, as well as positional parameters, are ignored. If an invalid value is specified for a valid keyword, the job is terminated by CJLI but the remaining jobs in the physical file are still executed. If the JOB card is omitted, the 'outer' interface provides the following default JOB card: //userid JOB Echo=Yes,Reply-to=Sender,Reply-via=Mail,Stat=Yes,PW="" 2.5. The EOJ control card The EOJ card indicates the end of a job; its syntax is very simple: //anything EOJ where 'anything' can be any valid label and is completely ignored. The 'outer' interface provides an EOJ card at the end of the physical job file, as well as before a new JOB card, if none was provided by the user. 2.6. The DD control card The DD control card allows you to define single or multi-line 'datasets' for use by the various commands in the job stream. The syntax of the DD card is the following: //ddname DD "single-line-constant" * *,EOF *,EOF,Res=Disk *,EOF,Res=Storage 'ddname' is the name the dataset is to be given. It must follow the general label naming convention, cannot be omitted and cannot appear more than once in the job stream. That is, datasets cannot be concatenated by means of several DD cards. "single-line-constant" denotes a single-line dataset, whose value is that of the first (quoted) argument. The length of this argument must not exceed 255 bytes. '*' denotes a multi-line dataset, whose successive lines immediately follow the DD card. Any number of lines can thus be included in a dataset, with a "/*" line indicating the end of the dataset. The JCL "DD DATA,DLM='xxxx'" is not implemented. Note that unlike JCL, CJLI does NOT end the dataset when a control card (ie one starting with "//") is encountered; the "/*" must always be specified. '*,EOF' denotes a multi-line dataset, whose successive lines immediately follow the DD card and end at the end of the PHYSICAL job file. This option is used when transmitting "unknown" data in a dataset which could a priori contain any kind of character string. Needless to say, there can be only one such dataset in the job file, and it must be the last dataset in the last job. 'Res=' indicates whether the dataset is to reside in storage ("Res=Storage") or on disk ("Res=Disk"). In some cases it may be necessary to keep a large dataset on disk to avoid running out of storage and to improve execution speed when a disk-file is to be generated anyway by the command using the dataset. The "Res=" keyword is therefore ignored on all datasets except the '*,EOF' one (if present), and causes a disk file to be generated with the remainder of the input deck. Please note that not all commands will support the "Res=Disk" option: commands which do not expect to receive a large dataset as input will usually expect to find it in storage and report an error when the "Res=Disk" option is used. For example, DISTRIBUTE fully supports "Res=Disk" while DELETE doesn't. An invalid dataset declaration causes the job to be terminated by CJLI with the remaining jobs in the physical file still being executed. 2.7. The //*MSG control card The "//*MSG" execution-time control card allows you to selectively halt/resume 'typing on the job output' or to discard messages which have been previously output during execution of the job. Note that the latter option has no effect when the output of the job is sent as interactive messages, as it is intended to control mail job output. The syntax of the "//*MSG" control card is: //*MSG option1,option2,... Valid options are: ON, OFF, FLUSH FLUSH discards all messages previously sent to the job output and leaves the message-receipt status unchanged. OFF turns message receipt off, as if "Reply-to=None" had been specified in the JOB card. ON turns message receipt back on. This does NOT override a possible "Reply-to=None" in the JOB card, though. 2.8. Special considerations This section contains more information on the trickiest parts of CJLI as well as some useful hints for application programmers. LISTSERV treats anything mailed to the LISTSERV userid as a set of commands to execute. Thus, as soon as an unknown command is encountered in the job stream, the whole physical job file (not just the current job) is immediately flushed and discarded. This avoids 'executing' hundreds of unknown commands and sending back a huge job output when a regular mailfile is sent to the LISTSERV userid by someone who thought it would be distributed to a list. Note that errors from known commands do not cause termination of the job -- only completely unknown commands such as "Hiya!!" would terminate the job. Although CJLI is based on the network standard 80-characters card images, LISTSERV accepts command jobs in several network formats, including Disk Dump and Netdata. In that case it will accept records of up to 255 characters as input, and you may find this very convenient when sending long commands to the server. Alternatively, continuation cards can be used to split long commands into several 80-characters cards. In that case you must insert a "// " string before the command text so that CJLI considers it as a control card and performs the required concatenation; it will then realize that the "card name" is unknown and transform the card into a regular command card. If you opt for that method, you will find the "//+" continuation card feature very convenient for a program (but not for a human person). This method is used by LISTSERV when transmitting "DISTRIBUTE" commands to other LISTSERVs. There is no limit at all on the final size of a control card, i.e., on the number of continuation cards you can specify; however, you must make sure that no line in any of the 'datasets' ever exceeds 255 characters. In particular, if the physical job file is sent in Netdata format, you must make sure that the file lrecl is not higher than 255. Note for VM servers: Due to internal coding considerations, it is recommended that physical job files be sent to LISTSERV in PUNCH format. This will make it easier for the 'outer' interface to detect the job file for what it is and will save some CPU time to the server, thereby improving job response time. Please keep in mind that DD lines longer than 80 characters cannot be sent in PUNCH format. For more information on how to send commands to LISTSERV for execution, see LISTSERV MEMO. 3. Relayed File Distribution and the DISTRIBUTE Command Note that DISTRIBUTE is also documented in RFC1429. 3.1. Introduction The Relayed File Distribution (also known as DISTRIBUTE) feature was developed in an attempt to provide an efficient and network-resources-saving means whereby files and mail could be distributed to a large number of persons on the network by ANY network user, without having to resort to predefined distribution lists (which have other advantages but are basically static). This chapter is composed of three independent sections. The first one (3.2) is a description of the distribution algorithm used by Relayed File Distribution. It is general enough to be accessible to inexpert computer users, but does not explain how to send a Relayed File Distribution Request (RFDR) to LISTSERV. The second section (3.3) gives a detailed technical description of RFDRs and assumes the reader is familiar with the basic concepts of the Commands-Jobs Language Interpreter (CJLI). More information about CJLI is available in Chapter 2, above. The third section (3.4 and 3.5) is a more practical tutorial regarding the construction of RFDR jobs. The general user who wishes only to learn how to set up simple RFDR jobs and who is not interested in the technical issues involved may therefore skip sections 3.2 and 3.3, moving directly to 3.4. Please note that while "RFDR Job" is actually the correct nomenclature for a job sent out using Relayed File Distribution, the more common (if not as entirely correct) term "DISTRIBUTE Job" may also be used herein, interchangeably, to describe such jobs. Starting with LISTSERV 1.8d, a new security validation feature has been added to the DISTRIBUTE command. The new feature changes the default behavior of DISTRIBUTE in two ways: Only a LISTSERV maintainer (i.e., a user who is identified in LISTSERV's site configuration file as a POSTMASTER=) or a "trusted" user (identified in LISTSERV's site configuration file in the DIST_ALLOWED_USERS= variable) may issue the DISTRIBUTE command; and The DISTRIBUTE command must be validated with the issuer's personal LISTSERV password (obtained with the PW ADD command). The new default can be relaxed back to the previous (pre-1.8d) behavior by setting the new DIST_SECURITY site configuration variable appropriately. See Appendix C of the Site Manager's Operations Manual for details. 3.2. What is Relayed File Distribution? Relayed File Distribution is a service provided by the ever-growing network of L-Soft LISTSERV servers to all users connected to the Internet who are allowed to send and receive files or messages. (The origins of the service were on BITNET and associated services where it was possible to send files, as opposed to messages.) The user desiring to send the message (whom we will call the 'sender') provides his nearest L-Soft LISTSERV host with a copy of the message to be distributed and the list of persons who are to receive it. He can optionally indicate the full name of these persons as well as his own full name, and they will be used in information messages and mail headers as appropriate. The LISTSERV hosts will then relay the file to each other as explained below and distribute the file to all the indicated recipients in the most efficient way they could 'manage'. The server that initially receives the file from the sender will first distribute the file to the (possible) recipients of its local node, which does not generate any network traffic. It will then examine the list of non-local recipients and determine, for each of them, which of the L-Soft LISTSERV servers is nearest.¹ Whenever it finds itself to be the nearest server, it distributes the file directly, and removes the recipient from the list. It then uses a rather complex algorithm to "route" the remaining recipients through its nearest servers, and transmits the request to them. This will be best illustrated by an example. User X@FRECP11 sends a file distribution request for the following list of people: Y@FRECP11,X@CEARN,Y@CEARN,X@CZHRZU1A,X@NEUVM1,X@IBACSATA, X@EARNET, X@PSUVM We will assume that a L-Soft LISTSERV server has been installed at the following sites: FRECP11, CEARN, DEARN, DKEARN, EARNET Finally we will assume the following topology, which will not necessarily reflect the actual network topology at the time you read this chapter (it has been simplified and nodes which do not participate in the transfer have been removed) CZHRZU1A | | V FRECP11 --> FRORS31 --> FRHEC11 --> CEARN --> DEARN --> DKEARN --> NEUVM1 | | | | | V | GWUVM --> PSUVM V EARNET --> IBACSATA The file is distributed by LISTSERV@FRECP11 to Y@FRECP11, and then forwarded to LISTSERV@CEARN, which will distribute to the two CEARN recipients and to X@CZHRZU1A. LISTSERV@CEARN then forwards one copy of the file to LISTSERV@DEARN and one to LISTSERV@EARNET, with the appropriate list of recipients. LISTSERV@EARNET distributes to the EARNET recipient and to X@IBACSATA. LISTSERV@DEARN distributes to X@PSUVM, and also to X@NEUVM1. LISTSERV@DKEARN does not receive anything because it would have served only a unique recipient, and in that case a direct send can only be better. The file has therefore crossed a total of 11 links. If it had been sent the normal way from FRECP11, it would have had to cross 32 links, i.e. thrice more. The reduction is very important because we assumed that a server is installed at all the central nodes in the network, which is not necessarily the case in practice. 3.3. A technical description of Relayed File Distribution Requests Relayed File Distribution Requests ("RFDR Jobs") must imperatively be transmitted as a commands-job file. It is recommended that you read Chapter 2 of this manual if you are not familiar with the basic concepts of CJLI. Beginning with 1.8d, by default DISTRIBUTE may only be executed by a LISTSERV maintainer (defined in the POSTMASTER= variable in the site configuration file) or a "trusted" user (identified in LISTSERV's site configuration file in the DIST_ALLOWED_USERS= variable) and requires a password (the invoker's personal LISTSERV password). The job sent to the server must contain: A JOB card with the "Echo=No" option to avoid tracing the unique command to the job output. The "Reply-via=" keyword can be set to "Message" if so desired, although this is not recommended. The job name can be anything you want, eg filename.filetype A DISTRIBUTE command with the appropriate arguments: DISTribute < <MAIL> <DD=ddname> <FROM=u@n | FROM=DD=ddname> <ACK=NOne | MAIL | MSG> <TO DD=ddname | TO u@n1 <...>> > <PRIOR=* | nn> <INFORM=MSG | MAIL> PW=password All the keywords (except PW=) can be omitted, but must be specified in the indicated order. The "PRIOR" and "INFORM" keywords are indepedent from the others and can appear anywhere in the command line. Beginning with LISTSERV 1.8d the DISTRIBUTE command requires a password (PW=password , above) for validation purposes. A description of the keywords follows: MAIL indicates that the file to be distributed is a mailfile, to be sent to the MAILER at the destination node. The contents of the file are proofread and the "mail origin" is verified so that users cannot send forged mail. MAIL-type distribution is 100% transparent to the mail recipient, which is not the case with file distribution -- a file would come from the LISTSERV userid instead of the sender's userid. The mailfile must contain a blank "To:" line where the actual name and network address of each recipient will be automatically inserted as the file is distributed to him. All DISTRIBUTE jobs sent from non-VM servers will be MAIL-type DISTRIBUTE jobs. Non-MAIL DISTRIBUTE is available only on VM. DD=ddname is the ddname corresponding to the data to be transmitted (ie the file or mailfile). The default is "DD=DATA" FROM=xxxx indicates either the network address of the file sender or the name of a dataset of which the first line indicates the network address and full name of the file sender, eg FROM=DD=XXX and //XXX DD "JPB@BIGNODE John P. Brown". The default is FROM= your-userid. This field contains the address you want bounces to go to (the RFC821 MAIL FROM: address) and it can be used to redirect bounces² that should not normally go to the user invoking DISTRIBUTE--eg, it can be set to a special address set up specifically to handle errors for this particular DISTRIBUTE job. Under 1.8c and earlier, only LISTSERV maintainers were allowed to use the FROM= field. General users were not allowed to specify an origin different from their own network address, so the FROM= option would have had no effect unless the user wanted to specify his full name. Under 1.8d and later, general users (those who are not otherwise defined in the DIST_ALLOWED_USERS site configuration parameter) do not have access to DISTRIBUTE to begin with. You do not have to indicate your name when distributing MAIL-type files -- put your name in the "From:"/"Sender:"/whatever field, as appropriate. ACK= indicates the amount of acknowledgement you want to get. The default is NOne and indicates you do not want to receive messages as the file is distributed. MAIL indicates mail acknowledgements while MSG indicate interactive messages. TO indicates the list of recipients for the file or mailfile. It can either be a list of network addresses ("TO u@n1 u@n2...."), which must not cause the physical command line to exceed 255 characters (use continuation cards in that case), or a ddname of which each line is a "userid@node <full name>" pair. The former is best suited to file distribution while the latter should be preferred for MAIL-type distribution since the recipient's full name will be inserted in the "To:" field of the mail file. The default is "TO DD=TO". Note that distributing a file to a LISTSERV userid will cause it to be interpreted as a command job for execution or a PUT request, as appropriate. Releases 1.5b and earlier did not accept such a destination for DISTRIBUTEd files and ignored the recipient. PRIOR is the network transmission priority you want to assign to the file. If specified, it can be either "*" or an integer number between 0 and 99, inclusive. "*" indicates that the file priority is left up to LISTSERV, which will use an internal algorithm to assign a reasonable priority to the file according to its number of records. INFORM (effective on VM only) specifies the media by which you want users to be informed of the arrival of the file. The default is MSG for interactive messages -- specify MAIL if you want LISTSERV to notify recipients via mail. Note that this option is ignored when MAIL distribution has been selected. Finally, a password is required to validate the command invoker for versions 1.8d and higher of LISTSERV. This is a standard LISTSERV personal password, set with the PW ADD command (q.q.v.). A dataset for the "FROM=DD=" keyword, if specified -- the DD "text" syntax is recommended since the dataset will contain only a single line of data. A dataset for the "TO DD=" keyword, if specified -- the DD * syntax is best suited to this dataset. This dataset contains network addresses (imperatively) and per-address options, one address/options per line. The per-address options are a real-name field and the keywords BSMTP or PROBE (BSMTP and PROBE are mutually-exclusive; only one can be specified per address). For instance a TO dataset could contain lines like janecustomer@abc.com jacktechie@def.edu Jack Techie johndoe@ghi.org BSMTP joe@example.com PROBE sue@example.edu Sue User BSMTP petergunn@example.edu Pete Gunn PROBE When the BSMTP option is specified, LISTSERV will combine the address along with any other addresses for which BSMTP is specified into a multi-recipient BSMTP envelope (which is much more efficient, since not using BSMTP tells LISTSERV to create a separate SMTP envelope for each address). For general DISTRIBUTE jobs (ie, non-mail-merge jobs), it is probably most efficient to specify BSMTP for all addresses in the job. When the PROBE option is specified, LISTSERV will process the address as a PROBE. This is most useful when using a backend list or a changelog file for error processing (see the section on "Advanced LISTSERV applications using DISTRIBUTE", below, for more information on these features; also see the sections on LISTSERV's address probing in the Site Manager's Operations Manual and/or in the List Owner's Manual). PROBE should not be used on one-off DISTRIBUTE jobs which do not require specialized error processing, as it has no particular usefulness otherwise. Again, BSMTP and PROBE are mutually-exclusive; you may specify one or the other but not both. Finally, a dataset for the "DD=" keyword, to contain the mail message or file. This should be the last dataset in the file and the DD *,EOF syntax should be used to ensure that the dataset is not prematurely ended by a possible "/*" card in the data. On VM, for storage space saving and performance considerations, the "Res=Disk" option should be specified on that DD card, ie: DD *,EOF,Res=Disk. Statistics have shown that this decreases the CPU time required to process the request by a factor of six. Mail-type RFDR under VM must IMPERATIVELY use raw-image (PUNCH) format for the mail dataset, since the header will be scanned and modified by the server. File-type RFDRs can use any type of network file format -- the contents of the "data" dataset will be sent "as is" to the recipients, without anything added on top or bottom of it. The only difference will be the RSCS file origin -- the file will come from the LISTSERV userid instead of the sender's userid. The file class, spool fileid and DIST-code are preserved, but the FORM is changed to QU-DIST to trigger the "quiet file transfer" feature installed in some RSCSs in the network. Non-BITNET recipients are routed to their gateway, as determined by the DOMAIN NAMES file. The first server in the chain that finds itself unable to determine the gateway distributes the file directly. Whenever non-mail file is distributed to a non-BITNET user, LISTSERV generates a standard mail envelope with the current date, sender's name and address, recipient's address and transfers it to the mailer. Any possible rejection mail will be sent directly to the sender by the mailing system (and not to the LISTSERV userid). 3.4. Creating and sending a mail RFDR ("DISTRIBUTE") job The instructions below are for sending plain-text messages to a one-time list via LISTSERV. If you wish to send MIME-encoded mail messages (for instance, to send encoded binaries or HTML mail), you must create and add the appropriate MIME headers along with the file (in the case of an encoded binary) to be sent. The syntax described here is for 1.8d running with the default of DIST_SECURITY=1 in the site configuration. We have not attempted to discuss the differences between 1.8d and previous versions of LISTSERV in writing RFDR jobs. A very basic RFDR job is shown below. This job would be sent to LISTSERV@NODE, where NODE is the NODE= site configuration setting for your local LISTSERV server. //WIDGET1 JOB DISTRIBUTE MAIL PW=XXXXXXXX //To DD * janecustomer@abc.com BSMTP email-address1@domain.com BSMTP email-address2@domain2.com BSMTP /* //Data DD *,EOF Date: Tue, 13 Jan 1998 12:25:00 -0400 From: joe.techie@xyz.com Subject: Release of XYZ's WIDGET Model 2 Reply-to: joe.techie@xyz.com To: Widget Customers <customers@widget.com> Your text here Below is a detailed description of each element of the job. //WIDGET1 JOB The JOB "card" is the required first line of any RFDR job. It signifies to LISTSERV that everything that follows in this particular message is part of the same job. In this case we have called the job WIDGET1. Each RFDR job should have a name, but if for some reason you don't want it to, you simply format the JOB card with a space between "//" and "JOB". DISTRIBUTE MAIL PW=XXXXXXXX This is the actual DISTRIBUTE command that tells LISTSERV that this is an RFDR job, and that specifically you want LISTSERV to distribute a piece of mail. PW=XXXXXXXX is part of the security mechanism described in 3.1. You replace XXXXXXXX with your personal LISTSERV password that you've previously set with LISTSERV's PW ADD command. //To DD * janecustomer@abc.com BSMTP email-address1@domain.com BSMTP email-address2@domain2.com BSMTP /* This part of the job tells LISTSERV to whom you want the mail distributed. There are a few options for the addresses themselves as you've probably noticed. The basic syntax for an address in the "To DD" is as follows: userid@host.domain [Personal Name] [BSMTP | PROBE] for instance, any of the following are correct:³ janecustomer@abc.com Jane Customer janecustomer@abc.com BSMTP janecustomer@abc.com PROBE janecustomer@abc.com Jane Customer PROBE The only part of this syntax that is required is the first token (userid@host.domain). You can follow it with the user's personal name, if available. Then you can add the token "BSMTP", which tells LISTSERV to use "Batch SMTP" processing for this address, or the token "PROBE", which tells LISTSERV to send the message to the specified address as a passive PROBE. Please note carefully that the BSMTP and PROBE options cannot be used together for the same address. If you use the BSMTP option, it isn't necessary to specify a "personal name" as LISTSERV won't use it for anything. BSMTP jobs are sent out in the same manner as regular LISTSERV list mail for users who have the FULLBSMTP user option set; that is, the To: line of the mail contains the address you have specified on the To: line in the data (message text) portion of the RFDR job. This is the most efficient way to send mail via DISTRIBUTE as it "bundles" recipients into a few large outgoing jobs rather than creating a separate outgoing job for each recipient. If you need to "personalize" each message then you do not want to use the BSMTP option. However, most large RFDR jobs will probably be sent using BSMTP since it is more efficient. If you do not use either BSMTP or PROBE and use just the e-mail address, e.g., janecustomer@abc.com or janecustomer@abc.com Jane Customer, the recipient's e-mail address will be specified in the To: field, regardless of what you define in the To: field header further down in the job. As noted above, this is an attractive way to do the mailings but uses FAR more resources since you are creating a separate outgoing copy of the message for each subscriber. The marketing benefits of personalized mail are obvious but not using BSMTP is something that you should test to see if the return is worth the cost. Utilizing BSMTP allows your job to be processed as if it were a traditional LISTSERV list and should be used when greatest efficiency is desired. The "/*" at the end of the "To DD" is required and may not be omitted. It tells LISTSERV where the DD ends. It must be on a line by itself. If it does not exist in the job stream, the job will fail. //Data DD *,EOF This card tells LISTSERV where the actual message begins. There is no end card for the "Data DD" as it ends at the end of the message you send the job in. This is important to note as it means that you must disable any signature file that normally is sent with your mail--otherwise it will appear as part of the RFDR job and be sent out! Date: Tue, 13 Jan 1998 12:25:00 -0400 From: joe.techie@xyz.com Subject: Release of XYZ's WIDGET Model 2 Reply-to: joe.techie@xyz.com To: Widget Customers <customers@widget.com> Following the "Data DD" job card are the RFC822 message headers for your message. The only required headers here are Date:, From:, and To:. However, while you do not technically need to provide a Subject: or a Reply-To: field, you will probably want to do so for completeness' sake. There is no specified order for these header fields so you may arrange them as suits you best. If you do not specify a value for the Date: line, but rather simply insert a "Date:" header without a following value, LISTSERV will insert its local time and date as of when it received the job for execution. In the following example: Date: From: joe.techie@xyz.com Subject: Release of XYZ's WIDGET Model 2 Reply-to: joe.techie@xyz.com To: Widget Customers <customers@widget.com> LISTSERV will fill the Date: header in by itself. Otherwise LISTSERV will accept any Date: field you provide without comment. If on the other hand you do not at least specify a "Date:" header line, no date will be inserted at all. If you do not specify a From: line, LISTSERV will insert a From: line containing the address of the command invoker (you!) unless you have specified a FROM= option in the DISTRIBUTE command (see either below or in the preceding technical section for specific DISTRIBUTE options), in which case the value of the DISTRIBUTE FROM= option will be inserted. It is probably wise in most cases to ensure that there is a From: line in the "Data DD" as otherwise the result may not be as you planned. If you do not specify a To: line, LISTSERV will insert a To: line as follows: To: Multiple recipients <LISTSERV@NODE> where "NODE" is the NODE= value for your LISTSERV server. As with the From: line, this is probably not what you want people to see, so you have a couple of options. For jobs which contain only BSMTP recipients, you should specify a To: value, which should point back to a real address in case someone decides to write back to it. For instance, in our example above, To: Widget Customers <customers@widget.com> and we'll assume that "customers@widget.com" is a customer-service address where people can write, or even an existing LISTSERV list out of which you've pulled a subset of addresses for this particular RFDR job. If you have all non-BSMTP and non-PROBE recipients, you can simply leave the To: field blank (as with Date:, above) and LISTSERV will fill the field in with the data you've provided in the "To DD". For instance, in the following example, Date: From: joe.techie@xyz.com Subject: Release of XYZ's WIDGET Model 2 Reply-to: joe.techie@xyz.com To: both the Date: and To: fields will be filled in by LISTSERV for you. Please note that the blank line at the end of the RFC822 headers is not a misprint. According to RFC822 you MUST provide a blank line between the last header and the beginning of the message body, and LISTSERV expects it to be there. If it is not there, the job will fail with an RFC822 parser error: > DISTRIBUTE MAIL Invalid RFC822 field found in mail header: "Your text here--forgot blank line". Mail not delivered. Finally, we get to the final element: the message body. Your text here This element is fairly self-explanatory; you simply add the actual body of the message you are sending out. Again, make sure that there is a blank line between the body of the message and the last RFC822 header as mentioned previously. 3.4.1. RFDR job options There are two options that might be of use in the JOB card (the first line of the RFDR job as noted above). These options are: 1. ECHO=NO This option suppresses the resource usage summary you would otherwise get upon completion of your delivery. Omitting it returns a mail message containing a summary like the following example to the e-mail address from which the RFDR job was sent. Job "WIDGET1" started on 10 Feb 1998 13:33:34 > DISTRIBUTE MAIL Job "WIDGET1" ended on 10 Feb 1998 13:33:36 Summary of resource utilization ------------------------------- CPU time: 0.000 sec Overhead CPU: 0.030 sec CPU model: 100MHz Pentium (64M) Job origin: joe.techie@XYZ.COM To use this option and suppress the summary from being sent, simply type ECHO=NO at the end of the JOB card, e.g., //WIDGET1 JOB ECHO=NO 2. PRIME=prime-time-specification This option controls when the job runs, if you do not want it to run immediately. This function of this option is identical to the function of the "Prime=" keyword setting for LISTSERV mailing lists, where "Prime=" defines (on a list by list basis) times during which LISTSERV should not process mail for the list. By default this option is set to "PRIME=Yes", meaning that the job can be processed at any time. If you set this option to "PRIME=No", it uses the value set globally by LISTSERV's PRIMETIME= site configuration variable, which by default is set to have no prime time (in other words, by default, mail can still be processed at any time even with "PRIME=No"). The PRIME= option for RFDR jobs can be set to an explicit time definition if necessary. For instance, you might have a very large RFDR job (e.g., a newsletter) that should not be processed until after midnight (when network traffic is low and more machine resources are generally available). RFDR jobs sent to LISTSERV during prime time are automatically held until non-prime time and then distributed normally, without requiring further intervention by anyone. For instance you could set the "PRIME=" option to PRIME="MON-FRI: 00:00-23:59" This means that you could send the job anytime during the regular work week--Monday morning through Friday afternoon--and know that it won't be distributed until Friday at midnight or shortly thereafter. As another example, if you want the job only to be processed between midnight and 6 AM on weekends, you might set it to PRIME="MON-FRI: 00:00-23:59; SAT-SUN: 06:00-23:59" Note that the specifications for PRIME= must be enclosed in quotes and the format (spaces) should be identical to the examples. VERY IMPORTANT: You should always leave yourself at the very least an hour processing window, and even more so for large jobs, since PRIME checks are done once each hour, on the hour, and very large jobs may require extended processing time. Also note that if running in Networked mode, even more time is required since jobs passed across the LISTSERV network for other servers to deliver also contain the PRIME setting, and depending on which time zone it is passed to, it may miss the "window" altogether. For example, PRIME="MON-FRI: 00:00-23:59; SAT-SUN: 06:00-23:59" leaves six hours of processing for your DISTRIBUTE job on Saturday and Sunday, i.e., 12 hours total processing time over the two days. So if the job does not get to, say, a server in Japan before the PRIME deadline on Saturday, it will be processed the next day during non-prime time. In actuality it is probably not necessary to use such small windows on the weekend, so PRIME="MON-FRI: 00:00-23:59" is probably all you would need for a job to be delivered over the weekend. To use the PRIME option, simply type it following the JOB operand. If you need to specify ECHO=, note that it must follow PRIME= or a syntax error will result. For example: //WIDGET1 JOB PRIME="MON-SUN: 09:00-23:59" ECHO=NO or //WIDGET1 JOB PRIME="MON-SUN: 09:00-23:59" Please make sure that you test the PRIME= option setting with test jobs first to make sure that the proper syntax is being used before sending mission critical jobs to be processed during off-prime hours. 3.4.2. DISTRIBUTE Command Options There are four options that might be of use in the DISTRIBUTE command (the second line of the RFDR job as noted above). These options are: 1. ACK=MAIL This option tells LISTSERV to confirm deliveries with an acknowledgement sent by mail (the default is not to send delivery confirmations, but rather to simply acknowledge receipt and processing of the job), as in the following example (note that ECHO=YES in the JOB card for this particular example): Job "WIDGET1" started on 10 Feb 1998 13:33:34 > DISTRIBUTE MAIL Mail delivered to janecustomer@abc.com Job "WIDGET1" ended on 10 Feb 1998 13:33:36 Summary of resource utilization ------------------------------- CPU time: 0.000 sec Overhead CPU: 0.030 sec CPU model: 100MHz Pentium (64M) Job origin: joe.techie@XYZ.COM Use of this option in the DISTRIBUTE command line is as follows: DISTRIBUTE MAIL ACK=MAIL PW=XXXXXXXX Other than the default ACK=NONE, the only other setting for this option is "MSG", which is obsolete except for NJE servers. 2. DEBUG=YES This option produces a report showing how the various recipients will be routed over the LISTSERV backbone, but WITHOUT actually delivering the message. Before sending out your actual posting, you may want to send the RFDR job with this option enabled to see how the mail will be routed. Examples of the use of this option in the DISTRIBUTE command line: DISTRIBUTE MAIL ACK=MAIL DEBUG=YES PW=XXXXXXXX DISTRIBUTE MAIL DEBUG=YES PW=XXXXXXXX 3. TRACE=YES This option produces a report showing how the various recipients will be routed over the LISTSERV backbone, and actually delivers the message (similar to DEBUG=YES but the message is actually delivered). Examples of the use of this option in the DISTRIBUTE command line: DISTRIBUTE MAIL ACK=MAIL TRACE=YES PW=XXXXXXXX DISTRIBUTE MAIL TRACE=YES PW=XXXXXXXX 4. FROM=netaddress This option can point to a particular address that will receive the bounced mail, notifications of delivery problems, etc. Specifically this option sets the RFC821 MAIL FROM: address to which all compliant mailers will bounce non-deliverable mail. The single parameter netaddress is an Internet address in the standard format (userid@host.domain). Examples of the use of this option in the DISTRIBUTE command line: DISTRIBUTE MAIL ACK=MAIL TRACE=YES FROM=john@example.com PW=XXXXXXXX DISTRIBUTE MAIL FROM=john@example.com PW=XXXXXXXX DISTRIBUTE MAIL ACK=MAIL FROM=john@example.com PW=XXXXXXXX DISTRIBUTE MAIL TRACE=YES FROM=john@example.com PW=XXXXXXXX 3.5. Advanced LISTSERV applications using DISTRIBUTE Depending on your specific needs, it is possible for sites to "get creative" and use traditional LISTSERV lists as "back-ends" in conjunction with DISTRIBUTE jobs to handle bounces in the same way they are handled for a regular LISTSERV list, without ever using the list itself for posts. Let's say that, to begin with, you collect addresses from a guestbook web page that you've set up for your company. You want to send out periodical updates to these people using RFDR "DISTRIBUTE" jobs but you know that you are going to have the usual 15-20% or more bad addresses from typos, pranksters, and plain old attrition. How do you find the bouncing addresses so you can purge them from your database without having to read each bounced message and determine what address actually bounced? Simply create a traditional list with the appropriate error handling setting desired (see other sections of this manual and of the List Owner's Manual for LISTSERV for more information). Set "Change-Log= Yes" and "Auto-Delete= Yes,Full-Auto,Delay(0)" in the list's header ("Change-Log" requires LISTSERV 1.8d and later). Use an ADD IMPORT job as detailed in chapter 4.3 of the List Owner's Manual to add all of the addresses from your database to the list. Then in your RFDR job, set the following in the FROM= option of the DISTRIBUTE command line: DISTRIBUTE MAIL FROM=owner-listname@yourdomain.com PW=XXXXXXXX where listname is the name of the list created. For instance, if the list was named bouncelist1 and the LISTSERV host name was LISTSERV.EXAMPLE.COM, then the DISTRIBUTE command would be as follows: DISTRIBUTE MAIL FROM=owner-bouncelist1@listserv.example.com PW=XXXXXXXX Once you have the bounce list set up (and we'll continue to use our bouncelist1 example), you simply check the BOUNCELIST1.CHANGELOG file for a listing of addresses LISTSERV has auto-deleted from bouncelist1 since you sent out your RFDR job. You can then use this changelog file to update your database and clean out the bad addresses. Do keep in mind, however, that in order for LISTSERV to take action on permanent bounces that come to the owner-bouncelist1 address and are, furthermore, in a format understandable by LISTSERV, the bouncing address(es) must also be listed as subscribers in the back-end bouncelist1 list (so do not forget to bulk add the addresses before sending the DISTRIBUTE job, as explained above). To take this a step further, you could also use this same back-end bounce processing list as a front-end list to handle automatic subscribe and unsubscribe requests for each mailing, allowing for total flexibility for both yourself and the customers being mailed to. 3.5.1. "List-free" bounce processing for one-shot lists (1.8d) Starting with LISTSERV 1.8d you can bounce-process a one-shot list without needing a back-end list for the bounces to come back to (as outlined above). To use the new "list-free" feature, you create and send a standard DISTRIBUTE job with the FROM= address set to OWNER-NOLIST-xxx@hostname (where 'xxx' can be anything and 'hostname' is LISTSERV's host name), for instance OWNER-NOLIST-MYDIST@LISTSERV.MYHOST.COM. While 'xxx' can be anything, OWNER-NOLIST@whatever doesn't work--you must provide '-xxx'. Within the job you then use PROBE for each address in the manner documented above. After sending the job, you get a NOLIST-xxx.CHANGELOG in LISTSERV's A directory with entries for every bounce. (In our example above, this file would be called NOLIST-MYLIST.CHANGELOG.)The entries read BOUNCE followed by the e-mail address. As there is no monitoring and no decision to delete, the entry does not read AUTODEL, and thus bounce reporting for "list-free" DISTRIBUTE jobs is passive. By definition, however, you can't monitor one-shot jobs; monitoring (if desired) must be provided by collating all the one-shot jobs into a big monitoring database, or something similar (all jobs from client such and such, etc). Note carefully that if you are running under unix with sendmail, you will have to patch sendmail in order for it to properly accept and route bouncing probe messages. As a convenience, L-Soft provides third-party sendmail patches for this purpose on its FTP site (in ftp://ftp.lsoft.com/listserv/unix/CONTRIB) but please be aware that L-Soft did not write and does not support these patches in any way; they are strictly for use at your own risk and you must contact the author(s) of the patches for help with them. 4. DBMS and mail-merge support 4.1. Overview Two of the most significant enhancements in version 1.8d of LISTSERV are DBMS and mail-merge support. In this chapter, you will find a brief description of these new features, along with installation instructions and a few samples. While the DBMS and mail-merge functions were designed to work together, they can also be used independently from each other. That is, you may find the DBMS support useful even if you have no need for mail-merge functionality, and likewise you can use the mail-merge functions without a DBMS back-end. Both functions require LISTSERV Classic or LISTSERV HPO, and are unavailable in LISTSERV Lite. Additionally, mail-merge requires that LSMTP Classic version 1.1b or later be installed as your outgoing mail transfer agent. 4.1.1. DBMS support LISTSERV's DBMS support allows you to: Direct LISTSERV to store subscriber information in a DBMS, on a list by list basis. That is, you may have a mix of traditional LISTSERV lists and DBMS lists. Furthermore, you can adjust the layout of your DBMS lists to match existing or current applications. You can map each list to a private table if this is what makes sense for you, or you can put all the lists in the same table, place related lists in one table, etc. You can add as many columns as you want to store additional information about subscribers. Use the DBMS as a back-end for mail-merge jobs. LISTSERV can execute arbitrary SQL SELECT statements to extract recipients from your DBMS, and make related information (name, country, account number, etc.) available for mail-merge operations. DBMS support is available through Microsoft's ODBC interface on Windows NT, and Oracle's OCI 8 interface on Windows NT, OpenVMS Alpha, Digital Unix, AIX and Solaris (SPARC only). 4.1.2. Why require Oracle 8? Oracle has introduced major changes to the OCI API in version 8. While many of the concepts remain similar, all the function calls have been renamed and most have a different calling sequence. Supporting both OCI 7 and OCI 8 would require more than a handful of #ifdef statements -- we would need to develop and support a separate interface for OCI 7. In addition, we would need to license Oracle 7 on all supported systems, which would significantly increase our development costs. While we recognise that there is a large Oracle 7 installed base, a year or so from now most of them will have upgraded to Oracle 8, thus the cost for developing an OCI 7 interface and purchasing six Oracle 7 licenses would have to be amortised over a period of about a year, leading to much higher licensing costs for the OCI 7 version. Furthermore, it is possible to use LISTSERV's OCI interface with an Oracle 7 server: the only component which must be at version 8 is the client, i.e. SQL*Net. As Oracle-based LISTSERV installations typically run on a separate, dedicated system, this simply means that you need to purchase a version 8 client license for the system in question. It is not necessary to upgrade the server to Oracle 8. Oracle 7 is also supported through the ODBC interface. 4.1.3. Mail-merge Documented Restriction: Note that LISTSERV's mail merge functionality REQUIRES the use of LSMTP Classic as the outgoing MTA. Mail merge does not work with sendmail, qmail, Post.Office, Netscape Mail Server, Microsoft Exchange, PMDF, MX, or any other MTA except L-Soft's LSMTP Classic mailer. Under unixes not supported by LSMTP Classic this may require that you set SMTP_FORWARD= accordingly in go.user, to point to a separate machine running LSMTP (for instance, a dedicated Windows NT LSMTP machine). Under OpenVMS or Windows NT can run LSMTP Classic either on the same machine (the preferred method), or on a separate machine if desired. The main point is that the outgoing mail-merge postings MUST be handled by LSMTP Classic. (LSMTP Lite does not support mail-merge.) LISTSERV's mail-merge support allows you to send individually customised messages to large numbers of recipients with very high throughput. The mail-merge functions support: Simple substitutions, such as "Dear &firstname;". Conditional blocks, such as a birthday greeting sent when the message happens to coincide with the recipient's birthday, or a warning when the balance of the account is negative. Special facilities to send promotional banners to a randomly generated subset of the recipients. For instance, you can indicate that a first banner should be sent to a random subset of 200 recipients, while another banner is sent to a randomly selected (but distinct) series of 500 recipients, and others receive a third banner, or no banner at all. Easy support for "few of many" topic subscription, such as a service offering news about movie actors (many registered actors, while most people will only want news about a handful of them). Full integration with the DBMS interface, allowing recipients to be selected through arbitrary SELECT statements, while every column that can be converted to a character string is made available as a mail-merge field. A simple bounce processing and collection system -- LISTSERV processes and decodes all bounces, and writes the failing addresses to a plain-text file. You can group related mailings in the same bounce file or use a separate file for each mailing, whichever makes the most sense in your context. As each message is sent in "probe" format, even non-standard bounces will be processed accurately, as long as the remote MTA sends bounces to the correct (RFC821 MAIL FROM:) address. Mail-merge support requires version 1.1b of LSMTP Classic (LSMTP Lite does not support mail-merge operations). LSMTP is currently available for Windows NT and OpenVMS Alpha, and will shortly be available for Digital Unix, AIX and Solaris (SPARC only) - the same systems for which the DBMS interface is available. Note that LSMTP and LISTSERV need not reside on the same hardware; by way of example, it would be possible to run the DBMS-enabled version of LISTSERV on any supported unix platform, with LSMTP Classic 1.1b running on Windows NT. 4.1.4. Why require LSMTP? The largest obstacle to the successful deployment of a full-scale mail-merge operation is performance. The feedback we received from the customers who participated in the design and early testing of this system was unanimous -- a mail-merge system would have no value to them unless it could be scaled to at least hundreds of thousands of recipients, if not millions. Unsurprisingly, while customers were willing to pay a moderate surcharge in hardware costs for the mail-merge functionality, they were unable to budget order of magnitude increases. Their requirement, and our design goal, was the ability to run a "normal" 6 figure mail-merge workload on a single high-end PC system. Performance requirements varied from as little as 10,000 hourly deliveries to a demanding 250,000/hour. Out of the box, a normal sendmail configuration running on a high-end PC system will not even reach the 10,000/hour mark. A mail-merge job leads to a large number of spool files (one recipient = one message), and a large spool is sendmail's bane. If you were to construct an artificial workload to demonstrate sendmail's technical flaws, it would be very close to a typical mail-merge workload with the usual 20% of bad addresses. To get an idea of the maximum mail-merge throughput from a system not specifically designed for this purpose, we conducted "brute force" (one recipient = one message) performance tests using LSMTP 1.1a on a 1-CPU, 200MHz Pentium Pro system. Using live traffic from our outsourcing customers and jobs in the 200-300,000 recipient range, we were able to sustain 150,000 hourly deliveries. While well within the target range of 10-250,000/hour, the resulting I/O rate was close to the reasonable practical limit of (then) current disk technology. With a (then) hypothetical 400MHz Pentium Pro CPU, we expected to achieve 175-200,000/hour rather than 300,000/hour. Clearly, the "brute force" approach could only be a short-term solution, as CPU performance improves much faster than I/O throughput. With the "one message = many recipients" approach we adopted in LSMTP 1.1b, we were able to achieve same-LAN throughput rates on the order of 1,000,000/hour on a 200MHz Pentium Pro. While LAN figures are meaningless as an estimate of actual performance with live addresses, in the present case they clearly indicate that I/O is no longer a bottleneck. We chose not to give customers the option of using sendmail, because the programming for a "brute force" approach is vastly different from LSMTP 1.1b's. Development and support costs for the mail-merge functions would have about doubled, whereas the only value of the "brute force" implementation would be in demonstrating that LSMTP is required for good performance. 4.2. Pre-installation tasks Before installing DBMS and mail-merge support, please review the following steps and make sure that your selected target system is ready to receive this update. DBMS support requires version 1.8d of LISTSERV Classic. Mail-merge support requires version 1.8d of LISTSERV Classic and version 1.1b of LSMTP Classic. If you are planning to use the DBMS interface, you must install vendor-supplied DBMS support files on the target machine before installing the LISTSERV update. For ODBC (Windows NT), the appropriate ODBC support files are shipped with the LISTSERV installation kit and are installed when you install LISTSERV if they do not already exist on your machine.⁴ For OCI, you need to install and configure the Oracle8 client files (SQL*Net et al.) This material is typically licensed and not freely redistributable, and thus does not come with the LISTSERV kit. If you are using Windows NT, you must install SP3 (4.0) or otherwise procure and install ODBC support (3.51) prior to installing the update, even if you do not plan to use the DBMS support features. ODBC is now a standard component of Windows NT, and applications that contain ODBC system calls will not launch if the ODBC DLLs are not present on the system. Again, NT will not launch the application at all, even if it does not make any ODBC call. As noted above, L-Soft ships these files with LISTSERV for your convenience, but the versions L-Soft ships may not be the latest versions. The simplest way to obtain the latest versions of the ODBC support files is to upgrade to NT 4.0 SP3. If you are planning to use the mail-merge functions, you must upgrade LSMTP to version 1.1b. While you can upgrade LISTSERV first, it is best to start with LSMTP. Upgrading LSMTP requires a 1.1b license key from your sales representative. If using the DBMS interface, you may want to create a DBMS username for LISTSERV in advance, and grant it the CREATE SESSION (mandatory) and CREATE TABLE (optional) privileges. If you are planning to create all tables yourself, you should not grant CREATE TABLE to LISTSERV's DBMS username. Note that a compiler is required to use the OCI interface on unix systems. L-Soft may not legally ship pre-linked executables containing the SQL*Net library. 4.2.1. Selecting a suitable DBMS product This section applies only to ODBC users, as OCI users will always be using Oracle, which does not exhibit any of the problems mentioned in this section. While L-Soft does not ordinarily recommend or endorse specific hardware or software brands, some database products (especially low-end PC offerings) may not be suitable for use together with LISTSERV. Without advocating one brand over another, L-Soft recommends the use of a DBMS that does not exhibit the problems mentioned below. All error messages are in reference to the diagnostics printed by the LISTSERV ODBC interface during startup. [FATAL] LIKE operator has no ESCAPE clause, errors will occur! This error indicates that the DBMS does not support any kind of "escape clause" for the LIKE operator. In practice, it means that whenever LISTSERV attempts a search containing a percent sign or underscore, the results will be incorrect (you may also get an ODBC error). This makes the DBMS unusable as a data store for LISTSERV lists. However, if you only plan to use the DBMS for mail-merge jobs, this restriction may be immaterial as LISTSERV will only be executing the SQL statements that you provide in the mail-merge job. SQL Server users should note that ESCAPE support was added in version 6.5. L-Soft will not support SQL Server 6.0 or older as a data store for LISTSERV lists. The Microsoft Access DBMS product also appears to have this restriction. [SEVERE] Max active stmt: 1 - expect uncommitted read & unrequested commits With a maximum of one active statement per transaction, the ODBC interface is unable to carry out typical SELECT ... UPDATE ... UPDATE ... COMMIT sequences using a single transaction, because the SELECT remains active until the COMMIT and prevents the execution of the UPDATE statements. To bypass this problem, the ODBC interface will use two transactions for these sequences. However, the two transactions will typically look like independent applications to the DBMS, and will suffer from "transaction isolation," a vital DBMS feature that permits shared database access by multiple unrelated applications. As LISTSERV expects that an update will be reflected in a subsequent search, whether it has been committed or not, the ODBC interface will be forced to commit updates before beginning a new SELECT, even when LISTSERV had not requested a commit. In addition, if the DBMS does not support row-level locking, the ODBC interface will hang when attempting to execute the UPDATE statement, because the table is locked by the transaction containing the SELECT! To avoid this, the ODBC interface may switch to the lowest level of transaction isolation, "uncommitted read." If LISTSERV is the only application writing to the tables containing the data, this will be of no consequence, otherwise LISTSERV may see uncommitted changes made by other applications. SQL Server artificially exhibits this condition. In reality, SQL Server does support multiple statements per transaction, but its ODBC driver reports otherwise, and will only allow one statement per transaction unless using dynamic cursors (or a related option). Dynamic cursors, however, will only work (with SQL Server) if the table contains a unique index; otherwise, the cursor is downgraded and leads to the one-statement behaviour. It is possible that third-party ODBC drivers may allow and report unlimited statements per transaction. [SEVERE] FOR UPDATE clause not supported, no locking will occur Some entry-level DBMS products may not support locking at all, or may only support it through a proprietary interface (rather than via SQL commands). In that case, LISTSERV will be unable to lock the rows it is in the process of updating, which may lead to incorrect behavior if you have other applications updating the tables. [SEVERE] '=' operator is case-insensitive, results may be incorrect! With some DBMS products, the equality operator is case-insensitive. While LISTSERV is generally case-insensitive, it does of course have the ability to make case-sensitive searches, and will do so on occasion. LISTSERV's built-in data management functions typically use the e-mail address as a unique, case-sensitive, primary index into the list, and LISTSERV assumes that it can use the e-mail address as a kind of "row ID" if the need arises. LISTSERV is not programmed to "doubt" the equality of a successful search for an e-mail address it had previously retrieved, as this is algorithmically impossible with its built-in functions. While this is a severe error in that it can lead to incorrect results, in practice it only has limited impact. SQL Server usually has case-insensitive equality configured by default. While it is possible to change this setting, this can only be done by reinstalling the product from scratch, which is usually unacceptable. In addition, the setting is global and affects all tables, all users, etc. In practice, the impact does not warrant the re-installation of an existing system. Cursor behaviour limits ability to commit before logical close This warning indicates that the DBMS will "forget" the results of any active SELECT statement whenever a transaction is committed. As LISTSERV's built-in data management functions do not have this restriction, LISTSERV will often commit changes while a search is in progress. For instance, if you issue the command SET XYZ-L DIGEST FOR *@AOL.COM, LISTSERV will search for users matching the pattern *@AOL.COM and, for each match, update the subscription options, send an e-mail message, and commit. This way, if the command is interrupted and re-executed, users who already received an e-mail notice will not receive a second copy. When this warning is printed, the ODBC interface will ignore any commit request from LISTSERV that would abort an active search. The transaction will always be committed when the LISTSERV command completes, so the final results will always be accurate. However, in extreme cases (such as a SET command updating every record in the database), the transaction might generate a very large amount of uncommitted data, and require a lot of rollback space. SQL Server users should note that while the SQL Server ODBC driver will cause this message to be displayed unless a non-standard ODBC call has been issued in advance, in practice it has no operational impact as the ODB

Table of Contents

1. LISTSERV's Archive Search Functions

2. LISTSERV "Commands-Job" Feature and CJLI Interpreter

3. Relayed File Distribution and the DISTRIBUTE Command

4. DBMS and mail-merge support

5. List Exits

6. The LISTSERV TCPGUI interface

Revision History