Software > LISTSERV Email List Management Software > Performance Comparison

LISTSERV Performance Comparison

Executive Summary
Majordomo vs LISTSERV Lite
LISTSERV Lite vs LISTSERV Classic
LISTSERV Classic vs LISTSERV HPO

Executive Summary

This document presents a series of tests comparing the respective performance of Majordomo, LISTSERV Lite, LISTSERV Classic and LISTSERV HPO (High Performance Option). Where these particular tests are concerned (see benchmark description below), the various products performed as follows:

LISTSERV Lite was always at least an order of magnitude faster than Majordomo – even for small (100 subscriber) lists.
LISTSERV Lite exhibited roughly the same scalability response as Majordomo. This means that, while LISTSERV Lite can handle lists an order of magnitude larger than Majordomo for a given level of performance, response time eventually degrades with the size of the list in about the same manner as with Majordomo.
LISTSERV Classic's enhanced algorithms dramatically improved response time (one to two orders of magnitude better than LISTSERV Lite) up to the "turning point" in its scalability curve, after which it retained its lead over LISTSERV Lite but experienced degraded performance.
LISTSERV HPO's state of the art algorithms delivered optimal performance with a near flat scalability curve all throughout the test.

Disclaimer

There is no standardized benchmark suite for mailing list managers (such as the TPC or SPEC benchmarks), nor is there any informal benchmark against which to measure this type of product. Due to their nature, mailing list managers are very difficult to compare to each other. To list but two of the most widely known obstacles:

Mailing list managers that offer a web interface or database functions (with online browsing and searching capabilities) usually require more system resources for a given set of lists than "bare" list managers. This is because the "bare" list managers are less functional and more difficult to use for novice users, and thus receive a smaller number of simpler end-user requests, ultimately trading cheap computer power for expensive manpower. While this is a business consideration rather than a technical issue, it makes it difficult to benchmark list managers in different weight classes against each other. These cross-class benchmarks, however, are the ones for which there is the biggest demand.
Even on a system dedicated to mailing lists, the response time that end-users see depends on overall system performance, and not just on the performance of individual components. In particular, some packages commonly used to implement mailing lists (Majordomo, sendmail, etc.) create large numbers of processes, which can dramatically impact overall system performance. This in turn varies with the particular operating system being used and with the system administrator's skills. The person running the benchmarks can easily skew the results of any live workload simulation by selecting a particular operating system over another, or by applying varying degrees of skill to operating system tuning, without touching the product being benchmarked. In some extreme cases, it is the operating system rather than the list management software which is being benchmarked.

Benchmark Description

In this document, we have used a simple benchmark, which has no other ambition than to give a rough idea of the capabilities of the various products being compared. Randomly generated mailing lists with increasing numbers of subscribers are created on the various mailing list managers, after which a single mail message is sent with a request to add 100 new users to the list. The products are configured or modified to suppress the delivery of any email confirmation to the new subscribers, as this would test the mail server rather than the mailing list manager. The benchmark measures the elapsed time for the completion of the 100 additions, on a dedicated system with no other activity. The test is run using a single process that fits entirely in physical storage; given the nature of the test, no operating system tuning is required to obtain good (nominal) performance, and none is allowed.

Test System

The test system we chose is a Sun SS5 running Solaris 2.5.1. The inclusion of Majordomo in the test required the use of a Unix system, and we chose Solaris because it is (as of this writing) the most popular Unix system among LISTSERV users. We decided to run the benchmark on a workstation system as this is what all but a handful of sites use for their mailing list workloads. The SS5 was chosen because we had convenient access to this machine and because it also happens to be the most popular system among Solaris LISTSERV users. In principle, a workstation with a faster CPU would result in lower absolute numbers with the same kind of pattern. A quick, partial test on a SS20 confirmed this.

The test was run using Majordomo 1.94.3 and LISTSERV version 1.8c (23 May 97 build). Note that the test works to Majordomo's advantage as LISTSERV maintains per-subscriber attributes and has more processing to do (default option rules, etc.) and much larger list files than Majordomo (due to the per-subscriber attributes). For this particular test, the LISTSERV list files were 5.5 times larger than Majordomo's. Finally, LISTSERV generally had more work to do in order to apply its more extensive security and access control functions, hostname aliasing for list lookup operations, and so forth – all features which, while useful, increase resource costs.

Majordomo vs LISTSERV Lite

We first ran the benchmark with list sizes of 1,000, 10,000 and 100,000 subscribers, for a coarse scalability comparison. The execution times (for 100 ADD requests) were as follows:

The ratio between Majordomo and LISTSERV Lite varied from 12:1 to 18:1. This is consistent with empirical observations and confirms the order of magnitude difference in performance between the two products. As expected, LISTSERV Classic and HPO outperformed Lite by a large factor, although it is difficult to see on this chart.

Since most Lite customers run smaller lists, we then took a closer look at the low end of the scale:

Even here, LISTSERV Lite outperforms Majordomo 15:1. The hashed red bar corresponds to a 100 subscriber list running under Majordomo, and even at this level Majordomo took 5.7 times longer than LISTSERV Lite did with 10 times as many subscribers. The threshold for sub-second response time is somewhat below 100 subscribers for Majordomo, and well above 1,000 subscribers for LISTSERV Lite. Classic and HPO are not visible on this chart.

LISTSERV Lite vs LISTSERV Classic

Having determined that Majordomo was not in the same performance range as the LISTSERV products, we removed it from the chart in order to take a closer look at the other results:

Although not quite as dramatic as the Majordomo vs Lite chart, we clearly have a repeat of the same situation, with the Lite execution time increasing in a straight line. This is called "linear growth" and means that it will take LISTSERV Lite (or Majordomo) 10 times longer to run the benchmark on a list with 10 times as many subscribers. While you may have heard the term "linear growth" used to refer to an ideal growth curve, in this context it is more like a worst-case scenario. The problem is that, if you have 10 times as many subscribers, you can expect your list manager to receive 10 times as many commands. If each of these commands then takes 10 times longer to execute, you will need 100 times more processing power. This is called "n² growth" or, in plain English, "bad news".

This being said, "bad news" is relative. After all, 100 times 0.0001 second is 0.01 second, and 100 times that is 1 second. If the list manager is fast to begin with, you can afford to take this factor of 100 hit once, or maybe even twice. In the present case, if your goal is a response time of roughly one second for a SUBSCRIBE request, LISTSERV Lite will deliver that with up to 10,000 subscribers. And if your list grows slowly, you may be able to upgrade your hardware from time to time to keep up with the growth rate. On the other hand, if your 10,000 subscriber newsletter becomes the most popular hit on the Internet, you will have a problem.

As a rule, L-Soft does not recommend the use of the Lite version with lists of significantly more than 1,000 subscribers. But this is a quality of service issue, and it is ultimately up to you to decide where to draw the line. For instance, if the subscribers are paying customers, you will probably want to offer a higher level of service than if you were running the list as a favor to a friend.

LISTSERV Classic vs LISTSERV HPO

Looking at the previous chart, one had the impression that the Classic version was again going to look like Majordomo once magnified, and that Classic was just going to be 10 times faster, being able to service a 100,000 subscriber list with a response time of about one second vs 10,000 for Lite. Well, let's zoom in and see what the two remaining curves look like:

From a distance, this does indeed look like a somewhat more bumpy repeat of the Majordomo curve, but notice the turning points at the 10,000 and 30,000 subscriber marks (there is actually a third and even sharper turning point at the 3,000 mark, but we would need to zoom in again to see it). These turning points correspond to optimizations that successively "give up" and stop "flattening" the curve.

Up until 3,000 subscribers, the curve is nearly flat, and the absolute numbers so low that performance is already optimal (it hardly matters if a command takes 0.001 or 0.01 second to execute). This corresponded roughly to the size of the largest active lists when the development of the HPO version began in 1994 (and this remains the case to date). Having provided optimal performance for all likely active lists, the first optimization, which would be too memory intensive for large lists, becomes more resource conscious and allows performance to degrade somewhat in the interest of conserving memory. Even in this constrained mode, the optimization remains effective, to a lesser extent, for another factor of 3 or so (i.e. roughly until the 10,000 mark). A number of other optimizations either give up or reach their design limits at various points between, very roughly, 5,000 and 30,000 subscribers, depending on operating system, hardware, etc. The combination of these effects leads to the turning point near the 10,000 mark. We believe the 30,000 turning point to be due to operating system considerations (disk caching, etc.) It was not present in another series of tests made on a different operating system.

All throughout the test, the HPO curve was almost completely flat (0.089 sec at 1,000 vs 0.122 at 100,000 – or about 0.001 sec per added subscriber). The HPO version uses advanced algorithms that scale effortlessly to much larger lists and is ideally suited to very large lists.

LISTSERV Overview Feature Checklist
Operating Systems Hardware Requirements
License Sizes and Pricing Technical Support
Remote Installation Renewal Details
Upgrade Center Test Licenses
LISTSERV Demo Request Information