It is critically important to test an email job thoroughly before sending it, especially if it is a large job being distributed to many recipients or a message with large amounts of data. Large jobs take up a lot of server time and space and affect many email hosts and individual recipients. Sending out a less than perfect message can create a very negative reaction among recipients and damage an institution’s credibility and reputation. As a safeguard, LISTSERV
Maestro will not allow an email job to be authorized before a test of the job has been sent out and approved. Any changes to contents, recipients, or sender require a new test to make sure that the changes were effective.
To test an email job, click on the Delivery Test icon on the Workflow diagram or the
Test link in the
Test Status section of the Summary diagram.
LISTSERV Maestro allows the user to use a previously entered group of test recipients, upload and download test recipient lists, or type in test recipient names and addresses. The test interface displays a table for entering test data. Each row represents a test recipient. The columns displayed depend on the columns defined in the recipient definition stage of the job.
Valid Internet email addresses are necessary for all the fields in the column that is specified as the email column. The test messages will be sent to these addresses. The same address may be used in more than one row.
The subject line for a test message can be changed. Changing the test subject line will not affect the subject line of the final message. The optional test line allows the user to enter a line of text that will appear above the body of the message when it is delivered. Use this to recognize test messages and different versions of test messages. If nothing is entered into the optional test line box, then no text is added to the test messages. A test line will
never be added to a final email job.
Notes: If the message is an HTML message with a text alternative, then a new option on the Delivery Test screen lets a separate email be sent to each test address with alternative text. If this new option,
Send additional plain text mail with alternative text, is checked, then the test mail will be sent out in two versions. The first version will be the normal HTML mail (including the text alternative) and the second version will be a plain text mail that contains only the alternative text. Each test delivery recipient will receive these two mails, allowing the test recipients to view both the normal HTML mail as well as what the alternative text would look like for a recipient where the email client strips away the HTML part and only displays the text alternative.
When sending a test for a job with Forward-to-a-Friend enabled, several copies of the test will be sent to each address: the normal message, the message as it will look when forwarded without a user-supplied preamble, and the message as it will look when forwarded with a user-supplied preamble. If the
Send additional plain text mail with alternative text option is also checked, then each of these test messages will be sent in both forms. This means that each test address may receive up to 6 different versions of the message.
Click [Send Delivery Test Messages] to continue. Click
[Save & Close] to save the test recipient data but not send the test message. Once the messages have been sent, review the results. It is usually a good idea to include the job owner and any team members in the test recipient list so test results can be viewed firsthand.
If the test results were not acceptable, select the
No option and then click
[OK]. This will prevent the job from being approved for delivery and allow any necessary changes to be made by bringing up the Job Details screen.
If the test results were acceptable, select the Yes option and then click
[OK]. This will allow the job to be authorized for delivery.
If you want to view and verify the delivery test results later, select the Verify Later option and click
[OK]. If this option is selected, the job can not yet be approved for delivery. The next time the
Delivery Test icon or link is clicked from the Job Details screen, the user will be returned to the Delivery Test screen. To approve the pending test, click the
Go directly to the validation page, without another delivery test link
under the
Advanced Options section to return to the Validation screen of the test that was saved for later verification. Click the
Yes option and then click
[OK] to proceed.
Tips: It is useful to have a good understanding of the actual recipient data when putting together a test. If the message contains merged fields, then study those fields in the actual data: what are the shortest and longest instances of those fields? Are those fields ever empty? Next, make sure to send tests that have values of the shortest and longest lengths, and empty fields if appropriate, to make sure that the message layout looks correct in all cases. It is sometimes necessary to change the layout or use conditional blocks to accommodate special cases. If using conditional blocks, make sure that the test cases exercise every possible condition, being especially cautious about the formatting when working with HTML messages.
The recommended method for testing mailings that have the recipients defined from a LISTSERV list and are being sent a standard list message is to create a clone of that list within LISTSERV (such as TEST-listname). Make sure that the entire list configuration is exactly duplicated, including customized templates (especially the top and bottom banners, if these are defined). Subscribe only the test recipients to the test list. On the Delivery Test screen, enter the full address of the test list as the sole test address.
Remember to also test both the From: address and the
Reply-To: address (if the latter is specified). To test the
From: address and the
Reply-To: addresses try replying to both from the test message and examine the results.
Once a test message containing tracked data has been delivered and the test recipients have opened the message and clicked on tracked links, then the Test Reports tab, available from the Job Details screen, becomes active. Click on the tab to open up a definition page to generate a “quick report” on the test message data. Quick Reports are predefined report types that create a temporary report based on the selected report settings and the active email job.
|
•
|
Event distribution over time – Produces a simple line graph showing the number of responses over time. Time is plotted on the x-axis of the chart and events are charted on the y-axis.
|
|
•
|
Sum of events – Produces a bar graph showing the number of recipients who clicked on each URL and/or opened up the message. The length of each bar shows the number of events.
|
|
•
|
Two types can be chosen for the report – Sum of unique events or Sum of all events. Sum of unique events records only a single event of each type for each recipient. This chart will register “0” (zero) events if blind tracking has been selected for the tracking type during the tracking definition because blind tracking cannot count any event as unique. Sum of all events counts all events triggered for each recipient.
|
|
•
|
Event details – Displays a table that lists the report type, lists the time period of the report, allows for the selection of encoding for downloading the data, and allows for the selection of the time zone to reference the downloaded data. Click the download link to download the raw data to a local drive. Raw data can be used in Microsoft ® Excel or third party statistical analysis software to create detailed reports. Event detail reports are available for personal, anonymous, unique, and blind tracking jobs.
|
Next, select the source data for the report type. Each data source will create a line or bar on the chart. There are up to five choices depending on the type of message (Plain Text or HTML), as well as the tracking events defined in the message:
|
•
|
Open-Up events – This option charts all the open-up events for a job. It is only available for HTML messages.
|
|
•
|
Click-Through events (all links bundled) – This option bundles all click-through events together into one variable charted on the report. For example, if three links were being tracked in a message, they would appear as one line or bar on the chart.
|
|
•
|
Click-Through events (all links separate) – This option creates separate variables for each click-through event that is tracked. For example, if three links were being tracked in a message, each of the three links would appear as a separate line or bar on the chart.
|
Other data sources can be added to the chart by selecting any or all of the Additional Statistics option buttons. These additional data sources can provide comparison data in the chart so that it is possible to see the total number of messages sent in relation to the number of open-up events that occurred, for example. Three
Additional Statistics options are available:
|
•
|
Number of Sent Messages – Adds the number of messages that were sent to LISTSERV as a constant reference value in the report.
|
|
•
|
Number of Unbounced Messages – Adds the number of unbounced messages as a constant reference in the report. This value shows the number of messages that were sent and not detected as bounces by LISTSERV. This option is only available if LISTSERV handles bounces automatically (selected during the sender definition).
|
|
•
|
Number of Bounced Messages – Adds the number of bounced messages as a constant value in the report. This value shows the number of messages that were sent but were detected as bounces by LISTSERV. This value can go up over time as new bounces are received by LISTSERV. This option is only available if LISTSERV handles bounces automatically (selected during the sender definition).
|
Make your selections, and then click [OK] to continue. A temporary report with the settings selected will be generated and displayed on the screen. This report will
not be saved (it can always be run again from the Test Reports tab).
Located at the bottom right side of the test report are four icons. Each icon opens a page with different functions and options. The currently selected icon will be highlighted with a blue border. Unavailable icons will appear grayed out. These icons appear on all test reports, quick reports, and regular report pages.
In order for an email job to be delivered, final authorization is required. After all previous requirements have been met – define recipients, define message, define tracking, define sender, delivery test, and delivery settings – a job may be authorized for delivery. The job owner always has permission to authorize a job, as do any other team members that have been granted permission.
To authorize an email job delivery, click on the Authorize Delivery icon on the Workflow Diagram or click the
[Authorize Delivery] button at the top of the Summary Diagram. The Authorize Delivery screen opens.
The Job Summary section lists all the selections that make up the email job. Above the [Authorize Delivery Now] and
[Cancel] buttons is a message that summarizes the selections made for the delivery. If this is an auto-repeat job, the auto-repeat sequence time delay will be listed as well as other notes on the screen. Normal jobs will have a shorter message summarizing the delivery schedule.
The Details link in the
Content section will display a copy of the message, alternative text, attachments, and tracking information. The message can be downloaded to a local file as well. The
Details link in the
Recipients section displays information regarding the recipients for the email job.
The Sender section lists all the selection you made while defining the sender’s information. If DomainKey signing is enabled, then LISTSERV Maestro interacts with LISTSERV to determine if the supplied sender address is supported by one of the DomainKeys that were deployed to the LISTSERV host when DomainKeys was configured. If the email job is configured for future delivery, then there is a considerable time window during which the administrator may opt to change the DomainKeys settings at the LISTSERV host. Therefore, if DomainKeys have been disabled during this time window, then the email job delivery will fail with an appropriate error message.
Clicking [Authorize Delivery Now] will do exactly that, and the job will be sent immediately or at the scheduled day and time. Clicking
[Cancel] will go back to the Job Details screen so that any part of the job may be edited, or the job can be deleted.
If a job is authorized for delivery at a future date and time, delivery authorization may be revoked by going to the Open Jobs screen and selecting the job. For more information, see Section 10
Open Jobs.
The scheduled delivery time shown on this screen is always displayed relative to the local time of the user. If a job is scheduled for a time zone different from the user’s local time, then the time displayed in the Job Summary may appear incorrect. For example, if the user enters the time 10:00 with time zone GMT as the scheduled delivery time, but the user’s location is not in the GMT time zone, the delivery time displayed on the Authorize Delivery screen will be different from the 10:00 time selected. This is because the user’s local time is in a different time zone than the scheduled delivery time. When converted to absolute time, the delivery time will be correct. Time zone differences between the local time and the delivery time may even show an “incorrect” day. If the user sets the scheduled delivery time to 0:30 at GMT on the first of January, the time will appear as 23:30 on the 31
st of December on this screen if the local time zone is one hour behind GMT.
Once the A/B-Split Job’s information is completed (green icons and checkmarks will let you know when everything is completed), and you are ready to authorize delivery, then click the
Mail Job menu and select
Authorize Delivery.
The Authorize Delivery screen opens, presenting you with a final overview of the job. Verify that everything is correct, and then click the
[Authorize Delivery Now] button.
Once the information for the A/B-Split Job with Sampling is completed (green icons and checkmarks will let you know when everything is completed), and you are ready to authorize delivery, then click on the Sampling Variant Jobs tab, click the
Mail Job menu, and select
Authorize Sampling Variant Delivery.
The Authorize Delivery screen opens, presenting you with a final overview of the job. Verify that everything is correct, and then click the
[Authorize Delivery Now] button. The sampling variant jobs are authorized and delivered first, while the main variant job is held back. You can then use the tracking results of the sampling variants to evaluate the "success" of each variant (just like a normal A/B-split job). With these results, you can copy the content of the most successful sampling variant job into the main variant. After this is done, you can authorize the delivery of the main variant.
Any auto-repeat job currently on the Open Jobs screen awaiting its scheduled delivery time can have its delivery authorization revoked just like a normal job. If authorization is revoked, the job will be put back into the Resume Jobs list, where it can be edited again.
If a job is re-authorized for future delivery, the job may have its standing changed in respect to the auto-repeat sequence it was part of before the authorization was revoked. Here are the possible standings:
|
•
|
The job is the first job of an auto-repeat sequence – This means that no delivery has taken place for this job because it was the first job of the sequence and was already revoked before its scheduled delivery time. When re-authorized, the job will simply continue to be the first (and still only) job of the same auto-repeat sequence it belonged to before.
|
|
•
|
The job is not the first job of the auto-repeat sequence but has not been changed since the delivery authorization was revoked – This means that this job is already an automatically created copy that is part of an auto-repeat sequence. The delivery authorization of this job was revoked, but the job has not changed since then. When re-authorized, the job can continue as part of the same auto-repeat sequence, or it can start a new auto-repeat sequence. This choice will have to be made on the Authorize Delivery screen.
|
|
•
|
The job is not the first job of the auto-repeat sequence and has been changed since the delivery authorization was revoked – This means that this job is already an automatically created copy that is part of an auto-repeat sequence. The delivery authorization of this job was revoked and then it was changed in some way. When re-authorized, the job will create a new auto-repeat sequence and it will no longer be part of the sequence it belonged to before the delivery authorization was revoked. This happens because the job is no longer an exact copy of the previous jobs in its original sequence. Instead, it will be the first (and still only) job of a new auto-repeat sequence.
|
Note: A job is defined as
changed since authorization was revoked if the recipients definition, content definition, tracking definition, or sender definition of the job has changed since the delivery authorization was revoked. If these four parts remained unchanged, the job is interpreted as unchanged. Changes on the Delivery Test or Delivery Settings screens are not interpreted as changes to the job.
