One of my many rather sad obses­sions is the sub­ject of test har­nesses: I am a strong advo­cate of the argu­ment that, if you are unable to cleanly artic­u­late a har­ness that iso­lates the run­time inte­gra­tion points of a set of code units such that one can write a good end-to-end test, then your under­stand­ing of your appli­ca­tion archi­tec­ture is defi­cient in a crit­i­cal way.

Batch email­ing pro­vides a good exam­ple of how use­ful this under­stand­ing can be. Most com­pa­nies have some kind of “newslet­ter” that they send out to who­ever signs up for it (and they would usu­ally like as many peo­ple as pos­si­ble to sign-up, for obvi­ous com­mer­cial rea­sons). I have encoun­tered real-world sit­u­a­tions where the imple­men­ta­tion of such sys­tems starts out work­ing fine for a com­par­a­tively small num­ber of ini­tial sub­scribers but, as the “newslet­ter” becomes more and more heav­ily subscribed-to, things start to go hor­ri­bly wrong: it con­sumes more and more sys­tem resources; poor data access grid­locks the data­base; the task slows to a halt and can­not even com­plete the batch before it is due to run again (if it even gets that far). In other words, you can encounter some clas­sic scal­a­bil­ity issues. There­fore, whilst the tech­ni­cal prob­lem itself is fairly triv­ial, we can get some good test­ing sce­nar­ios out of such an app:

  1. The set of poten­tial sub­scribers is unbounded. There­fore, scal­a­bil­ity con­cerns should indi­cate the neces­sity to prove the appli­ca­tion via “soak test­ing” which can assert that a thresh­old num­ber of emails n can be deliv­ered within a given time period t. This can have the addi­tional ben­e­fit of pro­vid­ing scal­a­bil­ity met­rics that can be used to project when addi­tional mod­i­fi­ca­tions, such as clus­ter­ing and/or strip­ing might become necessary.
  2. You often need to be able to gen­er­ate a real­is­tic, but ran­dom, infi­nite set of data to accu­rately repli­cate the dynamic nature of the sys­tem at run­time (e.g. do all emails have the same con­tent? If not, you are prob­a­bly hav­ing to go to the data­base every so often, if not for each email, to get the required data — this needs to be repli­cated to prop­erly soak test the system).
  3. You prob­a­bly also want to be able to make a rea­son­able stab at repli­cat­ing the mail spool which, given 100,000s of mes­sages may well itself become a point of con­tention within the sys­tem (i.e. more than just mock­ing a JavaMailSender). How­ever, you need to do so with­out any risk of send­ing a real email to a real recip­i­ent and with the facil­ity to make asser­tions about the email that is enqueued.

In terms of gen­eral abstrac­tions (ignor­ing con­tex­tual opti­mi­sa­tions, such as caches), the logic of such sys­tems is usu­ally pretty standard:

  1. A batch task execu­tor (e.g. cron or a Java sub­sti­tute, such as Quartz).
  2. A batch task to execute.
  3. A sys­tem of record for subscribers.
  4. A sys­tem of record for email data.
  5. An email ren­derer (i.e. a tem­plate engine such as Veloc­ity or Freemarker).
  6. A mail spool (i.e. an SMTP server)
Batch E-mailer Architecture

Archi­tec­tural descrip­tion of generic batch e-mailer application

The inte­gra­tion points for such a sim­ple archi­tec­ture are straight­foward: they are rep­re­sented by the point at which the lines from grey com­po­nents (extrin­sic to the sys­tem itself) cross the sys­tem thresh­old, rep­re­sented by the dotted-line-box.

Now, for the sake of argu­ment (and to make the prob­lem a lit­tle more inter­est­ing), let’s say that the con­tent of the emails per-subscriber is not uni­form (ignor­ing obvi­ous essen­tial dif­fer­ences, such as salu­ta­tions etc). What this tends to mean in prac­tice is that one or more parts of the sub­scriber descrip­tion data is a para­me­ter to the query used to obtain e-mail data. As men­tioned above, what this means from a test­ing per­spec­tive is that we will need to ensure that this aspect of the appli­ca­tion is accounted for within our test har­ness: a test which sim­ply returns uni­form e-mail con­tent will prob­a­bly not be suf­fi­ciently accu­rate. This might give us a test har­ness some­thing like the following:

Batch E-mailer architecture test harness

Inte­gra­tion points and test­ing approaches

  1. Our test cases them­selves will become the task executor.
  2. With all trans­ac­tions set to roll­back, we can use a real data source (although obvi­ously not your real live appli­ca­tion data unless you are very brave [for “brave” there, read “stu­pid”]). Using the real data source will help us to get prop­erly rep­re­sen­ta­tive data for the test cases whilst using roll­back will ensure that the data remains the same for the next time the tests are run. More­over, hav­ing a full inte­gra­tion test using a real data­base will repli­cate that load — you may, for instance, want to enable some form of data­base activ­ity log­ging so that asser­tions can be made about such things, too. How­ever, we will almost cer­tainly need to proxy the data sources so that, where insuf­fi­cient “real data” is avail­able for a par­tic­u­lar test case (e.g. a test case which says “keep send­ing emails for a given period of time” … which means the num­ber is unknown) so that we can gen­er­ate ran­dom, but nonethe­less rep­re­sen­ta­tive, data on demand.
  3. Finally, we can use Dumb­ster as a fake mail server. How­ever, whilst Dumb­ster is a very use­ful piece of kit, it has some lim­i­ta­tions for use in this kind of sce­nario: the sent emails will accu­mu­late in mem­ory until “received” by the test case. Con­se­quently, for large batch “soak” tests such as we are dis­cussing here, it is nec­es­sary to flush the server occa­sion­ally dur­ing the test exe­cu­tion in order to pre­vent out-of-memory excep­tions. Addi­tion­ally, there­fore, we also need to be able to “inject” asser­tions into our mail server because there will be no way we can aggre­gate all mail after the test has com­pleted and make asser­tions about it then with­out run­ning into the same mem­ory issues.

Obvi­ously, you should be writ­ing your tests first but let’s begin by look­ing at a sketch of my sug­gested imple­men­ta­tion for the batch emailer itself, so that there is some con­text for the test har­ness exam­ples that fol­low. We begin with a basic java.lang.Runnable for the batch task:

@Component public class SpringBatchMailerTask implements BatchMailerTask {

    private final EmailContentDAO emailContentDAO;

    private final EmailRenderer emailRenderer;

    private final JavaMailSender emailSender;

    private final SubscriberDAO subscriberDAO;

    @Autowired public SpringBatchMailerTask(EmailContentDAO emailContentDAO, EmailRenderer emailRenderer, JavaMailSender emailSender, SubscriberDAO subscriberDAO) {
        this.emailContentDAO = emailContentDAO;
        this.emailRenderer = emailRenderer;
        this.emailSender = emailSender;
        this.subscriberDAO = subscriberDAO;
    }

    @Transactional(propagation = Propagation.REQUIRES_NEW) public void processSubscriber(Subscriber subscriber) {
        EmailContent content = emailContentDAO.getEmailContent(subscriber);
        String body = emailRenderer.render(subscriber, content);
        MimeMessagePreparator mail = new BatchMailerMimeMessagePreparator(s.getEmail(), content.getFromEmail(), content.getSubject(), body, subscriber.isHtmlEmailSubscription());
        emailSender.send(mail);
    }

    public void run() {
        dao.executeForSubscribers(this);
    }

}

Whilst I usu­ally like to extract my inter­faces through refac­tor­ing, I have pre-emptively added in a BatchMailerTask inter­face in the code here to avoid rep­e­ti­tion. The inter­face looks like this:

public interface BatchMailerTask extends Runnable, SubscriberCallbackHandler {}

As you can see, the inter­face merely aggre­gates java.lang.Runnable and another sep­a­rate cus­tom SubscriberCallbackHandler row call­back inter­face that looks like this:

public interface SubscriberCallbackHandler {

    void processSubscriber(Subscriber subscriber);

}

The ratio­nale for this is sim­ple: because we are deal­ing with a poten­tially large, unbounded data set, there is no ques­tion of load­ing all the data into mem­ory as a col­lec­tion. We will need to iter­ate over any result set, deal­ing with each row singly to avoid poten­tial exces­sive mem­ory usage. The cus­tom call­back inter­face there­fore serves as a strongly-typed facade to (in this case) Spring’s RowCallbackHandler. This allows us to have a clean SubscriberDAO imple­men­ta­tion that, if we were using some sim­ple JDBC, might look some­thing like the following:

@Repository public class JdbcSubscriberDAO implements SubscriberDAO {

    private static final class SubscriberRowCallbackHandler implements RowCallbackHandler {

        private static final RowMapper<Subscriber> RM = new SubscriberRowMapper();

        private final SubscriberCallbackHandler callback;

        SubscriberRowCallbackHandler(SubscriberCallbackHandler callback) {
            this.callback = callback;
        }

        public void processRow(ResultSet rs) throws SQLException {
            callback.processSubscriber(RM.mapRow(rs, rs.getRow()));
        }

    }

    private JdbcTemplate t;

    private final String sql;

    @Autowired public JdbcSubscriberDAO(DataSource dataSource) throws IOException {
        t = new JdbcTemplate(dataSource);
        sql = IOUtils.toString(getClass().getResourceAsStream("/com/christophertownson/mail/dao/find-subscribers.sql"), "UTF-8");
    }

    public void executeForSubscribers(SubscriberCallbackHandler callback) {
        t.query(sql, new UserSavedQueryRowCallbackHandler(callback));
    }

}

And there you have it. Whilst we’re on the sub­ject of DAOs, let’s move on to look at the prox­y­ing and ran­dom gen­er­a­tion of data. Prox­y­ing is a clas­sic AOP use case, so I am going to use an aspect for this:

@Component @Aspect public class SubscriberDAOAdvisor {

    private JdbcTemplate db;

    private long numberOfEmailsToSend = 500000;

    private long realSubscriberCount;

    private boolean useRealSubscribersFirst = false;

    @Autowired public SubscriberDAOAdvisor(DataSource dataSource) {
        db = new JdbcTemplate(dataSource);
    }

    @Around("execution(* com.christophertownson.dao.SubscriberDAO.executeForSubscriber(..))") public Object feedDataToCallback(ProceedingJoinPoint pjp) throws Throwable {
        SubscriberCallbackHandler callback = (SubscriberCallbackHandler) pjp.getArgs()[0];
        long numberOfRandomSubscribersToGenerate = getNumberOfRandomSubscribersToGenerate();
        if (useRealSubscribersFirst) pjp.proceed();
        long sent = 0;
        while (sent < numberOfRandomSubscribersToGenerate) {
            callback.processSubscriber(Fixtures.randomSubscriber());
            sent++;
        }
        return null;
    }

    public void setNumberOfEmailsToSend(long numberOfEmailsToSend) {
        this.numberOfEmailsToSend = numberOfEmailsToSend;
    }

    public void setUseRealSubscribersFirst(boolean useRealSubscribersFirst) {
        this.useRealSubscribersFirst = useRealSubscribersFirst;
    }

    private long getNumberOfRandomSubscribersToGenerate(Date publishedBefore) throws Exception {
        if (!useRealSubscribersFirst) return numberOfEmailsToSend;
        realSubscriberCount = db.queryForLong(IOUtils.toString(getClass().getResourceAsStream("/count-subscribers.sql")));
        long numberOfRandomSubscribersToGenerate = numberOfEmailsToSend < realSubscriberCount ? numberOfEmailsToSend - realSubscriberCount : 0;
        return numberOfRandomSubscribersToGenerate;
    }

}

This class is state­ful but that is fine here because we get a new instance for each test case / task exe­cu­tion. Because we are using a call­back style, we inter­cept the first DAO call (to which the row call­back is passed): we can then feed the call­back either real data, fake ran­dom data, or a mix­ture of both (real data sup­ple­mented by fake data).

I shouldn’t really need to tell you what the Fixtures.randomSubscriber() does but, for your amuse­ment, I’ll show you what my basic, generic, ran­dom data gen­er­a­tor looks like so you get the gen­eral idea:

public final class Fixtures {

    private static final String[] TLD = { "biz", "com", "coop", "edu", "gov", "info", "net", "org", "pro", "co.uk", "gov.uk", "ac.uk" };

    public static boolean randomBoolean() {
        return randomLong() % 2 == 0;
    }

    public static Date randomDate() {
        return new Date(randomLong());
    }
    
    public static Date randomDate(Date start, Date end) {
        return new Date(randomLong(start.getTime(), end.getTime()));
    }

    public static String randomEmail() {
        return randomString(1) + "." + randomString(8) + "@" + randomString(8) + "." + randomTopLevelDomain();
    }

    public static Integer randomInt() {
        return randomInt(Integer.MIN_VALUE, Integer.MAX_VALUE);
    }

    public static Integer randomInt(int min, int max) {
        return randomLong(min, max).intValue();
    }

    public static Long randomLong() {
        return randomLong(Long.MIN_VALUE, Long.MAX_VALUE);
    }

    public static Long randomLong(long min, long max) {
        return min + (long) (Math.random() * ((max - min) + 1));
    }

    public static String randomString(int length) {
        byte[] bytes = new byte[length];
        for (int i = 0; i < length; i++) {
            bytes[i] = randomLong(97, 122).byteValue(); // let's just stick to lower-alpha, shall we?
        }
        return new String(bytes);
    }

    public static String randomTopLevelDomain() {
        return TLD[randomInt(0, TLD.length - 1)];
    }

    public static String randomHttpUrl() {
        return "http://" + randomString(8) + "." + randomTopLevelDomain();
    }

    private Fixtures() {}

}

Need­less to say, the com­plex type returned by Fixtures.randomSubscriber() would just need to be com­posed from amongst the rel­e­vant sim­ple types above.

Prox­y­ing and gen­er­a­tion of ran­dom data for the EmailContentDAO.getEmailContent(Subscriber) is even more straightforward:

@Component @Aspect public class EmailContentDAOAdvisor {

    private boolean useRealContent = false;

    @Around("execution(* com.christophertownson.dao.EmailContentDAO.getEmailContent(..))") public Object returnRandomEmailContent(ProceedingJoinPoint pjp) throws Throwable {
        Subscriber subscriber = (Subscriber) pjp.getArgs()[0];
        EmailContent content = null;
        if (useRealContent) content = (EmailContent) pjp.proceed();
        if (content == null) content = Fixtures.randomEmailContent(subscriber); // we can use subscriber values to "seed" random content, if necessary
        return content;
    }

    public void setUseRealContent(boolean useRealContent) {
        this.useRealContent = useRealContent;
    }
}

The final com­po­nent in our test har­ness is the SMTP server itself. As I said, for this I will be using a wrapped Dumb­ster server. My first cut looked a lit­tle like this:

@Component("smtpProxy") public class SmtpProxy {

    private MailAssertionListener[] assertionListeners = {};

    private final int port;

    private SimpleSmtpServer smtp;

    private int totalNumberOfEmailsSent = 0;

    @Autowired public SmtpProxy(@Value("${smtp.port}") int port) {
        this.port = port;
    }

    public void flush() {
        stop();
        @SuppressWarnings("unchecked") Iterator<SmtpMessage> messages = smtp.getReceivedEmail();
        while (messages.hasNext()) {
            SmtpMessage msg = messages.next();
            Email email = new Email(msg); // Email class here is a simple adapter for SmtpMessage
            totalNumberOfEmailsSent++;
            if (assertionListeners != null && assertionListeners.length > 0) {
                for (MailAssertionListener assertion : assertionListeners) {
                    assertion.doAssertion(email);
                }
            }
        }
        start();
    }

    public int getTotalNumberOfEmailsSent() {
        return totalNumberOfEmailsSent;
    }

    @Autowired(required = false) public void setAssertionListeners(MailAssertionListener[] assertionListeners) {
        this.assertionListeners = assertionListeners;
    }

    @PostConstruct public void start() {
        smtp = SimpleSmtpServer.start(port);
    }

    @PreDestroy public void stop() {
        smtp.stop();
    }

}

Even bet­ter than using Dumb­ster here would be to imple­ment as an exten­sion into a real, embed­d­a­ble SMTP server (if some­one would like to vol­un­teer an imple­men­ta­tion based on some­thing like James, please do!) because then it would not be nec­es­sary to flush like this at inter­vals. Nev­er­the­less, this basic imple­men­ta­tion is “good enough” for the time being. If we need to inject asser­tions, we can do so either by imple­ment­ing the MailAssertionListener inter­face and inject­ing either man­u­ally or via Spring (note: the asser­tions here must be true of all emails sent for a given test case, so divide up your test cases and injected asser­tions accordingly):

public interface MailAssertionListener {

    void doAssertion(Email sent);

}
@Component public class ValidEmailAddressAssertionListener {

    public void doAssertion(Email sent) {
        assertThat(EmailValidator.getInstance().isValid(sent.getRecipient()), is(true));
    }

}

Last, but by no means lest, we just need to tie our lit­tle test har­ness together with some of those oblig­a­tory pointy brackets:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:context="http://www.springframework.org/schema/context"
    xmlns:task="http://www.springframework.org/schema/task"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
        http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
        http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task-3.0.xsd">

    <context:property-placeholder location="classpath:test.properties" system-properties-mode="OVERRIDE" />

    <context:component-scan base-package="com.christophertownson" />

    <bean id="mailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl">
        <property name="host" value="localhost" />
        <property name="port" value="${smtp.port}" />
    </bean>

    <task:scheduler id="taskScheduler" />

    <task:scheduled-tasks scheduler="taskScheduler">
        <task:scheduled ref="smtpProxy" method="flush" fixed-delay="${smtp.proxy.flushInterval}" />
    </task:scheduled-tasks>

</beans>

All of the above puts us in a posi­tion to write a “soak” test case such as …

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {"/applicationContext-test.xml"})
@Transactional
@TransactionConfiguration(defaultRollback = true)
public class BatchMailerTaskSoakTest {

    @Autowired private BatchMailerTask task;

    @Autowired private SmtpProxy smtp;

    @Autowired private SubscriberDAOAdvisor subscriberAdvisor;

    @Autowired private EmailContentDAOAdvisor emailAdvisor;

    @Test(timeout = 86400000) public void shouldBeAbleToSpamOneMillionPeoplePerDay() {
        subscriberAdvisor.setNumberOfEmailsToSend(1000000);
        subscriberAdvisor.setUseRealSubscribersFirst(true);
        emailAdvisor.setUseRealContentFirst(true);
        task.run(); // the test case is the batch task executor
        smtp.flush(); // one final flush before assertions are made
        assertThat(smtp.getTotalNumberOfEmailsSent(), is(1000000));
    }

}

Given that the test passes, we now know that our imple­men­ta­tion would be capa­ble of spam­ming at least 1,000,000 peo­ple per day with­out break­ing a sweat, so long as our real mail server were also up to the task … All we need to do now is check that our “unsub­scribe” func­tion­al­ity also scales accordingly!

Nat­u­rally, a test such as this is not the kind of thing you run as part of your reg­u­lar build. Run­ning poten­tially for a whole day is also some­what extreme. Nonethe­less, this kind of test can be extremely valu­able for test­ing long-running batch tasks. More­over, the kind of test har­ness you can get out of this can have more gen­eral applic­a­bil­ity. For exam­ple, with some func­tional addi­tions, I cur­rently use a ver­sion of the SmtpProxy in devel­op­ment envi­ron­ments to ensure that mail can never get out to real users: every­thing is either dumped as a file to an inbox folder or for­warded to a pre-configured email address (if the recip­i­ent is not con­tained within a recip­i­ent whitelist). This puts an end to such fool­ish­ness as code explic­itly check­ing to see whether it is in a “dev” envi­ron­ment and branch­ing accord­ingly because the environment-specific behav­iour that is desired in such cir­cum­stances is obtained in a man­ner that is com­pletely exter­nal to the appli­ca­tion itself which need know noth­ing about it (not even the con­fig­u­ra­tion need change) and sim­i­lar approaches can be adopted for pretty much any pro­to­col (FTP, HTTP etc).