One of my many rather sad obsessions is the subject of test harnesses: I am a strong advocate of the argument that, if you are unable to cleanly articulate a harness that isolates the runtime integration points of a set of code units such that one can write a good end-to-end test, then your understanding of your application architecture is deficient in a critical way.
Batch emailing provides a good example of how useful this understanding can be. Most companies have some kind of “newsletter” that they send out to whoever signs up for it (and they would usually like as many people as possible to sign-up, for obvious commercial reasons). I have encountered real-world situations where the implementation of such systems starts out working fine for a comparatively small number of initial subscribers but, as the “newsletter” becomes more and more heavily subscribed-to, things start to go horribly wrong: it consumes more and more system resources; poor data access gridlocks the database; the task slows to a halt and cannot even complete the batch before it is due to run again (if it even gets that far). In other words, you can encounter some classic scalability issues. Therefore, whilst the technical problem itself is fairly trivial, we can get some good testing scenarios out of such an app:
- The set of potential subscribers is unbounded. Therefore, scalability concerns should indicate the necessity to prove the application via “soak testing” which can assert that a threshold number of emails
n can be delivered within a given time period t. This can have the additional benefit of providing scalability metrics that can be used to project when additional modifications, such as clustering and/or striping might become necessary.
- You often need to be able to generate a realistic, but random, infinite set of data to accurately replicate the dynamic nature of the system at runtime (e.g. do all emails have the same content? If not, you are probably having to go to the database every so often, if not for each email, to get the required data — this needs to be replicated to properly soak test the system).
- You probably also want to be able to make a reasonable stab at replicating the mail spool which, given 100,000s of messages may well itself become a point of contention within the system (i.e. more than just mocking a
JavaMailSender). However, you need to do so without any risk of sending a real email to a real recipient and with the facility to make assertions about the email that is enqueued.
In terms of general abstractions (ignoring contextual optimisations, such as caches), the logic of such systems is usually pretty standard:
- A batch task executor (e.g. cron or a Java substitute, such as Quartz).
- A batch task to execute.
- A system of record for subscribers.
- A system of record for email data.
- An email renderer (i.e. a template engine such as Velocity or Freemarker).
- A mail spool (i.e. an SMTP server)
Architectural description of generic batch e-mailer application
The integration points for such a simple architecture are straightfoward: they are represented by the point at which the lines from grey components (extrinsic to the system itself) cross the system threshold, represented by the dotted-line-box.
Now, for the sake of argument (and to make the problem a little more interesting), let’s say that the content of the emails per-subscriber is not uniform (ignoring obvious essential differences, such as salutations etc). What this tends to mean in practice is that one or more parts of the subscriber description data is a parameter to the query used to obtain e-mail data. As mentioned above, what this means from a testing perspective is that we will need to ensure that this aspect of the application is accounted for within our test harness: a test which simply returns uniform e-mail content will probably not be sufficiently accurate. This might give us a test harness something like the following:
Integration points and testing approaches
- Our test cases themselves will become the task executor.
- With all transactions set to rollback, we can use a real data source (although obviously not your real live application data unless you are very brave [for “brave” there, read “stupid”]). Using the real data source will help us to get properly representative data for the test cases whilst using rollback will ensure that the data remains the same for the next time the tests are run. Moreover, having a full integration test using a real database will replicate that load — you may, for instance, want to enable some form of database activity logging so that assertions can be made about such things, too. However, we will almost certainly need to proxy the data sources so that, where insufficient “real data” is available for a particular test case (e.g. a test case which says “keep sending emails for a given period of time” … which means the number is unknown) so that we can generate random, but nonetheless representative, data on demand.
- Finally, we can use Dumbster as a fake mail server. However, whilst Dumbster is a very useful piece of kit, it has some limitations for use in this kind of scenario: the sent emails will accumulate in memory until “received” by the test case. Consequently, for large batch “soak” tests such as we are discussing here, it is necessary to flush the server occasionally during the test execution in order to prevent out-of-memory exceptions. Additionally, therefore, we also need to be able to “inject” assertions into our mail server because there will be no way we can aggregate all mail after the test has completed and make assertions about it then without running into the same memory issues.
Obviously, you should be writing your tests first but let’s begin by looking at a sketch of my suggested implementation for the batch emailer itself, so that there is some context for the test harness examples that follow. We begin with a basic java.lang.Runnable for the batch task:
@Component public class SpringBatchMailerTask implements BatchMailerTask {
private final EmailContentDAO emailContentDAO;
private final EmailRenderer emailRenderer;
private final JavaMailSender emailSender;
private final SubscriberDAO subscriberDAO;
@Autowired public SpringBatchMailerTask(EmailContentDAO emailContentDAO, EmailRenderer emailRenderer, JavaMailSender emailSender, SubscriberDAO subscriberDAO) {
this.emailContentDAO = emailContentDAO;
this.emailRenderer = emailRenderer;
this.emailSender = emailSender;
this.subscriberDAO = subscriberDAO;
}
@Transactional(propagation = Propagation.REQUIRES_NEW) public void processSubscriber(Subscriber subscriber) {
EmailContent content = emailContentDAO.getEmailContent(subscriber);
String body = emailRenderer.render(subscriber, content);
MimeMessagePreparator mail = new BatchMailerMimeMessagePreparator(s.getEmail(), content.getFromEmail(), content.getSubject(), body, subscriber.isHtmlEmailSubscription());
emailSender.send(mail);
}
public void run() {
dao.executeForSubscribers(this);
}
}
Whilst I usually like to extract my interfaces through refactoring, I have pre-emptively added in a BatchMailerTask interface in the code here to avoid repetition. The interface looks like this:
public interface BatchMailerTask extends Runnable, SubscriberCallbackHandler {}
As you can see, the interface merely aggregates java.lang.Runnable and another separate custom SubscriberCallbackHandler row callback interface that looks like this:
public interface SubscriberCallbackHandler {
void processSubscriber(Subscriber subscriber);
}
The rationale for this is simple: because we are dealing with a potentially large, unbounded data set, there is no question of loading all the data into memory as a collection. We will need to iterate over any result set, dealing with each row singly to avoid potential excessive memory usage. The custom callback interface therefore serves as a strongly-typed facade to (in this case) Spring’s RowCallbackHandler. This allows us to have a clean SubscriberDAO implementation that, if we were using some simple JDBC, might look something like the following:
@Repository public class JdbcSubscriberDAO implements SubscriberDAO {
private static final class SubscriberRowCallbackHandler implements RowCallbackHandler {
private static final RowMapper<Subscriber> RM = new SubscriberRowMapper();
private final SubscriberCallbackHandler callback;
SubscriberRowCallbackHandler(SubscriberCallbackHandler callback) {
this.callback = callback;
}
public void processRow(ResultSet rs) throws SQLException {
callback.processSubscriber(RM.mapRow(rs, rs.getRow()));
}
}
private JdbcTemplate t;
private final String sql;
@Autowired public JdbcSubscriberDAO(DataSource dataSource) throws IOException {
t = new JdbcTemplate(dataSource);
sql = IOUtils.toString(getClass().getResourceAsStream("/com/christophertownson/mail/dao/find-subscribers.sql"), "UTF-8");
}
public void executeForSubscribers(SubscriberCallbackHandler callback) {
t.query(sql, new UserSavedQueryRowCallbackHandler(callback));
}
}
And there you have it. Whilst we’re on the subject of DAOs, let’s move on to look at the proxying and random generation of data. Proxying is a classic AOP use case, so I am going to use an aspect for this:
@Component @Aspect public class SubscriberDAOAdvisor {
private JdbcTemplate db;
private long numberOfEmailsToSend = 500000;
private long realSubscriberCount;
private boolean useRealSubscribersFirst = false;
@Autowired public SubscriberDAOAdvisor(DataSource dataSource) {
db = new JdbcTemplate(dataSource);
}
@Around("execution(* com.christophertownson.dao.SubscriberDAO.executeForSubscriber(..))") public Object feedDataToCallback(ProceedingJoinPoint pjp) throws Throwable {
SubscriberCallbackHandler callback = (SubscriberCallbackHandler) pjp.getArgs()[0];
long numberOfRandomSubscribersToGenerate = getNumberOfRandomSubscribersToGenerate();
if (useRealSubscribersFirst) pjp.proceed();
long sent = 0;
while (sent < numberOfRandomSubscribersToGenerate) {
callback.processSubscriber(Fixtures.randomSubscriber());
sent++;
}
return null;
}
public void setNumberOfEmailsToSend(long numberOfEmailsToSend) {
this.numberOfEmailsToSend = numberOfEmailsToSend;
}
public void setUseRealSubscribersFirst(boolean useRealSubscribersFirst) {
this.useRealSubscribersFirst = useRealSubscribersFirst;
}
private long getNumberOfRandomSubscribersToGenerate(Date publishedBefore) throws Exception {
if (!useRealSubscribersFirst) return numberOfEmailsToSend;
realSubscriberCount = db.queryForLong(IOUtils.toString(getClass().getResourceAsStream("/count-subscribers.sql")));
long numberOfRandomSubscribersToGenerate = numberOfEmailsToSend < realSubscriberCount ? numberOfEmailsToSend - realSubscriberCount : 0;
return numberOfRandomSubscribersToGenerate;
}
}
This class is stateful but that is fine here because we get a new instance for each test case / task execution. Because we are using a callback style, we intercept the first DAO call (to which the row callback is passed): we can then feed the callback either real data, fake random data, or a mixture of both (real data supplemented by fake data).
I shouldn’t really need to tell you what the Fixtures.randomSubscriber() does but, for your amusement, I’ll show you what my basic, generic, random data generator looks like so you get the general idea:
public final class Fixtures {
private static final String[] TLD = { "biz", "com", "coop", "edu", "gov", "info", "net", "org", "pro", "co.uk", "gov.uk", "ac.uk" };
public static boolean randomBoolean() {
return randomLong() % 2 == 0;
}
public static Date randomDate() {
return new Date(randomLong());
}
public static Date randomDate(Date start, Date end) {
return new Date(randomLong(start.getTime(), end.getTime()));
}
public static String randomEmail() {
return randomString(1) + "." + randomString(8) + "@" + randomString(8) + "." + randomTopLevelDomain();
}
public static Integer randomInt() {
return randomInt(Integer.MIN_VALUE, Integer.MAX_VALUE);
}
public static Integer randomInt(int min, int max) {
return randomLong(min, max).intValue();
}
public static Long randomLong() {
return randomLong(Long.MIN_VALUE, Long.MAX_VALUE);
}
public static Long randomLong(long min, long max) {
return min + (long) (Math.random() * ((max - min) + 1));
}
public static String randomString(int length) {
byte[] bytes = new byte[length];
for (int i = 0; i < length; i++) {
bytes[i] = randomLong(97, 122).byteValue(); // let's just stick to lower-alpha, shall we?
}
return new String(bytes);
}
public static String randomTopLevelDomain() {
return TLD[randomInt(0, TLD.length - 1)];
}
public static String randomHttpUrl() {
return "http://" + randomString(8) + "." + randomTopLevelDomain();
}
private Fixtures() {}
}
Needless to say, the complex type returned by Fixtures.randomSubscriber() would just need to be composed from amongst the relevant simple types above.
Proxying and generation of random data for the EmailContentDAO.getEmailContent(Subscriber) is even more straightforward:
@Component @Aspect public class EmailContentDAOAdvisor {
private boolean useRealContent = false;
@Around("execution(* com.christophertownson.dao.EmailContentDAO.getEmailContent(..))") public Object returnRandomEmailContent(ProceedingJoinPoint pjp) throws Throwable {
Subscriber subscriber = (Subscriber) pjp.getArgs()[0];
EmailContent content = null;
if (useRealContent) content = (EmailContent) pjp.proceed();
if (content == null) content = Fixtures.randomEmailContent(subscriber); // we can use subscriber values to "seed" random content, if necessary
return content;
}
public void setUseRealContent(boolean useRealContent) {
this.useRealContent = useRealContent;
}
}
The final component in our test harness is the SMTP server itself. As I said, for this I will be using a wrapped Dumbster server. My first cut looked a little like this:
@Component("smtpProxy") public class SmtpProxy {
private MailAssertionListener[] assertionListeners = {};
private final int port;
private SimpleSmtpServer smtp;
private int totalNumberOfEmailsSent = 0;
@Autowired public SmtpProxy(@Value("${smtp.port}") int port) {
this.port = port;
}
public void flush() {
stop();
@SuppressWarnings("unchecked") Iterator<SmtpMessage> messages = smtp.getReceivedEmail();
while (messages.hasNext()) {
SmtpMessage msg = messages.next();
Email email = new Email(msg); // Email class here is a simple adapter for SmtpMessage
totalNumberOfEmailsSent++;
if (assertionListeners != null && assertionListeners.length > 0) {
for (MailAssertionListener assertion : assertionListeners) {
assertion.doAssertion(email);
}
}
}
start();
}
public int getTotalNumberOfEmailsSent() {
return totalNumberOfEmailsSent;
}
@Autowired(required = false) public void setAssertionListeners(MailAssertionListener[] assertionListeners) {
this.assertionListeners = assertionListeners;
}
@PostConstruct public void start() {
smtp = SimpleSmtpServer.start(port);
}
@PreDestroy public void stop() {
smtp.stop();
}
}
Even better than using Dumbster here would be to implement as an extension into a real, embeddable SMTP server (if someone would like to volunteer an implementation based on something like James, please do!) because then it would not be necessary to flush like this at intervals. Nevertheless, this basic implementation is “good enough” for the time being. If we need to inject assertions, we can do so either by implementing the MailAssertionListener interface and injecting either manually or via Spring (note: the assertions here must be true of all emails sent for a given test case, so divide up your test cases and injected assertions accordingly):
public interface MailAssertionListener {
void doAssertion(Email sent);
}
@Component public class ValidEmailAddressAssertionListener {
public void doAssertion(Email sent) {
assertThat(EmailValidator.getInstance().isValid(sent.getRecipient()), is(true));
}
}
Last, but by no means lest, we just need to tie our little test harness together with some of those obligatory pointy brackets:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:task="http://www.springframework.org/schema/task"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task-3.0.xsd">
<context:property-placeholder location="classpath:test.properties" system-properties-mode="OVERRIDE" />
<context:component-scan base-package="com.christophertownson" />
<bean id="mailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl">
<property name="host" value="localhost" />
<property name="port" value="${smtp.port}" />
</bean>
<task:scheduler id="taskScheduler" />
<task:scheduled-tasks scheduler="taskScheduler">
<task:scheduled ref="smtpProxy" method="flush" fixed-delay="${smtp.proxy.flushInterval}" />
</task:scheduled-tasks>
</beans>
All of the above puts us in a position to write a “soak” test case such as …
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {"/applicationContext-test.xml"})
@Transactional
@TransactionConfiguration(defaultRollback = true)
public class BatchMailerTaskSoakTest {
@Autowired private BatchMailerTask task;
@Autowired private SmtpProxy smtp;
@Autowired private SubscriberDAOAdvisor subscriberAdvisor;
@Autowired private EmailContentDAOAdvisor emailAdvisor;
@Test(timeout = 86400000) public void shouldBeAbleToSpamOneMillionPeoplePerDay() {
subscriberAdvisor.setNumberOfEmailsToSend(1000000);
subscriberAdvisor.setUseRealSubscribersFirst(true);
emailAdvisor.setUseRealContentFirst(true);
task.run(); // the test case is the batch task executor
smtp.flush(); // one final flush before assertions are made
assertThat(smtp.getTotalNumberOfEmailsSent(), is(1000000));
}
}
Given that the test passes, we now know that our implementation would be capable of spamming at least 1,000,000 people per day without breaking a sweat, so long as our real mail server were also up to the task … All we need to do now is check that our “unsubscribe” functionality also scales accordingly!
Naturally, a test such as this is not the kind of thing you run as part of your regular build. Running potentially for a whole day is also somewhat extreme. Nonetheless, this kind of test can be extremely valuable for testing long-running batch tasks. Moreover, the kind of test harness you can get out of this can have more general applicability. For example, with some functional additions, I currently use a version of the SmtpProxy in development environments to ensure that mail can never get out to real users: everything is either dumped as a file to an inbox folder or forwarded to a pre-configured email address (if the recipient is not contained within a recipient whitelist). This puts an end to such foolishness as code explicitly checking to see whether it is in a “dev” environment and branching accordingly because the environment-specific behaviour that is desired in such circumstances is obtained in a manner that is completely external to the application itself which need know nothing about it (not even the configuration need change) and similar approaches can be adopted for pretty much any protocol (FTP, HTTP etc).