On September 17, 2011 christopher wrote: Handling Multiple File Uploads With Uploadify

More JavaScript fun this week, dur­ing which I have been play­ing around with Upload­ify, one of a slew of ready-made JavaScript + Flash solu­tions for han­dling mul­ti­ple file uploads in the browser. I have cho­sen to look at Upload­ify over the other alter­na­tives due to the fact that I find it per­son­ally to be the most sim­ple to use. Also, it is pro­vided as an exten­sion to jQuery, which is prob­a­bly my favourite JavaScript library. Last, but not least, it works … which is a qual­ity never to be under­es­ti­mated in software!

As usual, I am going to be demon­strat­ing its use in con­junc­tion with a Spring-Velocity web app. Rather than go through the whole process of set­ting up such an app again, I would refer you to my post on ActiveMQ, JMS & Ajax. The struc­ture of the project I talk about here is iden­ti­cal to that, but with fewer depen­den­cies, and, of course, you can down­load the code if you want to take a closer look. So let’s begin …

Let’s begin by imag­in­ing what our use case here might look like. For the sake of argu­ment, I am going to say that what we want is this:

  • The user needs to be able to upload one or more files.
  • The names of these files must match a nam­ing con­ven­tion from which the appli­ca­tion infers some kind of seman­tic value (which might be used to process them in dif­fer­ent ways, for example).
  • When the user uploads valid file(s), they should see a list of the files uploaded with the seman­tic infor­ma­tion we have parsed from them via their name(s).
  • When the user uploads invalid file(s), they should see infor­ma­tion about this error.

In my domain model, there­fore, I have 2 classes:

public enum UploadFileType {

    BAR(Pattern.compile("(?i)^[a-z0-9]+-bar\\.[a-z]+$")),
    BAZ(Pattern.compile("(?i)^[a-z0-9]+-baz\\.[a-z]+$")),
    FOO(Pattern.compile("(?i)^[a-z0-9]+-foo\\.[a-z]+$"));

    public static UploadFileType forName(String name) {
        for (UploadFileType type : values()) {
            if (type.p.matcher(name).matches()) return type;
        }
        throw new IllegalArgumentException(name + " does not match any of the patterns in the naming convention");
    }

    private Pattern p;

    private UploadFileType(Pattern p) {
        this.p = p;
    }

}

Because the logic required to parse my nam­ing con­ven­tion is sim­ple, I do not really need any kind of com­plex fac­tory … all I need is this sim­ple enum. You give it the file name in the forName(String) method and it returns the type. If no type is found, we throw an IllegalArgumentException which we will use later to man­age error flow. After this, I have a sim­ple Java bean to rep­re­sent the model for an UploadFile:

public class UploadFile {

    private final File file;

    private final String name;

    private final UploadFileType type;

    public UploadFile(File file, String name, UploadFileType type) {
        this.file = file;
        this.name = name;
        this.type = type;
    }

    @Override public boolean equals(Object obj) {
        return reflectionEquals(this, obj);
    }

    public File getFile() {
        return file;
    }

    public String getName() {
        return name;
    }

    public UploadFileType getType() {
        return type;
    }

    @Override public int hashCode() {
        return reflectionHashCode(this);
    }

    @Override public String toString() {
        return reflectionToString(this);
    }

}

Now, let’s assume I have a web page with a com­pletely stan­dard HTML form with a file input:

                <form id="upload_file" action="/upload" method="post" enctype="multipart/form-data">
                    <fieldset>
                        <legend>Choose the files to upload</legend>
                        <label for="file">File</label> <input id="file" name="file" type="file" /> <input type="submit" value="Upload" />
                    </fieldset>
                </form>

And, on the server-side, using Spring’s mul­ti­part form sup­port, I have a FileController that looks like this:

@Controller public class FileController {

    @RequestMapping(value = "/", method = RequestMethod.GET) public String getFileUploadViewName() {
        return "/fileupload";
    }

    @RequestMapping(value = "/upload", method = RequestMethod.POST) public UploadFile upload(@RequestParam("file") MultipartFile mf) throws IOException {
        String name = getName(mf.getOriginalFilename());
        File f = File.createTempFile(name, "." + getExtension(name));
        mf.transferTo(f);
        return new UploadFile(f, name, UploadFileType.forName(name));
    }

    @ExceptionHandler(IllegalArgumentException.class) @ResponseBody public String handleIllegalArgumentException(IllegalArgumentException e) {
        return "{\"error\":{\"message\":\"" + e.getMessage() + "\"}}";
    }

}

For the sake of clar­ity, here is what each of the three con­troller meth­ods do:

  1. getFileUploadViewName() sim­ply returns the Spring view name required to return the HTML file upload form. I have hard-coded this name for now but, in real life, you should prob­a­bly inject this as configuration.
  2. upload(MultipartFile) is the meat of the con­troller class: Spring will bind the mul­ti­part file from the file input of the sub­mit­ted form which has name="file". The only pro­cess­ing we are doing with this file for now is cre­at­ing a temp file from it and deter­min­ing its “type” by using the UploadFileType.forName(String) method. This will either return the type (in which case we have suc­cess­fully an UploadFile which we return as the model object) or it will throw the IllegalArgumentException in which case, our Spring error han­dler method is called …
  3. The handleIllegalArgumentException method is a Spring error han­dler method (indi­cated by the @ExceptionHandler anno­ta­tion. In such an event, all we do is return a hand-crafted JSON string as the response body with the excep­tion mes­sage in it … which is a bit lazy, I admit — but it will do for now.

With this in place, we have an end-to-end con­ven­tion HTML single-file upload mech­a­nism in place. So now it is time to dec­o­rate it with a lit­tle JavaScript magic so that the whole thing becomes a bit more usable and so that it becomes pos­si­ble for the user to upload mul­ti­ple files.

First, to pro­vide the user with the nec­es­sary feed­back in the page from the file uploads, I am going to add a few place­holder sec­tions to my HTML page within which I will dis­play data from the parsed JSON responses pro­vided by the controller …

        <section id="file_queue">
            <hgroup>
                <h1>File Upload Queue</h1>
                <div id="queue">
                </div>
            </hgroup>
        </section>
        <section id="files">
            <hgroup>
                <h1>Uploaded Files</h1>
                <ol>
                </ol>
            </hgroup>
        </section>
        <section id="errors">
            <hgroup>
                <h1>Errors</h1>
                <ol>
                </ol>
            </hgroup>
        </section>

We add, first, a sec­tion that will be used by Upload­ify to dis­play the queue of files as they are being uploaded with the ID “queue”. This is a built-in fea­ture of Upload­ify, so we don’t need to do any­thing spe­cial to get this queue dis­played, other than let Upload­ify know the ID of our “queue div” by set­ting the queueID prop­erty. After that, I have placed the 2 sec­tions that we will be writ­ing the info from the parsed JSON responses from the con­troller into — 1 for suc­cess­ful file uploads and 1 for errors.

Now I am just about ready to sprin­kle a lit­tle JavaScript on the page … But, stop! Before I do this there is ONE VERY IMPORTANT GOTCHA to deal with!

Now, due to the nature of the genius minds that work at Adobe, it is appar­ently impos­si­ble to set HTTP head­ers in requests made by Flash (I know, I know — what kind of non­sense is that?!) I am not a Flash expert, so I don’t know the ins-and-outs of this. How­ever, since Upload­ify (and pretty much any JavaScript-based mul­ti­file uploader) uses swfobject.js and Flash, this means that an Accept header can­not be set to request JSON responses. By default, as far as I can work out, Flash will always send an Accept header request­ing text/*. If you are using Spring’s ContentNegotiatingViewResolver (which I am in this exam­ple), you will need to work around this prob­lem. There are 3 options: the first is to use a .json file exten­sion on the request URI to indi­cate the required media type for the response this way. The sec­ond option is to use a request para­me­ter to achieve the same thing. The final way is to fil­ter the request and over­ride the Accept header for Upload­ify requests.

In real life, I would prob­a­bly just send Upload­ify requests to /upload.json and then the ContentNegotiatingViewResolver would know to send a JSON response. How­ever, for what­ever rea­son, you might not want or be able to change the URI in this way, so I will demon­strate the Old Skool servlet fil­ter approach (you might think you could also use a Spring HandlerInterceptor but that leads to call-order prob­lems with the ContentNegotiatingViewResolver — using a clas­sic fil­ter will ensure that the request is inter­cepted and wrapped before it even reaches Spring:

public class UploadifyFilter implements Filter {

    @SuppressWarnings({ "deprecation", "rawtypes" }) private static class UploadifyRequest implements HttpServletRequest {

        private final HttpServletRequest del;

        UploadifyRequest(HttpServletRequest delegate) {
            del = delegate;
        }

        @Override public String getHeader(String name) {
            return "Accept".equals(name) ? "application/json" : del.getHeader(name);
        }

        /* ... all other delegate methods not shown for brevity ... */

    }

    @Override public void init(FilterConfig filterConfig) throws ServletException {}

    @Override public void doFilter(ServletRequest req, ServletResponse resp, FilterChain chain) throws IOException, ServletException {
        if (req instanceof HttpServletRequest) chain.doFilter(new UploadifyRequest((HttpServletRequest) req), resp);
        else chain.doFilter(req, resp);
    }

    @Override public void destroy() {}

}

I then sim­ply wire this up in my web.xml to fil­ter requests sent to the /upload URI:

    <filter>
        <filter-name>uploadify</filter-name>
        <filter-class>com.christophertownson.foo.UploadifyFilter</filter-class>
    </filter>

    <filter-mapping>
        <filter-name>uploadify</filter-name>
        <url-pattern>/upload</url-pattern>
    </filter-mapping>

And now I am ready to receive Accept: text/* requests from the “Shock­wave Flash” user-agent and ensure that my app under­stands what content-type it should return. So let’s put in some JavaScript to make it all happen …

function init() {
    $('#file').uploadify({
        'script'         : '/upload', // URI to send the files to (could specify '/upload.json' here rather than using filter)
        'method'         : 'post', // request method to use
        'uploader'       : '/resources/uploadify.swf', // path to key uploadify files
        'expressInstall' : '/resources/expressInstall.swf', // path to key uploadify files
        'cancelImg'      : '/resources/cancel.png', // path to key uploadify files
        'auto'           : true, // automatically start uploading files (don't wait for user to push another button)
        'multi'          : true, // allow upload of multiple files
        'queueSizeLimit' : 10, // the number of files that can be placed into the queue at any 1 time
        'queueID'        : 'queue', // div ID to display the file upload queue in
        'fileDataName'   : 'file', // the request param name for the multipart file
        'fileExt'        : '*.txt;*.jpg;*.gif;*.png', // filter to use in the select files dialog
        'fileDesc'       : 'Upload Files (.txt, .jpg, .gif, .png)', // we MUST provide a description when we specify fileExt otherwise filter will NOT be applied
        // process JSON response to single file upload
        'onComplete'     : function(event, id, file, resp, data) {
            var obj = jQuery.parseJSON(resp);
            if (obj.error) {
                $('#errors ol').append('<li>' + obj.error.message + '</li>');
            } else if (obj.uploadFile) {
                $('#files ol').append('<li>' + obj.uploadFile.name + ' (Type: ' + obj.uploadFile.type +')</li>');
            }
        },
        // process HTTP 5xx errors etc
        'onError'        : function (event, id, file, err) {
            $('#errors ol').append('<li>' + err.type + ' Error: ' + err.info + '</li>')
        }
    });

    $('#upload_file').submit(function() {
        $('#file').uploadifyUpload();
        return false;
    });
}

$(document).ready(init);

It should be fairly evi­dent what this sim­ple JavaScript achieves but I will run through it briefly:

  1. It begins by con­fig­ur­ing some basic Upload­ify prop­er­ties includ­ing, per­haps most impor­tant for my pur­poses here 'multi' : true as this is what per­mits the upload of mul­ti­ple files.
  2. After that, we pro­vide an event call­back han­dler func­tion for the onComplete event — this event sig­nals com­ple­tion of the upload of an indi­vid­ual file within the queue and pro­vides access to the response from the server. In our case, this con­sists of a JSON string which we dese­ri­al­ize and append to the list of upload files or errors, as nec­es­sary. I also hook into the onError event to pro­vide sim­i­lar error han­dling for Upload­ify errors (rather than the log­i­cal excep­tions thrown by our con­troller that come back as JSON) … these are com­monly things like HTTP 5xx errors and so forth.
  3. Finally, I attach a lis­tener to the onSubmit event of the upload file form to pre­vent nor­mal sub­mis­sion and, instead, to get Upload­ify to han­dle the input.

There are many con­fig­u­ra­tion options and pos­si­bil­i­ties for event han­dling within the API. I would rec­om­mend read­ing the won­der­fully con­cise Upload­ify doc­u­men­ta­tion and it really is an excep­tion­ally easy piece of kit to use, espe­cially com­pared with the pain I recall try­ing to get some of the early ver­sions of the YUI uploader to work nicely a few years back … although the YUI3 Uploader looks a great improve­ment on pre­vi­ous ver­sions and is widely used and would also be well-worth considering.

On September 11, 2011 christopher wrote: ActiveMQ, JMS & Ajax: Involving the Browser in Event-Driven Systems

Event-driven archi­tec­ture is a big topic these days and with good rea­son. Aside from pro­vid­ing sig­nif­i­cant oppor­tu­ni­ties to increase scale and per­for­mance, given the right approach it can also make it eas­ier to divide sys­tems up into log­i­cal com­po­nents. The basic prin­ci­ple behind “event-driven” archi­tec­tures, as one might expect from the name, is asyn­chronic­ity: rather than view­ing your appli­ca­tion as a rigid sequence of processes, each being called in order as the result of a user action or sched­uled exe­cu­tion, com­po­nents pub­lish seman­tic events to which one or more other com­po­nents may be sub­scrib­ing and to which they may exe­cute logic (and per­haps pub­lish sub­se­quent events) in response. So, as you can prob­a­bly tell, like most buzz­words, “event-driven” really just tries to put a catchy name to an age-old con­cept; in this case “publish-subscribe” — but it does so, I would sug­gest, with the intent of help­ing devel­op­ers to get their head into the right space to think about how to actu­ally build sys­tems that are con­structed around such a mechanism.

In Java-land, we are for­tu­nate to have a sta­ble API with a num­ber of proven imple­men­ta­tions for publish-subscribe mes­sag­ing; namely: JMS, my favourite provider for which is Apache’s ActiveMQ. This is all well-and-good for han­dling our needs in this area on the server-side, but what about the user (sit­ting there with their browser) as an actor in this sys­tem? This ques­tion becomes very impor­tant when you con­sider that, described in abstract terms, the pri­mary use case, in my expe­ri­ence, for event-based mes­sag­ing sys­tems comes when you have users sub­mit­ting batches of work that demand inten­sive pro­cess­ing and require noti­fi­ca­tion of its completion.

In the past, one might have han­dled such sit­u­a­tions syn­chro­nously: the user would make the request through their browser, they would then sit and watch the “wheel of death” go around and around until, even­tu­ally, hope­fully, the request would be com­pleted. Of course, as soon as you got a few users doing this at the same time, your server became over­loaded and the whole thing ground to a halt. So you might have got as far as mak­ing this pro­cess­ing asyn­chro­nous, but then user instead would have to resort hit­ting the refresh but­ton and going “Is it done yet? Is it done yet? Is it done yet?” … or, worse, you would resort to that bête noire of office “noti­fi­ca­tion meth­ods”: email. Yuck!

Nowa­days, this kind of behav­iour is totally unnec­es­sary. Mak­ing the browser part of your event-driven sys­tem is a dod­dle and requires zero-alterations to your exist­ing JMS-based code (assum­ing that is what you are using). In this post, I would like to pro­vide a really basic exam­ple of how to send and receive JMS mes­sages using Active MQ’s Ajax sup­port. (Whilst there are bet­ter solu­tions on the hori­zon — most notably, in my view, Web­Sock­ets and STOMP — I have found them more than a lit­tle bit painful to inte­grate with exist­ing code to date, so I will focus on the more sta­ble solu­tion for the time being.)

Let’s begin by assum­ing that we have an appli­ca­tion that con­tains 1 “pro­cess­ing queue” and 1 “event topic”. With ActiveMQ, I can setup both a mes­sage bro­ker, the queue and the topic in a Spring appli­ca­tion con­text file:

    <amq:topic id="eventTopic" name="topic.events" />

    <amq:queue id="processingQueue" name="queue.process" />

    <amq:connectionFactory id="connectionFactory" brokerURL="vm://localhost" />

    <bean id="eventTopicTemplate" class="org.springframework.jms.core.JmsTemplate">
        <property name="connectionFactory" ref="connectionFactory" />
        <property name="defaultDestinationName" value="topic.events" />
    </bean>

Now, let’s say we have some kind of “ser­vice” that is respon­si­ble for doing this “inten­sive” asyn­chro­nous pro­cess­ing. The ser­vice will pick up mes­sages from the pro­cess­ing queue and, when the pro­cess­ing is com­plete, it will pub­lish a mes­sage to any inter­ested sub­scribers on the event topic say­ing that pro­cess­ing is com­plete. The API might look some­thing like this:

package com.christophertownson.foo;

public interface SomeService {

    void doProcessing(String msg);

}

For now, let’s make the most basic imple­men­ta­tion possible:

@Service("someService") public class SomeServiceImpl implements SomeService {

    private final JmsTemplate jms;

    @Autowired public SomeServiceImpl(@Qualifier("eventTopicTemplate") JmsTemplate eventTopicTemplate) {
        jms = eventTopicTemplate;
    }

    public void doProcessing(String msg) {
        jms.convertAndSend("We did something with your message: " + msg);
    }

}

All our imple­men­ta­tion does for now is send a mes­sage to the event topic to indi­cate that pro­cess­ing is “com­plete”. Any num­ber of com­po­nents may sub­scribe to this topic. In our case, the browser will ini­ti­ate pro­cess­ing requests by send­ing mes­sages to the pro­cess­ing queue and will sub­scribe to the event topic so that it can be noti­fied when the pro­cess­ing is com­plete and dis­play this to the user. But first we will wire up our ser­vice imple­men­ta­tion so that it will receive mes­sages sent to the pro­cess­ing queue:

    <jms:listener-container connection-factory="connectionFactory">
        <jms:listener destination="queue.process" ref="someService" method="doProcessing" />
    </jms:listener-container>

And that is it for the JMS-stuff on the server-side. Now let’s turn our atten­tion to the browser. I am going to assume that our user is inter­act­ing with our appli­ca­tion with some kind of “dash­board” web page. I am going to make my extremely beau­ti­ful (not!) dash­board look like this:

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8" />
        <title>Dashboard : Browser JMS Demo</title>
        <link rel="stylesheet" href="/resources/style.css" />
        <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.6.3/jquery.min.js" type="text/javascript"></script>
        <script type="text/javascript" src="/resources/amq_jquery_adapter.js"></script>
        <script type="text/javascript" src="/resources/amq.js"></script>
        <script type="text/javascript" src="/resources/behaviour.js"></script>
    </head>
    <body>
        <header>
            <hgroup>
                <h1>Browser JMS Demo Dashboard</h1>
            </hgroup>
        </header>
        <section id="send">
            <hgroup>
                <h1>Make something happen</h1>
                <form id="sendMessage" action="/process" method="post">
                    <fieldset>
                        <legend>Send a message to the processing queue.</legend>
                        <label for="msg">Message</label> <input type="text" id="msg" name="msg" maxlength="255" /> <input type="submit" value="Send" />
                    </fieldset>
                </form>
            </hgroup>
        </section>
        <section id="receive">
            <hgroup>
                <h1>Messages sent to the event topic</h1>
                <ol id="messages">
                </ol>
            </hgroup>
        </section>
        <footer>
            <p>Copyright &copy; Christopher Townson 2011</p><!-- yeah, that's right maaan: hands off my design! :p -->
        </footer>
    </body>
</html>

Note the use of the ActiveMQ JavaScript files on lines 8–9: these are pro­vided as part of any ActiveMQ dis­tri­b­u­tion pack­age, if you were won­der­ing where they come from. “Adapters” are also pro­vided for Pro­to­type and Dojo, but I am using JQuery in my exam­ple so opted for the JQuery adapter.

I will assume for now that you know how to setup some kind of con­troller to deliver this HTML file and will not go into the details of that here. As usual, I am using a Spring-Velocity setup to achieve this. How­ever, in this instance, that is not impor­tant because “con­trollers” are not really being used at all in this example.

So, let’s take a look at that behaviour.js file ref­er­enced on line 10. This is where I am putting the impor­tant stuff to actu­ally send and receive JMS messages …

function init() {
    var amq = org.activemq.Amq;
    amq.init({
        uri: 'amq',
        logging: true,
        timeout: 20
    });

    amq.addListener('theBrowser', 'topic.events', function(msg) {
        $('#messages').append('<li>' + msg.textContent + '</li>')
    });

    $('#sendMessage').submit(function() {
        var msg = $('#msg').val();
        amq.sendMessage('queue.process', msg);
        $('#sendMessage').after('<p id="ack">Sent message: "' + msg + '"');
        $("#ack").fadeOut(2000, function () {
            $("#ack").remove();
        });
        return false;
    });
}

$(document).ready(init);

Let’s go through this step-by-step …

  1. On lines 2–7, we instan­ti­ate and ini­tialise ActiveMQ.
  2. On lines 9–11, we setup lis­ten­ing for mes­sages on the event topic, pass­ing a client ID, the name of the des­ti­na­tion we want to lis­ten to (i.e. the event topic in this case) and a call­back to exe­cute when a mes­sage is received … I am just going to add the mes­sage to a list on the page for now.
  3. On lines 13–21, I inter­cept mes­sage form sub­mis­sions, send­ing a JMS mes­sage rather than con­tin­u­ing with a nor­mal HTTP form sub­mis­sion. As a lit­tle piece of sugar, I have a fade ani­ma­tion alert to say that the mes­sage has been sent after you sub­mit the form. (In real life, you should make this degrade grace­fully by hav­ing a han­dler on the server-side to send the JMS mes­sage and mak­ing your JavaScript code per­form an asyn­chro­nous request to that han­dler, rather than send­ing the mes­sage directly as I am doing here.)

The final piece of the jig­saw puz­zle is the ActiveMQ AjaxServlet. This is part of the activemq-web, which you will need to declare as a depen­dency in addi­tion to the usual activemq-core. Then we just need to declare this servlet in our web.xml as follows:

    <servlet>
        <servlet-name>amq</servlet-name>
        <servlet-class>org.apache.activemq.web.AjaxServlet</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>

    <servlet-mapping>
        <servlet-name>amq</servlet-name>
        <url-pattern>/amq/*</url-pattern>
    </servlet-mapping>

And, Lo and behold! Our dash­board page receives event noti­fi­ca­tions in the page with­out reload. Isn’t JavaScript magic?! Now, of course, there are all kinds of sub­tleties and sophis­ti­ca­tions that one can intro­duce, includ­ing mes­sage fil­ter­ing, brows­ing, con­ver­sion and so on. How­ever, first you need to con­sider care­fully what type of mes­sages you are send­ing to the browser. Ide­ally, in my view, these should nor­mally only con­sti­tute events that lead a call­back han­dler to update the data dis­played about some under­ly­ing per­sis­tent entity (because you don’t get these mes­sages again when the page is reloaded, obvi­ously). Also, there is much to con­sider about the for­mat of such mes­sages: using JSON as a JMS text mes­sage for­mat, rather than map or object mes­sages, makes a lot of sense when you look at things from this angle because it is then very easy to serialize/deserialize mes­sages in both the browser and in a MessageConverter in your Java code. Finally, as with all event-driven sys­tems, pay very close atten­tion to the point in time at which mes­sages are sent. For exam­ple, if, as I sug­gest, you are using these events to trig­ger an update to dis­played per­sis­tent state, you need to be damned sure that this state has actu­ally been updated before you pub­lish the “update your dis­play” event.

You can down­load the Maven project with the com­plete code for this demo. You can, of course, dis­trib­ute and change it in any way you like … but do share the results if you do. Enjoy!