Friday, August 28, 2009

Hudson for CI - Tips, Tricks and insights

Just started using Hudson recently and I'm wowed! It's head and shoulders above CruiseControl and things that I like a lot are

  1. Snappy web based config - felt great that I could set up a CI build with essentially the repo path alone

  2. Plugin system!

  3. Deep maven2 integration (though read on below that this isnt always what works)

  4. Trending data OOB - essentially giving you nice charts about how your build is doing over time


Now that I've said all the very nice things about it, here's a few things that were hard to figure out/or weren't immediately apparent. If your maven builds aggregates modules then you'll find the experience a bit challenging

  1. The generated site doesnt work: Basically, the link is to one of the modules' site instead of a link to the parent project. This apparently is a known issue and the solution on hudson user list is to run the site:deploy goal and have a link in the project description to point to that url

  2. Code coverage: none of the coverage tools (EMMA, clover etc) support code coverage over a multi module build. Since coverage is very important to me, I eventually resorted to having separate build jobs instead of using the default multi module support. Here's how my svn structure looks
    [sourcecode language="sh"]
    /trunk/basebuild #contains the parent pom
    /trunk/project1 # pom refers to ../basebuild/pom.xml
    /trunk/project2 # ditto here
    [/sourcecode]

    With the directory structure above, there are build jobs for project1 and project2. Each build job checks out both the project folder (/trunk/project1) and the basebuild folder so that the POM references work.
    One undesirable effect of this set up is that if project 2 depends on project 1, then project 1 build will have to install the artifact to the local repo for the project2 build to work.

  3. Findbugs plugin - Running maven builds with findbugs configured did a Out of Memory (OOM) and failed the build. I tried setting MAVEN_OPTS to -Xmx512M at a bunch of places and nothing worked. Eventually, it turned out that the right place to specify it is in the Hudson COnfigure job page in the build section!

  4. Violations plugin - This is a great little hudson plugin. However, I couldnt get this to work with a inherited POM setup above. Eventually resorted to using Findbugs and PMD hudson plugins individually.


I should mention that I'm running hudson 1.321 with the latest plugins. If you have any tips to share on running hudson - please do drop a link in the comments. Overall, a great big 'thank you' to the Hudson folks!

Wednesday, August 26, 2009

Recipe: Unit testing Apache CXF RESTful services

Recently, decided to use Apache CXF to expose a service with a RESTful API. Part of the reason for choosing REST had more to do with the fact that the client is going to be a mobile client. These days, though mobile devices stacks have come a long way and provide SOAP clients, it still seems prudent to not depend on a whole slew of technologies where plain 'ole HTTP and JSON might do the trick.
As I started exploring CXF, I liked the JAX-RS implementation and decided to go ahead with it - however, almost immediately, hit a snag when I went on to write test cases. Apache CXF documentation is not quite there and things do require some investigation - at least initially till you get a hang of the framework. As it took time to figure out the solution, it makes sense to share it on blogosphere. Here's how to go about writing unit tests:

Firstly, the service and the service implementation:

[sourcecode language="java"]
package com.aditi.blackberry.web;

import javax.ws.rs.FormParam;
import javax.ws.rs.GET;
import javax.ws.rs.HeaderParam;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.Response;

@Path("/chat")
@Produces("application/json")
public interface ChatWebService {

@POST
@Path("connect")
public Response connect(@FormParam("user")String username, @FormParam("pass")String password);
}
[/sourcecode]

The service implementation:

[sourcecode language="java"]
package com.aditi.blackberry.web;

import javax.ws.rs.Produces;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.Response.Status;

@Produces("application/json")
public class ChatWebServiceImpl implements ChatWebService {

public Response connect(String username, String password) {
if(username ==null || "".equals(username) ||
password ==null || "".equals(password)) {
return Response.status(Status.BAD_REQUEST).build();
}
String[] response = {username, password};
return Response.ok(response).build();
}
}
[/sourcecode]

The corresponding spring context xml (applicationContext.xml) is:

[sourcecode language="xml"]

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p"
xmlns:aop="http://www.springframework.org/schema/aop" xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:jaxrs="http://cxf.apache.org/jaxrs" xmlns:cxf="http://cxf.apache.org/core"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://www.springframework.org/schema/aop
http://www.springframework.org/schema/aop/spring-aop-2.5.xsd
http://www.springframework.org/schema/tx
http://www.springframework.org/schema/tx/spring-tx-2.5.xsd
http://cxf.apache.org/core http://cxf.apache.org/schemas/core.xsd
http://cxf.apache.org/jaxrs http://cxf.apache.org/schemas/jaxrs.xsd">





























[/sourcecode]

A few things to note here - logging is turned on using interceptors and the jaxrs server is defined. I'm also using flexJson to convert arbitrary objects to json - so a MessageBodyWriter bean is also injected into the jaxrs server node. The most important thing is that we havent included either the cxf-servlet.xml config for the cxf-extension-http-jetty.xml. Essentially, what we want to do is for the actual build, include cxf-servlet.xml and for the test runs, run the service on the bundled jetty server.

So, go ahead and define a applicationContext-web.xml:

[sourcecode language="xml"]
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">




[/sourcecode]

This is the context xml that we'll provide to the ContextLoaderListener in our web.xml.

For the test cases, define applicationContext-test.xml - this is the context xml which we'll load from the test cases.

[sourcecode language="xml"]
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">









[/sourcecode]

As you see, we also define a jaxrs:client for the test context xml.

There's one final issue to address - which is that we would ideally like the urls we use to access the service to be the same. The spring jaxrs:server binding takes an address attribute which defines the url the service is hosted on. For deployment onto an external container, this takes the form of "/myservice" - a path element relative to the context location. For the internal jetty hosted service, it takes the full http path (http://localhost:port/my/path/to/service). The easiest way is to have this set using a property reference in spring and have the applicationContext-web.xml and applicationContext-test.xml load different property files as shown in above.

For completeness, here's the web.xml:

[sourcecode language="xml"]

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd">

CXF REST Example
CXF REST Example


contextConfigLocation
classpath:/applicationContext-web.xml


org.springframework.web.context.ContextLoaderListener



CXFServlet
org.apache.cxf.transport.servlet.CXFServlet
1



CXFServlet
/*



[/sourcecode]

And finally, here's a junit test case:

base class:

[sourcecode language="java"]
package com.aditi.blackberry.web;

import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "classpath:/applicationContext-test.xml" })
public abstract class AbstractApiTest {

@Autowired
@Qualifier("chatclient")
protected ChatWebService proxy;
}
[/sourcecode]

A test case for the connect API:

[sourcecode language="java"]
package com.aditi.blackberry.web;

import javax.ws.rs.core.Response;
import javax.ws.rs.core.Response.Status;

import org.junit.Assert;
import org.junit.Test;

public class ConnectApiTest extends AbstractApiTest{
@Test
public void testConnect() {
Response resp = proxy.connect("raghu", "password");
Assert.assertTrue(resp.getStatus() == 200);
System.out.println(resp.getEntity().toString());
}
}
[/sourcecode]

Thursday, January 01, 2009

PIL vs Imagemagick

Decided that I want to timestamp my photo collection with the date from the exif data. Many digicams have an option to do this - unfortunately, my Panasonic DMC-LZ8 doesn't seem to do this. I knew imagemagick would do the trick, but thought it would be a good time to play around with PIL and python.

Here's my PIL effort - functional, but one that came with quite some amount of googling and trying to make sense of the PIL documentation which is inadequate at best.

[sourcecode language="python"]

from PIL import Image
from PIL import ImageFont, ImageDraw
from PIL.ExifTags import TAGS
from os.path import basename, dirname,join
import logging
import sys
import datetime
import time

# Important: I set out to write the image annotation in PIL - there's one serious drawback though. When saving
# the image, the exif data is'nt preserved.

logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s %(levelname)s %(message)s')
logger = logging.getLogger()
logger.level = logging.DEBUG

def readExif(image):
    info = image._getexif()
    ret ={}
    for tag,value in info.items():
        ret[TAGS.get(tag,tag)] = value
    dt = datetime.datetime (*time.strptime (ret['DateTime'],"%Y:%m:%d %H:%M:%S")[0:6])
    ret['DateTime'] = dt
    return ret

def annotateImage (file):
    i = Image.open(file)
    font = ImageFont.truetype("/usr/share/fonts/truetype/ttf-dejavu/DejaVuSans-Bold.ttf", 36)
    exif = readExif(i)
    draw = ImageDraw.Draw(i)
    width, height = i.size
    draw.text((width * 0.7, height - 100),exif['DateTime'].strftime("%a %d-%b-%Y  %l:%M %p"), font=font, fill='orange')
    outfile = join(dirname(file), "Ann_" + basename(file))
    i.save (outfile, quality=98)
    logger.debug (outfile + " saved")

if __name__== "__main__":
    logger.debug ("getting exif for " + sys.argv[1])
    for file in sys.argv[1:]:
        logger.debug ("Annotating " + file)
        annotateImage(file)

[/sourcecode]

Unfortunately, PIL has a fatal flaw - you can annotate the image and save it - but then the saved image doesn't retain the original image's exif metadata. I also tried the exiv2 library, but couldnt figure out a way to load the image, annotate it and then copy over the metadata. Googling around didn't turn up any intersting solutions - so if any of you have any ideas, please share.

Meanwhile, as I was getting tired of coaxing PIL to do what I want, I just wrote a a little bash script to do the same in imagemagick. Its as painless as it can be, comes with excellent documentation, hardly any gotchas, a world of options in case you feel creative and the job gets done in 10 mins. Here's the bash script below.

#! /bin/bash
# script adds a black 18px bottom border to the pic with the Exif datetime tag
# no safety checks :). Original pics are left untouched.
while [ "x$*" != "x" ]
do
file=$1;
shift;
outfile="$(dirname "$file")/Ann_$(basename "$file")"
echo $outfile
echo $file
date=$(identify -verbose "$file" | grep 'DateTime:'| sed 's/ Exif:DateTime: //;s/:/-/;s/:/-/')
date="$(date -d "$date" +"%a %d-%b-%Y %l:%M %p")"
convert "$file" -size 1x18 xc:Black -fill White -background Black -append -gravity Southeast -draw "text 0,0 '$date'" "$outfile"
done

Overall, the experience left me dissappointed and dissatisfied with PIL.