Maven, Artifactory and Hudson for Oracle OSB Continuous Integration

In the previous blog we discussed about using Ant and Hudson/Jenkins for continuous integration. What hasn't been explicitly discussed is about dependency management, that Maven can handsomely handle.

The benefits of using Maven instead of Ant:
1. standardization following best practices (e.g. directory structure) that leads to shorter/simpler configuration file (pom.xml), less maintenance, and higher reusability
2. transitive dependency management: Maven will find and solve the conflicts of the libraries needed. Perhaps you know this concept already if you've used ivy framework with Ant, but this concept is central in Maven so that lots of innovations has been implemented regarding this feature (e.g. enterprise repositories).
For example I just made adjustment and commited StudentRegistrationService-ver2.0 which depends on LDAPService-ver2.0 and hibernate-ver3.jar. When I deploy the StudentRegistrationService-ver2.0, Maven will include also the LDAPService-ver2.0 and hibernate-ver3.jar from a enterprise repository that stores all the libraries used in your company. If the build & test processes success, the artifact of my new StudentRegistrationService-ver2.0 will be included in the repository, so other services which consume my service will be able to use this version 2.0 of my service. Strong enough, you can specify the version dependencies using ranges (e.g. min version, max version), so I can specify that my service depends on LDAPService max version 2.0 (since I don't support the new interface of the newer LDAPService yet) and also depends on PaymentService min version 3.1.1 since there is a payment bug in the PaymentService version lower than 3.1.1. Here is an example of defining these dependencies in the pom.xml of the StudentRegistrationService:

<dependency>
<groupId>TUD</groupId>
<artifactId>LDAPService</artifactId>
<version>[0,2.0]</version>
</dependency>

<dependency>
<groupId>TUD</groupId>
<artifactId>PaymentService</artifactId>
<version>[3.1.1,)</version>
</dependency>
An illustration about how it works:


1. Using Hudson/Jenkins to let the svn commit trigger the Maven build
Please see the previous blog about how to install and setup Hudson/Jenkins.

For this example, I specify Hudson/Jenkins to pool the svn server every minute (set by the schedule "* * * * *" using cron format). When there is a new commit in the mysvnproject, the Maven "install" goal (along with its previous lifecycles phases i.e. compile, test) will be invoked.


2a. Checkout using mvn-scm plugin

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-scm-plugin</artifactId>
<version>1.1</version>
<configuration>
<username>username</username>
<password>password</password>
</configuration>
<execution>
<id>checkout</id>
<configuration>
<connectionUrl>mysvnserver</connectionUrl>
<checkoutDirectory>mysvndir</checkoutDirectory>
<excludes>folder2exclude/*</excludes>
</configuration>
<phase>compile</phase>
<goals>
<goal>checkout</goal>
</goals>
</execution>
</plugin>
2b. Build the OSB, you can use the same ant task as in my previous blog, wrapped with maven-antrun-plugin.

3a. Obtain the dependencies from the repositories, using dependency:copy-dependencies or dependency:copy.

3b. Deploy the OSB project and its dependencies, you can use the same ant task as in my previous blog, wrapped with maven-antrun-plugin.

4. Run SOAP UI web service test using maven-soapui-plugin (or alternatively you can use testrunner.bat / testrunner.sh similar to my other blog)

<plugin>
<groupId>eviware</groupId>
<artifactId>maven-soapui-plugin</artifactId>
<version>3.0</version>
<executions>
<execution>
<phase>test</phase>
<id>soapuitest</id>
<configuration>
<projectFile>${mysoapuitestfile}</projectFile>
<outputFolder>${testreportdir} </outputFolder>
<junitReport>true</junitReport>
<exportwAll>true</exportwAll>
<printReport>true</printReport>
<settingsFile>${soapuisettingfile}</settingsFile>
</configuration>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
5. Archieving the artifact using an enterprise repository.
The benefit of using enterprise repository:
• your developers don't have to search, download and install the libs manually
• it's faster & more reliable than downloading the libs from internet, the concept is similar to proxy server that cache the internet.
• it will store the artifacts of your company projects from ant/maven builds, so they will be readily available for testing and shipping
• web administration interface, search, backup, import/exports

I chose Artifactory as enterprise repository since it has more features than other products, such as: xpath search inside XML/POM, hudson integration (e.g. for build promotion), conn to ldap, cloud (saas) possibility, easy install (running in an embedded jetty server or as service in Windows/Linux.)
You can use Hudson artifactory plugin to integrate Artifactory to Hudson/Jenkins process.

I use 3 local repositories inside your Artifactory for different library categories:
• open source/ibibliolibraries (e.g. apache common jars), the Artifactory can download these automatically
• proprietary libraries (e.g. oracle jdbc jar), you need to install these manually (e.g. via Artifactory web interface)
• company libraries, you need to install these manually or via Hudson build as done in this example. For the company repository, I define such that the repository cab handle both the release/stable version (e.g. the PaymentService-ver3.1.1 which is already well tested and approved) as well as the snapshot version (e.g. I am not finished with my StudentRegistrationService-2.0 yet but I want to make it available for other projects which depend on it). For example in the artifactory.config.xml:

<localRepository>
<key>tud-repo</key>
<description>mycompany-libs</description>
<handleReleases>true</handleReleases>
<handleSnapshots>true</handleSnapshots>
</localRepository>

<localRepository>
<key>ibiblio-repo</key>
<description>stable-opensource-libs</description>
<handleReleases>true</handleReleases>
<handleSnapshots>false</handleSnapshots>
</localRepository>
You need to declare these repositories in your pom.xml (or with similar approach in settings.xml for all of your projects):

<repositories>
<repository>
<id>ibiblio-repo</id>
<url>http://myreposerver:port/artifactory/repo</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
<repository>
<id>tud-repo</id>
<url>http://myreposerver:port/artifactory/repo</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>ibiblio-repo</id>
<url>http://myreposerver:port/artifactory/repo</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
<pluginRepository>
<id>tud-repo</id>
<url>http://myreposerver:port/artifactory/repo</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories&gt
For the sake of clarity there are some details omitted from this blog. The concepts in this blog work also for non OSB projects (e.g. Java/J2EE applications).

Any comments are welcome :)

---

See also: http://soa-java.blogspot.nl/2011/03/soa-continuous-integration-test.html


References:
• Setting Up a Maven Repository
• Comparison Maven repository: Archiva, Artifactory, Nexus
• Amis blog: Soapui test with maven

Continuous integration for Oracle OSB projects using Ant and Hudson / Jenkins

Continuous integration (CI) is pervasive. This doesn't come from nothing, CI has many benefits, such as to avoid last minute integration hell and to improve software quality


The principles of Continuous integration
1. every commit to the SCM should be build so that you'll have an early feedback if the build breaks
2. automate build for consistency
3. automate deployment for consistency
4. automate test of the build artifact for consistency
5. archive the build artifact so that it'll be readily available (e.g. for further test)
6. keep the reports & metrics from build & test

You can also add additional steps for example when this commit breaks the test you can reject the commit by rolling back the deployment (using undeploy wlst task) & roll back the svn commit using merge backward:
svn merge -r currentver:previousver ; svn commit

This blog will show how to achieve these steps using Ant and Hudson/Jenkins. We use Ant since in many organizations Ant is already well adopted compared with Maven. In another blog we will discuss how to achieve the same goal using Maven and artifact repository (e.g. Artifactory or Nexus) which are handier than Ant particularly with respect to dependency management.

How it works:
1.Using Hudson/Jenkins to let the svn-commit triggers the Ant build
I chose Hudson/Jenkins since it's easy to use, has good features, scalable and recommended by many people (including folks working at Oracle). You can see Jenkins as a new version of Hudson, Jenkins was created to avoid legal problem with Oracle when Kohsuke Kawaguchi, the Hudson's creator, left Sun/Oracle.
Installing Hudson/Jenkins is easy, in Windows it can be run as a Window service. Hudson/Jenkins contains an embedeed Winstone servlet engine, so you can also run it using
java -jar hudson.war --httpPort=aportnumber
To install Hudson/Jenkins in Weblogic you need to add the deployment descriptor weblogic.xml to solve classpath conflicts of certain jars (depend on which version you install), also Hudson/Jenkins will not work in Weblogic servers with SOA/OSB extensions due to some conflicting settings.
You can add some plugins for example locale (to set languages), svn related plugins, trac (to recognize trac tag e.g. fixed#), cvs/svn browser viewer, html report, scp artifact repository, promote builds, among others.

You need to set some configurations such as JDK location, svn connection, Ant location, Maven location, Junit test report location, artifacts location, SMTP/email server for notifications.

For this example, I specify Hudson/Jenkins to pool the svn server every minute (set by the schedule "* * * * *" using cron format). When there is a new commit in the mysvnproject, the Ant ("main" target) will be invoked.

2. Checkout the new commited svn project using svntask:

<target name="checkout">
<delete failonerror="false" includeemptydirs="true" quiet="true"
dir="${servicename}" />
<svn username="${subversion.user}" password="${subversion.password}">
<checkout url="${subversion.path}" destPath="${servicename}" />
</svn>
</target>

Build this code to an OSB project using ConfigExport

<target name="makeosbjar" depends="deletemetadata">

<!-- osb jar compile -->
<java dir="${osb.eclipse.home}"
jar="${osb.eclipse.home}/plugins/org.eclipse.equinox.launcher_1.0.201.R35x_v20090715.jar"
fork="true"
failonerror="true"
maxmemory="768m">
<jvmarg line="-XX:MaxPermSize=256m" />
<arg line="-application com.bea.alsb.core.ConfigExport" />
<arg line="-data ${workdir}" />
<arg line="-configProject ${osb.config.project}" />
<arg line="-configJar ${jardir}/${packagename}" />
<arg line="-configSubProjects ${servicename}" />
<sysproperty key="weblogic.home" value="${osb.weblogic.home}" />
<sysproperty key="osb.home" value="${osb.home}" />
</java>
</target>

3. Deploy the jar to the OSB server using WLST import (a python script)

<target name="deployOSB">
<wlst fileName="${import.script}" debug="true" failOnError="false"
arguments="${wls.username} ${wls.password} ${wls.server} ${servicename} ${jardir}/${packagename} ${import.customFile}">
<script> <!-- run these before import.py -->
adminUser=sys.argv[1]
adminPassword=sys.argv[2]
adminUrl=sys.argv[3]
passphrase = "osb"
project=sys.argv[4]
importJar=sys.argv[5]
customFile=sys.argv[6]
connect(adminUser,adminPassword,adminUrl)
domainRuntime()
</script>
</wlst>
</target>

4. Run the SOAPUI web service test and generate the junit test report:
<target name="soapui-test">
<exec executable="cmd.exe" osfamily="windows" failonerror="false">
<arg line="/c ${testrunner.bat} -j -freports ${soapui.test}"/>
</exec>

<junitreport todir="${testreportdir}">
<fileset dir="reports">
<include name="TEST-*.xml"/>
</fileset>
<report format="frames" todir="${testreportdir}/html"/>
</junitreport>
</target>


Example of Hudson/Jenkins output:

Hudson also sent email notifications:

For the sake of clarity, there are some details omitted from this blog (e.g. the ant classpath for lib needed: svnant, svnjavahl, svnclientadapter, xmltask), these details will be found in the build.xml. Please download the build.xml, build.properties and the import.py wlst here.

The concepts in this blog work also for non OSB projects (e.g. Java/J2EE applications).

Source: Steve's blogs http://soa-java.blogspot.com/

Any comments are welcome :)

---

The file download is made possible by OpenDrive


References:

Using the Oracle Service Bus Plug-ins for Workshop for WebLogic http://docs.oracle.com/cd/E13159_01/osb/docs10gr3/eclipsehelp/tasks.html
Using WLST http://docs.oracle.com/cd/E15051_01/wls/docs103/config_scripting/using_WLST.html
Biemond's blog http://biemond.blogspot.com/2010/07/osb-11g-ant-deployment-scripts.html
How-to-deploy-Hudson-to-weblogic http://jenkins.361315.n4.nabble.com/How-to-deploy-Hudson-to-weblogic-td3246817.html

Using unix scripts or tools (e.g. sed , awk, find) in Windows

Suppose you work in Windows platforms but you miss the power of sed, awk, find, grep, or other handy tools in Unix, or you want to reuse your Linux scripts? Just install cygwin, it's a kind of Linux terminal for Windows (a simplified explanation.)

In cygwin, your Windows drives are located in the /cygdrive. For example if you want to delete all ".svn" directories in D:/My Workspace/ in cygwin you can type:
cd /cygdrive/d/My\ Workspace
rm -rf $(find . -type d -name .svn)

Which SVN client in Eclipse

Everytime I (re)install Eclipse, I need to remind myself which svn client should I choose. Basically there are 2 popular svn plugins in Eclipse: subclipse and subversive. Some people said that subversive has more features and handier (e.g. for merging). But when I used subversive I noticed that I couldn't use other svn client (e.g. subclipse, tortoise) nor other versions of the subversive. Because of this compatibility issue, I chose to use subclipse.


---
Update Sep 2011:
I used subversive since by default subclipse uses javahl (to connect to svn) which has problem with 64bit OS. I can use also svnkit instead of javahl but I don't have time now for work arounds.

Which svn client do you prefer to use? Please share in the comments.

The motivation for this note
Following a suggestion in the "97 Things Every Software Architect Should Know" that I should note my rationale so I don't have to redo research again every time I have to make similar decisions.