DIY Annotations

Standard

Since Java 5 there have been annotations in Java.  I wanted to make my own annotation just to see what it takes.  However, I found out that they were just interfaces.

There is the rub

Interfaces have no teeth behind them.  Some piece of code has to implement it.  I figured this is where the rubber hits the road and I really find a way to do this.

To start, I would need a purpose

I picked one recent hot topic, caching.  I didn’t want to implement JSR 109(JCache) but I didn’t want to do the typical “Hello World” either.  I picked implementing two annotations, one without any parameters and one with a parameter.  I also needed a caching provider.  Might as well bring a real caching library to the mix if I am going to do this.  It also follows my design philosophy to use products/libraries to reach  a goal instead of home spinning everything.  After careful consideration, I chose hazelcast to be my caching engine.  It is the fastest on the market and it is free.

More Decisions

After my purpose was chosen, I still needed to find out how to put teeth behind them.  After some digging around I found two methods:

Reflection

Almost every time I have used reflection, I have felt sorry for making such a clunky piece of code.  Plus, to do it the way I would like, I would have to create my own framework.  Sounds like a lot of work for two annotations.

Aspect Oriented Programming(AOP)

This was a perfect fit for what I wanted to do.  AOP deals in reducing boilerplate code into a single place.  This would be convenient and dovetails into caching because caching breaks down into the following steps:

  1. Check to see if this situation was done before.
  2. If so:
    1. retrieve the stored result
  3. if not:
    1. run the function
    2. store the result
  4. return the result

That maybe an oversimplification but in a nut shell it is true.  Like in all things, the devil is in the details.

Meanwhile, Back at the AOP Ranch

While I knew AOP was the place for me, I did not know much about it.  I found that Spring has an AOP library and that a well known library is AspectJ.  AspectJ is unfamiliar to me and needs a runtime engine to work.  I am much more familiar with Spring so I picked it.  As I dug into Spring’s AOP, I found that I had to delve into AspectJ’s annotations so I was stuck with AspectJ in some form or fashion anyway.

New Concepts, New Vocabulary

Writing aspects aren’t like writing objects.  They are objects but not really so, of course, a new set of terms are needed.  The ones I used are in the Spring AOP Documentation

I really needed to read the page a couple of times to grasp what is being said.  One is highly recommended to do the same or the rest of the post is going to sound like gibberish.

What Makes the Pointcut and How to Advise it

The pointcut design was easy since I was only interested in methods that had the annotation.  The advise it needed was the around advice because I needed to be able to circumvent calling the method if there was a matching call already done.

Finally the Code

Maven Pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.darylmathison</groupId>
    <artifactId>annotation-implementation</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <spring.version>4.2.4.RELEASE</spring.version>
    </properties>

    <description>
        This project is an example of how to implement an annotation via Spring AOP.
    </description>

    <scm>
        <url>https://github.com/darylmathison/annotation-implementation-example.git</url>
        <connection>scm:git:https://github.com/darylmathison/annotation-implementation-example.git</connection>
        <developerConnection>scm:git:git@github.com:darylmathison/annotation-implementation-example.git</developerConnection>
    </scm>

    <issueManagement>
        <system>GitHub</system>
        <url>https://github.com/darylmathison/annotation-implementation-example/issues</url>
    </issueManagement>

    <dependencies>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context</artifactId>
            <version>${spring.version}</version>
        </dependency>

        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-aop</artifactId>
            <version>${spring.version}</version>
        </dependency>

        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-test</artifactId>
            <version>${spring.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.aspectj</groupId>
            <artifactId>aspectjweaver</artifactId>
            <version>1.8.8</version>
        </dependency>

        <dependency>
            <groupId>com.hazelcast</groupId>
            <artifactId>hazelcast</artifactId>
            <version>3.6</version>
        </dependency>

        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>

    </dependencies>

    <reporting>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-project-info-reports-plugin</artifactId>
                <version>2.7</version>
                <reportSets>
                    <reportSet>
                        <reports>
                            <report>dependencies</report>
                            <report>index</report>
                            <report>project-team</report>
                            <report>issue-tracking</report>
                            <report>scm</report>
                        </reports>
                    </reportSet>
                </reportSets>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-report-plugin</artifactId>
                <version>2.18.1</version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-javadoc-plugin</artifactId>
                <version>2.10.3</version>
                <reportSets>
                    <reportSet>
                        <reports>
                            <report>javadoc</report>
                            <report>test-javadoc</report>
                        </reports>
                    </reportSet>
                </reportSets>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jxr-plugin</artifactId>
                <version>2.5</version>
                <configuration>
                    <linkJavadoc>true</linkJavadoc>
                </configuration>
                <reportSets>
                    <reportSet>
                        <reports>
                            <report>jxr</report>
                            <report>test-jxr</report>
                        </reports>
                    </reportSet>
                </reportSets>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-changelog-plugin</artifactId>
                <version>2.3</version>
                <configuration>
                    <type>range</type>
                    <range>90</range>
                </configuration>
            </plugin>
        </plugins>
    </reporting>
</project>

The Annotations

CacheMe

Cute name for a caching annotation, right?

package com.darylmathison.ai.annotation;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

/**
 * Created by Daryl on 2/19/2016.
 */
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD})
public @interface CacheMe {
}

CacheMeNow

package com.darylmathison.ai.annotation;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

/**
 * Created by Daryl on 2/19/2016.
 */
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD})
public @interface CacheMeNow {
    String key();
}

Spring Configuration

I decided to use Java based configuration instead of XML like I normally use for a change of pace. The EnableAspectJAutoProxy annotation is key to getting Spring AOP to start working. I was beside myself until I read this about this little jewel. Sometimes it is the easiest thing that burns a day.

AppConfig

package com.darylmathison.ai.config;

import com.darylmathison.ai.cache.CacheAspect;
import com.darylmathison.ai.service.FibonacciService;
import com.darylmathison.ai.service.FibonacciServiceImpl;
import com.hazelcast.config.Config;
import com.hazelcast.config.EvictionPolicy;
import com.hazelcast.config.MapConfig;
import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.EnableAspectJAutoProxy;

import java.util.HashMap;
import java.util.Map;

/**
 * Created by Daryl on 2/20/2016.
 */
@Configuration
@ComponentScan(basePackages = "com.darylmathison.ai")
@EnableAspectJAutoProxy
public class AppConfig {

    @Bean
    public Map<String, Object> cache() {
        Config config = new Config();
        MapConfig mapConfig = new MapConfig();
        mapConfig.setEvictionPercentage(50);
        mapConfig.setEvictionPolicy(EvictionPolicy.LFU);
        mapConfig.setTimeToLiveSeconds(300);
        Map<String, MapConfig> mapConfigMap = new HashMap<>();
        mapConfigMap.put("cache", mapConfig);
        config.setMapConfigs(mapConfigMap);

        HazelcastInstance instance = Hazelcast.newHazelcastInstance(config);
        return instance.getMap("cache");
    }

    @Bean
    public FibonacciService fibonacci() {
        return new FibonacciServiceImpl();
    }

    @Bean
    public CacheAspect cacheAspect() {
        return new CacheAspect();
    }
}

Service Code

Classic Spring based design needs a service right? Because Spring uses proxies to implement their AOP, it is highly advised to define an interface for the annotated class to implement.

FibonacciService

package com.darylmathison.ai.service;

/**
 * Created by Daryl on 2/20/2016.
 */
public interface FibonacciService {

    long calculate(int rounds);

    long calculateWithKey(int rounds);
}

FibonacciServiceImpl

package com.darylmathison.ai.service;


import com.darylmathison.ai.annotation.CacheMe;
import com.darylmathison.ai.annotation.CacheMeNow;

/**
 * Created by Daryl on 2/20/2016.
 */
public class FibonacciServiceImpl implements FibonacciService {

    @Override
    @CacheMe
    public long calculate(int rounds) {
        return sharedCalculate(rounds);
    }

    @Override
    @CacheMeNow(key = "now")
    public long calculateWithKey(int rounds) {
        return sharedCalculate(rounds);
    }

    private static long sharedCalculate(int rounds) {
        long[] lastTwo = new long[] {1, 1};

        for(int i = 0; i < rounds; i++) {
            long last = lastTwo[1];
            lastTwo[1] = lastTwo[0] + lastTwo[1];
            lastTwo[0] = last;
        }

        return lastTwo[1];
    }
}

AOP Stuff

This is heart of the annotation implementation. Everything else is support to do the source that follows.

SystemArch

According to Spring documentation, centralizing the pointcut definitions are a good idea.

package com.darylmathison.ai.cache;

import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Pointcut;

/**
 * Created by Daryl on 2/20/2016.
 */
@Aspect
public class SystemArch {

    @Pointcut("@annotation(com.darylmathison.ai.annotation.CacheMe)")
    public void cacheMeCut() {

    }

    @Pointcut("@annotation(com.darylmathison.ai.annotation.CacheMeNow)")
    public void cacheMeNowCut() {

    }
}

CacheAspect

The Around annotations take the full method names of the pointcut class to define what to advise. The advice for the CacheMeNow annotation includes an extra condition so the annotation can be defined so the key parameter can be read. There is a design bug in CacheMeNow that is revealed in the test code.

package com.darylmathison.ai.cache;

import com.darylmathison.ai.annotation.CacheMeNow;
import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.springframework.beans.factory.annotation.Autowired;

import java.util.Map;

/**
 * Created by Daryl on 2/20/2016.
 */
@Aspect
public class CacheAspect {

    @Autowired
    private Map<String, Object> cache;

    @Around("com.darylmathison.ai.cache.SystemArch.cacheMeCut()")
    public Object simpleCache(ProceedingJoinPoint proceedingJoinPoint) throws Throwable {
        StringBuffer keyBuffer = new StringBuffer();
        for(Object o: proceedingJoinPoint.getArgs()) {
            keyBuffer.append(o.hashCode());
        }
        String key = keyBuffer.toString();
        Object ret = cache.get(key);
        if(ret == null) {
            ret = proceedingJoinPoint.proceed();
            cache.put(key, ret);
        }
        return ret;
    }

    @Around("com.darylmathison.ai.cache.SystemArch.cacheMeNowCut() && @annotation(cacheMeNow)")
    public Object simpleCacheWithParam(ProceedingJoinPoint proceedingJoinPoint, CacheMeNow cacheMeNow) throws Throwable {
        Object ret = cache.get(cacheMeNow.key());
        if(ret == null) {
            ret = proceedingJoinPoint.proceed();
            cache.put(cacheMeNow.key(), ret);
        }
        return ret;
    }
}

Test Code

Driver code to show that the annotations do cause caching.

FibonacciTest

package com.darylmathison.ai.service;

import com.darylmathison.ai.config.AppConfig;
import org.junit.Assert;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

/**
 * Created by Daryl on 2/20/2016.
 */
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(classes = {AppConfig.class})
public class FibonacciTest {

    private static final int ROUNDS = 12;
    private static final long ANSWER = 377;

    @Autowired
    private FibonacciService fibonacci;

    @org.junit.Test
    public void testCalculate() throws Exception {
        long start = System.currentTimeMillis();
        Assert.assertEquals(ANSWER, fibonacci.calculate(ROUNDS));
        long middle = System.currentTimeMillis();
        Assert.assertEquals(ANSWER, fibonacci.calculate(ROUNDS));
        long end = System.currentTimeMillis();
        Assert.assertTrue((end - middle) < (middle - start));
    }

    @org.junit.Test
    public void testCalculateWithKey() throws Exception {
        Assert.assertEquals(ANSWER, fibonacci.calculateWithKey(ROUNDS));
        // This test should not pass
        Assert.assertEquals(ANSWER, fibonacci.calculateWithKey(13));
    }
}

Conclusion

Annotations do not have to be hard to implement. Using AOP programming, I was able to implement two annotations with little coding.  The code can be found here.

The One Thing They Should have Taught in School

Standard

One of the reasons I got into computers is because computers are not people.  The other reason is that life was boring and computers are challenging.  Part of the reason I got through college is because I was around people that understood computers better than I and I could learn from them.  As I have matured in my career I have found one glaring and now obvious fact that I wish was pounded into me during college.  It was indirectly taught with core classes but I needed a literal tell me straight statement.  If could talk to my former, younger self, here is what would I say.

Computers and People Coexist

You cannot get away from people no matter hard one tries.  One can surround themselves with computers and screens and it will not make a difference.  Any project worth doing nowadays is usually done in a team.  This means meetings, asking and answering questions and if one is blessed, training and mentoring.

The Ultimate User is a Person

After all the fancy multi-tiered design, implementation and the various types of testing done, that project will have a human in the loop making decisions and inputting data.  That means requirements come from … humans because humans are going to use it.  Requirements require understanding.  Understanding requires questions.  Questions require answers.  Question and answers require a time and place to ask and receive information.  Assigning a time and place to gather information is a meeting.  Meetings require people.

Teams Require People to Function

Teams are split up into areas of expertise.  These experts need each other to reach the same goal.  Reaching that goal requires communication.  Communication typically requires people.

Want a Job?

To become a professional software developer, one needs employment.  To become employed, one needs an interview.  Interviews are conversations convincing another human being about how much an expert one is.  One of the interviewer’s jobs is to convince that the candidate should work for the employer he/she represents.  Selling yourself means knowing a particular type of communication.  Communication could be between computers but not in an interview.  Interviews require people interacting.

Networking

I cannot tell one how many jobs I have landed because someone recommended me.  That is because I got my head out of the computer and talked computers with a teammate, enough said.

Well, There it is

That is what I would have told my college self if I could.

Beginner’s Guide to Hazelcast Part 7

Standard

This is a continuation of a series explaining how to use Hazelcast.  If one has not read the other six posts, please go to the Table Of Contents and read the other posts.

A Different Breed of Map

Hazelcast’s MultiMap breaks the normal mold of using java.util.Collection interfaces that have been used in former posts.  In fact, the concept of a MultiMap breaks the idea of a map altogether in my opinion.  While normal maps associate one key to one value, MultiMaps can map multiple values to the same key.  That is a really important concept, multiple values to the same key.  Values can be stored in two different collections, set or list.  These collections act like the collections of java.util.Collections library.

Is it Safe?

MultiMaps have a method to their madness.  In a normal map, multiple values per key can be stored but it has to be done manually.  That means getting a collection out of storage, doing any changes and then putting the collection back into storage.  This can be problematic for thread safety because the prior steps need to be done atomically or there is the possibility of stale or inconsistent data being read by other threads.  MultiMaps help with this problem by offering the following services:

  • One can add a value via a single put operation.
  • One can lock an entry by the key.  This is key (pun intended) because this means the developer does not have to keep track of a separate lock per entry.

Example

This example is a little different because I used Maven’s failsafe plugin as the main engine when running the examples.  Yes, I wrote two examples because I wanted to show two different ways of using a MultiMap.  One way is every thread getting their own playground, being assigned an unique key, or being assigned a shared playground or all of the threads sharing the same key.  This also is an example of how Hazelcast’s IdGenerator can be used as a method of creating thread safety in an application.

Pom File

Remember, this example code takes advantage of Apache’s Failsafe Maven plugin.  The failsafe plugin aids in automated integration tests by not killing the build at the first failure.  It is a fork of the surefire plugin.  I also have been experimenting with the reporting that is available with Maven.  Type “mvn site” at the command line and it will generate a website.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0>

    <groupId>com.darylmathison>
    <artifactId>hazelcast-multimap-example>
    <version>1.0-SNAPSHOT>
    <description>Examples of using Hazelcast Multimap>
    <developers>
        <developer>
            <name>Daryl Mathison>
            <id>dmathison>
            <roles>
                <role>developer>
            >
        >
    >

    <scm>
        <connection>scm:git:https://github.com/darylmathison/hazelcast-multimap-example.git>
        <url>https://github.com/darylmathison/hazelcast-multimap-example>
    >

    <properties>
        <maven.compiler.source>1.8>
        <maven.compiler.target>1.8>
        <project.build.sourceEncoding>UTF-8>
    >

    <dependencies>
        <dependency>
            <groupId>com.hazelcast>
            <artifactId>hazelcast>
            <version>3.4.2>
        >
        <dependency>
            <groupId>junit>
            <artifactId>junit>
            <version>4.12>
            <scope>test>
        >
    >

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins>
                <artifactId>maven-failsafe-plugin>
                <version>2.18.1>
                <executions>
                    <execution>
                        <goals>
                            <goal>integration-test>
                            <goal>verify>
                        >
                    >
                >
            >
        >
    >

    <reporting>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins>
                <artifactId>maven-project-info-reports-plugin>
                <version>2.7>
                <reportSets>
                    <reportSet>
                        <reports>
                            <report>dependencies>
                            <report>index>
                            <report>project-team>
                            <report>scm>
                        >
                    >
                >
            >
            <plugin>
                <groupId>org.apache.maven.plugins>
                <artifactId>maven-javadoc-plugin>
                <version>2.10.3>
                <reportSets>
                    <reportSet>
                        <reports>
                            <report>javadoc>
                            <report>test-javadoc>
                        >
                    >
                >
            >
            <plugin>
                <groupId>org.apache.maven.plugins>
                <artifactId>maven-surefire-report-plugin>
                <version>2.18.1>
            >
            <plugin>
                <groupId>org.apache.maven.plugins>
                <artifactId>maven-jxr-plugin>
                <version>2.5>
                <configuration>
                    <linkJavadoc>true>
                >
                <reportSets>
                    <reportSet>
                        <reports>
                            <report>jxr>
                            <report>test-jxr>
                        >
                    >
                >
            >
        >
    >
>

MultimapAccessThread

This is the base class for each of the access type threads.

package com.darylmathison.multimap;

import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.HazelcastInstanceAware;
import com.hazelcast.core.MultiMap;

import java.io.Serializable;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;

/**
 * Abstract class to access MultiMap.
 */
public abstract class MultiMapAccessThread implements Serializable, Runnable, HazelcastInstanceAware {
    protected com.hazelcast.core.HazelcastInstance instance;
    protected MultiMap<Long, Long> map;
    protected String mapName;
    protected Lock l = new ReentrantLock();

    public void setHazelcastInstance(HazelcastInstance instance) {
        l.lock();
        try {
            this.instance = instance;
            if (mapName != null && !mapName.isEmpty()) {
                map = instance.getMultiMap(mapName);
            }
        } finally {
            l.unlock();
        }
    }

    public String getMapName() {
        return mapName;
    }

    public void setMapName(String mapName) {
        l.lock();
        try {
            this.mapName = mapName;
        } finally {
            l.unlock();
        }
    }
}

IdMultiMapAccessThread

package com.darylmathison.multimap;

/**
 * This thread accesses only one "slot" in a multimap.
 */
public class IdMultiMapAccessThread extends MultiMapAccessThread {
    private Long id;

    @Override
    public void run() {
        l.lock();
        boolean shouldRun = (map != null && id != null);
        l.unlock();
        if(shouldRun) {
            for (long i = 0; i < 10; i++) {
                map.put(id, i);
            }
        }
    }

    public Long getId() {
        return id;
    }

    public void setId(Long id) {
        this.id = id;
    }
}

GroupMultiMapAccessThread

package com.darylmathison.multimap;

/**
 * Thread designed to share the same "slot" on a MultiMap.
 */
public class GroupMultiMapAccessThread extends MultiMapAccessThread {

    private static final long MAX = 10;

    private Long groupId;

    /**
     * When an object implementing interface Runnable is used
     * to create a thread, starting the thread causes the object's
     * run method to be called in that separately executing
     * thread.
     * 
     * The general contract of the method run is that it may
     * take any action whatsoever.
     *
     * @see Thread#run()
     */
    @Override
    public void run() {
        l.lock();
        boolean shouldRun = (groupId != null && map != null);
        l.unlock();
        if(shouldRun) {
            map.lock(groupId);
            try {
                if (map.get(groupId).isEmpty()) {
                    System.out.println("adding to list");
                    for (long i = 0; i < MAX; i++) {
                        map.put(groupId, i);
                    }
                } else {
                    System.out.println("nothing to add");
                }
            } finally {
                map.unlock(groupId);
            }
        }
    }

    public void setGroupId(Long groupId) {
        l.lock();
        this.groupId = groupId;
        l.unlock();
    }
}

HazelcastInstanceResource

This rule starts up and shuts down the Hazelcast instance needed for running the threads.

package com.darylmathison.multimap.test.rule;

import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.IExecutorService;
import org.junit.rules.ExternalResource;

/**
 * Created by Daryl on 4/27/2015.
 */
public class HazelcastInstanceResource extends ExternalResource {
    public static final String SERVICE_NAME = "Charlotte";
    HazelcastInstance instance;
    IExecutorService service;

    @Override
    protected void before() throws Throwable {
        super.before();
        instance = Hazelcast.newHazelcastInstance();
        service = instance.getExecutorService(SERVICE_NAME);
    }

    @Override
    protected void after() {
        super.after();
        service.shutdown();
        instance.shutdown();
    }

    public HazelcastInstance getInstance() {
        return instance;
    }

    public IExecutorService getService() {
        return service;
    }
}

IdMultiMapAccessIT

Here is an example of using the IdGenerator to create new “playgrounds” or keys for the threads to place data.

package com.darylmathison.multimap;

import com.darylmathison.multimap.test.rule.HazelcastInstanceResource;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.IExecutorService;
import com.hazelcast.core.IdGenerator;
import org.junit.ClassRule;
import org.junit.Test;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;

/**
 * Integration test for IdMultiMapAccessThread
 */
public class IdMultiMapAccessThreadIT {

    public static final String MAP_NAME = "idAccessMap";
    public static final String GEN_NAME = "singleAccess";
    public static final int NUM_THREADS = 10;

    @ClassRule
    public static HazelcastInstanceResource hazelcastInstanceResource = new HazelcastInstanceResource();

    @Test
    public void testIdThreads() {
        List threads = generateThreads(hazelcastInstanceResource.getInstance());
        List<Future<?>> futures = new ArrayList<>(NUM_THREADS);
        IExecutorService spinner = hazelcastInstanceResource.getService();
        for(IdMultiMapAccessThread thread: threads) {
            futures.add(spinner.submit(thread));
        }

        for(Future<?> future: futures) {
            try {
                future.get();
            } catch (InterruptedException | ExecutionException e) {
                e.printStackTrace();
            }
        }
    }

    private List generateThreads(HazelcastInstance instance) {
        IdGenerator gen = instance.getIdGenerator(GEN_NAME);
        List threads = new ArrayList<>(NUM_THREADS);
        for(int i = 0; i < NUM_THREADS; i++) {
            IdMultiMapAccessThread thread = new IdMultiMapAccessThread();
            thread.setMapName(MAP_NAME);
            thread.setId(gen.newId());
            threads.add(thread);
        }

        return threads;
    }
}

GroupMultiMapAccessThreadIT

This is an example of using an IdGenerator to create a shared playground or slot.

package com.darylmathison.multimap;

import com.darylmathison.multimap.test.rule.HazelcastInstanceResource;
import com.hazelcast.core.IExecutorService;
import com.hazelcast.core.IdGenerator;
import org.junit.ClassRule;
import org.junit.Test;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;

/**
 * GroupMultiMapAccessThread Integration Test
 */
public class GroupMultiMapAccessThreadIT {
    public static final int NUM_THREADS = 10;
    public static final String GEN_NAME = "groupIdGenerator";
    public static final String MAP_NAME = "groupMap";

    @ClassRule
    public static HazelcastInstanceResource hazelcastInstanceResource = new HazelcastInstanceResource();

    @Test
    public void testGroupMultiMapAccessThread() {
        List threads = createThreads();
        IExecutorService service = hazelcastInstanceResource.getService();
        List<Future<?>> futures = new ArrayList<>(NUM_THREADS);
        for(GroupMultiMapAccessThread thread: threads) {
            futures.add(service.submit(thread));
        }

        for(Future<?> future: futures) {
            try {
                future.get();
            } catch (InterruptedException | ExecutionException e) {
                e.printStackTrace();
            }
        }
    }

    private List createThreads() {
        List ret = new ArrayList<>(NUM_THREADS);
        IdGenerator gen = hazelcastInstanceResource.getInstance().getIdGenerator(GEN_NAME);
        Long groupId = gen.newId();
        for(int i = 0; i < NUM_THREADS; i++) {
            GroupMultiMapAccessThread thread = new GroupMultiMapAccessThread();
            thread.setMapName(MAP_NAME);
            thread.setGroupId(groupId);
            ret.add(thread);
        }

        return ret;
    }
}

Conclusion

In this post, Hazelcast’s MultiMap was profiled. It was shown that MultiMaps can store multiple values for a given key. It was also shown how a thread can share the data in a MultiMap or can store data for itself using the IdGenerator as a possible key generator.  The code can be found in GitHub here.

References

http://www.hazelcast.com
http://www.hazelcast.org
https://github.com/hazelcast/hazelcast

There is a Mojo in My Dojo (How to write a Maven plugin)

Standard

I have been up to my armpits involved using Maven at work.  For good number of developers I will hear, “So what.”  The difference is that I normally work in environments where I do not have access to the Internet directly.  So when I say I have been using Maven a lot, it means something.

Dependency Hell

To be fair, I have been using Maven casually in my examples.  I have found it to be more convenient to get downloads of dependencies and avoid “dependency hell.”  The situation where I have to download a library for a library that i am using.  For example, one has to download Hamcrest to use JUnit.  At home, put in the dependency for JUnit and Maven downloads Hamcrest for me because it is a dependency of JUnit.  If there was a dependency of Hamcrest, Maven would download that too.  When I am at work, I need to research what dependencies JUnit has and then research what dependencies the dependencies have.  I have avoided using libraries because of this very situation.

Situations Change

The change is because I am using Spring Roo at work.  Roo uses Maven to manage Spring dependencies that it needs to incorporate.  Because of this change, I set up a Nexus server on the development network and started the process of bringing over the dependencies from the Internet to the development network.  This got me learning about Maven.

What I Learned about Maven

After reading two books, Introducing Maven and Maven Build Customization, I got a pretty good idea about Maven and how to create the subject of this post.  I can go on and on about what I learned but I will keep it focused to what is needed to learn about Maven plugins. I do assume one has seen a pom file and run a few Maven builds from now on in the post.  If one has not, purchase the books I read or go to http://maven.apache.org first.

Maven is Plugin Rich

Maven is based on a plugin architecture.  Anything that does something in Maven is a plugin.  That goes from core functionality like compiling to creating sites.  As one can imagine, every plugin has certain things in common.

Maven is Package, Lifecycle, Phase and Goal Oriented

Maven is known for building something into a packaged item of some sort, for example a jar file.  That is obvious, that is one of the first lines of a pom file.  What may not be known is that there is a series of “phases” or “lifecycle” that happen to accomplish building the package (see what I did there).  In fact, one of those phases is named “package.”  The list of default phases in a lifecycle are as follows:

  1. validate
  2. generate-sources
  3. process-sources
  4. generate-resources
  5. process-resources
  6. compile
  7. process-classes
  8. generate-test-sources
  9. process-test-sources
  10. generate-test-resources
  11. process-test-resources
  12. test-compile
  13. process-test-classes
  14. test
  15. prepare-package
  16. package
  17. pre-integration-test
  18. integration-test
  19. post-integration-test
  20. verify
  21. install
  22. deploy

There is a lot of stuff going on in a Maven build!  All of that is being run by some sort of plugin.  Every plugin is made of goals which can be set to run at a certain phase of the lifecycle.  For example, the maven-jar-plugin’s jar goal is set to run in the package phase.

The Making of a Plugin

Now that one has a more in-depth knowledge of what is going on in a build, it is time to explain what is needed to create a Maven plugin.

Plugins Are Full of Mojos

What is a mojo?  Mojo is short for Maven plain Old Java Objects.  It is the smallest unit of a plugin Maven recognizes.  All plugins are made of mojos.  Each mojo is associated to a goal.  So for a plugin to have multiple goals, it needs multiple mojos.  The example I will show only has one mojo sadly but the example will also show best practices in testing a plugin.

Best Practices are the Only Practices Allowed

See what I did there to tie in with the Dojo deal in the title?  There is naming convention, unit testing and integration testing involved with writing plugins if one is inclined.  The naming convention is the most important so

  1. You don’t bust an Apache trademark
  2. Others know that one made a plugin.

 What is in a Name?

The naming convention for Apache’s plugins are maven-<title>-plugin.  For example, the jar plugin is named maven-jar-plugin.  For everyone else, the naming convention is <title>-maven-plugin.  For example, the example I created is named reminder-maven-plugin.  Another example that used when making this post is Spring Boot‘s plugin and it is named spring-boot-maven-plugin.  The source code to Spring Boot is here.  I forked it so I could peruse and abuse the code.  My fork can be found here.  If one wants to abuse it together, please fork my copy and send me a pull request when your particular piece of abuse is done.  Anyway if one uses Apache’s naming convention, it is a trademark infringement.  You have been warned.

Unit Testing

Automated unit and integration testing is important too.  Unit testing follows a little different directory pattern than normal unit testing so hang on.

The directory structure when doing a unit test of a plugin is

unit_test_plugin

Notice that all of the test directories are organized under the test directory.  What one is making is a little version of a project that will be using the plugin.  Under the test resources directory is a unit directory followed by the name of the unit in the child directory.  The goal is to test a single mojo at a time.  Since my example only has one mojo, I only set up one test.  There are other differences than the directory setup but that will be covered in the example section.

Integration Testing

I found that this testing will teach one the most about one’s particular plugin and how it works.  The goal is to test a certain situation as if it was part of an actual project build.  When I mean actual project build, I mean that there is even a temporary repository just for the integration build.  After reading about how to set up the tests, I borrowed heavily from spring-boot-maven-plugin’s integration test setup and mini pom files.  OK, I copied some of the files over to my example code.  Just informing one that Spring Boot did it right.  Just be safe a clone read only or fork their code just to be safe.  The directory structure is displayed below.

it_test_plugin

The integration tests are located not under the test directory but directly underneath the src directory in the it directory.  I could have done more integration tests but one is good enough for now.

Example

The example plugin was inspired by the fact that I am absent minded and need to be reminded of everything I do.  I thought of creating a wash-the-dogs-reminder-maven-plugin but I decided on a plain reminder-maven-plugin because then I could use it to remind me of anything I needed to do.

Pom File

 

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.darylmathison</groupId>
    <artifactId>reminder-maven-plugin</artifactId>
    <packaging>maven-plugin</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>reminder-maven-plugin Maven Mojo</name>
    <url>http://maven.apache.org</url>

    <properties>
        <mavenVersion>3.2.1</mavenVersion>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
    </properties>

    <dependencies>
        <!-- Maven dependencies -->
        <dependency>
            <groupId>org.apache.maven</groupId>
            <artifactId>maven-plugin-api</artifactId>
            <version>${mavenVersion}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.maven</groupId>
            <artifactId>maven-core</artifactId>
            <version>${mavenVersion}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.maven.plugin-tools</groupId>
            <artifactId>maven-plugin-annotations</artifactId>
            <version>3.2</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.maven</groupId>
            <artifactId>maven-compat</artifactId>
            <version>3.2.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.maven.plugin-testing</groupId>
            <artifactId>maven-plugin-testing-harness</artifactId>
            <version>3.1.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <configuration>
                        <source>1.8</source>
                        <target>1.8</target>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-plugin-plugin</artifactId>
                    <version>3.2</version>
                    <executions>
                        <execution>
                            <id>mojo-descriptor</id>
                            <goals>
                                <goal>descriptor</goal>
                            </goals>
                        </execution>
                    </executions>
                    <configuration>
                        <skipErrorNoDescriptorsFound>true</skipErrorNoDescriptorsFound>
                    </configuration>
                </plugin>
            </plugins>
        </pluginManagement>
    </build>
    <profiles>
        <profile>
            <id>run-its</id>
            <build>
                <plugins>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-invoker-plugin</artifactId>
                        <version>1.7</version>
                        <configuration>
                            <debug>true</debug>
                            <cloneProjectsTo>${project.build.directory}/it</cloneProjectsTo>
                            <cloneClean>true</cloneClean>
                            <pomIncludes>
                                <pomInclude>*/pom.xml</pomInclude>
                            </pomIncludes>
                            <addTestClassPath>true</addTestClassPath>
                            <postBuildHookScript>verify</postBuildHookScript>
                            <localRepositoryPath>${project.build.directory}/local-repo</localRepositoryPath>
                            <settingsFile>src/it/settings.xml</settingsFile>
                            <goals>
                                <goal>clean</goal>
                                <goal>compile</goal>
                                <goal>package</goal>
                            </goals>
                        </configuration>
                        <executions>
                            <execution>
                                <id>integration-test</id>
                                <goals>
                                    <goal>install</goal>
                                    <goal>run</goal>
                                </goals>
                            </execution>
                        </executions>
                    </plugin>
                </plugins>
            </build>
        </profile>
    </profiles>
</project>

 
As one can see, quite a few plugins and dependencies are needed to build one. There is one dependency of note here. This is the version of Junit. The version needs to be 3.8.1. This is because Maven extended the TestCase class to make it easier to unit test. That will be seen soon. Two plugins are of note, one is the maven-plugin-plugin and the other is the maven-invoker-plugin. The maven-plugin-plugin automates the process of creating a help goal for one’s plugin. The maven-invoker-plugin is used in the integration tests. Its function is to run Maven projects which is handy if one is running in a test pom.

ReminderMojo.java

 

package com.darylmathison;

import org.apache.maven.plugin.AbstractMojo;
import org.apache.maven.plugin.MojoExecutionException;
import org.apache.maven.plugins.annotations.LifecyclePhase;
import org.apache.maven.plugins.annotations.Mojo;
import org.apache.maven.plugins.annotations.Parameter;

import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.time.temporal.ChronoUnit;

@Mojo(name = "remind",
        defaultPhase = LifecyclePhase.PACKAGE,
        requiresOnline = false, requiresProject = true,
        threadSafe = false)
public class ReminderMojo extends AbstractMojo {

    @Parameter(property = "basedir", required = true)
    protected File basedir;

    @Parameter(property = "message", required = true)
    protected String message;

    @Parameter(property = "numOfWeeks", defaultValue = "6", required = true)
    protected int numOfWeeks;

    public void execute() throws MojoExecutionException {

        File timestampFile = new File(basedir, "timestamp.txt");
        getLog().debug("basedir is " + basedir.getName());
        if(!timestampFile.exists()) {
            basedir.mkdirs();
            getLog().info(message);
            timestamp(timestampFile);
        } else {
            LocalDateTime date = readTimestamp(timestampFile);
            date.plus(numOfWeeks, ChronoUnit.WEEKS);
            if(date.isBefore(LocalDateTime.now())) {
                getLog().info(message);
                timestamp(timestampFile);
            }
        }
    }

    private void timestamp(File file) throws MojoExecutionException {
        try(FileWriter w = new FileWriter(file)) {
            LocalDateTime localDateTime = LocalDateTime.now();
            w.write(localDateTime.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME));
        } catch (IOException e) {
            throw new MojoExecutionException("Error creating file " + file, e);
        }
    }

    private LocalDateTime readTimestamp(File file) throws MojoExecutionException {
        try(FileReader r = new FileReader(file)) {
            char[] buffer = new char[1024];
            int len = r.read(buffer);
            LocalDateTime date = LocalDateTime.parse(String.valueOf(buffer, 0, len));
            return date;
        } catch(IOException ioe) {
            throw new MojoExecutionException("Error reading file " + file, ioe);
        }
    }
}

 
This is the only Mojo in the plugin and as one can find, it is very simple but shows some of the cool features the mojo api provides. The class annotation defines that the name of the goal is “remind” and that it is not thread safe. It also defines the default phase is the package phase. The last thing I will mention is that any member variable can become a parameter. This becomes a parameter for the plugin of a goal.

ReminderMojoTest

 

package com.darylmathison;

import org.apache.maven.plugin.testing.AbstractMojoTestCase;

import java.io.File;

/**
 * Created by Daryl on 3/31/2015.
 */
public class ReminderMojoTest extends AbstractMojoTestCase {

    @Override
    protected void setUp() throws Exception {
        super.setUp();
    }

    @Override
    protected void tearDown() throws Exception {
        super.tearDown();
    }

    public void testJustMessage() throws Exception {
        File pom = getTestFile("src/test/resources/unit/reminder-mojo/pom.xml");
        assertNotNull(pom);
        assertTrue(pom.exists());
        ReminderMojo myMojo = (ReminderMojo) lookupMojo("remind", pom);
        assertNotNull(myMojo);
        myMojo.execute();
    }
}

 
Here is a basic unit test case of a mojo. The test class extends AbstractMojoTestCase to gain some functionality like getTestFile and lookupMojo. The following is the test pom file.

Unit Test Pom File

 

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.darylmathison.test</groupId>
    <artifactId>reminder-maven-plugin-test-reminder</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>reminder-maven-plugin Maven Mojo</name>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>com.darylmathison</groupId>
                <artifactId>reminder-maven-plugin</artifactId>
                <version>1.0-SNAPSHOT</version>
                <configuration>
                    <basedir>target/test-classes/unit/reminder-mojo</basedir>
                    <message>Wash the doggies</message>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

 
Just a mini version of the main pom file that defined the plugin.

Integration Test

This deserves its own section because it is really a separate Maven project within a Maven project. The main focus of this exercise is to see what the plugin will do and not anything else. The sample application is simple and just there for the Maven project to build. The other thing to note is that the pom file uses some filtering to match the groupId, artifactId, and version of the main plugin pom.

Pom File

 

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.darylmathison.it</groupId>
    <artifactId>new-timestamp</artifactId>
    <version>0.0.1.BUILD-SNAPSHOT</version>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>
    <build>
        <plugins>
            <plugin>
                <groupId>@project.groupId@</groupId>
                <artifactId>@project.artifactId@</artifactId>
                <version>@project.version@</version>
                <executions>
                    <execution>
                        <id>blah</id>
                        <goals>
                            <goal>remind</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <message>Wash the doggies</message>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

 

SampleApp

 

package java.test;

/**
 * Created by Daryl on 4/5/2015.
 */
public class SampleApp {
    public static void Main(String[] args) {
        System.out.println("out");
    }
}

 

Verify.groovy

 

System.out.println(basedir);
def file = new File(basedir, "timestamp.txt");
return file.exists();

 
The verify script is to make sure that the plugin does what it intended to do. It just checks for the existence of the timestamp.txt file because the plugin creates one when it cannot find a timestamp file. Maven checks for a true or false output of the verify script.

Conclusion

Wow! I covered a lot in this post. I went and gave an example of how to create a Maven plugin. I also showed how to test that plugin using best practices. I got the information between two books and an example of a real on going open source project. The example code is hosted on github here. This represents the first example in my new example home.

References