The Beginner’s Guide to Hazelcast Part 5

Standard

This is a continuation of a series of posts I have written about Hazelcast.  I highly suggest you read the other ones: Part 1, Part 2, Part 3 and Part 4.

Things That Makes One Go “Huh?”

This post will have no Hazelcast specific code in it.  Let me repeat that.  This post will have no Hazelcast specific code in it.  That is because the fine folks at Hazelcast produced a product that implements different standards.  This allows for a choice of clients.  One of those standards that Hazelcast implements is memcached.

What about JCache?

JCache (JSR 107) is just for Java.  Memcached protocol clients have been implemented across several languages so one is not nailed down to one language.  Implementing the memcached protocol was a smart move in my opinion because it makes Hazelcast more than a “Java thing.”

Why Use Hazelcast?

Excellent question!  If one can use any memcached server, why use Hazelcast.  Well, to tell you the truth, unless one is sharing a database between several servers, one may not even need caching!  If one does need a caching solution, here is why I would choose Hazelcast:

  1. Automatic, real time backups – I have not read of one Hazelcast datatype that is not backed up at least once.  Just stand up two instances, one off machine from the other, to get the full benefit.
  2. Security – If the servers that need to cache are across different networks, then the firewall rules can be easier with Hazelcast.  Lets say n servers are needing to cache data and n/2 of them are on the 192.168.1.x network and the other n/2 are on the 10.10.1.x network.  By setting one Hazelcast instance on either network, all n machines can be sharing a cache.  The Hazelcast instances can be configured to talk to just the instance on the other side.  That makes the firewall rule writer job easier because there only has to be a rule made for two servers rather than n machines  then the 192.168.1.x machines just talk to their Hazelcast node and the 10.10.1.x machines just talk to their Hazelcast node and let the Hazelcast instances do the rest of the work.

Example

I never like to show just a “ho hum” kind of example so I am going to show how a Java client can share data with a Python client.

Setup

I am using Java 1.7 and Python 3.4.  Unfortunately, neither language has memcached support out of the box so I went looking for already written clients.

Java

I found Spymemcached for Java.  I will be just skimming the surface of its abilities. It can be grabbed from Maven.  Here is the pom.xml file for the project:

 1 <?xml version="1.0" encoding="UTF-8"?>
 2 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 3     <modelVersion>4.0.0</modelVersion>
 4     <groupId>com.darylmathison</groupId>
 5     <artifactId>Memcache</artifactId>
 6     <version>1.0-SNAPSHOT</version>
 7     <packaging>jar</packaging>
 8     <properties>
 9         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
10         <maven.compiler.source>1.7</maven.compiler.source>
11         <maven.compiler.target>1.7</maven.compiler.target>
12     </properties>
13     <build>
14         <plugins>
15             <plugin>
16                 <groupId>org.apache.maven.plugins</groupId>
17                 <artifactId>maven-compiler-plugin</artifactId>
18                 <version>2.3.2</version>
19                 <configuration>
20                     <showDeprecation>true</showDeprecation>
21                 </configuration>
22             </plugin>
23             <plugin>
24                 <groupId>org.codehaus.mojo</groupId>
25                 <artifactId>exec-maven-plugin</artifactId>
26                 <version>1.3.2</version>
27                 <executions>
28                     <execution>
29                         <goals>
30                             <goal>java</goal>
31                         </goals>
32                     </execution>
33                 </executions>
34                 <configuration>
35                     <mainClass>com.darylmathison.memcache.Main</mainClass>
36                 </configuration>
37             </plugin>
38         </plugins>
39     </build>
40     <dependencies>
41         <dependency>
42             <groupId>net.spy</groupId>
43             <artifactId>spymemcached</artifactId>
44             <version>2.11.5</version>
45         </dependency>
46     </dependencies>
47 </project>

Python

Next, I found python3-memcached for Python. It uses the classic setup.py procedure to install.

Server

Not much of a cache if the server is missing. One can download Hazelcast at hazelcast.org/download, extract the contents, cd into the bin directory and run the server.bat or server script according to one’s OS. As setting up servers go, that is the easiest one I have ever done.

Situation

The “expensive” operation that I am trying to make cheaper is Fibonacci numbers. Because Python and Java can both understand unicode, the values are stored as unicode strings. The key is a unicode string of the number of the sequence or the number of rounds it takes to get there.

Code

Java

package com.darylmathison.memcache;

import java.io.IOException;
import java.net.InetSocketAddress;
import net.spy.memcached.MemcachedClient;

/**
 *
 * @author Daryl
 */
public class Main {

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) {
        try {
            MemcachedClient client = new MemcachedClient(new InetSocketAddress("localhost", 5701));
            for(int i = 2; i < 20; i++) {
                System.out.println("value of round " + i + " is " + fibonacci(i, client));
            }
            client.shutdown();
        } catch(IOException ioe) {
            ioe.printStackTrace();
        }
    }
    
    private static long fibonacci(int rounds, MemcachedClient client) {
        String cached = (String)client.get(String.valueOf(rounds));
        if(cached != null) {
            System.out.print("cached ");
            return Long.parseLong(cached);
        }
        
        long[] lastTwo = new long[] {1, 1};
        
        for(int i = 0; i < rounds; i++) {
            long last = lastTwo[1];
            lastTwo[1] = lastTwo[0] + lastTwo[1];
            lastTwo[0] = last;
        }
        
        client.set(String.valueOf(rounds), 360, String.valueOf(lastTwo[1]));
        return lastTwo[1];
     }
}

Python

Here is the Python client. As a pythonian, I tried to be as pythonic as possible.

import memcache

client = memcache.Client(['localhost:5701'])

def fibonacci(round):
    f = [1, 1, 1]
    
    for i in range(round):
        f[-1] = sum(f[:2])
        f[0], f[1] = f[1], f[2]
        
    return f[2]

def retrievefib(round):
    fib = client.get(str(round))
    if not fib:
        fib = fibonacci(round)
        client.set(str(round), str(fib))
    else:
        print("cached")
        
    return fib

def main():
    store = [ x for x in range(20) if x % 2 == 0]
    for i in store:
        retrievefib(i)
    
    for i in range(20):
        print(retrievefib(i))

if __name__ == "__main__":
    main()

Conclusion

Well, here is an example of Hazelcast as being the powerhouse behind the scenes. This is a place where I think it shines the most. One doesn’t have to create whole new crafty, distributed applications to take advantage of Hazelcast. All one has to do is use known practices and let Hazelcast do the hard work. The source for this post can be found here for the Java code and here for the Python code.

References

http://en.wikipedia.org/wiki/Fibonacci_number
https://code.google.com/p/spymemcached/
https://pypi.python.org/pypi/python3-memcached/1.51

How to Use Callable and FutureTask

Standard

Introduction

Since Java 1.5 there has been a new set of objects under java.util.concurrent.  This package has a number of different classes including thread queues.  I could have used those when I was programming with Java 1.2!  When I started looking at the new toys I became hesitant.  What is this Callable thing and what is the Future?  It turns out that there is nothing wrong with a Future and Callable.  In fact, they are what I have been hoping, looking for in my Java career.

Differences Between Callable and Runnable

Callable is what Runnable hoped to become.  Callable’s only method is “T call().”  What makes it so neat is that it returns something.  This is a step above having to create a getter for the answer to a task.  While this is cool, there needs to be a way to get at the returned value.

The Future is here

Future has a way to get the value out when the Callable is done.  The function is get() or get(long timeout, TimeUnit unit).  This is the equivalent of calling thread.join(); runnable.getValue() at the same time.

Example

I created a class called CounterCallable.  All it does is add numbers from the variable start to variable end.

CounterCallable

package org.mathison.futurecallable;

import java.util.concurrent.Callable;

/**
 *
 * @author Daryl
 */
public class CounterCallable implements Callable {

    private long start;
    private long end;

    public CounterCallable(long start, long end) {
        this.start = start;
        this.end = end;
    }

    @Override
    public SumTimeAnswer call() throws Exception {
        long sum = 0;
        long startTime = System.currentTimeMillis();
        for(long i = start; i <= end; i++){
            sum += i;
        }
        long endTime = System.currentTimeMillis();

        return new SumTimeAnswer(sum, endTime - startTime);
    }
}

SumTimeAnswer

Class SumTimeAnswer is really a simple getter class that holds the sum and the amount of time it took to do the operation.

package org.mathison.futurecallable;

/**
 *
 * @author Daryl
 */
public class SumTimeAnswer {
    private long timeToFinish;
    private long sum;

    public SumTimeAnswer(long sum, long timeToFinish) {
        this.sum = sum;
        this.timeToFinish = timeToFinish;
    }

    public long getTimeToFinish() {
        return timeToFinish;
    }

    public long getSum() {
        return sum;
    }
}

App

App is just a main class pulling everything together

package org.mathison.futurecallable;

import java.util.concurrent.CancellationException;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.FutureTask;

/**
 * Hello world!
 *
 */
public class App 
{
    public static final long BEGIN = 0;
    public static final long END = 100000;
    public static void main( String[] args )
    {
        FutureTask task = new FutureTask(new CounterCallable(BEGIN, END));
        FutureTask firstHalf = new FutureTask(new CounterCallable(BEGIN, END/2));
        FutureTask secondHalf = new FutureTask(new CounterCallable(END/2 + 1, END));
        
        ExecutorService pool = Executors.newSingleThreadExecutor();
        pool.submit(task);
        pool.submit(firstHalf);
        pool.submit(secondHalf);
        
        try {
            SumTimeAnswer taskAnswer = task.get();
            System.out.println("just one thread Time: " + taskAnswer.getTimeToFinish()
                + " Total: " + taskAnswer.getSum());
            
            
            SumTimeAnswer taskFirstAnswer = firstHalf.get();
            SumTimeAnswer taskSecondAnswer = secondHalf.get();
            long totalTime = taskFirstAnswer.getTimeToFinish() + taskSecondAnswer.getTimeToFinish();
            long totalSum = taskFirstAnswer.getSum() + taskSecondAnswer.getSum();
            System.out.println("Two thread time: " + totalTime + " Total: " + totalSum);
        } catch(CancellationException | InterruptedException | ExecutionException e) {
            e.printStackTrace();
        }
        pool.shutdown();
    }
}

Conclusion

In this post, classes Callable and FutureTask were used to demonstrate how to use the java.util.concurrent package.

BufferedReader of The EOF Kind

Standard

Introduction

I have been adding automated tests in projects that I have written at work.  One of these projects outputs CEF formatted messages to syslog.  The test required that /var/log/messages be read and compare the CEF message with the known good message.  I ran into a problem while reading the file.

What Problem?

Even a novice Java developer knows that reading a text file is no big deal, just chain the right streams together and the contents will be easily extracted.  I wanted to read the file line by line so I chained java.io.FileReader inside a java.io.BufferedReader.  Everything should work, right?  Indeed each line was dutifully read and was displaying on my log except the last line, the exact line that I was trying to compare.  I found the root of the cause in the javadocs themselves.  Here is the excerpt from the BufferedReader’s javadoc:

Reads a line of text. A line is considered to be terminated by any one of a line feed (‘\n’), a carriage return (‘\r’), or a carriage return followed immediately by a linefeed.
That sounds reasonable, at the end of a line, a line is returned.  This works 99.9% of the time but what if the line ends with an end of file(EOF)?  I will tell you what happens, a null is returned!  Remember that java.io.BufferedReader considers that a line ends in a line feed, carriage return or a carriage return followed by a line feed, not a EOF.  Syslog ends its logs with a EOF not a line terminator at the end of the last line.  This means java.io.BufferedReader as the javadoc states will not show the last line but return a null because there are no more end of line indicators before the end of the file.  This is a major problem for my test because I have to check the last line in most cases to validate output.  I am going to say this is an oversight for the makers of the Java libraries.

What To Do, What To Do

I obviously needed to modify the definition of what the end of a line was by java.io.BufferedReader.  My first thought was to extend BufferedReader so I started the checklist.  The class is not final but the buffer underneath was not visible.  This is a problem because the only way to get to the buffer is going through the java.io.Reader interface.  So my initial thought was to override the readLine method and use the read() method to get the contents one by one to parse each line.  That seemed a waste to make repeated method calls just to access a buffer.  It would be better to just read the buffer directly and increment a counter.  This meant that I would have to basically recreate BufferedReader.  “This should be no problem.  I am an experienced developer.  This will be easy!” I thought to myself.

A lesson Learned

Four days and 26 automated tests later, the beast was done.  While it is more efficient to to use a counter on an array instead of  repeated method calls, it did not make good time management sense.  Remember this was for an automated test, not code destined to be delivered to the customer.  A second or two extra per test run does not matter in the long run.  Getting it done in a quarter of the time so tests can be run does matter.  In my mind, the sooner a test is written, the more time can be saved and time is money.

The Solution

In theory, the concept is simple, the buffer is filled from the inner reader.  If the buffer is full, it is cleared and another set of data is read from the inner reader.  This repeats until the inner reader returns -1.  The main methods are skip(long n), read(char[] data, int offset, int length), readLine() and fillBuffer().

Skip(long n)

    @Override
    public long skip(long n) throws IOException {
        long numSkipped = 0;
        long leftToSkip = n;
        int lenRead = 0;
        while(leftToSkip > 0 && lenRead != -1) {
            if((offset + leftToSkip)  endIndex) {
                lenRead = fillBuffer();
                if(lenRead != -1) {
                    int amountBuffered = endIndex - offset;
                    long amountToSkip = (amountBuffered < leftToSkip)? 
                            amountBuffered:leftToSkip;
                    offset += amountToSkip;
                    numSkipped += amountToSkip;
                    leftToSkip -= amountToSkip;
                }
            }
        }

        return numSkipped;
    }

Read(char[] data, int offset, int length)

    @Override
    public int read(char[] cbuf, int off, int length) throws IOException {
        int totalRead = 0;
        boolean noReads = true;
        int targetOffset = off;
        int newTargetOffset = off + length;
        int readLen = 0;
        int lenToCopy = 0;
        int leftToCopy = length;
        
        while(targetOffset < newTargetOffset && readLen != -1) {
            if((offset + leftToCopy) < endIndex) {
                readLen = fillBuffer();
                if(readLen != -1) {
                    int amountBuffered = endIndex - offset;
                    lenToCopy = (amountBuffered < leftToCopy)? amountBuffered:leftToCopy;
                    System.arraycopy(buffer, offset, cbuf, targetOffset, lenToCopy);
                    noReads = false;
                    offset += lenToCopy;
                    targetOffset += lenToCopy;
                    leftToCopy -= lenToCopy;
                    totalRead += lenToCopy;
                }
            }
        }
        
        if(noReads) {
            totalRead = -1;
        }

        return totalRead;
    }

ReadLine()

    public String readLine() throws IOException {
        StringBuilder line = new StringBuilder();
        boolean foundCR = false;
        boolean foundLinefeed = false;
        boolean foundBoth = false;
        int readLen = 0;
        final char LINEFEED = '\n';
        final char CR = '\r';
        
        if(offset == endIndex) {
            readLen = fillBuffer();
        }
        
        while(!(foundCR || foundLinefeed || foundBoth) && readLen != -1) {
            if(buffer[offset] == CR) {
                foundCR = true;
                offset++;
                if(offset == endIndex) {
                    readLen = fillBuffer();
                    if(readLen != -1) {
                        if(buffer[offset] == LINEFEED) {
                            foundBoth = true;
                            offset ++;
                        } 
                    }
                } else if(buffer[offset] == LINEFEED) {
                    foundBoth = true;
                    offset++;
                } 
            } else if(buffer[offset] == LINEFEED) {
                foundLinefeed = true;
                offset ++;
            } else {
                line.append(buffer[offset]);
                offset++;
            }
            
            if(offset == endIndex) {
                readLen = fillBuffer();
            }
        }
        
        if(line.length() == 0) {
            return null;
        }
        
        return line.toString();
    }

FillBuffer()

    private int fillBuffer() throws IOException {
        int length;
        int lenRead = 0;
        long newOffset = offset + buffer.length;
        if(newOffset >= endIndex) {
            moveLeftoverToBeginning();
            endIndex = endIndex - offset;
            offset = 0;
            
            length = bufferSize - endIndex;
            lenRead = in.read(buffer, endIndex, length);
            if (lenRead != -1) {
               endIndex += lenRead;
            } else {
                //endIndex = offset;
            }
        } else if(newOffset < endIndex) {
            lenRead = 0;
        }
        return lenRead;
    }

Conclusion

In this blog entry, a custom BufferedReader is discussed. The reader includes EOF as a line terminator. This is to facilitate verifying the output of a CEF formatted syslog message. The link to see the rest of this BufferedReader and its tests, download the Maven project via git at https://github.com/darylmathison/buffered-reader-example.

Three Ways to Solve a Problem

Standard

Introduction

I wanted to show the differences of three different languages.  I made a palindrome detector with Java, Groovy and Python.  Technically the Python I used is CPython so it is not Java but JPython runs in the JVM so I am counting it as a Java related language.  Groovy is Java related because as of Java 1.6 and later, Groovy natively runs on the JVM.  Since this is a Java blog, Java had to be one of the languages used.  I will point out differences and similarities when all of the solutions are posted.  This will not be a explaination of each language.  There are many excellent resources out there.  For example, one place I find myself at a lot is http://www.python.org.  Another site I found myself for this post is http://groovy.codehaus.org/.

Solutions (code)

Requirements

The solution must analyze a string and see if it is a palindrome.  A palindrome is a word or phrase, that spelled the same forwards and backwards.  For example, 123321 is a palindrome.  Two strings will be used to test the solution, “123321” and “1234321.”  This covers the case of a palindrome with an odd number of characters and a even number of characters,  The solution returns the equivalent of a true when a palindrome is detected and the equivalent of false when one is not detected.

Java

public class Palindrome {

private static boolean isPalindrome(String testString) {

String firsthalf;

StringBuilder secondhalf;

int firstHalfIndex;

int secondHalfIndex;

int length = testString.length();

if(length % 2 == 0) {

firstHalfIndex = (int)(length/2);

firsthalf = testString.substring(0, firstHalfIndex);

secondhalf = new StringBuilder(testString.substring(firstHalfIndex));

} else if ( length % 2 == 1) {

firstHalfIndex = (int)(length/2);

secondHalfIndex = firstHalfIndex + 1;

firsthalf = testString.substring(0, firstHalfIndex);

secondhalf = new StringBuilder(testString.substring(secondHalfIndex));

} else {

firsthalf = “”;

secondhalf = new StringBuilder();

}

return firsthalf.equals(secondhalf.reverse().toString());

}

public static void main(String[] argv) {

System.out.println(isPalindrome(“123321”));

System.out.println(isPalindrome(“1234321”));

}

}

Groovy

def isPalindrome(testString){

def firsthalf

def secondhalf

def firstHalfIndex

def secondHalfIndex

if (testString.length() % 2 == 0) {

firstHalfIndex = testString.length()/2

firsthalf = testString[0..<firstHalfIndex]

secondhalf = testString[firstHalfIndex..<testString.length()]

} else if(testString.length() % 2 == 1) {

firstHalfIndex = testString.length()/2

firsthalf = testString[0..<firstHalfIndex]

secondHalfIndex = firstHalfIndex + 1

secondhalf = testString[secondHalfIndex..<testString.length()]

}

firsthalf == secondhalf.reverse()

}

println isPalindrome(“123321”)
println isPalindrome(“1234321”)

Python

def isPalindrome(test_string):

length = len(test_string)

first_half = None

second_half = None

if length % 2 == 0:

first_half_index = int(length/2)

first_half = test_string[:first_half_index]

second_half = test_string[first_half_index:]

elif length % 2 == 1:

first_half_index = int(length/2)

second_half_index = first_half_index + 1

first_half = test_string[:first_half_index]

second_half = test_string[second_half_index:]

return first_half == second_half[::-1]

def main():

print(isPalindrome(“123321”))

print(isPalindrome(“1234321”))

if __name__ == “__main__”:

main()

Thoughts

Similarities

What I found was the same was how easily it was to implement the same solution in three different languages.  Each one did it in their own idiom but it was done.  Each had the ability to figure out the length of the string, split the string by an index and reverse the string.  I am a sucker for slices so Groovy and Python were more fun for me.

Differences

I wasn’t aware of this before I started hitting the blogosphere but one of the complaints of Java is that it is too wordy.  Doing this exercise really exposed me to the wordiness of Java.  Groovy did the same work without having to use a StringBuilder to do a string reverse.  The only reason I did a main on the Python solution is because that is a way to be able to use a file as a library and be a standalone script (It is a Python “idiom” I once read.).  The concept of slices really made the code look neat and tidy for me.  Did I mention I am a sucker for slices? The Groovy reverse extension to Strings was really nice.  Being able to reverse the string with a slice was really cool in Python.

Conclusion

In this post, I compared three languages implementing the same solution.  While the problem was simple, each language succeeded in its own way.