Beginner’s Guide to Hazelcast Part 7

Standard

This is a continuation of a series explaining how to use Hazelcast.  If one has not read the other six posts, please go to the Table Of Contents and read the other posts.

A Different Breed of Map

Hazelcast’s MultiMap breaks the normal mold of using java.util.Collection interfaces that have been used in former posts.  In fact, the concept of a MultiMap breaks the idea of a map altogether in my opinion.  While normal maps associate one key to one value, MultiMaps can map multiple values to the same key.  That is a really important concept, multiple values to the same key.  Values can be stored in two different collections, set or list.  These collections act like the collections of java.util.Collections library.

Is it Safe?

MultiMaps have a method to their madness.  In a normal map, multiple values per key can be stored but it has to be done manually.  That means getting a collection out of storage, doing any changes and then putting the collection back into storage.  This can be problematic for thread safety because the prior steps need to be done atomically or there is the possibility of stale or inconsistent data being read by other threads.  MultiMaps help with this problem by offering the following services:

  • One can add a value via a single put operation.
  • One can lock an entry by the key.  This is key (pun intended) because this means the developer does not have to keep track of a separate lock per entry.

Example

This example is a little different because I used Maven’s failsafe plugin as the main engine when running the examples.  Yes, I wrote two examples because I wanted to show two different ways of using a MultiMap.  One way is every thread getting their own playground, being assigned an unique key, or being assigned a shared playground or all of the threads sharing the same key.  This also is an example of how Hazelcast’s IdGenerator can be used as a method of creating thread safety in an application.

Pom File

Remember, this example code takes advantage of Apache’s Failsafe Maven plugin.  The failsafe plugin aids in automated integration tests by not killing the build at the first failure.  It is a fork of the surefire plugin.  I also have been experimenting with the reporting that is available with Maven.  Type “mvn site” at the command line and it will generate a website.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0>

    <groupId>com.darylmathison>
    <artifactId>hazelcast-multimap-example>
    <version>1.0-SNAPSHOT>
    <description>Examples of using Hazelcast Multimap>
    <developers>
        <developer>
            <name>Daryl Mathison>
            <id>dmathison>
            <roles>
                <role>developer>
            >
        >
    >

    <scm>
        <connection>scm:git:https://github.com/darylmathison/hazelcast-multimap-example.git>
        <url>https://github.com/darylmathison/hazelcast-multimap-example>
    >

    <properties>
        <maven.compiler.source>1.8>
        <maven.compiler.target>1.8>
        <project.build.sourceEncoding>UTF-8>
    >

    <dependencies>
        <dependency>
            <groupId>com.hazelcast>
            <artifactId>hazelcast>
            <version>3.4.2>
        >
        <dependency>
            <groupId>junit>
            <artifactId>junit>
            <version>4.12>
            <scope>test>
        >
    >

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins>
                <artifactId>maven-failsafe-plugin>
                <version>2.18.1>
                <executions>
                    <execution>
                        <goals>
                            <goal>integration-test>
                            <goal>verify>
                        >
                    >
                >
            >
        >
    >

    <reporting>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins>
                <artifactId>maven-project-info-reports-plugin>
                <version>2.7>
                <reportSets>
                    <reportSet>
                        <reports>
                            <report>dependencies>
                            <report>index>
                            <report>project-team>
                            <report>scm>
                        >
                    >
                >
            >
            <plugin>
                <groupId>org.apache.maven.plugins>
                <artifactId>maven-javadoc-plugin>
                <version>2.10.3>
                <reportSets>
                    <reportSet>
                        <reports>
                            <report>javadoc>
                            <report>test-javadoc>
                        >
                    >
                >
            >
            <plugin>
                <groupId>org.apache.maven.plugins>
                <artifactId>maven-surefire-report-plugin>
                <version>2.18.1>
            >
            <plugin>
                <groupId>org.apache.maven.plugins>
                <artifactId>maven-jxr-plugin>
                <version>2.5>
                <configuration>
                    <linkJavadoc>true>
                >
                <reportSets>
                    <reportSet>
                        <reports>
                            <report>jxr>
                            <report>test-jxr>
                        >
                    >
                >
            >
        >
    >
>

MultimapAccessThread

This is the base class for each of the access type threads.

package com.darylmathison.multimap;

import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.HazelcastInstanceAware;
import com.hazelcast.core.MultiMap;

import java.io.Serializable;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;

/**
 * Abstract class to access MultiMap.
 */
public abstract class MultiMapAccessThread implements Serializable, Runnable, HazelcastInstanceAware {
    protected com.hazelcast.core.HazelcastInstance instance;
    protected MultiMap<Long, Long> map;
    protected String mapName;
    protected Lock l = new ReentrantLock();

    public void setHazelcastInstance(HazelcastInstance instance) {
        l.lock();
        try {
            this.instance = instance;
            if (mapName != null && !mapName.isEmpty()) {
                map = instance.getMultiMap(mapName);
            }
        } finally {
            l.unlock();
        }
    }

    public String getMapName() {
        return mapName;
    }

    public void setMapName(String mapName) {
        l.lock();
        try {
            this.mapName = mapName;
        } finally {
            l.unlock();
        }
    }
}

IdMultiMapAccessThread

package com.darylmathison.multimap;

/**
 * This thread accesses only one "slot" in a multimap.
 */
public class IdMultiMapAccessThread extends MultiMapAccessThread {
    private Long id;

    @Override
    public void run() {
        l.lock();
        boolean shouldRun = (map != null && id != null);
        l.unlock();
        if(shouldRun) {
            for (long i = 0; i < 10; i++) {
                map.put(id, i);
            }
        }
    }

    public Long getId() {
        return id;
    }

    public void setId(Long id) {
        this.id = id;
    }
}

GroupMultiMapAccessThread

package com.darylmathison.multimap;

/**
 * Thread designed to share the same "slot" on a MultiMap.
 */
public class GroupMultiMapAccessThread extends MultiMapAccessThread {

    private static final long MAX = 10;

    private Long groupId;

    /**
     * When an object implementing interface Runnable is used
     * to create a thread, starting the thread causes the object's
     * run method to be called in that separately executing
     * thread.
     * 
     * The general contract of the method run is that it may
     * take any action whatsoever.
     *
     * @see Thread#run()
     */
    @Override
    public void run() {
        l.lock();
        boolean shouldRun = (groupId != null && map != null);
        l.unlock();
        if(shouldRun) {
            map.lock(groupId);
            try {
                if (map.get(groupId).isEmpty()) {
                    System.out.println("adding to list");
                    for (long i = 0; i < MAX; i++) {
                        map.put(groupId, i);
                    }
                } else {
                    System.out.println("nothing to add");
                }
            } finally {
                map.unlock(groupId);
            }
        }
    }

    public void setGroupId(Long groupId) {
        l.lock();
        this.groupId = groupId;
        l.unlock();
    }
}

HazelcastInstanceResource

This rule starts up and shuts down the Hazelcast instance needed for running the threads.

package com.darylmathison.multimap.test.rule;

import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.IExecutorService;
import org.junit.rules.ExternalResource;

/**
 * Created by Daryl on 4/27/2015.
 */
public class HazelcastInstanceResource extends ExternalResource {
    public static final String SERVICE_NAME = "Charlotte";
    HazelcastInstance instance;
    IExecutorService service;

    @Override
    protected void before() throws Throwable {
        super.before();
        instance = Hazelcast.newHazelcastInstance();
        service = instance.getExecutorService(SERVICE_NAME);
    }

    @Override
    protected void after() {
        super.after();
        service.shutdown();
        instance.shutdown();
    }

    public HazelcastInstance getInstance() {
        return instance;
    }

    public IExecutorService getService() {
        return service;
    }
}

IdMultiMapAccessIT

Here is an example of using the IdGenerator to create new “playgrounds” or keys for the threads to place data.

package com.darylmathison.multimap;

import com.darylmathison.multimap.test.rule.HazelcastInstanceResource;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.IExecutorService;
import com.hazelcast.core.IdGenerator;
import org.junit.ClassRule;
import org.junit.Test;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;

/**
 * Integration test for IdMultiMapAccessThread
 */
public class IdMultiMapAccessThreadIT {

    public static final String MAP_NAME = "idAccessMap";
    public static final String GEN_NAME = "singleAccess";
    public static final int NUM_THREADS = 10;

    @ClassRule
    public static HazelcastInstanceResource hazelcastInstanceResource = new HazelcastInstanceResource();

    @Test
    public void testIdThreads() {
        List threads = generateThreads(hazelcastInstanceResource.getInstance());
        List<Future<?>> futures = new ArrayList<>(NUM_THREADS);
        IExecutorService spinner = hazelcastInstanceResource.getService();
        for(IdMultiMapAccessThread thread: threads) {
            futures.add(spinner.submit(thread));
        }

        for(Future<?> future: futures) {
            try {
                future.get();
            } catch (InterruptedException | ExecutionException e) {
                e.printStackTrace();
            }
        }
    }

    private List generateThreads(HazelcastInstance instance) {
        IdGenerator gen = instance.getIdGenerator(GEN_NAME);
        List threads = new ArrayList<>(NUM_THREADS);
        for(int i = 0; i < NUM_THREADS; i++) {
            IdMultiMapAccessThread thread = new IdMultiMapAccessThread();
            thread.setMapName(MAP_NAME);
            thread.setId(gen.newId());
            threads.add(thread);
        }

        return threads;
    }
}

GroupMultiMapAccessThreadIT

This is an example of using an IdGenerator to create a shared playground or slot.

package com.darylmathison.multimap;

import com.darylmathison.multimap.test.rule.HazelcastInstanceResource;
import com.hazelcast.core.IExecutorService;
import com.hazelcast.core.IdGenerator;
import org.junit.ClassRule;
import org.junit.Test;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;

/**
 * GroupMultiMapAccessThread Integration Test
 */
public class GroupMultiMapAccessThreadIT {
    public static final int NUM_THREADS = 10;
    public static final String GEN_NAME = "groupIdGenerator";
    public static final String MAP_NAME = "groupMap";

    @ClassRule
    public static HazelcastInstanceResource hazelcastInstanceResource = new HazelcastInstanceResource();

    @Test
    public void testGroupMultiMapAccessThread() {
        List threads = createThreads();
        IExecutorService service = hazelcastInstanceResource.getService();
        List<Future<?>> futures = new ArrayList<>(NUM_THREADS);
        for(GroupMultiMapAccessThread thread: threads) {
            futures.add(service.submit(thread));
        }

        for(Future<?> future: futures) {
            try {
                future.get();
            } catch (InterruptedException | ExecutionException e) {
                e.printStackTrace();
            }
        }
    }

    private List createThreads() {
        List ret = new ArrayList<>(NUM_THREADS);
        IdGenerator gen = hazelcastInstanceResource.getInstance().getIdGenerator(GEN_NAME);
        Long groupId = gen.newId();
        for(int i = 0; i < NUM_THREADS; i++) {
            GroupMultiMapAccessThread thread = new GroupMultiMapAccessThread();
            thread.setMapName(MAP_NAME);
            thread.setGroupId(groupId);
            ret.add(thread);
        }

        return ret;
    }
}

Conclusion

In this post, Hazelcast’s MultiMap was profiled. It was shown that MultiMaps can store multiple values for a given key. It was also shown how a thread can share the data in a MultiMap or can store data for itself using the IdGenerator as a possible key generator.  The code can be found in GitHub here.

References

http://www.hazelcast.com
http://www.hazelcast.org
https://github.com/hazelcast/hazelcast

Advertisements

Beginner’s Guide to Hazelcast Part 6

Standard

This is the sixth post in a series of posts about Hazelcast.  If one has not read the last five, please go to the table of contents post I have created to catch up.

Native Clients

After the last post I decided that I am going to go native.  Yep, I am going to demonstrate Hazelcast’s own Java client.  Java is not the only language native clients come in, C++ and C# flavors are available for the enterprise version.

Why Go Native?

It is a good question.  Native clients can keep one fixed into a product line without chance to escape.  Hazelcast rewards the one going native with the following:

  • The client is part of the cluster.  That means one can create places to stash data and listen for events going on in the cluster.  It also means that all of the tricks that have been discussed in my earlier posts can be used as a client.  This advantage cannot be underestimated.
  • The configuration file is similar.  This means that one doesn’t have to translate from the Hazelcast configuration file to client configuration file.  One can copy the file over and like magic it works.  The less translating one has to do, the less that gets lost.

The Any Client Rule of Thumb

Hazelcast clients are the easiest I have ever had the pleasure to set up and use.

Example

This simple example is a continuation of a theme started by the last post, caching expensive operations.

Pom File

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.darylmathison</groupId>
    <artifactId>HazelcastJavaClient</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.7</maven.compiler.source>
        <maven.compiler.target>1.7</maven.compiler.target>
    </properties>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.3.2</version>
                <configuration>
                    <showDeprecation>true</showDeprecation>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>1.3.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>java</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <mainClass>com.darylmathison.hazelcastjavaclient.Main</mainClass>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <dependencies>
        <dependency>
            <groupId>com.hazelcast</groupId>
            <artifactId>hazelcast-client</artifactId>
            <version>3.3.2</version>
        </dependency>
    </dependencies>

</project>

Client

This client becomes part of the cluster creates a IMap named “fibmap”. The fibonacci result is stored in the map if it hasn’t been calculated before. If one runs the client once, the results are stored in fibmap. The second time the client is run, the cached values are displayed.

package com.darylmathison.hazelcastjavaclient;

import com.hazelcast.client.HazelcastClient;
import com.hazelcast.core.HazelcastInstance;
import java.util.Map;

/**
 *
 * @author Daryl
 */
public class Main {

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) {
        HazelcastInstance instance = HazelcastClient.newHazelcastClient();
        Map<Long, Long> cacheMap = instance.getMap("fibmap");
        
        for(long i = 1; i <= 10L; i++) {
            System.out.println("value is " + fibonacci(i, cacheMap));
        }
        instance.shutdown();
    }
    
    private static long fibonacci(long rounds, Map<Long, Long> cacheMap) {
        Long cached = cacheMap.get(rounds);
        if(cached != null) {
            System.out.print("cached ");
            return cached;
        }
        
        long[] lastTwo = new long[] {1, 1};
        
        for(int i = 0; i < rounds; i++) {
            long last = lastTwo[1];
            lastTwo[1] = lastTwo[0] + lastTwo[1];
            lastTwo[0] = last;
        }
        
        cacheMap.put(rounds, lastTwo[1]);
        return lastTwo[1];
     }

}

Conclusion

In this post I discussed reasons to use Hazelcast’s native Java client. I also showed a quick example of how to use one. The code can be found here.

Reference

When it comes to Beginner’s Guide to Hazelcast. I always am looking at www.hazelcast.com and www.hazelcast.org.

Beginner’s Guide to Hazelcast Webinar Source Code Links

Standard

I promised during the question and answer section of my presentation that I would post my code so everyone can have access to it.  I am going to do one better and give links to all my Hazelcast source code examples.  They are on GitHub so one can do a clone and pull them down via their favorite git client.  I use Netbeans so if one is using Netbeans, one can open up the project contained within the checkout.

Beginner’s Guide to Hazelcast Part 1

https://github.com/darylmathison/hazelcast-simple-app-example

Beginner’s Guide to Hazelcast Part 2

https://github.com/darylmathison/hazelcast-collections-example

Beginner’s Guide to Hazelcast Part 3

https://github.com/darylmathison/hazelcast-primitives-example

Beginner’s Guide to Hazelcast Part 4

https://github.com/darylmathison/hazelcast-executor-service-example

Beginner’s Guide to Hazelcast Webinar

https://github.com/darylmathison/hazelcast-webinar-example

Broke 10,000 Views – I Have Some People to Thank

Standard

This may be a small amount to some people but I don’t care. As of now, My blog has broke the 10,000 view mark. I started this blog in 2010 but the number of views were small and I might have gotten 300 views in two years. Since I got serious about keeping this blog up in 2013, I have gone from that small 300 to 10,000 in two years!

I do like to thank my followers who have, I think, get to wait an unknown amount of time for original posts to come out. I do have some organizations to thank too. Without them, I would not have been able to do this. I discovered recently that Hazelcast has been re-blogging my posts no matter what I put up. I also would like to thank Java Code Geeks for noticing me and wanting me to become one of their partners. Java Code Geeks have been posting even some of my “garbage” posts.

Well, that is it. Expect another thank you at 20,000.

Beginner’s Guide to Hazelcast Part 2

Standard

This article continues the series that I have started featuring Hazelcast, a distributed, in-memory database.  If one has not read the first post, please click here.

Distributed Collections

Hazelcast has a number of distributed collections that can be used to store data.  Here is a list of them:

  • IList
  • ISet
  • IQueue

IList

IList is a collection that keeps the order of what is put in and can have duplicates.  In fact, it implements the java.util.List interface.  This is not thread safe and one must use some sort of mutex or lock to control access by many threads.  I suggest Hazelcast’s ILock.

ISet

ISet is a collection that does not keep order of the items placed in it.  However, the elements are unique.  This collection implements the java.util.Set interface.  Like ILists, this collection is not thread safe.  I suggest using the ILock again.

IQueue

IQueue is a collection that keeps the order of what comes in and allows duplicates.  It implements the java.util.concurrent.BlockingQueue so it is thread safe.  This is the most scalable of the collections because its capacity grows as the number of instances go up.  For instance, lets say there is a limit of 10 items for a queue.  Once the queue is full, no more can go in there unless another Hazelcast instance comes up, then another 10 spaces are available.  A copy of the queue is also made.  IQueues can also be persisted via implementing the interface QueueStore.

What They Have in Common

All three of them implement the ICollection interface.  This means one can add an ItemListener to them.  This lets one know when an item is added or removed.  An example of this is in the Examples section.

Scalablity

As scalability goes, ISet and IList don’t do that well in Hazelcast 3.x.  This is because the implementation changed from being map based  to becoming a collection in the MultiMap.  This means they don’t partition and don’t go beyond a single machine.  Striping the collections can go a long way or making one’s own that are based on the mighty IMap.  Another way is to implement Hazelcast’s spi.

Examples

Here is an example of an ISet, IList and IQueue.  All three of them have an ItemListener.  The ItemListener is added in the hazelcast.xml configuration file.  One can also add an ItemListener programmatically for those inclined.  A main class and the snippet of configuration file that configured the collection will be shown.

CollectionItemListener

I implemented the ItemListener interface to show that all three of the collections can have an ItemListener.   Here is the implementation
package hazelcastcollections;

import com.hazelcast.core.ItemEvent;
import com.hazelcast.core.ItemListener;

/**
*
* @author Daryl
*/
public class CollectionItemListener implements ItemListener {

@Override
public void itemAdded(ItemEvent ie) {
System.out.println(“ItemListener – itemAdded: ” + ie.getItem());
}

@Override
public void itemRemoved(ItemEvent ie) {
System.out.println(“ItemListener – itemRemoved: ” + ie.getItem());
}

}

ISet

Code

package hazelcastcollections.iset;

import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.ISet;

/**
*
* @author Daryl
*/
public class HazelcastISet {

/**
* @param args the command line arguments
*/
public static void main(String[] args) {
HazelcastInstance instance = Hazelcast.newHazelcastInstance();
HazelcastInstance instance2 = Hazelcast.newHazelcastInstance();
ISet set = instance.getSet(“set”);
set.add(“Once”);
set.add(“upon”);
set.add(“a”);
set.add(“time”);

ISet set2 = instance2.getSet(“set”);
for(String s: set2) {
System.out.println(s);
}

System.exit(0);
}

}

Configuration

hazelcastcollections.CollectionItemListener

IList

Code

package hazelcastcollections.ilist;

import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.IList;

/**
*
* @author Daryl
*/
public class HazelcastIlist {

/**
* @param args the command line arguments
*/
public static void main(String[] args) {
HazelcastInstance instance = Hazelcast.newHazelcastInstance();
HazelcastInstance instance2 = Hazelcast.newHazelcastInstance();
IList list = instance.getList(“list”);
list.add(“Once”);
list.add(“upon”);
list.add(“a”);
list.add(“time”);

IList list2 = instance2.getList(“list”);
for(String s: list2) {
System.out.println(s);
}
System.exit(0);
}

}

Configuration

hazelcastcollections.CollectionItemListener

 IQueue

Code

I left this one for last because I have also implemented a QueueStore.   There is no call on IQueue to add a QueueStore.  One has to configure it in the hazelcast.xml file.

package hazelcastcollections.iqueue;

import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.IQueue;

/**
*
* @author Daryl
*/
public class HazelcastIQueue {

/**
* @param args the command line arguments
*/
public static void main(String[] args) {
HazelcastInstance instance = Hazelcast.newHazelcastInstance();
HazelcastInstance instance2 = Hazelcast.newHazelcastInstance();
IQueue queue = instance.getQueue(“queue”);
queue.add(“Once”);
queue.add(“upon”);
queue.add(“a”);
queue.add(“time”);

IQueue queue2 = instance2.getQueue(“queue”);
for(String s: queue2) {
System.out.println(s);
}

System.exit(0);
}

}

QueueStore Code

package hazelcastcollections.iqueue;

import com.hazelcast.core.QueueStore;
import java.util.Collection;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;
import java.util.TreeSet;
/**
*
* @author Daryl
*/
public class QueueQStore implements QueueStore {

@Override
public void store(Long l, String t) {
System.out.println(“storing ” + t + ” with ” + l);
}

@Override
public void storeAll(Map<Long, String> map) {
System.out.println(“store all”);
}

@Override
public void delete(Long l) {
System.out.println(“removing ” + l);
}

@Override
public void deleteAll(Collection clctn) {
System.out.println(“deleteAll”);
}

@Override
public String load(Long l) {
System.out.println(“loading ” + l);
return “”;
}

@Override
public Map<Long, String> loadAll(Collection clctn) {
System.out.println(“loadAll”);
Map<Long, String> retMap = new TreeMap<>();
return retMap;
}

@Override
public Set loadAllKeys() {
System.out.println(“loadAllKeys”);
return new TreeSet<>();
}

}

Configuration

Some mention needs to be addressed when it comes to configuring the QueueStore.  There are three properties that do not get passed to the implementation.  The binary property deals with how Hazelcast will send the data to the store.  Normally, Hazelcast stores the data serialized and deserializes it before it is sent to the QueueStore.  If the property is true, then the data is sent serialized.  The default is false.  The memory-limit is how many entries are kept in memory before being put into the QueueStore.  A 10000 memory-limit means that the 10001st is being sent to the QueueStore.  At initialization of the IQueue, entries are being loaded from the QueueStore.  The bulk-load property is how many can be pulled from the QueueStore at a time.
10

hazelcastcollections.CollectionItemListener

hazelcastcollections.iqueue.QueueQStore
false 10000 500

 Conclusion

I hope one has learned about distributed collections inside Hazelcast.  ISet, IList and IQueue were discussed.  The ISet and IList only stay on the instance that they are created while the IQueue has a copy made, can be persisted and its capacity increases as the number of instances increase.  The code can be seen here.

References

The Book of Hazelcast: www.hazelcast.com

Hazelcast Documentation (comes with the hazelcast download)

Beginner’s Guide To Hazelcast Part 1

Standard

Introduction

I am going to be doing a series on Hazelcast. I learned about this product from Twitter. They decided to follow me and after some research into what they do, I decided to follow them. I tweeted that Hazelcast would be a great backbone for a distributed password cracker. This got some interest and I decided to go make one. A vice president of Hazelcast started corresponding with me and we decided that while a cracker was a good project, the community (and me) would benefit from having a series of posts for beginners. I have been getting a lot of good information in the book preview The Book of Hazelcast found on www.hazelcast.com.

What is Hazelcast?

Hazelcast is a distributed, in-memory database. There are projects all over the world using Hazelcast. The code is open source under the Apache License 2.0.

Features

There are a lot of features already built into Hazelcast. Here are some of them:

  • Auto discovery of nodes on a network
  • High Availablity
  • In memory backups
  • The ability to cache data
  • Distributed thread pools
    • Distributed Executor Service
  • The ability to have data in different partitions.
  • The ability to persist data asynchronously or synchronously.
  • Transactions
  • SSL support
  • Structures to store data:
    • IList
    • IMap
    • MultiMap
    • ISet
  • Structures for communication among different processes
    • IQueue
    • ITopic
  • Atomic Operations
    • IAtomicLong
  • Id Generation
    • IdGenerator
  • Locking
    • ISemaphore
    • ICondition
    • ILock
    • ICountDownLatch

Working with Hazelcast

Just playing around with Hazelcast and reading has taught me to assume these things.

  1. The data will be stored as an array of bytes. (This is not an assumption, I got this directly from the book)
  2. The data will go over the network.
  3. The data is remote.
  4. If the data is not in memory, it doesn’t exist.

Let me explain these assumptions:

The data will be stored as an array of bytes

I got this information from The Book of Hazelcast so it is really not an assumption. This is important because not only is the data stored that way, so is the key. This makes life very interesting if one uses something other than a primitive or a String as a key. The developer of hash() and equals() must think about it in terms of the key as an array of bytes instead of as a class.

The data will go over the network

This is a distributed database and so parts of the data will be stored in other nodes. There are also backups and caching that happen too. There are techniques and settings to reduce transferring data over the network but if one wants high availability, backups must be done.

The data is remote

This is a distributed database and so parts of the database will be stored on other nodes. I put in this assumption not to resign to the fact that the data is remote but to motivate designs that make sure operations are preformed where most of the data is located. If the developer is skilled enough, this can be kept to a minimum.

If the data is not in memory, it doesn’t exist

Do not forget that this is an in-memory database. If it doesn’t get loaded into memory, the database will not know that data is stored somewhere else. This database doesn’t persist data to bring it up later. It persists because the data is important. There is no bringing it back from disk once it is out of memory like a conventional database (MySQL) would do.

Data Storage

Java developers will be happy to know that Hazelcast’s data storage containers except one are extensions of the java.util.Collections interfaces. For example, an IList follows the same method contracts as java.util.List. Here is a list of the different data storage types:

  • IList – This keeps a number of objects in the order they were put in
  • IQueue – This follows BlockingQueue and can be used as alternative to a Message Queue in JMS. This can be persisted via a QueueStore
  • IMap – This extends ConcurrentMap. It can also be persisted by a MapStore. It also has a number of other features that I will talk about in another post.
  • ISet – The keeps a set of unique elements where order is not guaranteed.
  • MultiMap – This does not follow a typical map as there can be multiple values per key.

Example

Setup

For all the features that Hazelcast contains, the initial setup steps are really easy.

  1. Download the Hazelcast zip file at www.hazelcast.org and extract contents.
  2. Add the jar files found in the lib directory into one’s classpath.
  3. Create a file named hazelcast.xml and put the following into the file
 

    
        
    
    

Hazelcast looks in a few places for a configuration file:

  • The path defined by the property hazelcast.config
  • hazelcast.xml in the classpath if classpath is included in the hazelcast.config
  • The working directory
  • If all else fails, hazelcast-default.xml is loaded witch is in the hazelcast.jar.
  • If one dose not want to deal with a configuration file at all, the configuration can be done programmatically.

The configuration example here defines multicast for joining together.  It also defines the IMap “a.”

A Warning About Configuration

Hazelcast does not copy configurations to each node.  So if one wants to be able to share a data structure, it needs to be defined in every node exactly the same.

Code

This code brings up two nodes and places values in instance’s IMap using an IdGenerator to generate keys and reads the data from instance2.

package hazelcastsimpleapp;

import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.IdGenerator;
import java.util.Map;

/**
 *
 * @author Daryl
 */
public class HazelcastSimpleApp {

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) {
        HazelcastInstance instance = Hazelcast.newHazelcastInstance();
        HazelcastInstance instance2 = Hazelcast.newHazelcastInstance();
        
        Map map = instance.getMap("a");
        IdGenerator gen = instance.getIdGenerator("gen");
        for(int i = 0; i < 10; i++) {
            map.put(gen.newId(), "stuff " + i);
        }
        
        Map map2 = instance2.getMap("a");
        for(Map.Entry entry: map2.entrySet()) {
            System.out.printf("entry: %d; %s\n", entry.getKey(), entry.getValue());
        }
        
        System.exit(0);
    }
    
}

Amazingly simple isn’t it!  Notice that I didn’t even use the IMap interface when I retrieved an instance of the map.  I just used the java.util.Map interface.  This isn’t good for using the distributed features of Hazelcast but for this example, it works fine.

One can observe the assumptions at work here.  The first assumption is storing the information as an array of bytes.  Notice the data and keys are serializable.  This is important because that is needed to store the data.  The second and third assumptions hold true with the data being being accessed by the instance2 node.  The fourth assumption holds true because every value that was put into the “a” map was displayed when read.  All of this example can be found at https://github.com/darylmathison/hazelcast-simple-app-example.  The project was made using Netbeans 8.0.

Conclusion

An quick overview of the numerous features of Hazelcast were reviewed with a simple example showing IMap and IdGenerator.  A list of assumptions were discussed that apply when developing in a distributed, in-memory database environment.

References

The Book of Hazelcast. Download from http://www.hazelcast.com