vFabric GemFire 6.6.1 and 6.6.0 Release Notes

vFabric GemFire 6.6.0 | 14 SEPT 2011
vFabric GemFire 6.6.1 | 01 NOV 2011

Last Document Update: 04 NOV 2011

What's in the Release Notes

Note: To upgrade from vFabric GemFire 6.5 or later to the current version, see the "Upgrade vFabric GemFire" chapter in the vFabric GemFire User's Guide under Getting Started with vFabric GemFire. Also review product changes documented in What's New in vFabric GemFire 6.6.0 and What's New in vFabric GemFire 6.6.1, below, to determine what changes are required for you to migrate to vFabric GemFire 6.6. In particular, you must update your licensing and licensing configuration.

The release notes cover the following topics:

What's New in vFabric GemFire 6.6.1

The 6.6.1 release of GemFire includes the following changes and enhancements:

  • Bug fixes and usability enhancements to the GemFire licensing system. In addition, the section "vFabric GemFire Licensing" in the GemFire User Guide has been rewritten for clarity. See Resolved Issues and vFabric GemFire Licensing for details.
  • Bug fixes to the GemFire Hibernate Cache Module and HTTP Session Management Modules. See Resolved Issues and the Tools and Modules section of the GemFire User Guide for details.
  • Bug fixes to the gfsh utility. See Resolved Issues for details.

These changes and enhancements are in addition to those in GemFire 6.6.0.

What's New in vFabric GemFire 6.6.0

This section summarizes new functionality in vFabric GemFire 6.6.0 and refers you to more detailed information in the GemFire documentation.

Portable Data eXchange (PDX)Serialization

In addition to the existing DataSerializable option, GemFire 6.6 offers Portable Data eXchange Serialization (PDX), a more feature-rich option for data serialization.

Note: You cannot use PDX serialization for data that can be exchanged with C++ clients.

GemFire's PDX serialization format provides the following functionality:

  • Object versioning. Add or remove fields in your domain objects without having to stop the entire GemFire system. GemFire manages multiple versions of domain objects in the cache, allowing you to gradually introduce new versions of your applications.
  • Language independence. You can define serialization for your domain objects on a .NET client and store, query, and run functions against them on the servers without defining the object on the Java server. This allows you to completely manage the objects in the client without needing to create Java classes for the server side.
  • Reduced deserialization of serialized objects. The access methods for PdxSerializable objects allow you to examine specific fields of your domain object without deserializing the entire object. Depending on your object usage, this can cut down significantly on serialization and deserialization costs. The query engine automatically recognizes PDX objects, retrieves the PdxInstance of the object and uses just the fields it needs.
  • Automatic reflection-based serialization. You can configure your cache to serialize and deserialize domain objects automatically without adding any extra code to them.

In the GemFire User's Guide under Developing with vFabric GemFire, see the "Data Serialization" chapter.

Client-Initiated Transactions

GemFire now supports transactions that run from a client member. Previously, to perform a transaction from a client member, you needed to run a function on a server node that executed a transaction. In the GemFire User's Guide under Developing with vFabric GemFire, see the "Transactions" chapter.

Expanded Control over Partitioned Region Data Placement

By default, the physical location of data is abstracted away from the application. vFabric GemFire 6.5 gave you control over data colocated within a partitioned region and between partitioned regions. vFabric GemFire 6.6 gives you more control over data partitioning and placement, through these new features:

  • Fixed partitioning. You can control which members host the primary and secondary copies of each data entry. This function is useful in cases such as installations with heterogeneous hardware, or where the proximity of certain hardware elements to electronic exchanges may be critical. You implement fixed partitioned regions using the GemFire FixedPartitionResolver inside the existing custom partitioning framework.
  • Partitioned region redundancy zones. Redundancy zones let you separate primary and secondary copies by member groups, or zones. You assign each data host to a zone. Then GemFire places redundant copies in different redundancy zones, the same as it places redundant copies on different physical machines. You can use this to split data copies across different machine racks or networks, This option allows you to add members on the fly and use rebalancing to redistribute the data load, while preserving the redundancy zones you have specified.
  • Option to force use of different physical machines for redundant data copies. By default, GemFire places your primary and secondary data copies for you, avoiding placement of two copies on the same physical machine. If host members are not running on enough machines to keep different copies separate, GemFire places copies on the same physical machine. You can change this behavior so that GemFire only uses separate physical machines.

In the vFabric GemFire User's Guide under Developing with vFabric GemFire, see the "Partitioned Regions" chapter.

gfsh, New Command-Line Tool

vFabric GemFire has a new command line tool, gfsh (pronounced "g-fish"), for accessing and editing cached data. The tool includes a set of Unix-style commands through which you can easily access data, monitor system members, redirect outputs to files, run batch scripts, execute custom functions, and much more. You can also plug in custom commands tailored for your application. The tool is available in the bin directory of the product installation, as gfsh for Unix/Linux and gfsh.bat for Windows.

In the vFabric GemFire User's Guide, see gfsh under Deploying and Running vFabric GemFire.

Licensing Changes

The vFabric GemFire licensing model has changed to use VMware's licensing model. You no longer need to provide licensing information within client applications. The peer/server licensing has new configuration requirements. These are the primary licensing changes from prior releases:

  • Licensing model. Prior to this release, GemFire licenses were "node locked" to particular machines. Now, the model is based on your distributed system member topology. The following topology configurations apply to licensing for version 6.6:
    • Cache Servers. You need licensing for cache servers to run client/server installations and multi-site (WAN) installations. These licenses are called Data Management Node licenses. Physical servers hosting Data Management Nodes are licensed per-CPU.
    • Multisite (WAN). To use GemFire's WAN functionality, each Data Management Node license must be accompanied by a WAN Upgrade license.
    • Clients. Requires an Application Cache Node license indicating the maximum number of clients you will connect to each server. Alternatively, you can apply Unlimited Client Upgrade licenses to the Data Management Nodes.
    • Number of peer-only members. You need licensing for distributed system members that are not servers. These members use an Application Cache Node license.
  • License types.
    • Default evaluation license. The default evaluation license has not changed. As in version 6.5.0, GemFire comes bundled with a default evaluation license that allows you to run a distributed system with up to three members and with up to three clients for each cache server member that you run. The bundled default evaluation license was new in version 6.5.0.
    • Custom evaluation license. New. This is a special license with a fixed expiration date. You can get this license if you need more functionality for your evaluation than the default evaluation license allows.
    • Development and production licenses. GemFire no longer distinguishes between development and production licenses. Production licenses work for development and production.
  • License serial numbers. Instead of a license zip file, you now will use license serial numbers. You configure your licensing in the server and peer member's gemfire.properties files in the new license-application-cache or license-data-management properties.

    Example for peer member:

    Example for cache server:

  • Gemfire license command removed. The license command has been removed from the gemfire command-line tool.

For more information about GemFire 6.6 licensing, contact your account manager and visit the VMware Page on How to Buy GemFire. In the vFabric GemFire User's Guide under Deploying and Running vFabric GemFire, see the "Licensing" chapter.

If you obtain vFabric GemFire as part of a vFabric Cloud Application Platform package, refer to the licensing information in Getting Started with vFabric Cloud Application Platform.

Change to Order of Method Calls for Reconnecting Durable Clients

If you used durable clients before vFabric GemFire 6.6, you must change the order of your method calls on durable client reconnection. Before, you created your cache and regions, then called the Cache readyForEvents method, followed by your interest registration methods. You need to switch the order of the last two steps.

The new instructions are to program your durable client's reconnection to:

  1. Connect, initialize the client cache, regions, any cache listeners, and create and execute any durable continuous queries.
  2. Run all interest registration calls.
  3. Call ClientCache.readyForEvents so the server will replay stored events. If the ready message is sent earlier, the client may lose events.

The new code order follows:

ClientCache clientCache = ClientCacheFactory.create();
// Here, create regions, listeners, and CQs that are not defined in the cache.xml . . .
// Here, run all register interest calls before doing anything else

In the vFabric GemFire User's Guide under Developing with vFabric GemFire, see the "Events and Event Handling" chapter.

Updated Delta Propagation Statistics

Delta propagation statistics are consolidated into CachePerfStats and CacheClientProxyStatistics as follows:

  • Updated CachePerfStats. These delta statistics have been added to CachePerfStats:
    • deltaUpdates. Number of times entries in this cache have been updated using deltas.
    • deltaUpdatesTime. Time spent applying the received deltas to entries in this cache.
    • deltaFailedUpdates. Number of times entries in this cache failed to be updated using deltas.
    • deltasPrepared. Number of times deltas were prepared in this cache.
    • deltasPreparedTime. Time spent preparing deltas in this cache.
    • deltasSent. Number of times a delta was sent to remote caches. This does not include deltas sent from server to client.
    • deltaFullValuesSent. Number of times a full value was sent to a remote cache.
    • deltaFullValuesRequested. Number of times a full value was requested by this cache.
  • Updated CacheClientProxyStatistics. These delta statistics have been added to CacheClientProxyStatistics:
    • deltaMessagesSent. Number of client messages containing only deltas dispatched to the client. (This stat was previously called preparedDeltamessages.)
    • deltaFullMessagesSent. Number of client messages dispatched with full value in response to failed delta at client.

Delta propagation statistics have been removed from the following statistic types:

  • ClientStats
  • DistributionStats
  • PartitionedRegionStats
  • CacheServerStats
  • CacheClientUpdaterStats

Support for Query Bind Parameters from Clients

You can use query bind parameters in client queries to improve query performance.

Query Order By on Partitioned Regions

You can run a query using the order by clause on a partitioned region. Before this release, such query execution returned an exception.

  • Requirement for Using Order By Querying on Partitioned Regions. The fields specified in the order by clause must be part of the projection list.
    // This query works because p.status is part of projection list
    select distinct p.ID, p.status from /region p where p.ID > 5 order by p.status
    // This query works providing status is part of the value indicated by *
    select distinct * from /region where ID > 5 order by status
  • Execution Details for Order By Querying on Partitioned Regions. When an order by clause is used with a partition region query, the query is executed separately on each region host, the local query coordinator, and all remote members. The results are all gathered by the query coordinator. The cumulative result set is built by applying order by on the gathered results. If the limit clause is also used in the query, order by and limit are applied on each node before each node’s results are returned to the coordinator. Then the clauses are applied to the cumulative result set to get the final result set, which is returned to the calling application.

Index Support for Overflow Regions

You can now use indexes for querying on overflow regions with these caveats:

  • You must use synchronous index maintenance for the region. This is the default maintenance setting.
  • The index from clause must specify only one iterator and it must refer to the keys or entry values. The index cannot refer to the region’s entrySet.
  • The index data itself is not stored overflowed to disk

These example index creation calls do not work for overflow regions.

// There are two iterators in the from clause.
createIndex("secIdIndex", IndexType.FUNCTIONAL,"b.secId","/portfolios pf, pf.positions.values b");
// The from clause specifies the entrySet
createIndex("indx1", IndexType.FUNCTIONAL, "entries.value.getID", "/exampleRegion.entrySet() entries");

This example index creation call works for overflow regions. The index is on the region, specifies the region entries or region keys, and only has one iterator.

createIndex("pkidIndex", IndexType.FUNCTIONAL, "p.pkid", "/Portfolios p");
"indx1", IndexType.FUNCTIONAL, "ks.toString", "/portfolio.keys() ks");

In the vFabric GemFire Users’s Guide under Developing with vFabric GemFire, see the "Querying" chapter.

Enhancement to the like Predicate in Queries

In queries, the like predicate formerly supported only the special character ‘%’ at the end of the comparison string. With this release, the special characters ‘%’ and ‘_’ are supported at any position in the comparison string. likepredicate.

HA Support for FunctionServer.onServer(Pool pool)

FunctionService.onServer(Pool pool) now supports high availability (HA). If the method call fails because of server failure on the first server, GemFire retries it on other available servers until it succeeds or until no servers are available.

Statistics Type Included with Statistics Name

The statistics type is now appended to the statistics name.

cacheserver Script Changes

  • The cacheserver script had a bug that caused it to ignore the command-line arguments -server and -port. The script now uses the arguments as documented.
  • By default, if you run a cache server with no address or port settings, the server listens for clients on the default port on the server's machine host. The cacheserver command-line script now accepts a parameter, -disable-default-server, that causes it to run without listening for client connections. A cacheserver run in this way is a peer in its distributed system, but does not perform any server activities.

New Bind Address Setting for Gateway Hubs

You can now set a bind address for the gateway hub that is independent of any other bind address setting. Before this release, the gateway hub used the same bind address setting as the cache server. This new setting allows you to separate traffic for peer-to-peer, cache server, and gateway hubs to three different addresses. Set the gateway hub bind address in the cache.xml in the bind-address attribute. The com.gemstone.gemfire.cache.util.GatewayHub interface has comparable get and set methods. For example:

<gateway-hub id="LN" bind-address="" port="22220">

New JCA Resource Adapter

Gemfire participates in JTA transactions by implementing the Synchronization interface and registering itself with the JTA Transaction Manager to receive beforeCompletion and afterCompletion callbacks. In the beforeCompletion callback, preparatory work is done to see whether the transaction can proceed. If affirmative, GemFire does the appropriate locking to guarantee that GemFire will commit successfully in the afterCommit. The actual distribution of transaction state is done in the afterComplete callback, after the JTA transaction has completed.

In prior releases, a network or node failure could cause GemFire to miss some steps in distributing and committing the transaction events while the XAResources participating in the transaction successfully completed the transaction.

Now, GemFire includes a JCA Resource adapter for GemFire. This allows the JTA transaction coordinator (usually an application server container like WebLogic or WebSphere) to first send a prepare command to the participating XAResources before the commit. If the prepare phase completes successfully, then the coordinator issues a commit to the non XAResource (the GemFire JCA Resource adapter). The non XAresource participates in the transaction as a local transaction resource. If the non XA resource fails, the coordinator can roll back the entire transaction. This ensures data consistency across all resources participating in the transaction.

Improved OSGi and Java EE Support

The GEMFIRE/lib/gemfire.jar and GEMFIRE/lib/antlr.jar are now deployable as bundles within OSGi environments, including Virgo Server 2.1.1. For the recommended approach to using GemFire within OSGi, see the new OSGi Deployment Example in examples/osgi

This release includes improvements to GemFire’s use of classloaders. GemFire is more versatile at locating classes and other resources within container environments, including OSGi and Java EE.

JTA Support for Containers Other Than WebLogic and JBoss

In prior releases, JTA was supported only for WebLogic and JBoss. JTA support is now provided for many more application server containers, including the Cosminexus server from Hitachi.

Faster Startup and Recovery for Persistent Regions

vFabric GemFire becomes available more quickly during startup with persistent regions. It accelerates availability by using index files that are created and maintained during disk store persistence. At startup, GemFire first reads the entry keys from the index files, then reads the values asynchronously. If a member gets an entry before its value is loaded into memory, GemFire fetches it immediately from the data store. In addition to this functionality, this version of GemFire also reads data found in secondary buckets asynchronously. These two features improve overall startup performance.

Delta Propagation Optimized

Delta propagation of subscription events has been optimized to use less memory than before.

Negative hashCode Value No Longer Causes PR Single Hop to Fail

In GemFire 6.5.0, if the hasCode of a key was negative, it would cause failure of PR single hop access to that key. This problem has been fixed.

Improvements to PR Single Hop, Function Execution, and Bulk Operations

Client side metadata for PR single hop, function execution, and bulk operations is now more accurate. You should see operations being sent to the correct server more often than in prior releases.

Improvements to JMX Agent

  • The JMX Agent code is streamlined to reduce CPU usage.
  • The JMX Admin Agent Log Level is Now Configurable. The JMX agent registers with peer members to receive alerts above a certain level. The default level is ‘warning’.
    In the past, the AdminDistributedSystem MBean exposed this alert level as a read-only attribute, 'alertLevel'. This attribute is now writable and you can modify the alert level while the agent is running. Valid String values for this attribute are (case insensitive): warning, error, severe, off.

Query Optimization

Performance is improved for queries that use the order by clause, especially where an index is available on the order by expression.

Struct Object Serialization Optimization

The serialization of Struct objects is optimized.

Quick Start Example Changes

  • Overflowing Data to Disk Example. This example is updated to demonstrate compaction of overflow files.
  • Multiple Secure Client Example. The XML used in this example to initialize the server is updated to use predefined region shortcut refids such as REPLICATE and PARTITION.

In the vFabric GemFire User's Guide under Getting Started with vFabric GemFire, see the "Quick Start Examples" chapter.

Documentation Changes

The vFabric GemFire Java product documentation has changed significantly to improve ease of use. It now consists of one document, vFabric GemFire User's Guide, which is organized by functional area:

  • Getting Started with vFabric Gemfire. Verify system requirements, perform initial product setup, complete the tutorial and example walk-throughs, and understand GemFire features and functionality.
  • Basic Configuration and Programming. Configure GemFire distributed system members, member caches, regions, and data entries.
  • Topologies and Communication. Learn about GemFire system topologies and implement the topology you need. Perform related tasks for using bind addresses, grouping your servers into logical groups, and balancing communication load in client/server and WAN installations.
  • Developing with vFabric GemFire. Configure and manage your data and data events. Choose a data storage model such as partitioning or replication. Manage data storage and access, data loading, data eviction and expiration, data serialization, event handling, querying, transactions, and remote function execution.
  • Managing vFabric GemFire. Perform general administration of GemFire. Set up and manage memory use, disk storage for your data and event queues, security, network partitioning, and JMX with GemFire. Monitor, tune, and troubleshoot the system.
  • Deploying and Running vFabric GemFire. Deploy, start, and stop your GemFire system. Manage your configuration and output files. Learn about the gemfire and cacheserver command-line utilities and the GemFire locator process.
  • Tools and Modules. Learn about tools included with GemFire, such as the HTTP Session Management module.
  • Reference. Review general reference information that relates to all phases of GemFire programming and deployment: properties listings, system exceptions and failures, and memory requirements for cached data.
  • Glossary. Look up terms used in the documentation.

As part of the reorganization, client/server and multisite (WAN) documentation, which formerly had their own chapters, are now integrated into the rest of the topics. For example, information on configuring clients and servers to communicate is now under Topologies and Communication. Client interest registration and durable client management information is now under Developing with GemFire, in the "Events and Event Handling" chapter.

gemfire stats command

The gemfire stats command will now also print out the most recent value of a statistic. It is labelled "last=XXX".

cache.xml Enhancements

  • cache.xml now supports an "initializer" element that allows you to specify a declarable that will be run when you after all other cache.xml initialization is complete.
  • cache.xml now supports variable substitution.

Peers and Severs on Different Patch Releases Can Join Distributed System

Peers and servers can now join the same distributed system even if they are on different patch releases. This allows you to update one member at a time instead of needing to shutdown your entire distributed system. All member of the distributed system must have the same first two fields in their GemFire version; any of the other fields are allowed to be different. Fields are delimited with a '.'.

New 6.6 APIs

DataSerializer: write/readConcurrentHashMap

ClientCache.getCurrentServers(): Returns a set of the servers to which this client is currently connected



* Returns a set of the other non - administrative members in the distributed system.

* @since 6.6


public Set<DistributedMember> getMembers();


* Returns a set of the administrative members in the distributed system.

* @since 6.6


public Set<DistributedMember> getAdminMembers();


* Returns a set of the members in the distributed system that have the

* given region. For regions with local scope an empty set is returned.

* @param r a Region in the cache

* @since 6.6


public Set<DistributedMember> getMembers( Region r);

Resolved Issues

Issues Resolved in 6.6.1

For information about issues that have gone through the VMware bug tracking system and are resolved, see BugsFixedGemFire661.html

Issues Resolved in 6.6.0

For information about issues that have gone through the VMware bug tracking system and are resolved, see BugsFixedGemFire66.html

Cleaner Shutdown for Cache Servers

Occasionally, if an NPE was thrown during a DistributedSystem disconnect in the cache server process, the process would hang. This has been fixed, providing a cleaner shutdown of the cache server process.

Correction to gemfire encrypt-password Output

The command-line command, gemfire encrypt-password, produced output outside of the ascii range that did not work. This problem has been fixed.

Mismatched Cache Server and Agent Versions Work Together

Mismatched versions of cache servers and agents no longer negatively affect the cache server.

Corrected Gateway Event Conflation for Empty Regions

Gateway events on empty regions were not being conflated when conflation was enabled for the gateway. This problem has been resolved.

Corrected Null member-id in Authorization Request

Previously when using vFabric GemFire security, the authorization callback was invoked with a null member ID provided for the joining member. This problem has been resolved.

Backup of Disk Stores Prevents Deletion of Oplog Files

vFabric GemFire no longer permits oplog file deletion during disk store backup. Before this correction, oplog files could be deleted during backup, which could invalidate some disk stores.

Disk Stores Properly Handle putIfAbsent and Creates with NULL Values

putIfAbsent and create with NULL values are now treated correctly in the disk store.

Resolved Problem for Disk Stores When Running with Java 1.5

In vFabric GemFire running with Java 1.5, if multiple members backed up persistent disk stores to the same directory, one of the members may have failed to back up because it could not create the directory. This problem has been resolved.

CachePerfStats.nonSingleHopCnt Corrected

The statistic CachePerfStats.nonSingleHopCnt has been updated and correctly indicates the number of times the client performed non-single hop operations.

Correction to Key Query Retrieval on Overflow Regions with Key Indexes

This problem affected overflow regions with an index on the region keys. For these regions, queries written to retrieve the entry key contents would instead return the entry values. This problem has been resolved.

System Cleanup Finishes Correctly After Region Destroy

The system correctly finishes cleaning up if a member crashes or backs up while in the middle of destroying a region.

Improved Partitioned Region Single-Hop Behavior in Relaunched Servers

In prior versions, when servers with persistent partitioned regions were relaunched, the servers did not provide the clients with the correct initial single hop metadata. This resulted in extra network hops from the clients to access persistent partitioned regions. This problem has been fixed.

Partitioned Region Bucket Validation Added for Colocated Regions

vFabric GemFire now validates that the number of buckets in colocated partitioned regions is the same.

Improvement to Redundancy Recovery in Partitioned Regions

There was a problem with how GemFire satisfied partitioned region redundancy. The problem may have caused GemFire to bypass some members that should have been used for redundancy. This problem has been resolved.

Solution to NullPointerException in Recovery of Persistent Partitioned Regions with Eviction

This problem applied to persistent partitioned regions that are configured with eviction. When recovering region data from disk, GemFire threw a NullPointerException. Error messaging might have looked like this:

[error 2011/04/21 10:04:15.085 PDT tid=0x1] A DiskAccessException? has
occurred while writing to the disk for region /myPartitionedRegion. The
region will be closed. com.gemstone.gemfire.cache.DiskAccessException?:
For DiskStore?: DEFAULT: RuntimeException? in initializing the disk store
from the disk, caused by java.lang.NullPointerException?
com.gemstone.gemfire.internal.cache.DiskStoreImpl?.initializeOwner(DiskStoreImpl?.java:465) at
com.gemstone.gemfire.internal.cache.DiskRegion?.initializeOwner(DiskRegion?.java:216) at
com.gemstone.gemfire.internal.cache.DistributedRegion?.getInitialImageAndRecovery(DistributedRegion?.java:1077) at
com.gemstone.gemfire.internal.cache.DistributedRegion?.initialize(DistributedRegion?.java:972) at
This problem has been resolved.

Correction to Partitioned Region Transactions in Client/Server and WAN

Cache events for partitioned region transactions were not sent to clients or to remote WAN sites. This problem has been fixed.

Correction to Client Connection Errors for Missed Class Loads

In prior releases, when a client connected to a server, the server sent all instantiators that might be required for object deserialization, and the client loaded all classes. If classes were missing from the client CLASSPATH, the connection attempt threw a NoSubscriptionServersAvailableException. Now, classes are not loaded at client connection time. GemFire loads each class as needed, the first time the client deserializes the object. If GemFire does not find the class it needs to load, the deserialization attempt throws a ClassNotFoundException.

Correction to Client Backwards Compatibility

This version fixes a regression in backward compatibility for clients with some 6.5 servers.

Correct Result Set Returned for Queries Using Order By and Limit

Queries using both the order by and limit clauses applied first the limit, then the ordering, causing them to return inconsistent and incorrect results. Now, the query applies first the order by clause and then the limit clause, so the results returned are the first ones in the ordered list.

Locale-Specific Changes

Persistent disk stores did not work when the locale was specifically set to anything other than English or when the default locale was not English. This problem was due to file naming issues, and it has been resolved.

Corrected Initialization for Partitioned Region with Changed Region Configuration

GemFire stores metadata for partitioned regions, which it uses to initialize new members when they define a partitioned region. A bug caused region metadata to remain cached if all members that defined a region were stopped and other members of the system, which did not have the region defined, kept running. This situation caused problems if you tried to change the region’s configuration to one incompatible with the cached configuration. When the new version of the region was created, GemFire would detect the difference between the stored metadata and the new region configuration and would not initialize the region.

This problem resulted in startup messages like the following:

[info 2011/01/01 08:23:14.773 PDT pcs tid=0x1] Region PR requesting
initial image from alphi(30208) :32015/32792
[warning 2011/01/01 08:23:14.774 PDT pcs tid=0x1] Initialization failed
for Region /house2 java.lang.IllegalStateException?: The entry idle
timout set in RegionAttributes? is incompatible with entry idle timout
used by other distributed members. Exception in thread "main"
java.lang.IllegalStateException?: The entry idle timout set in
RegionAttributes? is incompatible with entry idle timout used by other
distributed members.
. . . .

This problem has been resolved.

Correction for NullPointerException on GemFire Startup

Periodically, GemFire hit a NullPointerException with messages in the logs:

[severe 2011/05/24 18:02:46.538 BST tid=0x27]
Uncaught exception in thread Thread[Thread-3 StatSampler,10,StatSamplerThreads]
at java.lang.Thread.run(Thread.java:619)

This probelm has been resolved. It was tracked with bug 41474 and support ticket 7824.

Short Type Indexes No Longer Cause an Exception on Entry Removal

If a region had a functional index on an attribute of type Short, vFabric GemFire threw an IndexMaintenanceException on entry removal. No problems were encountered with entry creation or update in the same region. The core error was:

java.lang.ClassCastException?: java.lang.Integer cannot be cast to
java.lang.Short at java.lang.Short.compareTo(Unknown Source) at
java.util.TreeMap?.getEntry(Unknown Source) at
java.util.TreeMap?.get(Unknown Source) at

The logged exception started with these lines:

exception in thread "main"
com.gemstone.gemfire.cache.query.internal.index.IMQException, caused by
com.gemstone.gemfire.cache.query.internal.index.IMQException at
RegionEntry?.java:681) at
com.gemstone.gemfire.internal.cache.AbstractRegionMap?.destroy(AbstractRegionMap?.java:778) at
com.gemstone.gemfire.internal.cache.LocalRegion?.mapDestroy(LocalRegion?.java:5352) at

Removing the functional index on the Short-type attribute allowed entry removal to work. This problem was tracked with bug 43519.

Region Entry Count Inconsistencies Addressed

vFabric GemFire has some problems in entry count management. Depending on your region configuration and the operations you perform, this could be evident to you in various ways, or not at all. You might see a mismatch between getKeysOnServer and Region.size results. Or your region LRU count might diverge from the region size. One significant source of this problem has been addressed. This problem is being tracked with bug 42228.

OutOfMemory Errors with Memory Consumed By Atomic50StatisticsImpl

A customer reported frequent OutOfMemory errors with 92% of the memory consumed by an object of the class com.gemstone.gemfire.internal.stats50.Atomic50StatisticsImpl. This problem was due to a memory leak that occurred when statistic-sampling-enabled was set to false or when the sampling interval was set very high. The problem was more likely to occur if many threads wrote statistics. This problem is resolved and was tracked with bug 42527.

Resolved Multiple Identical Results from Distinct Query

A bug in querying might have caused a query with DISTINCT specified to return multiple identical results. This problem is resolved.

Entry Remained After Clear of Persistent Region with Async Writes

When you put an entry into a persistent region that was configured for asynchronous disk writes, the put is first done in the region, then the put event is added to the async queue to be flushed to disk. Formerly, if the region was cleared while an entry put was being done and before the entry event was written to the queue, the event was not cleared along with the other enqueued events. The event was then enqueued and written to disk, and the entry could be recovered back into the region. This problem was fixed and ttracked with bug 43312.

Oplog Deletion Left Old Oplog Files in Windows Systems

In Windows systems only during oplog compaction, the compaction left empty crf and drf oplog files open and then attempted to delete them. This problem caused a failure with DiskAccessException in Windows systems. The probelm is resolved. It was tracked with bug 42797 and support ticket 7430.

NullPointerException During Index Maintenance

A bug in index maintenance caused an NPE if the index FROM clause had multiple iterators and if for any of the region entries, there were at least two tuples with the same index key. The problem is resolved.

Rebalancing Parent Persistent Colocated Partitioned Region Before Recreating Child Caused Unrecoverable Region

If two persistent partitioned regions were colocated, it was possible to recreate only the parent partitioned region and rebalance it. At that point, the child region's persistent data would no longer be colocated with the parent data, making the child region unrecoverable. The workaround to this was to recreate all persistent colocated partitioned regions before rebalancing the system. The problem is resolved. It was tracked with bug 43570.

Persistent Colocated Regions With recovery-delay>=0 Could Hang on Restart

Having several persistent partitioned regions that were colocated could result in a hang on restart if recovery-delay is set to a non zero value. The problem is resolved and was tracked with bug 43552.

OutOfMemoryException with Many Messages in the NakReceiverWindow of the JGroups NAKACK Protocol

A problem in JGroups message retransmission sometimes caused members of the distributed cache to accumulate messages and eventually run out of memory. This problem affected both multicast and non-multicast configurations of the product. The problem was caused by faulty size calculations in JGroups during the in gemfire.properties to 40000 bytes. The problem was tracked with bug 43652 and was resolved.

BucketMovedException was Thrown After All Results are Collected, After Last Result Is Sent

During function execution on partitioned region, a check is done for the local buckets' availability. During function execution, if the local buckets are not available, a BucketMovedException should be thrown. This check and exception throwing occurred after the last result was sent, when it should have been done before starting the function execution and before sending the results. The problem was tracked with bug 43630 and was resolved.

ConcurrentModificationException with Modification of System Properties During Initialization of GemFire Cache

Adding or modifying the system properties during the initialization of a GemFire Cache sometimes resulted in ConcurrentModificationException. The problem was tracked with bug 43686 and was resolved.

Hang in StateFlushOperation.flush Waiting for Replies While Targeting a Process That was No Longer Running

It was possible for a member of the peer-to-peer cache to hang while attempting to create a Region. The hang would have shown the process stuck in StateFlushOperation.flush() waiting for a message from a member that recently had left the distributed system. The 15-second warning would have shown that it was targeting that member and that it was waiting for replies from other members. The problem was tracked with bug 43583 and was resolved.

Exception String Did Not Provide Serializer ID

If the serializer that can deserialize an object is not registered, an IOException is thrown. The ID of the missing serializer was not provided in the Exception message string, and the message reported the serializer as “0”. The proper serializer ID is now provided in the exception string. The problem was tracked with bug 43741 and support ticket 7998.

Cleanup of Shutdown Log Messages

Spurious messages that were being logged during shutdown have been removed.

Client getAll Incorrectly Added Entries to the Local Cache

Client getAll no longer adds entries to the local cache for entries that do not exist on the server.

Standalone Members with Partitioned Regions Could Not Execute Functions on the Region

Standalone members that host partitioned regions can execute functions on the regions. In previous releases, a ClassCastException was thrown if this execution was attempted. The resolution allows single-node tests for partitioned regions.

Client Index Creation Unsupported

In vFabric GemFire 6.5, index creation on client regions threw an UnsupportedOperationException. The problem is resolved in this version.

Incorrect FunctionException from Client Function Execution

Under some failover scenarios, function execution from the client did not mark the last result as being the final response, and it erroneously threw a FunctionException. Now, the last result is marked as the final response to the function before it is written to the socket.

Incorrect Path Separators Used in Loading cache.xml

Correct '/' path separators are now used in all situations for loading the cache.xml.

Spurious UnsupportedException Thrown by Security Module

A spurious UnsupportedException was thrown in the security module. The problem was resolved.

ArrayIndexOutofBoundsException from getAll

An incorrect ArrayIndexOutofBoundsException was thrown by getAll. The problem was resolved.

OOME from Long-Running Locator

A memory leak in the locator service sometimes caused an out-of-memory exception (OOME) if the locator was left running for a long time. The problem was resolved.

InvalidToken Returned from Query

A problem in query indexing with asynchronous index maintenance caused some queries to return InvalidToken. The problem was tracked with bug 42922 and was resolved.

Multiple Order By Fields in Query Processed Randomly

A problem with querying caused multiple order by fields to be processed in a random order. The problem was resolved.

Problem With Primary Key Index Used with Limit or Order By

When a PrimaryKey index was used in a query that included limit or order by criteria, the query sometimes returned incorrect results. The problem was resolved.

Admin API Connect Can Fail With Some Containers

Connections could hang when the Admin API was used with JBoss and possibly other containers. The problem was tracked with bug 43153 and support ticket 7626. It was resolved.

Bad Message: "Current Connection count of 2 is greater than the 800 Max"

vFabric GemFire used this message incorrectly for unrelated problems. The message was revised to make sense. It was tracked with bug 43363.

Known Issues

Known Issues in 6.6.1

See BugNotesGemFire66.html.

Known Issues in 6.6.0

This section lists known issues in vFabric GemFire and its supported platforms. For issues that have been registered as bugs in the VMWare bug tracking system, see BugNotesGemFire66.html.

put May Update Local Cache in Spite of Exception

During a distributed put operation, if a remote cache fromDelta or fromData operation throws an exception that is returned to the put call, the entire operation should fail. In some cases, however, the update is performed in the cache where the put was invoked in spite of the thrown exception.

Locators Log Messages Indicating ClassNotFoundException

Locators may try unnecessarily to load classes that use custom data serializers or instantiators classes. If the classes are not available in its classpath, the locators may log warnings stating ClassNotFoundException. This problem is tracked with bug 43677 and support ticket 7940.

You Cannot Allow Multiple Threads to Destroy the Same Region

When multiple threads are destroying a region or aborting region creation, a race condition in the setting and checking of the region's destroyed status can cause the region mapping to be nulled out multiple times. This problem in turn can corrupt statistics and static data. Workaround: Write your code so that:

  • destroyRegion acquires the destroyLock
  • close and localDestroyRegion do not acquire the destroyLock

Using IPv6: RMI Does Not Allow Server to Bind to a link-local Address

All IPv6 link-local addresses start with "fe80:*". GemFire JMX agent binds to the local host unless an rmi-bind-address is provided as a system property, in the gemfire.properties file, or through the command line while start the agent. On many common configurations running Windows operating system, this causes an issue as ava.net.InetAddress.getLocalHost() returns an IPv6 link-local address and the user cannot correct the problem easily.

Workaround: Provide an rmi-bind-address explicitly.

On systems running Windows, use the command ipconfig /all tool to list all valid IP addresses.

For Linux/AIX/Solaris, use the command ifconfig -a tool to list all valid addresses for a machine.

The following command starts an agent and provides an RMI bind address so that it does not conflict with IP addresses used by the system:

.\agent.bat start rmi-bind-address=fdf0:76df:b0ed:9449::17

This issue occurs commonly on systems running Windows and rarely on systems running other operating systems.

JMX Agent

  • CLASSPATH must include activation.jar to run GemFire with Java 1.5. This requirement applies to email alerts. The activation.jar is available in the JavaBeans Activation Framework download.
  • AdminException: Failed to refresh statistics. You may receive a few occurrences of this exception. These can be safely ignored if, in the same timeframe, there is this log statement:
    Processing client membership event from for client with
    id: running on host:

    These exceptions could appear in the logs until the cleanup event received from the server is completely processed.

  • Agent becomes inaccessible over JMX if rmi-registry-enabled is set to false. The Agent boolean rmi-registry-enabled indicates whether the JMX Agent should start the RMI Registry or use an external RMI Registry. The default value for this is true and the RMI Registry is started by the agent. If this property is set to false, the rmi-bind-address must be set for the agent to start up. However, even though the Agent starts, it can not be accessed over JMX.
  • NullPointerException is observed sometimes in JMX Admin Agent logs if the member(s) are continuously & rapidly joining/leaving the busy GemFire Distributed System.  
  • JMX Admin Agent Requires CLASSPATH to Include Java Classes for Non-Standard Keys and Values. The JMX agent can't detect servers when servers have regions with non-standard Java Classes used as Key and/or Value constraints. JMX Admin Agent requires these constraint classes to be present in its CLASSPATH.

Saving Statistic Alert Definitions in a JMX Admin Agent

While making changes to the statistic alert definitions of a JMX agent, if you do not have write permissions to the agent.ser file or to the directory that contains the agent.ser file, a warning is displayed indicating that the user cannot save additional definitions to the file.

Locators Must Be Used for Member Discovery While Using NIC Bonding and GemFire

When using NIC-bonded Ethernet cards on a Linux platform, regardless of the NIC bonding mode, do not use multicast-based member discovery. NIC bonding on Linux platforms causes issues with GemFire multicast member discovery capability, and members might not locate each other, especially during member startup.

Workaround: Always use locator-based member discovery.

Weblogic Server and UTF Encoding

Some Weblogic servers cannot process XML files with a UTF encoding specified. Workaround: To disable the UTF encoding, make sure the cache.xml file specifies and not .

Acrobat Reader 6.0 Links Always Go to First Page

There is a known problem with Acrobat Reader 6.0 that causes links between pdf files to always go to the first page.

Workaround: If you encounter this problem while viewing the documentation pdfs, you can try these steps. Open Acrobat 6, open the Edit menu, select Preferences, click on General, then uncheck the "Open cross-document links in same window" option.