VMware

vFabric GemFire 6.6.2 Release Notes

vFabric GemFire 6.6.2 | 08 MAR 2012

Last Document Update: 13 JUN 2012

What's in the Release Notes

The release notes cover the following topics:

What's New in vFabric GemFire 6.6.2

The VMware® vFabric™ GemFire® 6.6.2 release includes the following changes and enhancements. These changes and enhancements are in addition to those in GemFire 6.6.0 and 6.6.1.

Querying Enhancements

vFabric GemFire 6.6.2 enhances the native querying feature as follows. (See also the Querying chapter in the GemFire User's Guide under Developing with vFabric GemFire.)

  • Faster performance rates for queries on partitioned regions and for queries that use indexes.
  • Ability to query a partitioned region on a single node. Execute a query on a specific subset of data located in a partitioned region. By providing a RegionFunctionContext when you execute the query, you can target the specific node that hosts the partitioned region data. This ability improves query performance significantly. See New and Modified APIs.
  • Improved query and index creation. You no longer need to specify an index type when creating an index. You can use the QueryService's createKeyIndex method to create an index on keys. See New and Modified APIs.
  • Query language support for COUNT and ORDER BY keywords. Write queries that use the COUNT keyword, and use ORDER BY when querying a partitioned region.
  • Query debugging at query or system level. Enable query debugging on the query level by adding the <trace> keyword before your query string. Alternately, enable query debugging at the system level by setting the gemfire.Query.VERBOSE system property. For example:
    • programmatically: System.setProperty("gemfire.Query.VERBOSE", "true");
    • cacheserver startup script: cacheserver start -J-Dgemfire.Query.VERBOSE=true
  • New query-dependent and query-independent index statistics.
    • Query-dependent index statistics
      • numUses: Number of times the Index has been used for querying.
      • useTime: Total time during the use of the Index for querying.
      • usesInProgress: Current number of uses of the index in progress or current number of concurrent threads accessing the index for querying. Concurrent use of an index is allowed for different queries.
    • Query-independent index statistics
      • numKeys: Number of keys stored in the Index currently.
      • numValues: Number of values stored in the Index currently. One key can have multiple values.
      • numUpdates: Number of updates applied and completed on the Index while inserting, modifying, or deleting corresponding data in GemFire.
      • updateTime: Total time taken in applying and completing updates on the Index.
      • updatesInProgress: Current number of updates in progress on the Index. Concurrent updates on an index are allowed.

Gateway Concurrency and Ordering Updates

You can configure gateway hubs and gateway event listeners to use multiple concurrent event processing queues. This configuration enhancement improves gateway processing throughput and performance in multi-site configurations. You can make these configuration changes by using cache.xml or Java APIs.

This change adds the following gateway and gateway hub configuration APIs:

  • GatewayHub
    public Gateway addGateway(String id, int concurrencyLevel) throws GatewayException;
  • Gateway
    public int getConcurrencyLevel();
    public void setOrderPolicy(OrderPolicy orderPolicy);
    public OrderPolicy getOrderPolicy();
    

See also the Multi-site Configuration chapter under "Topologies and Communication" and the Events and Event Handling chapter under "Developing with vFabric GemFire".

Portable Data eXchange (PDX) Serialization Enhancements

vFabric GemFire 6.6.2 enhances the PDX Serialization feature as follows. (See also the Data Serialization chapter in the GemFire User's Guide under "Developing with vFabric GemFire").

Optimized serialization of enum, java.math.BigInteger, java.math.BigDecimal, java.util.UUID, and java.sql.Timestamp objects

vFabric GemFire now serializes enums, java.math.BigInteger, java.math.BigDecimal, java.util.UUID, and java.sql.Timestamp objects in an optimized manner. These optimizations do not change any public APIs, but they do change the serialized wire format of these objects. GemFire 6.6.2 optimizes serialization of these objects unless the system property gemfire.serializationVersion is set to an older version of GemFire. If you are performing a rolling upgrade to 6.6.2, modify this system property on all 6.6.2 members to maintain backwards compatibility until all members are upgraded to 6.6.2. (See Upgrading to GemFire 6.6.2 for other upgrade considerations.)

Notes on enum serialization:

  • If an enum is referenced (directly or indirectly) by an object that is PDX-serialized, the serialized form of the enum is optimized, unless the system property gemfire.serializationVersion is set to a version of GemFire older than 6.6.2.
  • Enum instances that are not referenced (directly or indirectly) in an object being pdx serialized are still optimized but to a lesser degree.
  • Enums are still encoded as an object field when they are referenced directly in a PDX serialized object. This characteristic makes them variable length.
  • The number of bytes required to serialize an enum the new optimized way is 3 to 7 bytes. The number of bytes required to serialize an enum using the older, unoptimized way is 121 bytes. This number is an estimation and depends on the class name, package name, enum constant name, field count, and field names.
  • The serialization of enums does not depend on the ordinal value of the enum. It is acceptable to serialize an enum constant with an ordinal value of 1 and deserialize it in a context in which its value is 2. All that matters is that the symbolic name of the enum does not change.
  • If more than 16,777,215 unique enum constants are serialized, an exception is thrown.

Optimized deserialization through caching of last class used

In previous releases of vFabric GemFire, any deserialization could do a Class.forName call to find the current version of the class available from the current loaders. This ability slowed deserialization unnecessarily. Now when deserializing an object, GemFire 6.6.2 retains (with a weak reference) the last class that was loaded for a particular class name. Subsequent deserializations use the cached class instead of attempting to load the class again. To use the older version of deserialization, set the system property gemfire.loadClassOnEveryDeserialization to true.

Autoserialization (ReflectionBasedAutoSerializer) performance enhancements

Autoserialization performs as fast or faster than PdxSerializable. A number of classes already have special standard serialization code. For example, they may implement Externalizable or the private writeObject method. In previous 6.6 releases, the autoserializer attempted to serialize these classes and ignored their special serialization code. In 6.6.2 the ReflectionBasedAutoSerializer detects these special methods and passes these instances to standard java serialization.

Ability to create PdxInstances using PdxInstanceFactory

You can use the PdxInstanceFactory to create a PdxInstance from raw data when the domain class is not available on the server. This ability is especially useful when you need an instance of a domain class for plug-in code such as a function or a loader.

PdxInstance support for enums

In previous releases when an enum was deserialized and read-serialized set to true, GemFire still required the enum class to be present, and the object was deserialized into an instance of the enum class. In 6.6.2, if read-serialized is set to true, then when an enum is deserialized, it is deserialized as a PdxInstance instead of a domain class instance.

PdxInstance has new methods to detect whether a PdxInstance is an enum (isEnum) and to obtain the name of the class that a PdxInstance represents (getClassName). In addition, the RegionService has a new API that allows you to create a PdxInstance that represents an enum:

 public PdxInstance createPdxEnum(String className, String enumName);

Note: Because the query engine implicitly sets read-serialized to true, the query engine no longer needs to load enum classes.

Additional validation capability for PdxWriter

If you use the PdxWriter instead of using autoserialization to serialize your objects, you may make some mistakes in your use of PdxWriter. To perform extra PdxWriter validation, set the system property gemfire.validatePdxWriters to true. Exceptions are thrown when PdxWriter mistakes are detected. You should only set this property to true when debugging new code because it slows performance.

Function APIs and PartitionResolver updated to support autoserialization

Prior to 6.6.2, some public vFabric GemFire APIs had methods that either returned or took parameters typed as Serializable. GemFire also had classes with generic parameters constrained to be an instance of Serializable. Pdx and DataSerializer allow classes that do not implement Serializable to be serialized by GemFire. However, those classes did not work with these APIs because the classes required the API input and output to be an instance of Serializable.

To resolve this limitation, the following APIs are revised to use Object instead of Serializable. If you have application code that calls the following APIs, you need to recompile your application code for 6.6.2. If you have application code that implements or extends these APIs, you need to first modify your code to use Object instead of Serializable and then recompile for 6.6.2.

  • PartitionResolver
    public Object getRoutingObject(EntryOperation<K,V> opDetails);
  • Execution
    public Execution withArgs(Object args);
    public Execution withCollector(ResultCollector<?, ?> rc);
    public ResultCollector<?, ?> execute(String functionId)
       throws FunctionException;
    public ResultCollector<?, ?> execute(Function function)
       throws FunctionException;
    public ResultCollector<?, ?> execute(String functionId, boolean hasResult)
       throws FunctionException;
    public ResultCollector<?, ?> execute(String functionId, boolean hasResult,
       boolean isHA) throws FunctionException;
    public ResultCollector<?, ?> execute(String functionId, boolean hasResult,
       boolean isHA, boolean optimizeForWrite) throws FunctionException;
  • FunctionContext
    public Object getArguments();
    public <T> ResultSender<T> getResultSender();
  • ResultCollector
    public interface ResultCollector<T,S> {...}
  • ResultSender
    public interface ResultSender<T> {...}
  • ExecuteFunctionOperationContext
    public Object getResult();
    public void setResult(Object oneResult);

Ability to check the portability of objects before serialization (PdxWriter and ReflectionBasedAutoSerializer)

When using the GemFire 6.6 Java APIs, it is possible to serialize data that cannot be deserialized in .NET. In 6.6.2 the PdxWriter (writeObject method) and the ReflectionBasedAutoSerializer can check whether the objects that they are serializing are portable to .NET. You can set the check-portability flag when calling the ReflectionBasedAutoSerializer constructor. If this flag is set to true, the ReflectionBasedAutoSerializer throws a NonPortableClassException when attempting to serialize a non-portable object.

New PDX serialization statistics

DistributionStats has these new pdx stats:

  • pdxSerializations: Total number of pdx serializations.
  • pdxSerializedBytes: Total number of bytes produced by pdx serialization.
  • pdxDeserializations: Total number of pdx deserializations.
  • pdxDeserializedBytes: Total number of bytes read by pdx deserialization.
  • pdxInstanceDeserializations: Total number of times getObject has been called on a PdxInstance.
  • pdxInstanceDeserializationTime: Total amount of time, in nanoseconds, spent deserializing PdxInstances by calling getObject.
  • pdxInstanceCreations: Total number of times a PdxInstance has been created by deserialization.

Algorithm Update for Network Partition Detection

Network partitioning detection now uses a quorum-based weighting system to detect network partitions in a distributed system. See Network Partitioning.

Support for Resuming and Suspending Transactions

You can suspend and resume transactions. This ability is useful when a transaction must perform multiple functions calls before it can commit.

See also the Transactions chapter in the GemFire User's Guide under Developing with vFabric GemFire.

64-bit support for GemFire GFMon and Data Browser Tools

GemFire GFMon and Data Browser can now run on 64-bit operating systems.

Documentation Changes

The vFabric GemFire User's Guide has been updated with the following changes for 6.6.2:

  • Rewrote and reorganized the Querying chapter. Updated contents for new APIs. See Querying.
  • Added Gateway concurrency and Order Policy documentation. See the Multi-site Configuration chapter under "Topologies and Communication" and the Events and Event Handling chapter under "Developing with vFabric GemFire."
  • Updated the Data Serialization chapter to reflect PDX Serialization updates. See Data Serialization.
  • Revised the Network Partitioning chapter to reflect the new algorithm. See Network Partitioning.
  • Updated Transactions chapter to add support for suspending and resuming transactions. See Transactions.
  • Fixed various documentation bugs and issues.

The GemFire Tools User Guide has been updated with the following change for 6.6.2:

  • Updated to reflect 64-bit support for GemFire GFMon and Data Browser Tools

New and Modified APIs

This section provides a summary of the new and modified Java APIs in GemFire 6.6.2 and provides links to the relevant API documentation pages.

Query and Index Creation

Gateway Concurrency and Ordering

PDX

Transactions

Upgrading to GemFire 6.6.2

To upgrade from an earlier version of GemFire to the current version, see the Upgrading vFabric GemFire topic in the vFabric GemFire User's Guide under "Getting Started with vFabric GemFire." Also review product changes documented in What's New in vFabric GemFire 6.6.2. If you are upgrading to GemFire 6.6.2 from a version of GemFire earlier than 6.6.0, see also the vFabric GemFire 6.6 Release Notes to determine what changes are required for you to migrate to vFabric GemFire 6.6.X.

Before you upgrade from any earlier release to vFabric GemFire 6.6.2, be aware of the following:

  • Check your application code to see if you are subclassing any modified APIs listed in Function APIs and PartitionResolver updated to support autoserialization. If you have application code that implements or extends these APIs, then you need to first modify your code to use Object instead of Serializable and then recompile for 6.6.2. If you are only calling these APIs, then you only need to recompile your applications for 6.6.2.

  • During a rolling upgrade, members that are running a vFabric GemFire version older than 6.6.2 and have network partitioning detection enabled (enable-network-partition-detection=true) may not receive proper notification if a network partition event is detected by the new 6.6.2 algorithm. It is recommended that you only perform a rolling upgrade during a window of low network activity, or that you avoid a rolling upgrade by upgrading the entire cluster at once while offline. If a network partition event does occur during a rolling upgrade, restart the cluster with GemFire 6.6.2 running on all nodes.

Before you upgrade from vFabric GemFire 6.6.0 or 6.6.1 to vFabric GemFire 6.6.2, also be aware of the following considerations in addition those listed above:

  • GemFire 6.6.2 introduces a new serialization format for enums, BigInteger, BigDecimal, UUID, and Timestamp object types. To ensure compatibility with older GemFire processes, you can use the system property gemfire.serializationVersion to specify the older format version. The value of this property is a GemFire product version. Valid values are strings that begin with "6.6.0" or "6.6.1"-- any other strings throw an exception. If you have clients or servers running both 6.6.0 and 6.6.1, set the value to the oldest version.

    For example, to be safe during a rolling upgrade from 6.6.1 to 6.6.2, set -Dgemfire.serializationVersion=6.6.1 on each 6.6.2 member. After all members (including clients) have been upgraded to 6.6.2, you can remove or unset this property. Removing this property allows you to take advantage of the optimized serialization performance.

  • The shut-down-all command is not supported on a system that contains a mix of members running both vFabric Gemfire 6.6.1 and 6.6.2. The shut-down-all command may hang if there are any persistent colocated regions stored on members running both vFabric GemFire 6.6.1 and 6.6.2.

Resolved Issues

For a list of bugs that are fixed in GemFire 6.6.2 and that are resolved in the VMware bug tracking system, see BugsFixedGemFire662.html. In addition, the following issue is resolved:

NullPointerException in GFMon

In GFMon, a NullPointerException was observed occasionally during rendering of charts. This issue is resolved.

Known Issues

For a list of issues that have been registered as bugs in the VMware bug tracking system, see BugNotesGemFire662.html.