Skip to end of metadata
Go to start of metadata

Goals

The goals for this work are fairly well-stated in KULRICE-4816 but I will include them here as well:

  1. SOAP-based interface to the service registry
  2. Replace serialized form of service definition with XML/JAXB-based form
  3. Remove "dead" indicator from the registry, move the "dead service" state to the client app so it's not in the registry.
  4. Add "version" information into the registry.
  5. API to get version for a service.
  6. Implement a model by which applications automatically have a version and their services are published under that version.
  7. Client application configures a registry "url" in order to configure it. Build in backend support for "loading" the service that points to the registry.

Issues with Rice 1.0.x KSB

In order to implement the enhancements required for this work item, some refactoring had to be done on the Kuali Service Bus first. In Rice 1.0.x, the service bus contains three main components:

  • RemoteResourceServiceLocator - handles proxies to all "remote" services that it has read from the service registry
  • RemotedServiceRegistry - handles all services that a client application has published to the registry
  • ServiceRegistry - reads and writes directly from the service registry tables in the database

On the surface this breakdown makes sense, however there are some issues with it:

  • The functionality of RemoteResourceServiceLocator and RemotedServiceRegistry is somewhat tangled.
  • The RemoteResourceServiceLocator actually calls into the RemotedServiceRegistry for various operations, but in certain situations it's required to go explicitly to the RemotedServiceRegistry.
  • The ServiceRegistry is accessible only via direct database connections.
  • These classes share a common object called ServiceDefinition which is used both during service publishing and when reading services from the registry. However, these are two slightly different situations that need to be considered.

In order to get around some of these issues, for 2.0 we chose to do a re-implementation of the KSB apis in order to address some of these issues.

KSB Architecture Redesign and Reimplementation

Overview

The KSB refactoring for version compatibility focuses on the design and implementation of the following components:

  1. Service Registry
  2. Service Bus
  3. Service Export Manager

Service Registry

The Service Registry contains information about all services which have registered with the service bus from all instances and applications that are "connected" to the bus. The ServiceRegistry is a remotable implementation of the registry (available as a SOAP endpoint) which allows for registering, removing, and querying on information about these service endpoints. It is important in Rice 2.0 that this service is accessible remotely because this allows client applications to connect to it through a remote interface instead of through direct database connections as was done in Rice 1.0.x. The registry is intended to be deployed and available as a separate service which all clients who are joined up to the "bus" can connect into.

In general, Java client applications will not interact directly with the ServiceRegistry, instead they will leverage the ServiceBus api (see below) which manages a client's state in respect to the central registry. Non-java clients could take advantage of the SOAP apis provided for the service registry if they would like to publish their own services or query registry state in order to locate services.

Because the registry is now accessed remotely, a discovery process has to be put into place that allows for client applications to "connect" to it (see Service Registry Discovery below). This allows for the registry to be "bootstrapped" into existence from the perspective of the client. The registry itself is a service, but obviously it is not possible to use the registry in order to discover the available endpoints for the registry.

Service Registry Data Model

The data model for the Service Registry consists of three data elements:

  1. ServiceInfo - includes standard configuration information about a service that has been published in the registry
  2. ServiceDescriptor - includes an XML form of the ServiceConfiguration which consists of detailed service configuration information
  3. ServiceEndpoint - includes both the ServiceInfo and the ServiceDescriptor for a service

Each of these is a concrete immutable class which implements a contract interface that defines the various pieces of data on each and what the invariants are. Contract interfaces for each can be reviewed below:

As can been seen from this data model, the service endpoint is separated into two parts, the service information and the service descriptor. The ServiceInfo contains the service descriptor id that can be used to load the descriptor when it is needed. This is important as it allows the client-side service bus functionality to load all the online ServiceInfo from the registry and then only load the ServiceDescriptor for services it actually needs to use. This was implemented similarly in previous versions of Rice but took advantage of lazy loading of proxy references in the OJB layer. Since this functionality has moved from the ORM layer up to the service layer (in order to facilitate version compatibility) we are implementing a "manual" lazy loading of this information through the service layer.

The Service Registry "Service"

The ServiceRegistry interface defines the contract for the registry and specifies the various operations that are supported. A summary of supported operations include:

  • Get ServiceInfo for all of the services that are "online"
  • Get ServiceInfo for all online services with a particular name
  • Get all ServiceInfo in the registry, regardless of status
  • Load the ServiceDescriptor for a given descriptor id
  • Publish a ServiceEndpoint
  • Remove a ServiceEndpoint
  • Remove and publish as a single operation - used by the client-side registry "diff" process
  • Update service status to the given status
  • Take an instance offline based on the instance id - sets all statuses for services with that instance id to OFFLINE

The full specification for the ServiceRegistry interface can be found here: http://fisheye.kuali.org/browse/~raw,r=20802/rice/trunk/ksb/api/src/main/java/org/kuali/rice/ksb/api/registry/ServiceRegistry.java

The concept of a service being "online" has to do with the status code on the service's ServiceInfo. These are defined by the ServiceEndpointStatus enumeration which defines three valid values:

  • ONLINE - Indicates the service is online and available to receieve requests.
  • OFFLINE - Indicates the service has been taken offline, most likely as the result of the instance hosting the service being taken offline (i.e. for maintenance or otherwise)
  • DISABLED - Indicates the service has been disabled because the registry has detected that it, or it's host instance is defective or not processing requests properly.

Note that the DISABLED status is not currently used by the KSB but is meant to be leveraged by future functionality which will help with keeping entries in the service registry as viable as possible.

The registry is accessible through a SOAP-based interface which allows for both java and non-java clients to both query for information about services that have been published, as well as publishing their own services to the registry.

Service Registry Discovery

Because the ServiceRegistry is itself now a service, a process must be put in place to "bootstrap" a connection to the registry. For other services, the process to acquire information is to obtain it from the registry, however, for the registry itself we need to use another mechanism in order to bring it online. To accomplish this, the client application simply has to configure a URL in a similar fashion to the following:

Currently, this connector is only configured to understand a SOAP interface to the service registry which is secured by digital signatures. This is the only type of interface to the registry that the standalone server currently publishes.

Additionally, only a single URL to the registry can be configured at the current time. If someone wants to do load balancing amongst potential registry endpoints, then a hardware or software load balancer could be configured to do this.

Service Bus

The ServiceBus handles maintaining a client application's "view" of the registry. It periodically synchronizes with the registry to find newly available services or ones that were removed. Additionally, a client application can "publish" it's own services here and this piece will handle installing those services into the ServiceRegistry. Note that the ServiceBus implementations runs embedded within the client application and is therefore only relevant for Java applications. Non-java applications can still interact with the remote ServiceRegistry as discussed previously, but would need to code any behavior similar to the ServiceBus by hand if they wanted to replicate it.

ServiceBus Collaborators

The ServiceBus works with three main objects:

  1. ServiceDefinition
  2. ServiceConfiguration
  3. Endpoint

ServiceDefinnition - The ServiceDefinition contains information about a service to be published, as well a reference to the service endpoint instance to publish. This definition is typically used in conjuction with the ServiceBusExporter and is sent to the ServiceExportManager to handle making an endpoint available for the service (see below for more information about both of these). There are generally different implementations of the service definition depending on the type of service being published (i.e. SoapServiceDefinition, RestServiceDefinition, etc.)

ServiceConfiguration - The ServiceConfiguration contains all information about the configuration of the service including service name, endpoint url, version, type, messaging configuration, security policies, as well as protocol specific configuration. There are generally different implementations of the service configuration, such as SoapServiceConfiguration, RestServiceConfiguration, etc. These are serialized to an XML form and stored on the ServiceDescriptor in the registry as was discussed previously.

Endpoint - The Endpoint contains a reference to the ServiceConfiguration for a service as well as a proxy to the service endpoint that can be invoked. This service can be cast to the appropriate service interface in order to invoke the desired operations.

Service Acquisition

The ServiceBus is primarily interact with through a method called "getService" that takes a service name and returns a reference to a service with that name that can be invoked. The reference returned from this method is actually an Endpoint but the service proxy can be obtained by invoking getService() on the endpoint.

The bus will give priority to locally available services with the given name over remote services with the same name. If necessary (in the case that the service is remote), the service reference it returns will be wrapped in a proxy that handles failover which is triggered during invocation failure scenarios. When a failover event is encountered, the proxy will go back to the ServiceBus to attempt to locate another endpoint with the same service name and attempt to invoke that instead. It will continue this process until it finds a valid service to invoke, if none of the services work, then the service invocation will fail.

Publishing Services

Services are generally published by creating a ServiceDefinition and "exporting" it using a ServiceBusExporter. This is often done via spring configuration as in the following example showing the publication of the ServiceRegistry service:

Ultimately, the ServiceBusExporter simply delegates to the "publishService" method on ServiceBus which handles publishing and exporting the service. So using this operation on the bus is another way to publish a service directly without using a ServiceBusExporter.

ServiceBus Synchronization with the Registry

This synchronization process is handled by a component called the ServiceRegistryDiffCalculator. This component contains the logic related to looking at a client application's local view of the registry and services it has published and making sure that the client's view is synchronized with the registry. It also ensures that the registry's understanding of services available at the client is consistent. It does this by interacting with the ServiceRegistry service.

The client-side bus will be synchronized with the remote ServiceRegistry once upon startup (startup will fail if the synchronization fails or the remote registry is unavailable) and then subsequent synchronization is handled by a background thread which executes periodically. This refresh period defaults to 30 seconds but is configurable by setting a parameter as follows:

Integration with the ResourceLoader Framework

The ResourceLoader framework integrates with the ServiceBus by providing a special ResourceLoader implementation which delegates it's "getService" method to the corresponding "getService" method on the ServiceBus. This resource loader is placed on the bottom of the resource loader stack to ensure that potentially remoteable services are always accessed last before checking for locally available implementations of the service being looked up.

Additional ServiceBus Operations

Besides the operations already mentioned, there are many other operations on the ServiceBus interface. The ServiceBus specification includes operations like the following:

  • Get the instance id for this service bus client (see terminology section below for more information on the instance id)
  • Get all endpoints for a given service name
  • Get the remote endpoints for a given service name
  • Get the local endpoint for a given service name
  • Get all local endpoints
  • Get an available endpoint for a given service name
  • Get a service proxy for the given service name
  • Publish a service
  • Remove a service
  • Manually invoke synchronization with the registry

The full specification of the ServiceBus interface can be found here: http://fisheye.kuali.org/browse/~raw,r=20493/rice/trunk/ksb/api/src/main/java/org/kuali/rice/ksb/api/bus/ServiceBus.java

Service Export Manager

The ServiceExportManager is a component that is communicated with during service publishing through the ServiceBus. It interfaces with a set of ServiceExporter implementations to create an endpoint proxy that can receive and process incoming requests to invoke that service endpoint. This proxy handles things like security and ultimately delegates the call down to the underlying service implementation that was published.

The ServiceExportManager also knows how to translate an incoming URL from an HttpServletRequest into a service endpoint to invoke. The KSBDispatcherServlet then delegates incoming requests down to the ServiceExportManager in order to invoke the appropriate endpoint during service invocation.

Other Important Terminology

To wrap up a few other pieces, there is some other important terminology. All applications have an application id (this was previously called service namespace). An application id is an identifier for an application, though it's intended that more than one instance of an application could exist (i.e. In the typical clustered use case). There is additionally the concept of an "instance", every instance has an instanceId which uniquely identifies it. By default (if not manually set) this value is generated as a combination of the server's ip address and the application id (effectively meaning that you can't deploy the same application more than once on the same ip unless you customize how the instance id is generated).

KSB Configuration

The KSB is configured using the KSBConfigurer class. It supports two run modes:

  1. LOCAL - local run mode should only be used by the KSB registry server and/or the Rice standalone server, in this mode the application connects via direct database connections to the registry
  2. REMOTE - the run mode that all client applications should use, this connects to the service registry through a remote SOAP interface

There are additionally some other configuration parameters that can be configured in different ways to control how the KSB operates:

message.persistence

  • indicates whether or not the client application will persist outgoing messages for reliable messaging
  • if "true", messageDataSource and nonTransactionalDatasource are required on the KSBConfigurer
  • if "false", no messageDataSource on nonTransactionalDatasource is required

dev.mode

  • if "true", don't publish or read from remote registry and no "connection" to remote registry attempted
    • it's not required to set the registry url in this case and the remote registry won't be accessed
  • if "false" then the KSB will integrate with the remote registry

rice.ksb.batch.mode

  • if "true", still need to make connection to registry, but don't "publish" services to it

Architecture Diagram

Gliffy Zoom Zoom ksb-2.0-architecture

KSB Impact when moving from 1.0.x to 2.0

Many of the changes made to the KSB for this refactoring may be impacting to existing code. A summary is below:

File

Details

org.kuali.rice.ksb.messaging.KSBExporter

Moved to org.kuali.rice.ksb.api.bus.support.ServiceBusExporter

org.kuali.rice.ksb.messaging.PropertyConditionalKSBExporter

Moved to org.kuali.rice.ksb.api.bus.support.PropertyConditionalServiceBusExporter

org.kuali.rice.ksb.messaging.JavaServiceDefinition

Moved to org.kuali.rice.ksb.api.bus.support.JavaServiceDefinition

org.kuali.rice.ksb.messaging.RESTServiceDefintion

Moved to org.kuali.rice.ksb.api.bus.support.RestServiceDefinition

org.kuali.rice.ksb.messaging.SOAPServiceDefintion

Moved to org.kuali.rice.ksb.api.bus.support.SoapServiceDefinition

RemoteResourceServiceLocatorImpl, RemotedServiceRegistry

Use org.kuali.rice.ksb.api.bus.ServiceBus instead

org.kuali.rice.ksb.messaging.PropertyConditionalKSBExporter

Moved to org.kuali.rice.ksb.api.bus.support.PropertyConditionalServiceBusExporter

  • No labels