Distributed Data Framework Architecture

Any ingest attempts that fail inside the Catalog Framework (whether the failure comes from the Catalog Framework itself, pre-ingest plugin failures, or issues with the Catalog Provider) will be logged to a separate log file for ease of error handling. The file is located at <DDF_HOME>/data/log/ingest_error.log and will log the Metacards that fail, their ID and Title name, and the stack trace associated with their failure. By default, successful ingest attempts are not logged. However, that functionality can be achieved by setting the log level of the ingestLogger to DEBUG (note that enabling DEBUG can cause a non-trivial performance hit).

The Query Service Endpoint, the Catalog Framework, and the CatalogProvider are key components for processing a query request as well. The Endpoint bundle contains a Web service that exposes the interface to query for Metacards. The Endpoint calls the CatalogFramework to execute the operations of its specification. The CatalogFramework relies on the CatalogProvider to execute the actual query. Optional PreQuery and PostQuery Catalog Plugins may be invoked by the CatalogFramework to modify the query request/response prior to the Catalog Provider processing the query request and providing the query response. If a CatalogProvider is not configured and no other remote Sources are configured, a fault will be returned. It is possible to have only remote Sources configured and no local CatalogProvider configured and be able to execute queries to specific remote Sources by specifying the site name(s) in the query request.

The Catalog Framework optionally provides caching of products, so future requests to retrieve the same product will be serviced much quicker. If caching is enabled, each time a retrieve product request is received, the Catalog Framework will look in its cache (default location <DDF_HOME>/data/product-cache) to see if the product has been cached locally. If it has, the product is retrieved from the local site and returned to the client, providing a much quicker turnaround because remote product retrieval and network traffic was avoided. If the requested product is not in the cache, the product is retrieved from the Source (local or remote) and cached locally while returning the product to the client. The caching to a local file of the product and the streaming of the product to the client are done simultaneously so that the client does not have to wait for the caching to complete before receiving the product. If errors are detected during the caching, caching of the product will be abandoned, and the product will be returned to the client. 

The Catalog Framework attempts to detect any network problems during the product retrieval, e.g., long pauses where no bytes are read implying a network connection was dropped. (The amount of time defined as a "long pause" is configurable, with the default value being five seconds.) The Catalog Framework will attempt to retrieve the product up to a configurable number of times (default = three), waiting for a configurable amount of time (default = 10 seconds) between each attempt, trying to successfully retrieve the product. If the Catalog Framework is unable to retrieve the product, an error message is returned to the client.

If the admin has enabled the Always Cache When Canceled option, caching of the product will occur even if the client cancels the product retrieval so that future requests will be serviced quickly. Otherwise, caching is canceled if the user cancels the product download.

As part of the caching of products, the Catalog Framework also posts events to the OSGi notification framework. Information includes when the product download started, whether the download is retrying or failed (after the number of retrieval attempts configured for product caching has been exhausted), and when the download completes. These events are retrieved by the Search UI and presented to the user who initiated the download.

The Catalog API is an OSGi bundle (catalog-core-api) that contains the Java interfaces for the Catalog components and implementation classes for the Catalog Framework, Operations, and Data components.

The Standard Catalog Framework provides the reference implementation of a Catalog Framework that implements all requirements of the Catalog API.  CatalogFrameworkImpl is the implementation of the DDF Standard Catalog Framework.

The Standard Catalog Framework is the core class of DDF. It provides the methods for create, update, delete, and resource retrieval (CRUD) operations on the Sources. By contrast, the Fanout Catalog Framework only allows for query and resource retrieval operations, no catalog modifications, and all queries are enterprise-wide.

Use this framework if:

It is possible to have only remote Sources configured with no local CatalogProvider configured and be able to execute queries to specific remote sources by specifying the site name(s) in the query request.

The Standard Catalog Framework also maintains a list of ResourceReaders for resource retrieval operations. A resource reader is matched to the scheme (i.e., protocol, such as file://) in the URI of the resource specified in the request to be retrieved.

Site information about the catalog provider and/or any federated source(s) can be retrieved using the Standard Catalog Framework. Site information includes the source’s name, version, availability, and the list of unique content types currently stored in the source (e.g., NITF). If no local catalog provider is configured, the site information returned includes site info for the catalog framework with no content types included.

The Catalog Framework Camel Component supports creating, updating, and deleting metacards using the Catalog Framework from a Camel route.

The Atom Query Response Transformer transforms a query response into an Atom 1.0 feed. The Atom transformer maps a QueryResponse object as described in the Query Result Mapping.

The CSW Query Response Transformer transforms a query response into a CSW-formatted document.

The GeoJSON input transformer is responsible for translating GeoJSON into a Catalog metacard.

GeoJSON Metacard Transformer translates a metacard into GeoJSON.

The GeoJSON Query Response Transformer translates a query response into a GeoJSON-formatted document.

The KML Metacard Transformer is responsible for translating a metacard into a KML-formatted document. The KML will contain an HTML description that will display in the pop-up bubble in Google Earth. The HTML contains links to the full metadata view as well as the resource.

The KML Query Response Transformer translates a query response into a KML-formatted document. The KML will contain an HTML description for each metacard that will display in the pop-up bubble in Google Earth. The HTML contains links to the full metadata view as well as the resource.

The KML Style Mapper provides the ability for the KMLTransformer to map a KML Style URL to a metacard based on that metacard’s attributes. For example, if a user wanted all JPEGs to be blue, the KML Style Mapper provides the ability to do so. This would also allow an administrator to configure metacards from each source to be different colors.

The configured style URLs are expected to be HTTP URLs. For more information on style URL’s, refer to the KML Reference .

The KML Style Mapper supports all basic and extended metacard attributes. When a style mapping is configured, the resulting transformed KML contain a <styleUrl> tag pointing to that style, rather than the default KML style supplied by the KMLTransformer.

The Metadata Metacard Transformer returns the Metacard.METADATA attribute when given a metacard. The MIME Type returned is text/xml.

The PDF Input Transformer is responsible for translating a PDF document into a Catalog Metacard.

The PPTX Input Transformer translates Microsoft PowerPoint (OOXML only) documents into Catalog Metacards, using Apache Tika for basic metadata and Apache POI for thumbnail creation. The PPTX Input Transformer ingests PPTX documents into the DDF Content Repository and the Metadata Catalog, and adds a thumbnail of the first page in the PPTX document.

The PPTX Input Transformer will take precedence over the Tika Input Transformer for PPTX documents.

The Query Response Transformer Consumer is responsible for translating a query response into a Catalog Metacard.

The Resource Metacard Transformer retrieves a resource associated with a metacard.

The Thumbnail Metacard Transformer retrieves the thumbnail bytes of a Metacard by returning the Metacard.THUMBNAIL attribute value.

The Tika Input Transformer is the default input transformer responsible for translating Microsoft Word, Microsoft Excel, Microsoft PowerPoint, OpenOffice Writer, and PDF documents into Catalog records. This input transformer utilizes Apache Tika to provide basic support for these mime types. The metadata common to all these document types, e.g., creation date, author, last modified date, etc., is extracted and used to create the catalog record. The Tika Input Transformer’s main purpose is to ingest these types of content into the Metadata Catalog.

The Tika input transformer is most basic input transformer and the last to be invoked. This allows any registered input transformers that are more specific to a document type to be invoked instead of this rudimentary input transformer.

The video input transformer Creates Catalog metacards from certain video file types. Currently, it is handles MPEG-2 transport streams as well as MPEG-4, AVI, MOV, and WMV videos. This input transformer uses Apache Tika to extract basic metadata from the video files and applies more sophisticated methods to extract more meaningful metadata from these types of video.

The XML Input Transformer is responsible for translating an XML document into a Catalog Metacard.

The XML metacard transformer is responsible for translating a metacard into an XML-formatted document. The metacard element that is generated is an extension of gml:AbstractFeatureType, which makes the output of this transformer GML 3.1.1 compatible.

The XML Query Response Transformer is responsible for translating a query response into an XML-formatted document. The metacard element generated is an extension of gml:AbstractFeatureCollectionType, which makes the output of this transformer GML 3.1.1 compatible.

The DDF Mime Type Mapper is the core implementation of the DDF Mime API. It provides access to all MimeTypeResolvers within DDF, which provide mapping of mime types to file extensions and file extensions to mime types.

These are mime types not supported by the default TikaMimeTypeResolver.

As a MimeTypeResolver, the Custom Mime Type Resolver will provide methods to map the file extension to the corresponding mime type, and vice versa.

The TikaMimeTypeResolver is a MimeTypeResolver that is implemented using the Apache Tika open source product.

Using the Apache Tika content analysis toolkit, the TikaMimeTypeResolver provides support for resolving over 1300 mime types, but not all mime types yield the same quality metadata.

The TikaMimeTypeResolver is assigned a default priority of -1 to insure that it is always invoked last by the MimeTypeMapper. This insures that any custom MimeTypeResolvers that may be installed will be invoked before the TikaMimeTypeResolver.

The TikaMimeTypeResolver provides the bulk of the default mime type support for DDF.

Pre-delivery plugins are invoked before any security rules are applied.  This is an opportunity to take any action before authorization, including but not limited to:

Policy plugins are invoked to set up the policy for a request/response.  This provides an opportunity to attach custom requirements on operations or individual metacards. All the 'requirements' from each Policy plugin will be combined into a single policy that will be included in the request/response. Access plugins will be used to act on this combined policy.

Access plugins are invoked directly after the Policy plugins have been successfully executed.  This is an opportunity to either stop processing or modify the request/response based on policy information.

Pre-ingest plugins are invoked before an ingest operation is sent to the catalog. They are not run on a query. This is an opportunity to take any action on the ingest request, including but not limited to:

Post-ingest plugins are invoked after data has been created, updated, or deleted in a Catalog Provider.

Post-Process Plugins are invoked after a metacard has been created, updated, or deleted and committed to the Catalog. They are the last plugins to run and are triggered by a Post-Ingest Plugin. Post-Process plugins are well-suited for asynchronous tasks. See the Asynchronous Processing Framework for more information about how Post-Process Plugins are used.

Pre-query plugins are invoked before a query operation is sent to any of the Sources.  This is an opportunity to take any action on the query, including but not limited to:

Pre-federated-query plugins are invoked before a federated query operation is sent to any of the Sources.  This is an opportunity to take any action on the query, including but not limited to:

Post-query plugins are invoked after a query has been executed successfully, but before the response is returned to the endpoint.  This is an opportunity to take any action on the query response, including but not limited to:

Post-federated-query plugins are invoked after a federated query has been executed successfully, but before the response is returned to the endpoint.  This is an opportunity to take any action on the query response, including but not limited to:

Pre-Resource plugins are invoked before a request to retrieve a resource is sent to a Source.  This is an opportunity to take any action on the request, including but not limited to:

Post-resource plugins are invoked after a resource has been retrieved, but before it is returned to the endpoint.  This is an opportunity to take any action on the response, including but not limited to:

Pre-Create storage plugins are invoked immediately before an item is created in the content repository.

Post-Create storage plugins are invoked immediately after an item is created in the content repository.

Pre-Update storage plugins are invoked immediately before an item is updated in the content repository.

Post-Update storage plugins are invoked immediately after an item is updated in the content repository.

Pre-subscription plugins are invoked before a Subscription is activated by an Event Processor.  This is an opportunity to take any action on the Subscription, including but not limited to:

Pre-delivery plugins are invoked before a Delivery Method is invoked on a Subscription.  This is an opportunity to take any action before event delivery, including but not limited to:

The Catalog Backup Plugin is used to enable data backup of the catalog and the metacards it contains.

The Catalog Policy Plugin configures the attributes required for users to perform Create, Read, Update, and Delete operations on the catalog.

The Checksum plugin creates a unique checksum for resources input into the system to identify updated content.

The client info plugin injects request-specific network information into request properties, such as Remote IP Address, Remote Host Name, Servlet Scheme, and Servlet Context.

The Content URI Access Plugin prevents a Metacard’s resource URI from being overridden by an incoming UpdateRequest.

The Event Processor creates, updates, and deletes subscriptions for event notification. These subscriptions optionally specify a filter criteria so that only events of interest to the subscriber are posted for notification.

As metacards are created, updated, and deleted, the Catalog’s Event Processor is invoked (as a post-ingest plugin) for each of these events. The Event Processor applies the filter criteria for each registered subscription to each of these ingest events to determine if they match the criteria.

For more information on creating subscriptions, see Creating a Subscription.

The Expiration Date plugin adds or updates expiration dates which can be used later for archiving old data.

The Filter Plugin performs filtering on query responses as they pass through the framework.

Each metacard result can contain security attributes that are pulled from the metadata record after being processed by a PolicyPlugin that populates this attribute. The security attribute is a Map containing a set of keys that map to lists of values. The metacard is then processed by a filter plugin that creates a KeyValueCollectionPermission from the metacard’s security attribute. This permission is then checked against the user subject to determine if the subject has the correct claims to view that metacard. The decision to filter the metacard eventually relies on the installed Policy Decision Point (PDP). The PDP that is being used returns a decision, and the metacard will either be filtered or allowed to pass through.

How a metacard gets filtered is left up to any number of FilterStrategy implementations that might be installed. Each FilterStrategy will return a result to the filter plugin that says whether or not it was able to process the metacard, along with the metacard or response itself. This allows a metacard or entire response to be partially filtered to allow some data to pass back to the requester. This could also include filtering any resources sent back to a requester.

The security attributes populated on the metacard are completely dependent on the type of the metacard. Each type of metacard must have its own PolicyPlugin that reads the metadata being returned and then returns the appropriate attributes.

In the above example, the user’s claims are represented very simply and are similar to how they would actually appear in a SAML 2 assertion. Each of these user (or subject) claims will be converted to a KeyValuePermission object. These permission objects will be implied against the permission object generated from the metacard record. In this particular case, the metacard might be allowed if the policy is configured appropriately because all of the permissions line up correctly.

The GeoCoder Plugin is a pre-ingest plugin that is responsible for populating the Metacard’s Location.COUNTRY_CODE attribute if the Metacard has an associated location. If there is a valid country code for the Metacard, it will be in ISO 3166-1 alpha-3 format. If the metacard’s country code is already populated, the plugin will not override it. The GeoCoder relies on either the WebService or Offline Gazetteer to retrieve country code information.

The Historian Policy Plugin protects metacard history from being edited or deleted by users without the history role (a http://schemas.xmlsoap.org/ws/2005/05/identity/claims/role of system-history).

The JPEG2000 Thumbnail converter creates thumbnails from images ingested in jpeg2000 format.

The Metacard Attribute Security Policy Plugin combines existing metacard attributes to make new attributes and adds them to the metacard. For example, if a metacard has two attributes, sourceattribute1 and sourceattribute2, the values of the two attributes could be combined into a new attribute, destinationattribute1. The sourceattribute1 and sourceattribute2 are the source attributes and destinationattribute1 is the destination attribute.

There are two way to combine the values of source attributes. The first, and most common, is to take all of the attribute values and put them together. This is called the union. For example, if the source attributes sourceattribute1 and sourceattribute2 had the values:

sourceattribute1 = MASK, VESSEL

sourceattribute2 = WIRE, SACK, MASK

…​the union would result in the new attribute destinationattribute1:

destinationattribute1 = MASK, VESSEL, WIRE, SACK

The other way to combine attributes is use the values common to all of the attributes. This is called the intersection. Using our previous example, the intersection of sourceattribute1 and sourceattribute2 would create the new attribute destinationattribute1

destinationattribute1 = MASK

because only MASK is common to all of the source attributes.

The policy plugin could also be used to rename attributes. If there is only one source attribute, and the combination policy is union, then the attribute’s values are effectively renamed to the destination attribute.

The Metacard Backup File Storage Provider is a storage provider that will store backed-up metacards in a specified file system location.

The Metacard Backup S3 Storage Provider is a storage provider that will store backed up metacards in the specified S3 bucket and key.

The Metacard Groomer Pre-Ingest plugin makes modifications to CreateRequest and UpdateRequest metacards.

Use this pre-ingest plugin as a convenience to apply basic rules for your metacards. 

This plugin makes the following modifications when metacards are in a CreateRequest:

In an UpdateRequest, the same operations are performed as a CreateRequest, except:

The Metacard Ingest Network Plugin allows the conditional insertion of new attributes on metacards during ingest based on network information from the ingest request; including IP address and hostname.

For the extent of this section, a 'rule' will refer to a configured, single instance of this plugin.

This post-query plugin updates the resource size attribute of each metacard in the query results if there is a cached file for the resource and it has a size greater than zero; otherwise, the resource size is unmodified and the original result is returned.

Use this post-query plugin as a convenience to return query results with accurate resource sizes for cached products. 

The Metacard Validity Filter Plugin determines whether metacards with validation errors or warnings are filtered from query results.

The Metacard Validity Marker Pre-Ingest plugin modifies the metacards contained in create and update requests.

The plugin runs each metacard in the CreateRequest and UpdateRequest against each registered MetacardValidator service.

The operation plugin validates the subject’s security attributes to ensure they are adequate to perform the operation.

The Point of Contact Policy Plugin is a PreUpdate plugin that will check if the point-of-contact attribute has changed. If it does, then it adds a policy to that metacard’s policy map that cannot be implied. This will deny such an update request, which essentially makes the point-of-contact attribute read-only.

The Processing Post Ingest Plugin is responsible for submitting catalog Create, Update, and Delete (CUD) requests to the Processing Framework.

The Resource URI Policy Plugin configures the attributes required for users to set the resource URI when creating a metacard or alter the resource URI when updating an existing metacard in the catalog.

The Security Audit Plugin is used to allow the auditing of specific metacard attributes. Any time a metacard attribute listed in the configuration is updated, a log will be generated in the security log.

The Security Logging Plugin logs operations to the security log.

The Security Plugin identifies the subject for an operation.

The Tags Filter Plugin updates queries without filters for tags, and adds a default tag of resource. For backwards compatibility, a filter will also be added to include metacards without any tags attribute.

The Video Thumbnail Plugin provides the ability to generate thumbnails for video files stored in the Content Repository.

It is an implementation of both the PostCreateStoragePlugin and PostUpdateStoragePlugin interfaces. When installed, it is invoked by the Catalog Framework immediately after a content item has been created or updated by the Storage Provider.

This plugin uses a custom 32-bit LGPL build of FFmpeg (a video processing program) to generate thumbnails. When this plugin is installed, it places the FFmpeg executable appropriate for the current operating system in <DDF_HOME>/bin_third_party/ffmpeg. When invoked, this plugin runs the FFmpeg binary in a separate process to generate the thumbnail. The <DDF_HOME>/bin_third_party/ffmpeg directory is deleted when the plugin is uninstalled.

The Workspace Access Plugin prevents non-owner users from changing workspace permissions.

The Workspace Pre-Ingest Plugin verifies that a workspace has an associated email to enable sharing and assigns that email as "owner".

The Workspace Sharing Policy Plugin collects attributes for a workspace to identify the appropriate policy to apply to allow sharing.

The XML Attribute Security Policy Plugin parses XML metadata contained within a metacard for security attributes on any number of XML elements in the metadata. The configuration for the plugin contains one field for setting the XML elements that will be parsed for security attributes and the other two configurations contain the XML attributes that will be pulled off of those elements. The Security Attributes (union) field will compute the union of values for each attribute defined and the Security Attributes (intersection) field will compute the intersection of values for each attribute defined.

A metacard type indicates the attributes available for a particular metacard. It is a model used to define the attributes of a metacard, much like a schema.

A metacard type indicates the attributes available for a particular type of data. For example, an image may have different attributes than a PDF document, so each could be defined to have their own metacard type.

The MetacardTypeRegistry allows DDF components, primarily catalog providers and sources, to make available the MetacardTypes that they support.  It maintains a list of all supported MetacardTypes in the CatalogFramework, so that other components such as Endpoints, Plugins, and Transformers can make use of those MetacardTypes.  The MetacardType is essential for a component in the CatalogFramework to understand how it should interpret a metacard by knowing what attributes are available in that metacard. 

For example, an endpoint receiving incoming metadata can perform a lookup in the MetacardTypeRegistry to find a corresponding MetacardType.  The discovered MetacardType will then be used to help the endpoint populate a metacard based on the specified attributes in the MetacardType.  By doing this, all the incoming metadata elements can then be available for processing, cataloging, and searching by the rest of the CatalogFramework.

MetacardTypes should be registered with the MetacardTypeRegistry.  The MetacardTypeRegistry makes those MetacardTypes available to other DDF CatalogFramework components.  Other components that need to know how to interpret metadata or metacards should look up the appropriate MetacardType from the registry.  By having these MetacardTypes available to the CatalogFramework, these components can be aware of the custom attributes. 

The MetacardTypeRegistry is accessible as an OSGi service.  The following blueprint snippet shows how to inject that service into another component:

The reference to this service can then be used to register new MetacardTypes or to lookup existing ones. 

Typically, new MetacardTypes will be registered by CatalogProviders or sources indicating they know how to persist, index, and query attributes from that type.  Typically, Endpoints or InputTransformers will use the lookup functionality to access a MetacardType based on a parameter in the incoming metadata.  Once the appropriate MetacardType is discovered and obtained from the registry, the component will know how to translate incoming raw metadata into a DDF Metacard.

An attribute is a single field of a metacard, an instance of an attribute type. Attributes are typically indexed for searching by a source or catalog provider.

The quickest way to create a Metacard is to extend or construct the MetacardImpl object.  MetacardImpl is the most commonly used and extended Metacard implementation in the system because it provides a convenient way for developers to retrieve and set Attributes without having to create a new MetacardType (see below). MetacardImpl uses BASIC_METACARD as its MetacardType.