Developers will build or extend the functionality of the applications. 

DDF includes several extension points where external developers can add functionality to support individual use cases.

DDF is written in Java and uses many open source libraries. DDF uses OSGi to provide modularity, lifecycle management, and dynamic services. OSGi services can be installed and uninstalled while DDF is running. DDF development typically means developing new OSGi bundles and deploying them to the running DDF. A complete description of OSGi is outside the scope of this documentation. For more information about OSGi, see the OSGi Alliance website This link is outside the DDF documentation.

Architecture Diagram
Architecture Diagram
Important

If developing for a Highly Available Cluster of DDF, see High Availability Guidance.

1. Catalog Framework API

Catalog Architecture
Catalog Architecture
Catalog Framework Architecture
Catalog Framework Architecture

The CatalogFramework is the routing mechanism between catalog components that provides integration points for the Catalog Plugins. An endpoint invokes the active Catalog Framework, which calls any configured Pre-query or Pre-ingest plug-ins. The selected federation strategy calls the active Catalog Provider and any connected or federated sources. Then, any Post-query or Post-ingest plug-ins are invoked. Finally, the appropriate response is returned to the calling endpoint.

The Catalog Framework wires all Catalog components together.

It is responsible for routing Catalog requests and responses to the appropriate target. 

Endpoints send Catalog requests to the Catalog Framework. The Catalog Framework then invokes Catalog PluginsTransformers, and Resource Components as needed before sending requests to the intended destination, such as one or more Sources

The Catalog Framework decouples clients from service implementations and provides integration points for Catalog Plugins and convenience methods for Endpoint developers.

1.1. Catalog API Design

The Catalog is composed of several components and an API that connects them together. The Catalog API is central to DDF’s architectural qualities of extensibility and flexibility.  The Catalog API consists of Java interfaces that define Catalog functionality and specify interactions between components.  These interfaces provide the ability for components to interact without a dependency on a particular underlying implementation, thus allowing the possibility of alternate implementations that can maintain interoperability and share developed components. As such, new capabilities can be developed independently, in a modular fashion, using the Catalog API interfaces and reused by other DDF installations.

1.1.1. Ensuring Compatibility

The Catalog API will evolve, but great care is taken to retain backwards compatibility with developed components. Compatibility is reflected in version numbers.

1.1.2. Catalog Framework Sequence Diagrams

Because the Catalog Framework plays a central role to Catalog functionality, it interacts with many different Catalog components. To illustrate these relationships, high-level sequence diagrams with notional class names are provided below. These examples are for illustrative purposes only and do not necessarily represent every step in each procedure.

Ingest Request Data Flow
Ingest Request Data Flow

The Ingest Service Endpoint, the Catalog Framework, and the Catalog Provider are key components of the Reference Implementation. The Endpoint bundle implements a Web service that allows clients to create, update, and delete metacards. The Endpoint calls the CatalogFramework to execute the operations of its specification. The CatalogFramework routes the request through optional PreIngest and PostIngest Catalog Plugins, which may modify the ingest request/response before/after the Catalog Provider executes the ingest request and provides the response.  Note that a CatalogProvider must be present for any ingest requests to be successfully processed, otherwise a fault is returned.

This process is similar for updating catalog entries, with update requests calling the update(UpdateRequest) methods on the Endpoint, CatalogFramework, and Catalog Provider. Similarly, for deletion of catalog entries, the delete requests call the delete(DeleteRequest) methods on the Endpoint, CatalogFramework, and CatalogProvider.

1.1.2.1. Error Handling

Any ingest attempts that fail inside the Catalog Framework (whether the failure comes from the Catalog Framework itself, pre-ingest plugin failures, or issues with the Catalog Provider) will be logged to a separate log file for ease of error handling. The file is located at <DDF_HOME>/data/log/ingest_error.log and will log the Metacards that fail, their ID and Title name, and the stack trace associated with their failure. By default, successful ingest attempts are not logged. However, that functionality can be achieved by setting the log level of the ingestLogger to DEBUG (note that enabling DEBUG can cause a non-trivial performance hit).

Tip

To turn off logging failed ingest attempts into a separate file, execute the following via the command line console

log:set
 ERROR ingestLogger
1.1.2.2. Query
Query Request Data Flow
Query Request Data Flow

The Query Service Endpoint, the Catalog Framework, and the CatalogProvider are key components for processing a query request as well. The Endpoint bundle contains a Web service that exposes the interface to query for Metacards. The Endpoint calls the CatalogFramework to execute the operations of its specification. The CatalogFramework relies on the CatalogProvider to execute the actual query. Optional PreQuery and PostQuery Catalog Plugins may be invoked by the CatalogFramework to modify the query request/response prior to the Catalog Provider processing the query request and providing the query response. If a CatalogProvider is not configured and no other remote Sources are configured, a fault will be returned. It is possible to have only remote Sources configured and no local CatalogProvider configured and be able to execute queries to specific remote Sources by specifying the site name(s) in the query request.

1.1.2.3. Product Retrieval

The Query Service Endpoint, the Catalog Framework, and the CatalogProvider are key components for processing a retrieve product request. The Endpoint bundle contains a Web service that exposes the interface to retrieve products, also referred to as Resources. The Endpoint calls the CatalogFramework to execute the operations of its specification. The CatalogFramework relies on the Sources to execute the actual product retrieval. Optional PreResource and PostResource Catalog Plugins may be invoked by the CatalogFramework to modify the product retrieval request/response prior to the Catalog Provider processing the request and providing the response.  It is possible to retrieve products from specific remote Sources by specifying the site name(s) in the request.

1.1.2.4. Product Caching

The Catalog Framework optionally provides caching of products, so future requests to retrieve the same product will be serviced much quicker. If caching is enabled, each time a retrieve product request is received, the Catalog Framework will look in its cache (default location <DDF_HOME>/data/product-cache) to see if the product has been cached locally. If it has, the product is retrieved from the local site and returned to the client, providing a much quicker turnaround because remote product retrieval and network traffic was avoided. If the requested product is not in the cache, the product is retrieved from the Source (local or remote) and cached locally while returning the product to the client. The caching to a local file of the product and the streaming of the product to the client are done simultaneously so that the client does not have to wait for the caching to complete before receiving the product. If errors are detected during the caching, caching of the product will be abandoned, and the product will be returned to the client. 

The Catalog Framework attempts to detect any network problems during the product retrieval, e.g., long pauses where no bytes are read implying a network connection was dropped. (The amount of time defined as a "long pause" is configurable, with the default value being five seconds.) The Catalog Framework will attempt to retrieve the product up to a configurable number of times (default = three), waiting for a configurable amount of time (default = 10 seconds) between each attempt, trying to successfully retrieve the product. If the Catalog Framework is unable to retrieve the product, an error message is returned to the client.

If the admin has enabled the Always Cache When Canceled option, caching of the product will occur even if the client cancels the product retrieval so that future requests will be serviced quickly. Otherwise, caching is canceled if the user cancels the product download.

1.1.2.5. Product Download Status

As part of the caching of products, the Catalog Framework also posts events to the OSGi notification framework. Information includes when the product download started, whether the download is retrying or failed (after the number of retrieval attempts configured for product caching has been exhausted), and when the download completes. These events are retrieved by the Search UI and presented to the user who initiated the download.

1.1.3. Catalog API

The Catalog API is an OSGi bundle (catalog-core-api) that contains the Java interfaces for the Catalog components and implementation classes for the Catalog Framework, Operations, and Data components.

1.1.3.1. Catalog API Search Interfaces

The Catalog API includes two different search interfaces.

Search UI Application Search Interface

The DDF Search UI application provides a graphic interface to return results and locate them on an interactive globe or map.

SSH Search Interface

Additionally, it is possible to use a client script to remotely access DDF via SSH and send console commands to search and ingest data.

1.1.3.2. Catalog Search Result Objects

Data is returned from searches as Catalog Search Result objects. This is a subtype of Catalog Entry that also contains additional data based on what type of sort policy was applied to the search. Because it is a subtype of Catalog Entry, a Catalog Search Result has all Catalog Entry’s fields such as metadata, effective time, and modified time. It also contains some of the following fields, depending on type of search, that are populated by DDF when the search occurs:

Distance

Populated when a point-radius spatial search occurs. Numerical value that indicates the result’s distance from the center point of the search.

Units

Populated when a point-radius spatial search occurs. Indicates the units (kilometer, mile, etc.) for the distance field.

Relevance

Populated when a contextual search occurs. Numerical value that indicates how relevant the text in the result is to the text originally searched for.

1.1.3.3. Search Programmatic Flow

Searching the catalog involves three basic steps:

  1. Define the search criteria (contextual, spatial, or temporal).

    1. Optionally define a sort policy and assign it to the criteria.

    2. For contextual search, optionally set the fuzzy flag to true or false (the default value for the Metadata Catalog fuzzy flag is true, while the portal default value is false).

    3. For contextual search, optionally set the caseSensitive flag to true (the default is that caseSensitive flag is NOT set and queries are not case sensitive). Doing so enables case sensitive matching on the search criteria. For example, if caseSensitive is set to true and the phrase is “Baghdad” then only metadata containing “Baghdad” with the same matching case will be returned. Words such as “baghdad”, “BAGHDAD”, and “baghDad” will not be returned because they do not match the exact case of the search term.

  2. Issue a search.

  3. Examine the results.

1.1.3.4. Sort Policies

Searches can also be sorted according to various built-in policies. A sort policy is applied to the search criteria after its creation but before the search is issued. The policy specifies to the DDF the order the Catalog search results should be in when they are returned to the requesting client. Only one sort policy may be defined per search.

There are three policies available.

Table 1. Sort Policies
Sort Policy Sorts By Default Order Available for

Temporal

The catalog search result’s effective time field

Newest to oldest

All Search Types

Distance

The catalog search result’s distance field

Nearest to farthest

Point-Radius Spatial searches

Relevance

The catalog search result’s relevance field

Most to least relevant

Contextual

If no sort policy is defined for a particular search, the temporal policy will automatically be applied.

1.1.3.5. Product Retrieval

The DDF is used to catalog resources. A Resource is a URI-addressable entity that is represented by a Metacard. Resources may also be known as products or data. Resources may exist either locally or on a remote data store.

Examples of Resources
  • NITF image

  • MPEG video

  • Live video stream

  • Audio recording

  • Document

Product Retrieval Services
  • SOAP Web services

  • DDF JSON

  • DDF REST

The Query Service Endpoint, the Catalog Framework, and the CatalogProvider are key components for processing a retrieve product request. The Endpoint bundle contains a Web service that exposes the interface to retrieve products, also referred to as Resources. The Endpoint calls the CatalogFramework to execute the operations of its specification. The CatalogFramework relies on the Sources to execute the actual product retrieval. Optional PreResource and PostResource Catalog Plugins may be invoked by the CatalogFramework to modify the product retrieval request/response prior to the Catalog Provider processing the request and providing the response. It is possible to retrieve products from specific remote Sources by specifying the site name(s) in the request.

Note
Product Caching

Existing DDF clients are able to leverage product caching due to the product cache being implemented in the DDF. Enabling the product cache is an administrator function.

Product Caching is enabled by default.

To configure product caching:

  1. Navigate to the Admin Console.

  2. Select Catalog.

  3. Select Configuration.

  4. Select Resource Download Settings.

See Resource Download Settings configurations for all possible configurations.

Product Retrieval Request
Product Retrieval Request
1.1.3.6. Notifications and Activities

DDF can send/receive notifications of "Activities" occuring in the system.

1.1.3.6.1. Notifications

Currently, the notifications provide information about product retrieval only.

1.1.3.6.2. Activities

Activity events include the status and progress of actions that are being performed by the user, such as searches and downloads.

1.2. Included Catalog Frameworks, Associated Components, and Configurations

These catalog frameworks are available in a standard DDF installation:

Standard Catalog Framework

Reference implementation of a Catalog Framework that implements all requirements of the Catalog API.

Catalog Framework Camel Component

Supports creating, updating, and deleting metacards using the Catalog Framework from a Camel route.

1.2.1. Standard Catalog Framework

The Standard Catalog Framework provides the reference implementation of a Catalog Framework that implements all requirements of the Catalog API.  CatalogFrameworkImpl is the implementation of the DDF Standard Catalog Framework.

The Standard Catalog Framework is the core class of DDF. It provides the methods for create, update, delete, and resource retrieval (CRUD) operations on the Sources. By contrast, the Fanout Catalog Framework only allows for query and resource retrieval operations, no catalog modifications, and all queries are enterprise-wide.

Use this framework if:

  • access to a catalog provider is required to create, update, and delete catalog entries.

  • queries to specific sites are required.

  • queries to only the local provider are required.

It is possible to have only remote Sources configured with no local CatalogProvider configured and be able to execute queries to specific remote sources by specifying the site name(s) in the query request.

The Standard Catalog Framework also maintains a list of ResourceReaders for resource retrieval operations. A resource reader is matched to the scheme (i.e., protocol, such as file://) in the URI of the resource specified in the request to be retrieved.

Site information about the catalog provider and/or any federated source(s) can be retrieved using the Standard Catalog Framework. Site information includes the source’s name, version, availability, and the list of unique content types currently stored in the source (e.g., NITF). If no local catalog provider is configured, the site information returned includes site info for the catalog framework with no content types included.

1.2.1.1. Installing the Standard Catalog Framework

The Standard Catalog Framework is bundled as the catalog-core-standardframework feature and can be installed and uninstalled using the normal processes described in Configuration.

1.2.1.2. Configuring the Standard Catalog Framework

These are the configurable properties on the Standard Catalog Framework.

See Catalog Standard Framework configurations for all possible configurations.

Table 2. Standard Catalog Framework Exported Services
Registered Interface Service Property Value

ddf.catalog.federation.FederationStrategy

shortname

sorted

org.osgi.service.event.EventHandler

event.topics

ddf/catalog/event/CREATED, ddf/catalog/event/UPDATED, ddf/catalog/event/DELETED

ddf.catalog.CatalogFramework

ddf.catalog.event.EventProcessor

ddf.catalog.plugin.PostIngestPlugin

Table 3. Standard Catalog Framwork Imported Services
Registered Interface Availability Multiple

ddf.catalog.plugin.PostFederatedQueryPlugin

optional

true

ddf.catalog.plugin.PostIngestPlugin

optional

true

ddf.catalog.plugin.PostQueryPlugin

optional

true

ddf.catalog.plugin.PostResourcePlugin

optional

true

ddf.catalog.plugin.PreDeliveryPlugin

optional

true

ddf.catalog.plugin.PreFederatedQueryPlugin

optional

true

ddf.catalog.plugin.PreIngestPlugin

optional

true

ddf.catalog.plugin.PreQueryPlugin

optional

true

ddf.catalog.plugin.PreResourcePlugin

optional

true

ddf.catalog.plugin.PreSubscriptionPlugin

optional

true

ddf.catalog.plugin.PolicyPlugin

optional

true

ddf.catalog.plugin.AccessPlugin

optional

true

ddf.catalog.resource.ResourceReader

optional

true

ddf.catalog.source.CatalogProvider

optional

true

ddf.catalog.source.ConnectedSource

optional

true

ddf.catalog.source.FederatedSource

optional

true

ddf.cache.CacheManager

 

false

org.osgi.service.event.EventAdmin

 

false

1.2.2. Catalog Framework Camel Component

The Catalog Framework Camel Component supports creating, updating, and deleting metacards using the Catalog Framework from a Camel route.

URI Format
catalog:framework
1.2.2.1. Message Headers
1.2.2.1.1. Catalog Framework Producer
Header Description

operation

the operation to perform using the Catalog Framework (possible values are CREATE | UPDATE | DELETE)

1.2.2.2. Sending Messages to Catalog Framework Endpoint
1.2.2.2.1. Catalog Framework Producer

In Producer mode, the component provides the ability to provide different inputs and have the Catalog framework perform different operations based upon the header values.  

For the CREATE and UPDATE operation, the message body can contain a list of metacards or a single metacard object. 

For the DELETE operation, the message body can contain a list of strings or a single string object. The string objects represent the IDs of metacards to be deleted.  The exchange’s "in" message will be set with the affected metacards. In the case of a CREATE, it will be updated with the created metacards. In the case of the UPDATE, it will be updated with the updated metacards and with the DELETE it will contain the deleted metacards.

Table 4. Catalog Framework Camel Component Operations
Header Message Body (Input) Exchange Modification (Output)

operation = CREATE

List<Metacard> or Metacard

exchange.getIn().getBody() updated with List of Metacards created

operation = UPDATE

List<Metacard> or Metacard

exchange.getIn().getBody() updated with List of Metacards updated

operation = DELETE

List<String> or String (representing metacard IDs)

exchange.getIn().getBody() updated with List of Metacards deleted

Note

If there is an exception thrown while the route is being executed, a FrameworkProducerException will be thrown causing the route to fail with a CamelExecutionException.

1.2.2.2.2. Samples

This example demonstrates:

  1. Reading in some sample data from the file system.

  2. Using a Java bean to convert the data into a metacard.

  3. Setting a header value on the Exchange.

  4. Sending the Metacard to the Catalog Framework component for ingesting.

1
2
3
4
5
6
7
8
<route>
 <from uri="file:data/sampleData?noop=true“/>
    <bean ref="sampleDataToMetacardConverter" method="covertToMetacard"/>\
   <setHeader headerName="operation">
  <constant>CREATE</constant>
 </setHeader>
    <to uri="catalog:framework"/>
</route>

2. Transformers

Transformers
Transformers

Transformers transform data to and from various formats. Transformers are categorized by when they are invoked and used. The existing types are Input transformers, Metacard transformers, and Query Response transformers. Additionally, XSLT transformers are provided to aid in developing custom, lightweight Metacard and Query Response transformers.

Transformers are utility objects used to transform a set of standard DDF components into a desired format, such as into PDF, GeoJSON, XML, or any other format. For instance, a transformer can be used to convert a set of query results into an easy-to-read GeoJSON format (GeoJSON Transformer) or convert a set of results into a RSS feed that can be easily published to a URL for RSS feed subscription. Transformers can be registered in the OSGi Service Registry so that any other developer can access them based on their standard interface and self-assigned identifier, referred to as its "shortname." Transformers are often used by endpoints for data conversion in a system standard way. Multiple endpoints can use the same transformer, a different transformer, or their own published transformer.

Warning

The current transformers only work for UTF-8 characters and do not support Non-Western Characters (for example, Hebrew). It is recommend not to use international character sets, as they may not be displayed properly.

Communication Diagram
Communication Diagram

Transformers are used to alter the format of a resource or its metadata to or from the catalog’s metacard format.

Types of Transformers
Input Transformers

Input Transformers create metacards from input. Once converted to a Metacard, the data can be used in a variety of ways, such as in an UpdateRequest, CreateResponse, or within Catalog Endpoints or Sources. For instance, an input transformer could be used to receive and translate XML into a Metacard so that it can be placed within a CreateRequest to be ingested within the Catalog. Input transformers should be registered within the Service Registry with the interface ddf.catalog.transform.InputTransformer to notify Catalog components of any new transformers.

Metacard Transformers

Metacard Transformers translate a metacard from catalog metadata to a specific data format.

Query Response Transformers

Query Response transformers convert query responses into other data formats.

2.1. Available Input Transformers

The following input transformers are available in a standard installation of DDF:

GeoJSON Input Transformer

Translates GeoJSON into a Catalog metacard.

PDF Input Transformer

Translates a PDF document into a Catalog Metacard.

PPTX Input Transformer

Translates Microsoft PowerPoint (OOXML only) documents into Catalog Metacards.

Registry Transformer

Creates Registry metacards from ebrim messages and translates a Registry metacard. (used by the Registry application)

Tika Input Transformer

Translates Microsoft Word, Microsoft Excel, Microsoft PowerPoint, OpenOffice Writer, and PDF documents into Catalog records.

Video Input Transformer

Creates Catalog metacards from certain video file types.

XML Input Transformer

Translates an XML document into a Catalog Metacard.

2.2. Available Metacard Transformers

The following metacard transformers are available in a standard installation of DDF:

GeoJSON Metacard Transformer

Translates a metacard into GeoJSON.

KML Metacard Transformer

Translates a metacard into a KML-formatted document.

KML Style Mapper

Maps a KML Style URL to a metacard based on that metacard’s attributes.

Metadata Metacard Transformer

returns the Metacard.METADATA attribute when given a metacard.

Registry Transformer

Creates Registry metacards from ebrim messages and translates a Registry metacard. (used by the Registry application)

Resource Metacard Transformer

Retrieves the resource bytes of a metacard by returning the product associated with the metacard.

Thumbnail Metacard Transformer

Retrieves the thumbnail bytes of a Metacard by returning the Metacard.THUMBNAIL attribute value.

XML Metacard Transformer

Translates a metacard into an XML-formatted document.

2.3. Available Query Response Transformers

The following query response transformers are available in a standard installation of DDF:

Atom Query Response Transformer

Transforms a query response into an Atom 1.0 feed.

CSW Query Response Transformer

Transforms a query response into a CSW-formatted document.

GeoJSON Query Response Transformer

Translates a query response into a GeoJSON-formatted document.

KML Query Response Transformer

Translates a query response into a KML-formatted document.

Query Response Transformer Consumer

Translates a query response into a Catalog Metacard.

XML Query Response Transformer

Translates a query response into an XML-formatted document.

2.4. Transformers Details

Availability and configuration details of available transformers.

2.4.1. Atom Query Response Transformer

The Atom Query Response Transformer transforms a query response into an Atom 1.0 feed. The Atom transformer maps a QueryResponse object as described in the Query Result Mapping.

2.4.1.1. Installing the Atom Query Response Transformer

The Atom Query Response Transformer is installed by default with a standard installation.

2.4.1.2. Configuring the Atom Query Response Transformer

The Atom Query Response Transformer has no configurable properties.

2.4.1.3. Using the Atom Query Response Transformer

Use this transformer when Atom is the preferred medium of communicating information, such as for feed readers or federation. An integrator could use this with an endpoint to transform query responses into an Atom feed.

For example, clients can use the OpenSearch Endpoint. The client can query with the format option set to the shortname, atom.

Sample OpenSearch Query with Atom Specified as Return Format
http://{FQDN}:{PORT}/services/catalog/query?q=ddf?format=atom

Developers could use this transformer to programmatically transform QueryResponse objects on the fly.

Sample Atom Feed from QueryResponse object
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
 <feed xmlns="http://www.w3.org/2005/Atom" xmlns:os="http://a9.com/-/spec/opensearch/1.1/">
    <title type="text">Query Response</title>
    <updated>2017-01-31T23:22:37.298Z</updated>
    <id>urn:uuid:a27352c9-f935-45f0-9b8c-5803095164bb</id>
    <link href="#" rel="self" />
    <author>
        <name>Organization Name</name>
    </author>
    <generator version="2.1.0.20130129-1341">ddf123</generator>
    <os:totalResults>1</os:totalResults>
    <os:itemsPerPage>10</os:itemsPerPage>
    <os:startIndex>1</os:startIndex>
    <entry xmlns:relevance="http://a9.com/-/opensearch/extensions/relevance/1.0/" xmlns:fs="http://a9.com/-/opensearch/extensions/federation/1.0/"
        xmlns:georss="http://www.georss.org/georss">
        <fs:resultSource fs:sourceId="ddf123" />
        <relevance:score>0.19</relevance:score>
        <id>urn:catalog:id:ee7a161e01754b9db1872bfe39d1ea09</id>
        <title type="text">F-15 lands in Libya; Crew Picked Up</title>
        <updated>2013-01-31T23:22:31.648Z</updated>
        <published>2013-01-31T23:22:31.648Z</published>
        <link href="http://123.45.67.123:8181/services/catalog/ddf123/ee7a161e01754b9db1872bfe39d1ea09" rel="alternate" title="View Complete Metacard" />
        <category term="Resource" />
        <georss:where xmlns:gml="http://www.opengis.net/gml">
            <gml:Point>
                <gml:pos>32.8751900768792 13.1874561309814</gml:pos>
            </gml:Point>
        </georss:where>
        <content type="application/xml">
            <ns3:metacard xmlns:ns3="urn:catalog:metacard" xmlns:ns2="http://www.w3.org/1999/xlink" xmlns:ns1="http://www.opengis.net/gml"
                xmlns:ns4="http://www.w3.org/2001/SMIL20/" xmlns:ns5="http://www.w3.org/2001/SMIL20/Language" ns1:id="4535c53fc8bc4404a1d32a5ce7a29585">
                <ns3:type>ddf.metacard</ns3:type>
                <ns3:source>ddf.distribution</ns3:source>
                <ns3:geometry name="location">
                    <ns3:value>
                        <ns1:Point>
                            <ns1:pos>32.8751900768792 13.1874561309814</ns1:pos>
                        </ns1:Point>
                    </ns3:value>
                </ns3:geometry>
                <ns3:dateTime name="created">
                    <ns3:value>2013-01-31T16:22:31.648-07:00</ns3:value>
                </ns3:dateTime>
                <ns3:dateTime name="modified">
                    <ns3:value>2013-01-31T16:22:31.648-07:00</ns3:value>
                </ns3:dateTime>
                <ns3:stringxml name="metadata">
                    <ns3:value>
                        <ns6:xml xmlns:ns6="urn:sample:namespace" xmlns="urn:sample:namespace">Example description.</ns6:xml>
                    </ns3:value>
                </ns3:stringxml>
                <ns3:string name="metadata-content-type-version">
                    <ns3:value>myVersion</ns3:value>
                </ns3:string>
                <ns3:string name="metadata-content-type">
                    <ns3:value>myType</ns3:value>
                </ns3:string>
                <ns3:string name="title">
                    <ns3:value>Example title</ns3:value>
                </ns3:string>
            </ns3:metacard>
        </content>
    </entry>
</feed>
Table 5. Atom Query Response Transformer Result Mapping
XPath to Atom XML Value

/feed/title

"Query Response"

/feed/updated

ISO 8601 dateTime of when the feed was generated

/feed/id

Generated UUID URN This link is outside the DDF documentation

/feed/author/name

Platform Global Configuration organization

/feed/generator

Platform Global Configuration site name

/feed/generator/@version

Platform Global Configuration version

/feed/os:totalResults

SourceResponse Number of Hits

/feed/os:itemsPerPage

Request’s Page Size

/feed/os:startIndex

Request’s Start Index

/feed/entry/fs:resultSource/@fs:sourceId

Source Id from which the Result came. Metacard.getSourceId()

/feed/entry/relevance:score

Result’s relevance score if applicable. Result.getRelevanceScore()

/feed/entry/id

urn:catalog:id:<Metacard.ID>

/feed/entry/title

Metacard.TITLE

/feed/entry/updated

ISO 8601 dateTime of Metacard.MODIFIED

/feed/entry/published

ISO 8601 dateTime of Metacard.CREATED

/feed/entry/link[@rel='related']

URL to retrieve underlying resource (if applicable and link is available)

/feed/entry/link[@rel='alternate']

Link to alternate view of the Metacard (if a link is available)

/feed/entry/category

Metacard.CONTENT_TYPE

/feed/entry//georss:where

GeoRSS GML of every Metacard attribute with format AttributeFormat.GEOMETRY

/feed/entry/content

Metacard XML generated by DDF.catalog.transform.MetacardTransformer with shortname=xml. If no transformer found, /feed/entry/content/@type will be text and Metacard.ID is displayed

<content type="text">4e1f38d1913b4e93ac622e6c1b258f89</content>


2.4.2. CSW Query Response Transformer

The CSW Query Response Transformer transforms a query response into a CSW-formatted document.

2.4.2.1. Installing the CSW Query Response Transformer

The CSW Query Response Transformer is installed by default with a standard installation in the Spatial application.

2.4.2.2. Configuring the CSW Query Response Transformer

The CSW Query Response Transformer has no configurable properties.


2.4.3. GeoJSON Input Transformer

The GeoJSON input transformer is responsible for translating GeoJSON into a Catalog metacard.

Table 6. GeoJSON Input Transformer Usage
Schema Mime-types

N/A

application/json

2.4.3.1. Installing the GeoJSON Input Transformer

The GeoJSON Input Transformer is installed by default with a standard installation.

2.4.3.2. Configuring the GeoJSON Input Transformer

The GeoJSON Input Transformer has no configurable properties.

2.4.3.3. Using the GeoJSON Input Transformer

Using the REST Endpoint, for example, HTTP POST a GeoJSON metacard to the Catalog. Once the REST Endpoint receives the GeoJSON Metacard, it is converted to a Catalog metacard.

Example HTTP POST of a Local metacard.json File Using the Curl Command
curl -X POST -i -H "Content-Type: application/json" -d "@metacard.json" https://{FQDN}:{PORT}/services/catalog
2.4.3.4. Conversion to a Metacard

A GeoJSON object consists of a single JSON object. This can be a geometry, a feature, or a FeatureCollection. The GeoJSON input transformer only converts "feature" objects into metacards because feature objects include geometry information and a list of properties. A geometry object alone does not contain enough information to create a metacard. Additionally, the input transformer currently does not handle FeatureCollections.

Important
Cannot create Metacard from this limited GeoJSON
1
2
3
{ "type": "LineString",
 "coordinates": [ [100.0, 0.0], [101.0, 1.0] ]
 }

The following sample will create a valid metacard:

Sample Parseable GeoJson (Point)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
{
    "properties": {
        "title": "myTitle",
        "thumbnail": "CA==",
        "resource-uri": "http://example.com",
        "created": "2012-09-01T00:09:19.368+0000",
        "metadata-content-type-version": "myVersion",
        "metadata-content-type": "myType",
        "metadata": "<xml></xml>",
        "modified": "2012-09-01T00:09:19.368+0000"
    },
    "type": "Feature",
    "geometry": {
        "type": "Point",
        "coordinates": [
            30.0,
            10.0
        ]
    }
}

In the current implementation, Metacard.LOCATION is not taken from the properties list as WKT, but instead interpreted from the geometry JSON object. The geometry object is formatted according to the GeoJSON standard. Dates are in the ISO 8601 standard. White space is ignored, as in most cases with JSON. Binary data is accepted as Base64. XML must be properly escaped, such as what is proper for normal JSON.

Currently, only Required Attributes are recognized in the properties.

2.4.3.4.1. Metacard Extensibility

GeoJSON supports custom, extensible properties on the incoming GeoJSON using DDF’s extensible metacard support. To have those customized attributes understood by the system, a corresponding MetacardType must be registered with the MetacardTypeRegistry. That MetacardType must be specified by name in the metacard-type property of the incoming GeoJSON. If a MetacardType is specified on the GeoJSON input, the customized properties can be processed, cataloged, and indexed.

Sample GeoJSON input
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
{
    "properties": {
        "title": "myTitle",
        "thumbnail": "CA==",
        "resource-uri": "http://example.com",
        "created": "2012-09-01T00:09:19.368+0000",
        "metadata-content-type-version": "myVersion",
        "metadata-content-type": "myType",
        "metadata": "<xml></xml>",
        "modified": "2012-09-01T00:09:19.368+0000",
        "min-frequency": "10000000",
        "max-frequency": "20000000",
        "metacard-type": "ddf.metacard.custom.type"
 },
    "type": "Feature",
    "geometry": {
        "type": "Point",
        "coordinates": [
            30.0,
            10.0
        ]
    }
}

When the GeoJSON Input Transformer gets GeoJSON with the MetacardType specified, it will perform a lookup in the MetacardTypeRegistry to obtain the specified MetacardType in order to understand how to parse the GeoJSON. If no MetacardType is specified, the GeoJSON Input Transformer will assume the default MetacardType. If an unregistered MetacardType is specified, an exception will be returned to the client indicating that the MetacardType was not found.

2.4.3.5. Usage Limitations of the GeoJSON Input Transformer

The GeoJSON Input Transformer does not handle multiple geometries.


2.4.4. GeoJSON Metacard Transformer

GeoJSON Metacard Transformer translates a metacard into GeoJSON.

2.4.4.1. Installing the GeoJSON Metacard Transformer

The GeoJSON Metacard Transformer is not installed by default with a standard installation.

To install:

  1. Navigate to the Admin Console.

  2. Select the System tab.

  3. Select the Features tab.

  4. Install the catalog-transformer-json feature.

2.4.4.2. Configuring the GeoJSON Metacard Transformer

The GeoJSON Metacard Transformer has no configurable properties.

2.4.4.3. Using the GeoJSON Metacard Transformer

The GeoJSON Metacard Transformer can be used programmatically by requesting a MetacardTransformer with the id geojson. It can also be used within the REST Endpoint by providing the transform option as geojson.

Example REST GET Method with the GeoJSON Metacard Transformer
https://{FQDN}:{PORT}/services/catalog/0123456789abcdef0123456789abcdef?transform=geojson
Example REST GET Output from the GeoJSON Metacard Transformer
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
{
    "properties":{
        "title":"myTitle",
        "thumbnail":"CA==",
        "resource-uri":"http:\/\/example.com",
        "created":"2012-08-31T23:55:19.518+0000",
        "metadata-content-type-version":"myVersion",
        "metadata-content-type":"myType",
        "metadata":"<xml>text<\/xml>",
        "modified":"2012-08-31T23:55:19.518+0000",
        "metacard-type": "ddf.metacard"
    },
    "type":"Feature",
    "geometry":{
        "type":"LineString",
        "coordinates":[
            [
                30.0,
                10.0
            ],
            [
                10.0,
                30.0
            ],
            [
                40.0,
                40.0
            ]
        ]
    }
}

2.4.5. GeoJSON Query Response Transformer

The GeoJSON Query Response Transformer translates a query response into a GeoJSON-formatted document.

2.4.5.1. Installing the GeoJSON Query Response Transformer

The GeoJSON Query Response Transformer is installed by default with a standard installation in the Catalog application.

2.4.5.2. Configuring the GeoJSON Query Response Transformer

The GeoJSON Query Response Transformer has no configurable properties.


2.4.6. KML Metacard Transformer

The KML Metacard Transformer is responsible for translating a metacard into a KML-formatted document. The KML will contain an HTML description that will display in the pop-up bubble in Google Earth. The HTML contains links to the full metadata view as well as the product.

2.4.6.1. Installing the KML Metacard Transformer

The KML Metacard Transformer is installed by default with a standard installation in the Spatial Application.

2.4.6.2. Configuring the KML Metacard Transformer

The KML Metacard Transformer has no configurable properties.

2.4.6.3. Using the KML Metacard Transformer

Using the REST Endpoint for example, request a metacard with the transform option set to the KML shortname.

KML Metacard Transformer Example Output
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<kml xmlns:ns2="http://www.google.com/kml/ext/2.2" xmlns="http://www.opengis.net/kml/2.2" xmlns:ns4="urn:oasis:names:tc:ciq:xsdschema:xAL:2.0" xmlns:ns3="http://www.w3.org/2005/Atom">
  <Placemark id="Placemark-0103c77e66d9428d8f48fab939da528e">
    <name>MultiPolygon</name>
    <description>&lt;!DOCTYPE html&gt;
    &lt;html&gt;
      &lt;head&gt;
        &lt;meta content="text/html; charset=windows-1252" http-equiv="content-type"&gt;
        &lt;style media="screen" type="text/css"&gt;
          .label {
            font-weight: bold
          }
          .linkTable {
width: 100% }
          .thumbnailDiv {
            text-align: center
          }
          img {
            max-width: 100px;
            max-height: 100px;
            border-style:none
          }
    &lt;/style&gt;
  &lt;/head&gt;
  &lt;body&gt;
        &lt;div class="thumbnailDiv"&gt;&lt;a href="http://{FQDN}:{PORT}/services/catalog/sources/ddf.distribution/0103c77e66d9428d8f48fab939da528e?transform=resource"&gt;&lt;img alt="Thumnail" src="data:image/jpeg;charset=utf-8;base64, CA=="&gt;&lt;/a&gt;&lt;/div&gt;
    &lt;table&gt;
      &lt;tr&gt;
        &lt;td class="label"&gt;Source:&lt;/td&gt;
        &lt;td&gt;ddf.distribution&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td class="label"&gt;Created:&lt;/td&gt;
        &lt;td&gt;Wed Oct 30 09:46:29 MDT 2013&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
      &lt;td class="label"&gt;Effective:&lt;/td&gt;
        &lt;td&gt;2014-01-07T14:58:16-0700&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/table&gt;
    &lt;table class="linkTable"&gt;
      &lt;tr&gt;
        &lt;td&gt;&lt;a href="http://{FQDN}:{PORT}/services/catalog/sources/ddf.distribution/0103c77e66d9428d8f48fab939da528e?transform=html"&gt;View Details...&lt;/a&gt;&lt;/td&gt;
        &lt;td&gt;&lt;a href="http://{FQDN}:{PORT}/services/catalog/sources/ddf.distribution/0103c77e66d9428d8f48fab939da528e?transform=resource"&gt;Download...&lt;/a&gt;&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/table&gt;
  &lt;/body&gt;
&lt;/html&gt;
</description>
    <TimeSpan>
      <begin>2014-01-07T21:58:16</begin>
    </TimeSpan>
    <Style id="bluenormal">
      <LabelStyle>
        <scale>0.0</scale>
      </LabelStyle>
      <LineStyle>
        <color>33ff0000</color>
        <width>3.0</width>
      </LineStyle>
      <PolyStyle>
        <color>33ff0000</color>
        <fill xsi:type="xs:boolean" xmlns:xs="http://www.w3.org/2001/XMLSchema"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">true</fill>
      </PolyStyle>
      <BalloonStyle>
<text>&lt;h3&gt;&lt;b&gt;$[name]&lt;/b&gt;&lt;/h3&gt;&lt;table&gt;&lt;tr&gt;&lt;td
width="400"&gt;$[description]&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;</text>
      </BalloonStyle>
    </Style>
    <Style id="bluehighlight">
      <LabelStyle>
        <scale>1.0</scale>
      </LabelStyle>
      <LineStyle>
        <color>99ff0000</color>
        <width>6.0</width>
      </LineStyle>
      <PolyStyle>
        <color>99ff0000</color>
        <fill xsi:type="xs:boolean" xmlns:xs="http://www.w3.org/2001/XMLSchema"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">true</fill>
      </PolyStyle>
      <BalloonStyle>
        <text>&lt;h3&gt;&lt;b&gt;$[name]&lt;/b&gt;&lt;/h3&gt;&lt;table&gt;&lt;tr&gt;&lt;td width="400"&gt;$[description]&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;</text>
      </BalloonStyle>
    </Style>
    <StyleMap id="default">
      <Pair>
        <key>normal</key>
        <styleUrl>#bluenormal</styleUrl>
      </Pair>
      <Pair>
        <key>highlight</key>
        <styleUrl>#bluehighlight</styleUrl>
      </Pair>
    </StyleMap>
    <MultiGeometry>
      <Point>
        <coordinates>102.0,2.0</coordinates>
      </Point>
      <MultiGeometry>
        <Polygon>
          <outerBoundaryIs>
            <LinearRing>
              <coordinates>102.0,2.0 103.0,2.0 103.0,3.0 102.0,3.0 102.0,2.0</coordinates>
            </LinearRing>
          </outerBoundaryIs>
        </Polygon>
        <Polygon>
100.8,0.2
    <outerBoundaryIs>
      <LinearRing>
        <coordinates>100.0,0.0 101.0,0.0 101.0,1.0 100.0,1.0 100.0,0.0 100.2,0.2 100.8,0.8 100.2,0.8 100.2,0.2</coordinates>
      </LinearRing>
    </outerBoundaryIs>
  </Polygon>
</MultiGeometry>
</Placemark>
</kml>

2.4.7. KML Query Response Transformer

The KML Query Response Transformer translates a query response into a KML-formatted document. The KML will contain an HTML description for each metacard that will display in the pop-up bubble in Google Earth. The HTML contains links to the full metadata view as well as the product.

2.4.7.1. Installing the KML Query Response Transformer

The spatial-kml-transformer feature is installed by default in the Spatial Application.

2.4.7.2. Configuring the KML Query Response Transformer

The KML Query Response Transformer has no configurable properties.

2.4.7.3. Using the KML Query Response Transformer

Using the OpenSearch Endpoint, for example, query with the format option set to the KML shortname: kml.

KML Query Response Transformer URL
http://{FQDN}:{PORT}/services/catalog/query?q=schematypesearch&format=kml
KML Query Response Transformer Example Output
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<kml xmlns:ns2="http://www.google.com/kml/ext/2.2" xmlns="http://www.opengis.net/kml/2.2" xmlns:ns4="urn:oasis:names:tc:ciq:xsdschema:xAL:2.0" xmlns:ns3="http://www.w3.org/2005/Atom">
  <Document id="f0884d8c-cf9b-44a1-bb5a-d3c6fb9a96b6">
    <name>Results (1)</name>
    <open xsi:type="xs:boolean" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">false</open>
    <Style id="bluenormal">
      <LabelStyle>
        <scale>0.0</scale>
      </LabelStyle>
      <LineStyle>
        <color>33ff0000</color>
        <width>3.0</width>
      </LineStyle>
      <PolyStyle>
        <color>33ff0000</color>
        <fill xsi:type="xs:boolean" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">true</fill>
      </PolyStyle>
      <BalloonStyle>
        <text>&lt;h3&gt;&lt;b&gt;$[name]&lt;/b&gt;&lt;/h3&gt;&lt;table&gt;&lt;tr&gt;&lt;td width="400"&gt;$[description]&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;</text>
      </BalloonStyle>
    </Style>
    <Style id="bluehighlight">
      <LabelStyle>
        <scale>1.0</scale>
      </LabelStyle>
      <LineStyle>
        <color>99ff0000</color>
        <width>6.0</width>
      </LineStyle>
      <PolyStyle>
        <color>99ff0000</color>
        <fill xsi:type="xs:boolean" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">true</fill>
      </PolyStyle>
      <BalloonStyle>
        <text>&lt;h3&gt;&lt;b&gt;$[name]&lt;/b&gt;&lt;/h3&gt;&lt;table&gt;&lt;tr&gt;&lt;td width="400"&gt;$[description]&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;</text>
      </BalloonStyle>
    </Style>
    <StyleMap id="default">
      <Pair>
        <key>normal</key>
        <styleUrl>#bluenormal</styleUrl>
      </Pair>
      <Pair>
        <key>highlight</key>
        <styleUrl>#bluehighlight</styleUrl>
      </Pair>
    </StyleMap>
    <Placemark id="Placemark-0103c77e66d9428d8f48fab939da528e">
      <name>MultiPolygon</name>
      <description>&lt;!DOCTYPE html&gt;
&lt;html&gt;
  &lt;head&gt;
    &lt;meta content="text/html; charset=windows-1252" http-equiv="content-type"&gt;
    &lt;style media="screen" type="text/css"&gt;
      .label {
        font-weight: bold
      }
      .linkTable {
width: 100% }
      .thumbnailDiv {
        text-align: center
} img {
        max-width: 100px;
        max-height: 100px;
        border-style:none
      }
    &lt;/style&gt;
  &lt;/head&gt;
  &lt;body&gt;
        &lt;div class="thumbnailDiv"&gt;&lt;a
href="http://{FQDN}:{PORT}/services/catalog/sources/ddf.distribution/0103c77e66d9428d8f
48fab939da528e?transform=resource"&gt;&lt;img alt="Thumnail"
src="data:image/jpeg;charset=utf-8;base64, CA=="&gt;&lt;/a&gt;&lt;/div&gt;
    &lt;table&gt;
      &lt;tr&gt;
        &lt;td class="label"&gt;Source:&lt;/td&gt;
        &lt;td&gt;ddf.distribution&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td class="label"&gt;Created:&lt;/td&gt;
        &lt;td&gt;Wed Oct 30 09:46:29 MDT 2013&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td class="label"&gt;Effective:&lt;/td&gt;
        &lt;td&gt;2014-01-07T14:48:47-0700&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/table&gt;
    &lt;table class="linkTable"&gt;
      &lt;tr&gt;
        &lt;td&gt;&lt;a
href="http://{FQDN}:{PORT}/services/catalog/sources/ddf.distribution/0103c77e66d9428d8f
48fab939da528e?transform=html"&gt;View Details...&lt;/a&gt;&lt;/td&gt;
        &lt;td&gt;&lt;a
href="http://{FQDN}:{PORT}/services/catalog/sources/ddf.distribution/0103c77e66d9428d8f
48fab939da528e?transform=resource"&gt;Download...&lt;/a&gt;&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/table&gt;
  &lt;/body&gt;
&lt;/html&gt;
</description>
      <TimeSpan>
        <begin>2014-01-07T21:48:47</begin>
      </TimeSpan>
      <styleUrl>#default</styleUrl>
      <MultiGeometry>
        <Point>
          <coordinates>102.0,2.0</coordinates>
        </Point>
        <MultiGeometry>
          <Polygon>
            <outerBoundaryIs>
              <LinearRing>
                <coordinates>102.0,2.0 103.0,2.0 103.0,3.0 102.0,3.0
102.0,2.0</coordinates>
              </LinearRing>
100.8,0.2
  </outerBoundaryIs>
</Polygon>
<Polygon>
  <outerBoundaryIs>
    <LinearRing>
      <coordinates>100.0,0.0 101.0,0.0 101.0,1.0 100.0,1.0 100.0,0.0 100.2,0.2
        100.8,0.8 100.2,0.8 100.2,0.2</coordinates>
              </LinearRing>
            </outerBoundaryIs>
          </Polygon>
        </MultiGeometry>
      </MultiGeometry>
    </Placemark>
  </Document>
</kml>

2.4.8. KML Style Mapper

The KML Style Mapper provides the ability for the KMLTransformer to map a KML Style URL to a metacard based on that metacard’s attributes. For example, if a user wanted all JPEGs to be blue, the KML Style Mapper provides the ability to do so. This would also allow an administrator to configure metacards from each source to be different colors.

The configured style URLs are expected to be HTTP URLs. For more information on style URL’s, refer to the KML Reference This link is outside the DDF documentation.

The KML Style Mapper supports all basic and extended metacard attributes. When a style mapping is configured, the resulting transformed KML contain a <styleUrl> tag pointing to that style, rather than the default KML style supplied by the KMLTransformer.

2.4.8.1. Installing the KML Style Mapper

The KML Style Mapper is installed by default with a standard installation in the Spatial Application in the spatial-kml-transformer feature.

2.4.8.2. Configuring the KML Style Mapper

The properties below describe how to configure a style mapping. The configuration name is Spatial KML Style Map Entry.

See KML Style Mapper configurations for all possible configurations.

KML Style Mapper Example Values
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
<xmlns="http://www.opengis.net/kml/2.2"
  xmlns:ns4="urn:oasis:names:tc:ciq:xsdschema:xAL:2.0"
xmlns:ns3="http://www.w3.org/2005/Atom">
  <Placemark id="Placemark-0103c77e66d9428d8f48fab939da528e">
    <name>MultiPolygon</name>
    <description>&lt;!DOCTYPE html&gt;
&lt;html&gt;
  &lt;head&gt;
    &lt;meta content="text/html; charset=windows-1252" http-equiv="content-type"&gt;
    &lt;style media="screen" type="text/css"&gt;
      .label {
        font-weight: bold
      }
      .linkTable {
width: 100% }
      .thumbnailDiv {
        text-align: center
} img {
        max-width: 100px;
        max-height: 100px;
        border-style:none
      }
    &lt;/style&gt;
  &lt;/head&gt;
  &lt;body&gt;
        &lt;div class="thumbnailDiv"&gt;&lt;a
href="http://{FQDN}:{PORT}/services/catalog/sources/ddf.distribution/0103c77e66d9428d8f48fab939da528e?transform=resource"&gt;&lt;img alt="Thumnail"
src="data:image/jpeg;charset=utf-8;base64, CA=="&gt;&lt;/a&gt;&lt;/div&gt;
    &lt;table&gt;
      &lt;tr&gt;
        &lt;td class="label"&gt;Source:&lt;/td&gt;
        &lt;td&gt;ddf.distribution&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td class="label"&gt;Created:&lt;/td&gt;
        &lt;td&gt;Wed Oct 30 09:46:29 MDT 2013&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
        &lt;td class="label"&gt;Effective:&lt;/td&gt;
        &lt;td&gt;2014-01-07T14:58:16-0700&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/table&gt;
    &lt;table class="linkTable"&gt;
      &lt;tr&gt;
        &lt;td&gt;&lt;a
href="http://{FQDN}:{PORT}/services/catalog/sources/ddf.distribution/0103c77e66d9428d8f48fab939da528e?transform=html"&gt;View Details...&lt;/a&gt;&lt;/td&gt;
        &lt;td&gt;&lt;a href="http://{FQDN}:{PORT}/services/catalog/sources/ddf.distribution/0103c77e66d9428d8f48fab939da528e?transform=resource"&gt;Download...&lt;/a&gt;&lt;/td&gt;
      &lt;/tr&gt;
    &lt;/table&gt;
  &lt;/body&gt;
&lt;/html&gt;
</description>
    <TimeSpan>
      <begin>2014-01-07T21:58:16</begin>
    </TimeSpan>
 <styleUrl>http://example.com/kml/style#sampleStyle</styleUrl>
    <MultiGeometry>
      <Point>
        <coordinates>102.0,2.0</coordinates>
      </Point>
      <MultiGeometry>
        <Polygon>
          <outerBoundaryIs>
            <LinearRing>
              <coordinates>102.0,2.0 103.0,2.0 103.0,3.0 102.0,3.0
102.0,2.0</coordinates>
            </LinearRing>
          </outerBoundaryIs>
        </Polygon>
        <Polygon>
100.8,0.2
<outerBoundaryIs>
  <LinearRing>
    <coordinates>100.0,0.0 101.0,0.0 101.0,1.0 100.0,1.0 100.0,0.0 100.2,0.2
      100.8,0.8 100.2,0.8 100.2,0.2</coordinates>
  </LinearRing>
    </outerBoundaryIs>
  </Polygon>
</MultiGeometry>
</MultiGeometry>
</Placemark>
</kml>

2.4.9. Metadata Metacard Transformer

The Metadata Metacard Transformer returns the Metacard.METADATA attribute when given a metacard. The MIME Type returned is text/xml.

2.4.9.1. Installing the Metadata Metacard Transformer

The Metadata Metacard Transformer is installed by default in a standard installation with the Catalog application.

2.4.9.2. Configuring the Metadata Metacard Transformer

The Metadata Metacard Transformer has no configurable properties.

2.4.9.3. Using the Metadata Metacard Transformer

The Metadata Metacard Transformer can be used programmatically by requesting a metacard transformer with the id metadata. It can also be used within the REST Endpoint by providing the transform option as metadata.

Example REST GET method with the Metadata Metacard Transformer
http://{FQDN}:{PORT}/services/catalog/0123456789abcdef0123456789abcdef?transform=metadata

2.4.10. PDF Input Transformer

The PDF Input Transformer is responsible for translating a PDF document into a Catalog Metacard.

Table 7. PDF Input Transformer Usage
Schema Mime-types

N/A

application/pdf

2.4.10.1. Installing the PDF Input Transformer

The PDF Transformer is installed by default with a standard installation in the Catalog application.

2.4.10.2. Configuring the PDF Input Transformer

To configure the PDF Input Transformer:

  1. Navigate to the Catalog application.

  2. Select the Configuration tab.

  3. Select the PDF Input Transformer.

These configurations are available for the PDF Input Transformer:

See PDF Input Transformer configurations for all possible configurations.

2.4.11. PPTX Input Transformer

The PPTX Input Transformer translates Microsoft PowerPoint (OOXML only) documents into Catalog Metacards, using Apache Tika for basic metadata and Apache POI for thumbnail creation. The PPTX Input Transformer ingests PPTX documents into the DDF Content Repository and the Metadata Catalog, and adds a thumbnail of the first page in the PPTX document.

The PPTX Input Transformer will take precedence over the Tika Input Transformer for PPTX documents.

Table 8. PPTX Input Transformer Usage
Schema Mime-types

N/A

application/vnd.openxmlformats-officedocument.presentationml.presentation

2.4.11.1. Installing the PPTX Input Transformer

This transformer is installed by default with a standard installation in the Catalog application.

2.4.11.2. Configuring the PPTX Input Transformer

The PPTX Input Transformer has no configurable properties. '''

2.4.12. Query Response Transformer Consumer

The Query Response Transformer Consumer is responsible for translating a query response into a Catalog Metacard.

2.4.12.1. Installing the Query Response Transformer Consumer

The Query Response Transformer Consumer is installed by default with a standard installation in the Catalog application.

2.4.12.2. Configuring the Query Response Transformer Consumer

The Query Response Transformer Consumer has no configurable properties.

2.4.13. Registry Transformer

The Registry Transformer creates Registry metacards from ebrim messages. It also returns the ebrim message from the metacard metadata.

2.4.13.1. Installing the Registry Transformer

The Registry Transformer is installed with the Registry application.

  1. Install Registry application.

2.4.13.2. Configuring the Registry Transformer

The Registry Transformer has no configurable properties.

2.4.14. Resource Metacard Transformer

The Resource Metacard Transformer retrieves a resource associated with a metacard.

2.4.14.1. Installing the Resource Metacard Transformer

The Resource Metacard Transformer is installed by default in a standard installation with the Catalog application as the feature catalog-transformer-resource.

2.4.14.2. Configuring the Resource Metacard Transformer

The Resource Metacard Transformer has no configurable properties.

2.4.14.3. Using the Resource Metacard Transformer

Endpoints or other components can retrieve an instance of the Resource Metacard Transformer using its id resource.

Sample Resource Metacard Transformer Blueprint Reference Snippet
<reference id="metacardTransformer" interface="ddf.catalog.transform.MetacardTransformer" filter="(id=resource)"/>

2.4.15. Thumbnail Metacard Transformer

The Thumbnail Metacard Transformer retrieves the thumbnail bytes of a Metacard by returning the Metacard.THUMBNAIL attribute value.

2.4.15.1. Installing the Thumbnail Metacard Transformer

This transformer is installed by default with a standard installation in the Catalog application.

2.4.15.2. Configuring the Thumbnail Metacard Transformer

The Thumbnail Metacard Transformer has no configurable properties.

2.4.15.3. Using the Thumbnail Metacard Transformer

Endpoints or other components can retrieve an instance of the Thumbnail Metacard Transformer using its id thumbnail.

Sample Blueprint Reference Snippet
1
<reference id="metacardTransformer" interface="ddf.catalog.transform.MetacardTransformer" filter="(id=thumbnail)"/>

The Thumbnail Metacard Transformer returns a BinaryContent object of the Metacard.THUMBNAIL bytes and a MIME Type of image/jpeg.


2.4.16. Tika Input Transformer

The Tika Input Transformer is the default input transformer responsible for translating Microsoft Word, Microsoft Excel, Microsoft PowerPoint, OpenOffice Writer, and PDF documents into Catalog records. This input transformer utilizes Apache Tika to provide basic support for these mime types. The metadata common to all these document types, e.g., creation date, author, last modified date, etc., is extracted and used to create the catalog record. The Tika Input Transformer’s main purpose is to ingest these types of content into the Metadata Catalog.

The Tika input transformer is most basic input transformer and the last to be invoked. This allows any registered input transformers that are more specific to a document type to be invoked instead of this rudimentary input transformer.

Table 9. Tika Input Transformer Usage
Schema Mime-types

N/A

This basic transformer can ingest many file types. See All Formats Supported.

2.4.16.1. Installing the Tika Input Transformer

This transformer is installed by default with a standard installation in the Catalog.

2.4.16.2. Configuring the Tika Input Transformer

The properties below describe how to configure the Tika input transformer.

See Tika Input Transformer configurations for all possible configurations.

2.4.17. Video Input Transformer

The video input transformer Creates Catalog metacards from certain video file types. Currently, it is handles MPEG-2 transport streams as well as MPEG-4, AVI, MOV, and WMV videos. This input transformer uses Apache Tika to extract basic metadata from the video files and applies more sophisticated methods to extract more meaningful metadata from these types of video.

Table 10. Video Input Transformer Usage
Schema Mime-types

N/A

  • video/avi

  • video/msvideo

  • video/vnd.avi

  • video/x-msvideo

  • video/mp4

  • video/MP2T

  • video/mpeg

  • video/quicktime

  • video/wmv

  • video/x-ms-wmv

2.4.17.1. Installing the Video Input Transformer

This transformer is installed by default with a standard installation in the Catalog application.

2.4.17.1.1. Configuring the Video Input Transformer

The Video Input Transformer has no configurable properties.

2.4.18. XML Input Transformer

The XML Input Transformer is responsible for translating an XML document into a Catalog Metacard.

Table 11. XML Input Transformer Usage
Schema Mime-types

urn:catalog:metacard

text/xml

2.4.18.1. Installing the XML Input Transformer

The XML Input Transformer is installed by default with a standard installation in the Catalog application.

2.4.18.2. Configuring the XML Input Transformer

The XML Input Transformer has no configurable properties.

2.4.19. XML Metacard Transformer

The XML metacard transformer is responsible for translating a metacard into an XML-formatted document. The metacard element that is generated is an extension of gml:AbstractFeatureType, which makes the output of this transformer GML 3.1.1 compatible.

2.4.19.1. Installing the XML Metacard Transformer

This transformer comes installed by default with a standard installation in the Catalog application.

To install or uninstall manually, use the catalog-transformer-xml feature.

2.4.19.2. Configuring the XML Metacard Transformer

The XML Metacard Transformer has no configurable properties.

2.4.19.3. Using the XML Metacard Transformer

Using the REST Endpoint for example, request a metacard with the transform option set to the XML shortname.

XML Metacard Transformer URL
https://{FQDN}:{PORT}/services/catalog/ac0c6917d5ee45bfb3c2bf8cd2ebaa67?transform=xml
Table 12. Metacard to XML Mappings
Metacard Variables XML Element

id

metacard/@gml:id

metacardType

metacard/type

sourceId

metacard/source

all other attributes

metacard/<AttributeType>[name='<AttributeName>']/value
For instance, the value for the metacard attribute named "title" would be found at metacard/string[@name='title']/value

XML Adapted Attributes (AttributeTypes)
  • boolean

  • base64Binary

  • dateTime

  • double

  • float

  • geometry

  • int

  • long

  • object

  • short

  • string

  • stringxml


2.4.20. XML Query Response Transformer

The XML Query Response Transformer is responsible for translating a query response into an XML-formatted document. The metacard element generated is an extension of gml:AbstractFeatureCollectionType, which makes the output of this transformer GML 3.1.1 compatible.

2.4.20.1. Installing the XML Query Response Transformer

This transformer is installed by default with a standard installation in the Catalog application. To uninstall, uninstall the catalog-transformer-xml feature.

2.4.20.2. Configuring the XML Query Response Transformer

To configure the XML Query Response Transformer:

  1. Navigate to the Admin Console.

  2. Select the Catalog application.

  3. Select the Configuration tab.

  4. Select the XML Query Response Transformer.

See XML Query Response Transformer configurations for all possible configurations.

2.4.20.3. Using the XML Query Response Transformer

Using the OpenSearch Endpoint, for example, query with the format option set to the XML shortname xml.

XML Query Response Transformer Query Example
http://{FQDN}:{PORT}/services/catalog/query?q=input?format=xml
XML Query Response Transformer Example Output
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns3:metacards xmlns:ns1="http://www.opengis.net/gml" xmlns:ns2="http://www.w3.org/1999/xlink" xmlns:ns3="urn:catalog:metacard" xmlns:ns4="http://www.w3.org/2001/SMIL20/" xmlns:ns5="http://www.w3.org/2001/SMIL20/Language">
    <ns3:metacard ns1:id="000ba4dd7d974e258845a84966d766eb">
        <ns3:type>ddf.metacard</ns3:type>
        <ns3:source>southwestCatalog1</ns3:source>
        <ns3:dateTime name="created">
          <ns3:value>2013-04-10T15:30:05.702-07:00</ns3:value>
        </ns3:dateTime>
        <ns3:string name="title">
            <ns3:value>Input 1</ns3:value>
        </ns3:string>
    </ns3:metacard>
    <ns3:metacard ns1:id="00c0eb4ba9b74f8b988ef7060e18a6a7">
        <ns3:type>ddf.metacard</ns3:type>
        <ns3:source>southwestCatalog1</ns3:source>
        <ns3:dateTime name="created">
          <ns3:value>2013-04-10T15:30:05.702-07:00</ns3:value>
        </ns3:dateTime>
        <ns3:string name="title">
            <ns3:value>Input 2</ns3:value>
        </ns3:string>
    </ns3:metacard>
</ns3:metacards>

2.5. Mime Type Mapper

The MimeTypeMapper is the entry point in DDF for resolving file extensions to mime types, and vice versa.

MimeTypeMappers are used by the ResourceReader to determine the file extension for a given mime type in aid of retrieving a product. MimeTypeMappers are also used by the FileSystemProvider in the Catalog Framework to read a file from the content file repository.

The MimeTypeMapper maintains a list of all of the MimeTypeResolvers in DDF.

The MimeTypeMapper accesses each MimeTypeResolver according to its priority until the provided file extension is successfully mapped to its corresponding mime type. If no mapping is found for the file extension, null is returned for the mime type. Similarly, the MimeTypeMapper accesses each MimeTypeResolver according to its priority until the provided mime type is successfully mapped to its corresponding file extension. If no mapping is found for the mime type, null is returned for the file extension.

For files with no file extension, the MimeTypeMapper will attempt to determine the mime type from the contents of the file. If it is unsuccessful, the file will be ingested as a binary file.

DDF Mime Type Mapper

Core implementation of the DDF Mime API.

2.5.1. DDF Mime Type Mapper

The DDF Mime Type Mapper is the core implementation of the DDF Mime API. It provides access to all MimeTypeResolvers within DDF, which provide mapping of mime types to file extensions and file extensions to mime types.

2.5.1.1. Installing the DDF Mime Type Mapper

The DDF Mime Type Mapper is installed by default with a standard installation in the Platform application.

2.5.1.2. Configuring DDF Mime Type Mapper

The DDF Mime Type Mapper has no configurable properties.


2.6. Mime Type Resolver

A MimeTypeResolver is a DDF service that can map a file extension to its corresponding mime type and, conversely, can map a mime type to its file extension.

MimeTypeResolvers are assigned a priority (0-100, with the higher the number indicating the higher priority). This priority is used to sort all of the MimeTypeResolvers in the order they should be checked to map a file extension to a mime type (or vice versa). This priority also allows custom MimeTypeResolvers to be invoked before default MimeTypeResolvers by setting custom resolver’s priority higher than the default.

MimeTypeResolvers are not typically invoked directly. Rather, the MimeTypeMapper maintains a list of MimeTypeResolvers (sorted by their priority) that it invokes to resolve a mime type to its file extension (or to resolve a file extension to its mime type).

Custom Mime Type Resolver

The Custom Mime Type Resolver is a MimeTypeResolver that defines the custom mime types that DDF will support.

Tika Mime Type Resolver

Provides support for resolving over 1300 mime types.

2.6.1. Custom Mime Type Resolver

These are mime types not supported by the default TikaMimeTypeResolver.

Table 13. Custom Mime Type Resolver Default Supported Mime Types
File Extension Mime Type

nitf

image/nitf

ntf

image/nitf

json

json=application/json;id=geojson

As a MimeTypeResolver, the Custom Mime Type Resolver will provide methods to map the file extension to the corresponding mime type, and vice versa.

2.6.1.1. Installing the Custom Mime Type Resolver

One Custom Mime Type Resolver is configured and installed for the image/nitf mime type. This custom resolver is bundled in the mime-core-app application and is part of the mime-core feature.

Additional Custom Mime Type Resolvers can be added for other custom mime types.

2.6.1.1.1. Configuring the Custom Mime Type Resolver

The configurable properties for the Custom Mime Type Resolver are accessed from the MIME Custom Types configuration in the Admin Console.

  • Navigate to the Admin Console.

  • Select the Platform application.

  • Select Configuration.

  • Select MIME Custom Types.

Managed Service Factory PID

  • Ddf_Custom_Mime_Type_Resolver

See Custom Mime Type Resolver configurations for all possible configurations.


2.6.2. Tika Mime Type Resolver

The TikaMimeTypeResolver is a MimeTypeResolver that is implemented using the Apache Tika open source product.

Using the Apache Tika content analysis toolkit, the TikaMimeTypeResolver provides support for resolving over 1300 mime types, but not all mime types yield the same quality metadata.

The TikaMimeTypeResolver is assigned a default priority of -1 to insure that it is always invoked last by the MimeTypeMapper. This insures that any custom MimeTypeResolvers that may be installed will be invoked before the TikaMimeTypeResolver.

The TikaMimeTypeResolver provides the bulk of the default mime type support for DDF.

2.6.2.1. Installing the Tika Mime Type Resolver

The TikaMimeTypeResolver is bundled as the mime-tika-resolver feature in the mime-tika-app application.

This feature is installed by default.

2.6.2.1.1. Configuring the Tika Mime Type Resolver

The Tika Mime Type Resolver has no configurable properties.


3. Catalog Plugins

Catalog Architecture: Catalog Plugins
Catalog Architecture: Catalog Plugins

Plugins are additional tools to use to add additional business logic at certain points, depending on the type of plugin.

The Catalog Framework calls Catalog Plugins to process requests and responses as they enter and leave the Framework. 

3.1. Types of Plugins

Plugins can be designed to run before or after certain processes. They are often used for validation, optimization, or logging. Many plugins are designed to be called at more than one time. See Catalog Plugin Compatibility.

Pre-Authorization Plugins

Perform any changes needed before security rules are applied.

Policy Plugins

Allows or denies access to the Catalog operation or response.

Access Plugins

Used to build policy information for requests.

Pre-Ingest Plugins

Perform any changes to a metacard prior to ingest.

Post-Ingest Plugins

Perform actions after ingest is completed.

Post-Process Plugins

Performs additional processing after ingest.

Pre-Query Plugins

Perform any changes to a query before execution.

Pre-Federated-Query Plugins

Perform any changes to a federated query before execution.

Post-Query Plugins

Perform any changes to a response after query completes.

Post-Federated-Query Plugins

Perform any changes to a response after federated query completes.

Pre-Resource Plugins

Perform any changes to a request associated with a metacard prior to download.

Post-Resource Plugins

Perform any changes to a resource after download.

Pre-Create Storage Plugins

Perform any changes before creating a resource.

Post-Create Storage Plugins

Perform any changes after creating a resource.

Pre-Update Storage Plugins

Perform any changes before updating a resource.

Post-Update Storage Plugins

Perform any changes after updating a resource.

Pre-Subscription Plugins

Perform any changes before creating a subscription.

Pre-Delivery Plugins

Perform any changes before delivering a subscribed event.

Plugins are called in a specific order during different operations. Custom Plugins can be added to the chain for special use cases.

Query Request Plugin Call Order
Query Request Plugin Call Order
Create Request Plugin Call Order
Create Request Plugin Call Order
Update Request Plugin Call Order
Update Request Plugin Call Order
Delete Request Plugin Call Order
Delete Request Plugin Call Order
Resource Request Plugin Call Order
Resource Request Plugin Call Order
Storage Create Request Plugin Call Order
Storage Create Request Plugin Call Order
Storage Update Request Plugin Call Order
Storage Update Request Plugin Call Order
Table 14. Catalog Plugin Compatibility
Plugin Pre-Authorization Plugins Policy Plugins Access Plugins Pre-Ingest Plugins Post-Ingest Plugins Pre-Query Plugins Post-Query Plugins Post-Process Plugins

Catalog Backup Plugin

x

Catalog Metrics Plugin

x

x

x

Catalog Policy Plugin

x

Client Info Plugin

x

Content URI Access Plugin

x

Event Processor

x

Expiration Date Pre-Ingest Plugin

x

Filter Plugin

x

GeoCoder Plugin

x

Historian Policy Plugin

x

Identification Plugin

x

x

JPEG2000 Thumbnail Converter

x

Metacard Attribute Security Policy Plugin

x

Metacard Backup File Storage Provider

x

Metacard Backup S3 Storage Provider

x

Metacard Groomer

x

Metacard Resource Size Plugin

x

Metacard Validity Filter Plugin

x

Metacard Validity Marker

x

Metacard Ingest Network Plugin

x

Operation Plugin

x

Point of Contact Policy Plugin

x

Processing Post-Ingest Plugin

x

Registry Policy Plugin

x

Resource URI Policy Plugin

x

Security Audit Plugin

x

Security Logging Plugin

x

x

x

x

Security Plugin

x

Source Metrics Plugin

x

x

x

Workspace Access Plugin

x

Workspace Pre-Ingest Plugin

x

Workspace Sharing Policy Plugin

x

XML Attribute Security Policy Plugin

x

Table 15. Catalog Plugin Compatibility, Cont.
Plugin Pre-Federated-Query Plugins Post-Federated-Query Plugins Pre-Resource Plugins Post-Resource Plugins Pre-Create Storage Plugins Post-Create Storage Plugins Pre-Update Storage Plugins Post-Update Storage Plugins Pre-Subscription Plugins Pre-Delivery Plugins

Catalog Metrics Plugin

x

Checksum Plugin

x

x

Resource Usage Plugin

x

x

Security Logging Plugin

x!

x

x

x

x

x

x

x

Source Metrics Plugin

x

Video Thumbnail Plugin

x

x

3.1.1. Pre-Authorization Plugins

Pre-delivery plugins are invoked before any security rules are applied.  This is an opportunity to take any action before authorization, including but not limited to:

  • logging.

  • adding network-specific information.

  • adding user-identifying information.

3.1.1.1. Available Pre-Authorization Plugins
Client Info Plugin

Injects request-specific network information into a request.

Metacard Ingest Network Plugin

Adds attributes for network info from ingest request.

3.1.2. Policy Plugins

Policy plugins are invoked to set up the policy for a request/response.  This provides an opportunity to attach custom requirements on operations or individual metacards. All the 'requirements' from each Policy plugin will be combined into a single policy that will be included in the request/response. Access plugins will be used to act on this combined policy.

3.1.2.1. Available Policy Plugins
Catalog Policy Plugin

Configures user attributes required for catalog operations.

Historian Policy Plugin

Protects metacard history from being edited by users without the history role.

Metacard Attribute Security Policy Plugin

Collects attributes into a security field for the metacard.

Metacard Validity Filter Plugin

Determines whether to filter metacards with validation errors or warnings.

Point of Contact Policy Plugin

Adds a policy if Point of Contact is updated.

Registry Policy Plugin

Defines user access polices for registry operations.

Resource URI Policy Plugin

Configures required user attributes for setting or altering a resource URI.

Workspace Sharing Policy Plugin

Collects attributes for a workspace to identify the appropriate policy to allow sharing.

XML Attribute Security Policy Plugin

Finds security attributes contained in a metacard’s metadata.

3.1.3. Access Plugins

Access plugins are invoked directly after the Policy plugins have been successfully executed.  This is an opportunity to either stop processing or modify the request/response based on policy information.

3.1.3.1. Available Access Plugins
Content URI Access Plugin

Prevents a Metacard’s resource URI from being overridden by an incoming UpdateRequest.

Filter Plugin

Performs filtering on query responses as they pass through the framework.

Operation Plugin

Validates a user or subject’s security attributes.

Security Audit Plugin

Audits specific metacard attributes.

Security Plugin

Identifies the subject for an operation.

Workspace Access Plugin

Prevents non-owner users from changing workspace permissions.

3.1.4. Pre-Ingest Plugins

Ingest Plugin Flow
Ingest Plugin Flow

Pre-ingest plugins are invoked before an ingest operation is sent to the catalog. They are not run on a query. This is an opportunity to take any action on the ingest request, including but not limited to:

  • validation.

  • logging.

  • auditing.

  • optimization.

  • security filtering.

3.1.4.1. Available Pre-Ingest Plugins
Expiration Date Pre-Ingest Plugin

Adds or updates expiration dates for the resource.

GeoCoder Plugin

Populates the Location.COUNTRY_CODE attribute if the Metacard has an associated location.

Identification Plugin

Manages IDs on registry metacards.

Metacard Groomer

Modifies metacards when created or updated.

Metacard Validity Marker

Modifies metacards when created or ingested according to metacard validator services.

Security Logging Plugin

Logs operations to the security log.

Workspace Pre-Ingest Plugin

Verifies that a workspace has an associated email to enable sharing.

3.1.5. Post-Ingest Plugins

Query Plugin Flow
Query Plugin Flow

Post-ingest plugins are invoked after data has been created, updated, or deleted in a Catalog Provider.

3.1.5.1. Available Post-Ingest Plugins
Catalog Backup Plugin

Enables backup of the catalog and its metacards.

Catalog Metrics Plugin

Captures metrics on catalog operations.

Event Processor

Creates, updates, and deletes subscriptions.

Identification Plugin

Manages IDs on registry metacards.

Metacard Backup File Storage Provider

Stores backed-up metacards.

Metacard Backup S3 Storage Provider

Stores backed-up metacards in a specified S3 bucket and key.

Processing Post-Ingest Plugin

Submits catalog Create, Update, or Delete requests to the Processing Framework.

Security Logging Plugin

Logs operations to the security log.

Source Metrics Plugin

Captures metrics on catalog operations.

3.1.6. Post-Process Plugins

Note

This code is experimental. While this interface is functional and tested, it may change or be removed in a future version of the library.

Post-Process Plugins are invoked after a metacard has been created, updated, or deleted and committed to the Catalog. They are the last plugins to run and are triggered by a Post-Ingest Plugin. Post-Process plugins are well-suited for asynchronous tasks. See the Asynchronous Processing Framework for more information about how Post-Process Plugins are used.

3.1.7. Pre-Query Plugins

Pre-query plugins are invoked before a query operation is sent to any of the Sources.  This is an opportunity to take any action on the query, including but not limited to:

  • validation.

  • logging.

  • auditing.

  • optimization.

  • security filtering.

3.1.7.1. Available Pre-Query Plugins
Catalog Metrics Plugin

Captures metrics on catalog operations.

Security Logging Plugin

Logs operations to the security log.

Source Metrics Plugin

Captures metrics on catalog operations.

3.1.8. Pre-Federated-Query Plugins

Pre-federated-query plugins are invoked before a federated query operation is sent to any of the Sources.  This is an opportunity to take any action on the query, including but not limited to:

  • validation.

  • logging.

  • auditing.

  • optimization.

  • security filtering.

3.1.8.1. Available Pre-Federated-Query Plugins
Security Logging Plugin

Logs operations to the security log.

Tags Filter Plugin

Updates queries without filters.

3.1.9. Post-Query Plugins

Post-query plugins are invoked after a query has been executed successfully, but before the response is returned to the endpoint.  This is an opportunity to take any action on the query response, including but not limited to:

  • logging.

  • auditing.

  • security filtering/redaction.

  • deduplication.

3.1.9.1. Available Post-Query Plugins
Catalog Metrics Plugin

Captures metrics on catalog operations.

JPEG2000 Thumbnail Converter

Creates thumbnails for jpeg2000 images.

Metacard Resource Size Plugin

Updates the resource size attribute of a metacard.

Security Logging Plugin

Logs operations to the security log.

Source Metrics Plugin

Captures metrics on catalog operations.

3.1.10. Post-Federated-Query Plugins

Post-federated-query plugins are invoked after a federated query has been executed successfully, but before the response is returned to the endpoint.  This is an opportunity to take any action on the query response, including but not limited to:

  • logging.

  • auditing.

  • security filtering/redaction.

  • deduplication.

3.1.11. Pre-Resource Plugins

Pre-Resource plugins are invoked before a request to retrieve a resource is sent to a Source.  This is an opportunity to take any action on the request, including but not limited to:

  • validation.

  • logging.

  • auditing.

  • optimization.

  • security filtering.

3.1.11.1. Available Pre-Resource Plugins
Resource Usage Plugin

Monitors and limits system data usage.

Security Logging Plugin

Logs operations to the security log.

3.1.12. Post-Resource Plugins

Post-resource plugins are invoked after a resource has been retrieved, but before it is returned to the endpoint.  This is an opportunity to take any action on the response, including but not limited to:

  • logging.

  • auditing.

  • security filtering/redaction.

3.1.12.1. Available Post-Resource Plugins
Catalog Metrics Plugin

Captures metrics on catalog operations.

Resource Usage Plugin

Monitors and limits system data usage.

Security Logging Plugin

Logs operations to the security log.

Source Metrics Plugin

Captures metrics on catalog operations.

3.1.13. Pre-Create Storage Plugins

Pre-Create storage plugins are invoked immediately before an item is created in the content repository.

3.1.13.1. Available Pre-Create Storage Plugins
Checksum Plugin

Creates a unique checksum for ingested resources.

Security Logging Plugin

Logs operations to the security log.

3.1.14. Post-Create Storage Plugins

Post-Create storage plugins are invoked immediately after an item is created in the content repository.

3.1.14.1. Available Post-Create Storage Plugins
Security Logging Plugin

Logs operations to the security log.

Video Thumbnail Plugin

Generates thumbnails for video files.

3.1.15. Pre-Update Storage Plugins

Pre-Update storage plugins are invoked immediately before an item is updated in the content repository.

3.1.15.1. Available Pre-Update Storage Plugins
Checksum Plugin

Creates a unique checksum for ingested resources.

Security Logging Plugin

Logs operations to the security log.

3.1.16. Post-Update Storage Plugins

Post-Update storage plugins are invoked immediately after an item is updated in the content repository.

3.1.16.1. Available Post-Update Storage Plugins
Security Logging Plugin

Logs operations to the security log.

Video Thumbnail Plugin

Generates thumbnails for video files.

3.1.17. Pre-Subscription Plugins

Pre-subscription plugins are invoked before a Subscription is activated by an Event Processor.  This is an opportunity to take any action on the Subscription, including but not limited to:

  • validation.

  • logging.

  • auditing.

  • optimization.

  • security filtering.

3.1.18. Pre-Delivery Plugins

Pre-delivery plugins are invoked before a Delivery Method is invoked on a Subscription.  This is an opportunity to take any action before event delivery, including but not limited to:

  • logging.

  • auditing.

  • security filtering/redaction.

3.2. Catalog Plugin Details

Installation and configuration details listed by plugin name.

3.2.1. Catalog Backup Plugin

The Catalog Backup Plugin is used to enable data backup of the catalog and the metacards it contains.

Warning
Catalog Backup Plugin Considerations

Using this plugin may impact performance negatively.

3.2.1.1. Installing the Catalog Backup Plugin

The Catalog Backup Plugin is installed by default with a standard installation in the Catalog application.

3.2.1.2. Configuring the Catalog Backup Plugin

To configure the Catalog Backup Plugin:

  1. Navigate to the Admin Console.

  2. Select Catalog application.

  3. Select Configuration tab.

  4. Select Backup Post-Ingest Plugin.

See Catalog Backup Plugin configurations for all possible configurations.

3.2.1.3. Usage Limitations of the Catalog Backup Plugin
  • May affect performance.

  • Must be installed prior to ingesting any content.

  • Once enabled, disabling may cause incomplete backups.


3.2.2. Catalog Metrics Plugin

The Catalog Metrics Plugin captures metrics on catalog operations. These metrics can be viewed and analyzed using the Metrics Reporting Application in the Admin Console.

3.2.2.2. Installing the Catalog Metrics Plugin

The Catalog Metrics Plugin is installed by default with a standard installation in the Catalog application.

3.2.2.3. Configuring the Catalog Metrics Plugin

The Catalog Metrics Plugin has no configurable properties.


3.2.3. Catalog Policy Plugin

The Catalog Policy Plugin configures the attributes required for users to perform Create, Read, Update, and Delete operations on the catalog.

3.2.3.1. Installing the Catalog Policy Plugin

The Catalog Policy Plugin is installed by default with a standard installation in the Catalog application.

3.2.3.2. Configuring the Catalog Policy Plugin

To configure the Catalog Policy Plugin:

  1. Navigate to the Admin Console.

  2. Select Catalog application.

  3. Select Configuration tab.

  4. Select Catalog Policy Plugin.

See Catalog Policy Plugin configurations for all possible configurations.


3.2.4. Checksum Plugin

The Checksum plugin creates a unique checksum for resources input into the system to identify updated content.

3.2.4.1. Installing the Checksum Plugin

The Checksum is installed by default with a standard installation in the Catalog application.

3.2.4.2. Configuring the Checksum Plugin

The Checksum Plugin has no configurable properties.


3.2.5. Client Info Plugin

The client info plugin injects request-specific network information into request properties, such as Remote IP Address, Remote Host Name, Servlet Scheme, and Servlet Context.

3.2.5.2. Installing the Client Info Plugin

The Client Info Plugin is installed by default with a standard installation in the Catalog application.

3.2.5.3. Configuring the Client Info Plugin

The Client Info Plugin has no configurable properties.


3.2.6. Content URI Access Plugin

The Content URI Access Plugin prevents a Metacard’s resource URI from being overridden by an incoming UpdateRequest.

3.2.6.1. Installing the Content URI Access Plugin

The Content URI Access Plugin is installed by default with a standard installation in the Catalog application.

3.2.6.2. Configuring the Content URI Access Plugin

The Content URI Access Plugin has no configurable properties.


3.2.7. Event Processor

The Event Processor creates, updates, and deletes subscriptions for event notification. These subscriptions optionally specify a filter criteria so that only events of interest to the subscriber are posted for notification.

As metacards are created, updated, and deleted, the Catalog’s Event Processor is invoked (as a post-ingest plugin) for each of these events. The Event Processor applies the filter criteria for each registered subscription to each of these ingest events to determine if they match the criteria.

For more information on creating subscriptions, see Creating a Subscription.

3.2.7.1. Installing the Event Processor

The Event Processor is installed by default with a standard installation in the Catalog application.

3.2.7.2. Configuring the Event Processor

The Event Processor has no configurable properties.

3.2.7.3. Usage Limitations of the Event Processor

The Standard Event processor currently broadcasts federated events and should not. It should only broadcast events that were generated locally, all other events should be dropped. See DDF-3151 for status.


3.2.8. Expiration Date Pre-Ingest Plugin

The Expiration Date plugin adds or updates expiration dates which can be used later for archiving old data.

3.2.8.1. Installing the Expiration Date Pre-Ingest Plugin

The Expiration Date Pre-Ingest Plugin is not installed by default with a standard installation. To install:

  1. Navigate to the Admin Console.

  2. Select the Catalog application.

  3. Select the Configuration tab.

  4. Select the Expiration Data Pre-Ingest Plugin.

3.2.8.2. Configuring the Expiration Date Pre-Ingest Plugin

To configure the Expiration Date Pre-Ingest Plugin:

  1. Navigate to the Admin Console.

  2. Select the Catalog application.

  3. Select the Configuration tab.

  4. Select the Expiration Date Pre-Ingest Plugin.

See Expiration Date Plugin configurations for all possible configurations.


3.2.9. Filter Plugin

The Filter Plugin performs filtering on query responses as they pass through the framework.

Each metacard result can contain security attributes that are pulled from the metadata record after being processed by a PolicyPlugin that populates this attribute. The security attribute is a Map containing a set of keys that map to lists of values. The metacard is then processed by a filter plugin that creates a KeyValueCollectionPermission from the metacard’s security attribute. This permission is then checked against the user subject to determine if the subject has the correct claims to view that metacard. The decision to filter the metacard eventually relies on the installed Policy Decision Point (PDP). The PDP that is being used returns a decision, and the metacard will either be filtered or allowed to pass through.

How a metacard gets filtered is left up to any number of FilterStrategy implementations that might be installed. Each FilterStrategy will return a result to the filter plugin that says whether or not it was able to process the metacard, along with the metacard or response itself. This allows a metacard or entire response to be partially filtered to allow some data to pass back to the requester. This could also include filtering any products sent back to a requester.

The security attributes populated on the metacard are completely dependent on the type of the metacard. Each type of metacard must have its own PolicyPlugin that reads the metadata being returned and then returns the appropriate attributes.

Example (represented as simple XML for ease of understanding):
1
2
3
4
5
6
7
8
9
10
<metacard>
    <security>
        <map>
            <entry assertedAttribute1="A,B" />
            <entry assertedAttribute2="X,Y" />
            <entry assertedAttribute3="USA,GBR" />
            <entry assertedAttribute4="USA,AUS" />
        </map>
    </security>
</metacard>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<user>
    <claim name="subjectAttribute1">
        <value>A</value>
        <value>B</value>
    </claim>
    <claim name="subjectAttribute2">
        <value>X</value>
        <value>Y</value>
    </claim>
    <claim name="subjectAttribute3">
        <value>USA</value>
    </claim>
    <claim name="subjectAttribute4">
        <value>USA</value>
    </claim>
</user>

In the above example, the user’s claims are represented very simply and are similar to how they would actually appear in a SAML 2 assertion. Each of these user (or subject) claims will be converted to a KeyValuePermission object. These permission objects will be implied against the permission object generated from the metacard record. In this particular case, the metacard might be allowed if the policy is configured appropriately because all of the permissions line up correctly.

3.2.9.1. Installing the Filter Plugin

The Filter Plugin is installed by default with a standard installation in the Catalog application.

3.2.9.2. Configuring the Filter Plugin

The Filter Plugin has no configurable properties.


3.2.10. GeoCoder Plugin

The GeoCoder Plugin is a pre-ingest plugin that is responsible for populating the Metacard’s Location.COUNTRY_CODE attribute if the Metacard has an associated location. If there is a valid country code for the Metacard, it will be in ISO 3166-1 alpha-3 format. If the metacard’s country code is already populated, the plugin will not override it. The GeoCoder relies on either the WebService or Offline Gazetteer to retrieve country code information.

Warning

For a polygon or polygons, this plugin takes the center point of the bounding box to assign the country code.

3.2.10.1. Installing the GeoCoder Plugin

The GeoCoder Plugin is installed by default with the Spatial application, when the WebService or Offline Gazetteer is started.

3.2.10.2. Configuring the GeoCoder Plugin

To configure the GeoCoder Plugin:

  1. Navigate to the Admin Console.

  2. Select Spatial application.

  3. Select Configuration tab.

  4. Select GeoCoder Plugin.

These are the available configurations:

See GeoCoder Plugin configurations for all possible configurations.


3.2.11. Historian Policy Plugin

The Historian Policy Plugin protects metacard history from being edited or deleted by users without the history role (a http://schemas.xmlsoap.org/ws/2005/05/identity/claims/role of system-history).

3.2.11.1. Installing the Historian Policy Plugin

The Historian is installed by default with a standard installation in the Catalog application.

3.2.11.2. Configuring the Historian Policy Plugin

The Historian Policy Plugin has no configurable properties.


3.2.12. Identification Plugin

The Identification Plugin assigns IDs to registry metacards and adds/updates IDs on create and update.

3.2.12.1. Installing the Identification Plugin

The Identification Plugin is not installed by default in a standard installation. It is installed by default with the Registry application.

3.2.12.2. Configuring the Identification Plugin

The Identification Plugin has no configurable properties.


3.2.13. JPEG2000 Thumbnail Converter

The JPEG2000 Thumbnail converter creates thumbnails from images ingested in jpeg2000 format.

3.2.13.1. Installing the JPEG2000 Thumbnail Converter

The JPEG2000 Thumbnail Converter is installed by default with a standard installation in the Catalog application.

3.2.13.2. Configuring the JPEG2000 Thumbnail Converter

The JPEG2000 Thumbnail Converter has no configurable properties.


3.2.14. Metacard Attribute Security Policy Plugin

The Metacard Attribute Security Policy Plugin combines existing metacard attributes to make new attributes and adds them to the metacard. For example, if a metacard has two attributes, sourceattribute1 and sourceattribute2, the values of the two attributes could be combined into a new attribute, destinationattribute1. The sourceattribute1 and sourceattribute2 are the source attributes and destinationattribute1 is the destination attribute.

There are two way to combine the values of source attributes. The first, and most common, is to take all of the attribute values and put them together. This is called the union. For example, if the source attributes sourceattribute1 and sourceattribute2 had the values:

sourceattribute1 = MASK, VESSEL

sourceattribute2 = WIRE, SACK, MASK

…​the union would result in the new attribute destinationattribute1:

destinationattribute1 = MASK, VESSEL, WIRE, SACK

The other way to combine attributes is use the values common to all of the attributes. This is called the intersection. Using our previous example, the intersection of sourceattribute1 and sourceattribute2 would create the new attribute destinationattribute1

destinationattribute1 = MASK

because only MASK is common to all of the source attributes.

The policy plugin could also be used to rename attributes. If there is only one source attribute, and the combination policy is union, then the attribute’s values are effectively renamed to the destination attribute.

3.2.14.1. Installing the Metacard Attribute Security Policy Plugin

The Metacard Attribute Security Policy Plugin is installed by default with a standard installation in the Catalog application.

See Metacard Attribute Security Policy Plugin configurations for all possible configurations.


3.2.15. Metacard Backup File Storage Provider

The Metacard Backup File Storage Provider is a storage provider that will store backed-up metacards in a specified file system location.

3.2.15.1. Installing the Metacard Backup File Storage Provider

To install the Metacard Backup File Storage Provider

  1. Navigate to the Admin Console.

  2. Select the System tab.

  3. Select the Features tab.

  4. Install the catalog-metacard-backup-filestorage feature.

3.2.15.2. Configuring the Metacard Backup File Storage Provider

To configure the Metacard Backup File Storage Provider

  1. Navigate to the Admin Console.

  2. Select Catalog application.

  3. Select Configuration tab.

  4. Select Metacard Backup File Storage Provider.

See Metacard Backup File Storage Provider configurations for all possible configurations.


3.2.16. Metacard Backup S3 Storage Provider

The Metacard Backup S3 Storage Provider is a storage provider that will store backed up metacards in the specified S3 bucket and key.

3.2.16.1. Installing the Metacard S3 File Storage Provider

To install the Metacard Backup File Storage Provider

  1. Navigate to the System tab.

  2. Select the Features tab.

  3. Install the catalog-metacard-backup-s3storage feature.

3.2.16.2. Configuring the Metacard S3 File Storage Provider

To configure the Metacard Backup S3 Storage Provider:

  1. Navigate to the Admin Console.

  2. Select Catalog application.

  3. Select Configuration tab.

  4. Select Metacard Backup S3 Storage Provider.

See Metacard Backup S3 Storage Provider configurations for all possible configurations.


3.2.17. Metacard Groomer

The Metacard Groomer Pre-Ingest plugin makes modifications to CreateRequest and UpdateRequest metacards.

Use this pre-ingest plugin as a convenience to apply basic rules for your metacards. 

This plugin makes the following modifications when metacards are in a CreateRequest:

  • Overwrites the Metacard.ID field with a generated, unique, 32 character hexadecimal value if missing or if the resource URI is not a catalog resource URI.

  • Sets Metacard.CREATED to the current time stamp if not already set.

  • Sets Metacard.MODIFIED to the current time stamp if not already set.

  • Sets Core.METACARD_CREATED to the current time stamp if not present.

  • Sets Core.METACARD_MODIFIED to the current time stamp.

In an UpdateRequest, the same operations are performed as a CreateRequest, except:

  • If no value is provided for Metacard.ID in the new metacard, it will be set using the UpdateRequest ID if applicable.

3.2.17.1. Installing the Metacard Groomer

The Metacard Groomer is included in the catalog-core-plugins feature. It is not recommended to uninstall this feature.

3.2.17.2. Configuring the Metacard Groomer

The Metacard Groomer has no configurable properties.


3.2.18. Metacard Ingest Network Plugin

The Metacard Ingest Network Plugin allows the conditional insertion of new attributes on metacards during ingest based on network information from the ingest request; including IP address and hostname.

For the extent of this section, a 'rule' will refer to a configured, single instance of this plugin.

3.2.18.2. Installing the Metacard Ingest Network Plugin

The Metacard Ingest Network Plugin is installed by default during a standard installation in the Catalog application.

3.2.18.3. Configuring the Metacard Ingest Network Plugin

To configure the Metacard Ingest Network Plugin:

  • Navigate to the Admin Console.

  • Select the Catalog application.

  • Select the Configuration tab.

  • Select the label Metacard Ingest Network Plugin to setup a network rule.

See Metacard Ingest Network Plugin configurations for all possible configurations.

Multiple instances of the plugin can be configured by clicking on its configuration title within the configuration tab of the Catalog app. Each instance represents a conditional statement, or a 'rule', that gets evaluated for each ingest request. For any request that meets the configured criteria of a rule, that rule will attempt to transform its list of key-value pairs to become new attributes on all metacards in that request.

The rule is divided into two fields: "Criteria" and "Expected Value". The "Criteria" field features a drop-down list containing the four elements for which equality can be tested:

  • IP Address of where the ingest request came from

  • Host Name of where the ingest request came from

  • Scheme that the ingest request arrived on, for example, http vs https

  • Context Path that the ingest request arrived on, for example, /services/catalog

In order for a rule to evaluate to true and the attributes be applied, the value in the "Expected Value" field must be an exact match to the actual value of the selected criteria. For example, if the selected criteria is "IP Address" with an expected value of "192.168.0.1", the rule only evaluates to true for ingest requests coming from "192.168.0.1" and nowhere else.

Important
Check for IPv6
Verify your system’s IP configuration. Rules using "IP Address" may need to be written in IPv6 format.

The key-value pairs within each rule should take the following form: "key = value" where the "key" is the name of the attribute and the "value" is the value assigned to that attribute. Whitespace is ignored unless it is within the key or value. Multi-valued attributes can be expressed in comma-separated format if necessary.

Examples of Valid Attribute Assignments
contact.contributor-name = John Doe
contact.contributor-email = john.doe@example.net
language = English
language = English, French, German
security.access-groups = SJ202, SR 101, JS2201
3.2.18.3.1. Useful Attributes

The following table provides some useful attributes that may commonly be set by this plugin:

Table 16. Useful Attributes
Attribute Name Expected Format Multi-Valued

expiration

ISO DateTime

no

description

Any String

no

metacard.owner

Any String

no

language

Any String

yes

security.access-groups

Any String

yes

security.access-individuals

Any String

yes

3.2.18.4. Usage Limitations of the Metacard Ingest Network Plugin
  • This plugin only works for ingest (create requests) performed over a network; data ingested via command line does not get processed by this plugin.

  • Any attribute that is already set on the metacard will not be overwritten by the plugin.

  • The order of execution is not guaranteed. For any rule configuration where two or more rules add different values for the same attribute, it is undefined what the final value for that attribute will be in the case where more than one of those rules evaluates to true.


3.2.19. Metacard Resource Size Plugin

This post-query plugin updates the resource size attribute of each metacard in the query results if there is a cached file for the product and it has a size greater than zero; otherwise, the resource size is unmodified and the original result is returned.

Use this post-query plugin as a convenience to return query results with accurate resource sizes for cached products. 

3.2.19.1. Installing the Metacard Resource Size Plugin

The Metacard Resource Size Plugin is installed by default with a standard installation.

3.2.19.2. Configuring the Metacard Resource Size Plugin

The Metacard Resource Size Plugin has no configurable properties.


3.2.20. Metacard Validity Filter Plugin

The Metacard Validity Filter Plugin determines whether metacards with validation errors or warnings are filtered from query results.

3.2.20.2. Installing the Metacard Validity Filter Plugin

The Metacard Validity Filter Plugin is installed by default with a standard installation in the Catalog application.


3.2.21. Metacard Validity Marker

The Metacard Validity Marker Pre-Ingest plugin modifies the metacards contained in create and update requests.

The plugin runs each metacard in the CreateRequest and UpdateRequest against each registered MetacardValidator service.

Note

This plugin can make it seem like ingested products are not successfully ingested if a user does not have permissions to access invalid metacards. If an ingest did not fail, there are no errors in the ingest log, but the expected results do not show up after a query, verify either that the ingested data is valid or that the Metacard Validity Filter Plugin is configured to show warnings and/or errors.

3.2.21.2. Installing Metacard Validity Marker

This plugin is installed by default with a standard installation in the Catalog application.

3.2.21.4. Using Metacard Validity Marker

Use this pre-ingest plugin to validate metacards against metacard validators, which can check schemas, schematron, or any other logic. 


3.2.22. Operation Plugin

The operation plugin validates the subject’s security attributes to ensure they are adequate to perform the operation.

3.2.22.1. Installing the Operation Plugin

The Operation Plugin is installed by default with a standard installation in the Catalog application.

3.2.22.2. Configuring the Operation Plugin

The Operation Plugin has no configurable properties.


3.2.23. Point of Contact Policy Plugin

The Point of Contact Policy Plugin is a PreUpdate plugin that will check if the point-of-contact attribute has changed. If it does, then it adds a policy to that metacard’s policy map that cannot be implied. This will deny such an update request, which essentially makes the point-of-contact attribute read-only.

3.2.23.2. Installing the Point of Contact Policy Plugin

The Point of Contact Policy Plugin is installed by default with a standard installation in the Catalog application.

3.2.23.3. Configuring the Point of Contact Policy Plugin

The Point of Contact Policy Plugin has no configurable properties.


3.2.24. Processing Post-Ingest Plugin

The Processing Post Ingest Plugin is responsible for submitting catalog Create, Update, and Delete (CUD) requests to the Processing Framework.

3.2.24.2. Installing the Processing Post-Ingest Plugin

The Processing Post-Ingest Plugin is not installed by default with a standard installation, but is installed by default when the in-memory Processing Framework is installed.

3.2.24.3. Configuring the Processing Post-Ingest Plugin

The Processing Post-Ingest Plugin has no configurable properties.


3.2.25. Registry Policy Plugin

The Registry Policy Plugin defines the policies for user access to registry entries and operations.

3.2.25.1. Installing the Registry Policy Plugin

The Registry Policy Plugin is not installed by default on a standard installation. It is installed with the Registry application.

3.2.25.2. Configuring the Registry Policy Plugin

The Registry Policy Plugin can be configured from the Admin Console:

  1. Navigate to the Admin Console.

  2. Select the Registry application.

  3. Select the Configuration tab.

  4. Select Registry Policy Plugin.

See Registry Policy Plugin configurations for all possible configurations.


3.2.26. Resource URI Policy Plugin

The Resource URI Policy Plugin configures the attributes required for users to set the resource URI when creating a metacard or alter the resource URI when updating an existing metacard in the catalog.

3.2.26.1. Installing the Resource URI Policy Plugin

The Resource URI Policy Plugin is installed by default with a standard installation in the Catalog application.

3.2.26.2. Configuring the Resource URI Policy Plugin

To configure the Resource URI Policy Plugin:

  1. Navigate to the Admin Console.

  2. Select Catalog application.

  3. Select Configuration tab.

  4. Select Resource URI Policy Plugin.

See Resource URI Policy Plugin configurations for all possible configurations.


3.2.27. Resource Usage Plugin

The Resource Usage Plugin monitors and limits data usage, and enables cancelling long-running queries.

3.2.27.1. Installing the Resource Usage Plugin

The Resource Usage Plugin is not installed by default with a standard installation. It is installed with the Resource Management application.

3.2.27.2. Configuring the Resource Usage Plugin

The Resource Usage Plugin can be configured from the Admin Console:

  1. Navigate to the Admin Console.

  2. Select the Resource Management application.

  3. Select the Configuration tab.

  4. Select Data Usage.

See Resource Usage Plugin configurations for all possible configurations.


3.2.28. Security Audit Plugin

The Security Audit Plugin is used to allow the auditing of specific metacard attributes. Any time a metacard attribute listed in the configuration is updated, a log will be generated in the security log.

3.2.28.1. Installing the Security Audit Plugin

The Security Audit Plugin is installed by default with a standard installation in the Catalog application.


3.2.29. Security Logging Plugin

The Security Logging Plugin logs operations to the security log.

3.2.29.1. Installing Security Logging Plugin

The Security Logging Plugin is installed by default in a standard installation in the Security application.

3.2.29.2. Enhancing the Security Log

The security log contains attributes related to the subject acting on the system. To add additional attributes related to the subject to the logs, append the attribute’s key to the comma separated values assigned to security.logger.extra_attributes in /etc/custom.system.properties.


3.2.30. Security Plugin

The Security Plugin identifies the subject for an operation.

3.2.30.1. Installing the Security Plugin

The Security Plugin is installed by default with a standard installation in the Catalog application.

3.2.30.2. Configuring the Security Plugin

The Security Plugin has no configurable properties.


3.2.31. Source Metrics Plugin

The Source Metrics Plugin captures metrics on catalog operations. These metrics can be viewed and analyzed using the Metrics Reporting Application in the Admin Console.

3.2.31.2. Installing the Source Metrics Plugin

The Source Metrics Plugin is installed by default with a standard installation in the Catalog application.

3.2.31.3. Configuring the Source Metrics Plugin

The Source Metrics Plugin has no configurable properties.


3.2.32. Tags Filter Plugin

The Tags Filter Plugin updates queries without filters for tags, and adds a default tag of resource. For backwards compatibility, a filter will also be added to include metacards without any tags attribute.

3.2.32.2. Installing the Tags Filter Plugin

The Tags Filter Plugin is installed by default with a standard installation in the Catalog application.

3.2.32.3. Configuring the Tags Filter Plugin

The Tags Filter Plugin has no configurable properties.


3.2.33. Video Thumbnail Plugin

The Video Thumbnail Plugin provides the ability to generate thumbnails for video files stored in the Content Repository.

It is an implementation of both the PostCreateStoragePlugin and PostUpdateStoragePlugin interfaces. When installed, it is invoked by the Catalog Framework immediately after a content item has been created or updated by the Storage Provider.

This plugin uses a custom 32-bit LGPL build of FFmpeg (a video processing program) to generate thumbnails. When this plugin is installed, it places the FFmpeg executable appropriate for the current operating system in <DDF_HOME>/bin_third_party/ffmpeg. When invoked, this plugin runs the FFmpeg binary in a separate process to generate the thumbnail. The <DDF_HOME>/bin_third_party/ffmpeg directory is deleted when the plugin is uninstalled.

Note

Prebuilt FFmpeg binaries are provided for Linux, Mac, and Windows only.

3.2.33.1. Installing the Video Thumbnail Plugin

The Video Thumbnail Plugin is installed by default with a standard installation in the Catalog application.

3.2.33.2. Configuring the Video Thumbnail Plugin

To configure the Video Thumbnail Plugin:

  1. Navigate to the Admin Console.

  2. Select the Catalog application.

  3. Select the Configuration tab.

  4. Select the Video Thumbnail Plugin.

See Video Thumbnail Plugin configurations for all possible configurations.


3.2.34. Workspace Access Plugin

The Workspace Access Plugin prevents non-owner users from changing workspace permissions.

3.2.34.2. Installing the Workspace Access Plugin

The Workspace Access Plugin is installed by default with a standard installation in the Catalog application.

3.2.34.3. Configuring the Workspace Access Plugin

The Workspace Access Plugin has no configurable properties.


3.2.35. Workspace Pre-Ingest Plugin

The Workspace Pre-Ingest Plugin verifies that a workspace has an associated email to enable sharing and assigns that email as "owner".

3.2.35.2. Installing the Workspace Pre-Ingest Plugin

The Workspace Pre-Ingest Plugin is installed by default with a standard installation in the Catalog application.

3.2.35.3. Configuring the Workspace Pre-Ingest Plugin

The Workspace Pre-Ingest Plugin has no configurable properties.


3.2.36. Workspace Sharing Policy Plugin

The Workspace Sharing Policy Plugin collects attributes for a workspace to identify the appropriate policy to apply to allow sharing.

3.2.36.2. Installing the Workspace Sharing Policy Plugin

The Workspace Sharing Policy Plugin is installed by default with a standard installation in the Catalog application.

3.2.36.3. Configuring the Workspace Sharing Policy Plugin

The Workspace Sharing Policy Plugin has no configurable properties.


3.2.37. XML Attribute Security Policy Plugin

The XML Attribute Security Policy Plugin parses XML metadata contained within a metacard for security attributes on any number of XML elements in the metadata. The configuration for the plugin contains one field for setting the XML elements that will be parsed for security attributes and the other two configurations contain the XML attributes that will be pulled off of those elements. The Security Attributes (union) field will compute the union of values for each attribute defined and the Security Attributes (intersection) field will compute the intersection of values for each attribute defined.

3.2.37.1. Installing the XML Attribute Security Policy Plugin

The XML Attribute Security Policy Plugin is installed by default with a standard installation in the Security application.


4. Data

Catalog Architecture Diagram: Data
Catalog Architecture Diagram: Data

The Catalog stores and translates Metadata, which can be transformed into many data formats, shared, and queried. The primary form of this metadata is the metacard.  A Metacard is a container for metadata.  CatalogProviders accept Metacards as input for ingest, and Sources search for metadata and return matching Results that include Metacards.

4.1. Metacards

A metacard is a single instance of metadata in the Catalog (an instance of a metacard type) which generally contains general information about the product, such as the title of the product, the product’s geo-location, the date the product was created and/or modified, the owner or producer, and/or the security classification. 

4.1.1. Metacard Type

A metacard type indicates the attributes available for a particular metacard. It is a model used to define the attributes of a metacard, much like a schema.

A metacard type indicates the attributes available for a particular type of data. For example, an image may have different attributes than a PDF document, so each could be defined to have their own metacard type.

4.1.1.1. Default Metacard Type and Attributes

Most metacards within the system are created using the default metacard type or a metacard type based on the default type. The default metacard type of the system can be programmatically retrieved by calling ddf.catalog.data.impl.MetacardImpl.BASIC_METACARD. The name of the default MetacardType can be retrieved from ddf.catalog.data.MetacardType.DEFAULT_METACARD_TYPE_NAME.

The default metacard type has the following required attributes. Though the following attributes are required on all metacard types, setting their values is optional except for ID.

Note

It is highly recommended when referencing a default attribute name to use the ddf.catalog.data.types.* interface constants whenever possible. Mapping to a normalized taxonomy allows for higher quality transformations between different formats and for improved federation. This neutral profile facilitates improved search and discovery across disparate data types.

Warning

Every Source should at the very least return an ID attribute according to Catalog API. Other fields may or may not be applicable, but a unique ID must be returned by a source.

4.1.1.2. Extensible Metacards

Metacard extensibility is achieved by creating a new MetacardType that supports attributes in addition to the required attributes listed above.

Required attributes must be the base of all extensible metacard types. 

Warning

Not all Catalog Providers support extensible metacards. Nevertheless, each Catalog Provider should at least have support for the default MetacardType; i.e., it should be able to store and query on the attributes and attribute formats specified by the default metacard type. Catalog providers are neither expected nor required to store attributes that are not in a given metacard’s type.

Consult the documentation of the Catalog Provider in use for more information on its support of extensible metacards.

Often, the BASIC_METACARD MetacardType does not provide all the functionality or attributes necessary for a specific task. For performance or convenience purposes, it may be necessary to create custom attributes even if others will not be aware of those attributes. One example could be if a user wanted to optimize a search for a date field that did not fit the definition of CREATEDMODIFIEDEXPIRATION, or EFFECTIVE. The user could create an additional java.util.Date attribute in order to query the attribute separately. 

Metacard objects are extensible because they allow clients to store and retrieve standard and custom key/value Attributes from the Metacard.  All Metacards must return a MetacardType object that includes an AttributeDescriptor for each Attribute, indicating it’s key and value type. AttributeType support is limited to those types defined by the Catalog.

New MetacardType implementations can be made by implementing the MetacardType interface.

4.1.2. Metacard Type Registry

Warning

The MetacardTypeRegistry is experimental.  While this component has been tested and is functional, it may change as more information is gathered about what is needed and as it is used in more scenarios.

The MetacardTypeRegistry allows DDF components, primarily catalog providers and sources, to make available the MetacardTypes that they support.  It maintains a list of all supported MetacardTypes in the CatalogFramework, so that other components such as Endpoints, Plugins, and Transformers can make use of those MetacardTypes.  The MetacardType is essential for a component in the CatalogFramework to understand how it should interpret a metacard by knowing what attributes are available in that metacard. 

For example, an endpoint receiving incoming metadata can perform a lookup in the MetacardTypeRegistry to find a corresponding MetacardType.  The discovered MetacardType will then be used to help the endpoint populate a metacard based on the specified attributes in the MetacardType.  By doing this, all the incoming metadata elements can then be available for processing, cataloging, and searching by the rest of the CatalogFramework.

MetacardTypes should be registered with the MetacardTypeRegistry.  The MetacardTypeRegistry makes those MetacardTypes available to other DDF CatalogFramework components.  Other components that need to know how to interpret metadata or metacards should look up the appropriate MetacardType from the registry.  By having these MetacardTypes available to the CatalogFramework, these components can be aware of the custom attributes. 

The MetacardTypeRegistry is accessible as an OSGi service.  The following blueprint snippet shows how to inject that service into another component:

MetacardTypeRegistry Service Injection
1
2
3
4
5
6
<bean id="sampleComponent" class="ddf.catalog.SampleComponent">
    <argument ref="metacardTypeRegistry" />
</bean>

<!-- Access MetacardTypeRegistry -->
<reference id="metacardTypeRegistry" interface="ddf.catalog.data.MetacardTypeRegistry"/>

The reference to this service can then be used to register new MetacardTypes or to lookup existing ones. 

Typically, new MetacardTypes will be registered by CatalogProviders or sources indicating they know how to persist, index, and query attributes from that type.  Typically, Endpoints or InputTransformers will use the lookup functionality to access a MetacardType based on a parameter in the incoming metadata.  Once the appropriate MetacardType is discovered and obtained from the registry, the component will know how to translate incoming raw metadata into a DDF Metacard.

4.1.3. Attributes

An attribute is a single field of a metacard, an instance of an attribute type. Attributes are typically indexed for searching by a source or catalog provider.

4.1.3.1. Attribute Types

An attribute type indicates the attribute format of the value stored as an attribute.  It is a model for an attribute.

4.1.3.1.1. Attribute Format

An enumeration of attribute formats are available in the catalog. Only these attribute formats may be used.

Table 17. Attribute Formats
AttributeFormat Description

BINARY

Attributes of this attribute format must have a value that is a Java byte[] and AttributeType.getBinding() should return Class<Array>of byte.

BOOLEAN

Attributes of this attribute format must have a value that is a Java boolean.

DATE

Attributes of this attribute format must have a value that is a Java date.

DOUBLE

Attributes of this attribute format must have a value that is a Java double.

FLOAT

Attributes of this attribute format must have a value that is a Java float.

GEOMETRY

Attributes of this attribute format must have a value that is a WKT-formatted Java string.

INTEGER

Attributes of this attribute format must have a value that is a Java integer.

LONG

Attributes of this attribute format must have a value that is a Java long.

OBJECT

Attributes of this attribute format must have a value that implements the serializable interface.

SHORT

Attributes of this attribute format must have a value that is a Java short.

STRING

Attributes of this attribute format must have a value that is a Java string and treated as plain text.

XML

Attributes of this attribute format must have a value that is a XML-formatted Java string.

4.1.3.1.2. Attribute Naming Conventions

Catalog taxonomy elements follow the naming convention of group-or-namespace.specific-term, except for extension fields outside of the core taxonomy. These follow the naming convention of ext.group-or-namespace.specific-term and must be namespaced. Nesting is not permitted.

4.1.3.2. Result

A single "hit" included in a query response.

A result object consists of the following:

  • a metacard.

  • a relevance score if included.

  • distance in meters if included.

4.1.4. Creating Metacards

The quickest way to create a Metacard is to extend or construct the MetacardImpl object.  MetacardImpl is the most commonly used and extended Metacard implementation in the system because it provides a convenient way for developers to retrieve and set Attributes without having to create a new MetacardType (see below). MetacardImpl uses BASIC_METACARD as its MetacardType.  

4.1.4.1. Limitations

A given developer does not have all the information necessary to programmatically interact with any arbitrary source.  Developers hoping to query custom fields from extensible Metacards of other sources cannot easily accomplish that task with the current API. A developer cannot question a source for all its queryable fields. A developer only knows about the MetacardTypes which that individual developer has used or created previously. 

The only exception to this limitation is the Metacard.ID field, which is required in every Metacard that is stored in a source. A developer can always request Metacards from a source for which that developer has the Metacard.ID value.  The developer could also perform a wildcard search on the Metacard.ID field if the source allows.

4.1.4.2. Processing Metacards

As Metacard objects are created, updated, and read throughout the Catalog, care should be taken by all catalog components to interrogate the MetacardType to ensure that additional Attributes are processed accordingly.

4.1.4.3. Basic Types

The Catalog includes definitions of several basic types all found in the ddf.catalog.data.BasicTypes class.

Table 18. Basic Types
Name Type Description

BASIC_METACARD

MetacardType

Represents all required Metacard Attributes.

BINARY_TYPE

AttributeType

A Constant for an AttributeType with AttributeType.AttributeFormat.BINARY.

BOOLEAN_TYPE

AttributeType

A Constant for an AttributeType with AttributeType.AttributeFormat.BOOLEAN.

DATE_TYPE

AttributeType

A Constant for an AttributeType with AttributeType.AttributeFormat.DATE.

DOUBLE_TYPE

AttributeType

A Constant for an AttributeType with AttributeType.AttributeFormat.DOUBLE.

FLOAT_TYPE

AttributeType

A Constant for an AttributeType with AttributeType.AttributeFormat.FLOAT.

GEO_TYPE

AttributeType

A Constant for an AttributeType with AttributeType.AttributeFormat.GEOMETRY.

INTEGER_TYPE

AttributeType

A Constant for an AttributeType with AttributeType.AttributeFormat.INTEGER.

LONG_TYPE

AttributeType

A Constant for an AttributeType with AttributeType.AttributeFormat.LONG.

OBJECT_TYPE

AttributeType

A Constant for an AttributeType with AttributeType.AttributeFormat.OBJECT.

SHORT_TYPE

AttributeType

A Constant for an AttributeType with AttributeType.AttributeFormat.SHORT.

STRING_TYPE

AttributeType

A Constant for an AttributeType with AttributeType.AttributeFormat.STRING.

XML_TYPE

AttributeType

A Constant for an AttributeType with AttributeType.AttributeFormat.XML.


5. Operations

catalog architecture operations

The Catalog provides the capability to query, create, update, and delete metacards; retrieve resources; and retrieve information about the sources in the enterprise.

Each of these operations follow a request/response paradigm. The request is the input to the operation and contains all of the input parameters needed by the Catalog Framework’s operation to communicate with the Sources. The response is the output from the execution of the operation that is returned to the client, which contains all of the data returned by the sources. For each operation there is an associated request/response pair, e.g., the QueryRequest and QueryResponse pair for the Catalog Framework’s query operation.

All of the request and response objects are extensible in that they can contain additional key/value properties on each request/response. This allows additional capability to be added without changing the Catalog API, helping to maintain backwards compatibility.


6. Resources