Distributed Data Framework Documentation: Complete Documentation

Many values used in descriptions are customizable and should be changed for specific use cases. These values are denoted by < >, and by [[ ]] when within XML syntax. When using a real value, the placeholder characters should be omitted.

Java objects, lines of code, or file properties are denoted with the Monospace font style. Example: ddf.catalog.CatalogFramework

Some hyperlinks (e.g., /admin) within the documentation assume a locally running installation of DDF.  Simply change the hostname if accessing a remote host.

Hyperlinks that take the user away from the DDF documentation are marked with an external link () icon.

The most current DDF documentation is available at DDF Documentation .

Only these capabilities are supported in a Highly Available Cluster. For a detailed list of features, look at the ha.json file located in <DDF_HOME>/etc/profiles/.

Service standards are implemented within Endpoints and/or Sources. Standards marked Experimental are functional and have been tested, but are subject to change or removal during the incubation period.

DDF has extended capabilities to extract rich metadata from many common data formats if those attributes are populated in the source document. See appendix for a complete list of file formats that can be ingested with limited metadata coverage. Metadata standards use XML or JSON, or both.

DDF makes use of these security standards to protect the system and interactions with it.

Follow the instructions outlined here: Java Requirements.

If DDF is being installed on a remote server that has no user interface, the hostname will need to be updated in the configuration files and certificates.

DDF comes with a populated truststore containing entries for many public certificate authorities, such as Go Daddy and Verisign. It also includes an entry for the DDF Demo Root CA. This entry is a self-signed certificate used for testing. It enables DDF to run immediately after unzipping the distribution. The keys and certificates for the DDF Demo Root CA are included as part of the DDF distribution. This entry must be removed from the truststore before DDF can operate securely.

If using the Demo CA is not desired, DDF supports creating self-signed certificates with a self-signed certificate authority. This is considered an advanced configuration.

Creating self-signed certificates involves creating and configuring the files that contain the certificates. In DDF, these files are generally Java Keystores (jks) and Certificate Revocation Lists (crl). This includes commands and tools that can be used to perform these operations.

For this example, the following tools are used:

After changing the certificates it will be necessary to update the system user and the org.codice.ddf.system.hostname property with the value of either the FQDN or the IP.

FQDNs should be used wherever possible. In the absence of DNS, however, IP addresses can be used.

Replace localhost with the FQDN or the IP in <DDF_HOME>/etc/users.properties, <DDF_HOME>/etc/users.attributes, and <DDF_HOME>/etc/custom.system.properties.

Finally, restart the DDF instance. Navigate to the Admin Console to test changes.

In the event the system is unable to write to the security.log file, DDF must be configured to fall back to report the error in the application log:

If you want to change the location of your systems security backup log from the default location: <DDF_HOME>/data/log/securityBackup.log, follow the next two steps:

DDF has been tested on the following operating systems and with the following browsers. Other operating systems or browsers may be used but have not been officially tested.

For a runtime system:

For a development system:

If JDK was installed:

If JRE was installed:

Restrict access to sensitive files by ensuring that the only users with access privileges are administrators.

Within the <DDF_HOME>, a directory is created named ddf-2.26.17. This directory will be referred to in the documentation as <DDF_HOME>.

DDF uses certificates in two ways:

To ensure proper configuration of keystore, truststore, and certificates, follow the options below according to situation.

Jump to the steps referenced in the diagram:

At this point, DDF should be configured and running with a Solr Catalog Provider. New features (endpoints, services, and sites) can be added as needed.

Verification is achieved by checking that all of the DDF bundles are in an Active state (excluding fragment bundles which remain in a Resolved state).

Execute the following command to display the status of all the DDF bundles:

After successfully completing these steps, the DDF is ready to be configured.

During DDF installation, the major directories and files shown in the table below are created, modified, or replaced in the destination directory.

Upon startup, complete installation from either the Admin Console or the Command Console.

Below is a table listing all of the default ports that DDF uses and a description of what they are used for. Firewalls will need to be configured to open these ports in order for external systems to communicate with DDF.

Below are steps listing all of the changes that DDF requires to run on Internet Explorer 11 and several additional considerations to keep in mind.

These steps are handled differently for the initial setup of a Highly Available Cluster.

Individual features can also be added via the Command Console.

The console output should show an entry similar to the following:

Global configuration settings are configured via the properties file custom.system.properties. These properties can be manually set by editing this file or set via the initial configuration from the Admin Console.

These properties are available to be used as variable parameters in input url fields within the Admin Console. For example, the url for the local csw service (https://{FQDN}:{PORT}/services/csw) could be defined as:

This variable version is more verbose, but will not need to be changed if the system host, port or root context changes.

The DDF is configured using .config files. Like the Karaf .cfg files, these configuration files must be located in the <DDF_HOME>/etc/ directory. Unlike the Karaf .cfg files, .config files must follow the naming convention that includes the configuration persistence ID (PID) that they represent. The filenames must be the pid with a .config extension. This type of configuration file also supports lists within configuration values (metatype cardinality attribute greater than 1) and String, Boolean, Integer, Long, Float, and Double values.

The main purpose of the configuration files is to allow administrators to pre-configure DDF without having to use the Admin Console. In order to do so, the configuration files need to be copied to the <DDF_HOME>/etc directory after DDF zip has been extracted.

Upon start up, all the .config files located in <DDF_HOME>/etc are automatically read and processed. DDF monitors the <DDF_HOME>/etc directory for any new .config file that gets added. As soon as a new file is detected, it is read and processed. Changes to these configurations from the Admin Console or otherwise are persisted in the original configuration file in the <DDF_HOME>/etc directory.

Unauthenticated access to a secured DDF system is provided by the Guest user. By default, DDF allows guest access.

Because DDF does not know the identity of a Guest user, it cannot assign security attributes to the Guest. The administrator must configure the attributes and values (i.e. the "claims") to be assigned to Guests. The Guest Claims become the default minimum attributes for every user, both authenticated and unauthenticated. Even if a user claim is more restrictive, the guest claim will grant access, so ensure the guest claim is only as permissive as necessary.

The Guest user is uniquely identified with a Principal name of the format Guest@UID. The unique identifier is assigned to a Guest based on its source IP address and is cached so that subsequent Guest accesses from the same IP address within a 30-minute window will get the same unique identifier. To support administrators' need to track the source IP Address for a given Guest user, the IP Address and unique identifier mapping will be audited in the security log.

If using REST services or connecting to REST sources, several configuration options are available.

DDF can be configured to support an external SAML IdP or no IdP at all. The following diagram shows the configuration options.

Use the Encryption Service, from the Command Console to set passwords for your LDAP server. Then change the LDAP Bind User Password in the Security STS LDAP and Roles Claims Handler configurations to use the encrypted password.

A claim is an additional piece of data about a principal that can be included in a token along with basic token data. A claims manager provides hooks for a developer to plug in claims handlers to ensure that the STS includes the specified claims in the issued token.

Claims handlers convert incoming user credentials into a set of attribute claims that will be populated in the SAML assertion. For example, the LDAPClaimsHandler takes in the user’s credentials and retrieves the user’s attributes from a backend LDAP server. These attributes are then mapped and added to the SAML assertion being created. Integrators and developers can add more claims handlers that can handle other types of external services that store user attributes.

See the Security STS LDAP and Roles Claims Handler for all possible configurations.

By default, all system users are located in the <DDF_HOME>/etc/users.properties and <DDF_HOME>/etc/users.attributes files. The default users included in these two files are "admin" and "localhost". The users.properties file contains username, password, and role information; while the users.attributes file is used to mix in additional attributes. The users.properties file must also contain the user corresponding to the fully qualified domain name (FQDN) of the system where DDF is running. This FQDN user represents this host system internally when making decisions about what operations the system is capable of performing. For example, when performing a DDF Catalog Ingest, the system’s attributes will be checked against any security attributes present on the metacard, prior to ingest, to determine whether or not the system should be allowed to ingest that metacard.

Additionally, the users.attributes file can contain user entries in a regex format. This allows an administrator to mix in attributes for external systems that match a particular regex pattern. The FQDN user within the users.attributes file should be filled out with attributes sufficient to allow the system to ingest the expected data. The users.attributes file uses a JSON format as shown below:

For this example, the "admin" user will end up with two additional claims of "test" and "test1" with values of "testValue" and [ "testing1", "testing2", "testing3" ] respectively. Also, any host matching the regex ".host." would end up with the claim "reg" with the single value of "ex". The "localhost" user would have no additional attributes mixed in.

If you have integrated DDF with your existing security infrastructure, then you may want to limit access to parts of the DDF based on user roles/groups.

Limit access to the Admin Console to those users who need access. To set access restrictions on the Admin Console, consult the organization’s security architecture to identify specific realms, authentication methods, and roles required.

The default security configuration uses a property file located at <DDF_HOME>/etc/users.properties to store users and passwords. A hardened system will remove this file and manage all users externally, via an LDAP server or by other means.

Once DDF is configured to use an external user (such as LDAP), remove the users.properties file from the <DDF_HOME>/etc directory. Use of a users.properties file should be limited to emergency recovery operations and replaced as soon as effectively possible.

The deletion of the default users in the users.properties file can be done automatically after 72 hours. This feature can be found at Admin Console → Admin → Default Users Deletion Scheduler → Enable default users automatic deletion.

DDF can be configured to prevent login without a valid PKI certificate.

For hardening purposes, it is recommended to implement a way to verify a Certificate Revocation List (CRL) at least daily or an Online Certificate Status Protocol (OCSP) server.

The default catalog provider for DDF is Solr. If using another catalog provider, see Changing Catalog Providers.

This scenario describes how to reconfigure DDF to use a different catalog provider.

This scenario assumes DDF is already running.

By default, the STS server, STS client and the rest of the services use the system property org.codice.ddf.system.hostname which is defaulted to 'localhost' and not to the fully qualified domain name of the DDF instance. Assuming the DDF instance is providing these services, the configuration must be updated to use the fully qualified domain name as the service provider. If the DDF is being accessed from behind a proxy or load balancer, set the system property org.codice.ddf.external.hostname to the hostname users will be using to access the DDF.

This can be changed during Initial Configuration or later by editing the <DDF_HOME>/etc/custom.system.properties file.

DDF performs several types of validation on metadata ingested into the catalog. Depending on need, configure DDF to act on the warnings or errors discovered.

To configure product caching:

See Resource Download Settings configurations for all possible configurations.

The Content Directory Monitor (CDM) provides the capability to easily add content and metacards into the Catalog by placing a file in a directory.

The Platform UI configuration contains the settings for displaying messages to users at login or in banners in the headers and footers of all pages. For, example this configuration can provide warnings that system usage is monitored or controlled.