NIEM API 2.0

This is a Java Spring Boot REST API and backend implementation for NIEM tool functionality. It includes support for NIEM and NIEM-based community models, search, transformations, NIEM subset migrations, and validation.

Provide existing NIEM tool capabilities:

• as open source code for NIEMOpen
• for use with both NIEM reference and user data models
• via an API to make functionality easily accessible to other developers and to avoid tool lock-in
• without requiring users to install and run code locally on their own systems

Support future NIEM model management:

• Maintain and update the NIEM data model
• Build artifacts necessary for publishing NIEM model packages
• Support harmonization work in the NBAC
• Support rapid prototyping for model and NDR updates

Maintain legacy support for older NIEM versions

Support multiple serializations of NIEM

• Leverage CMF and CMF transformations as a modular approach to multi-format support
• Provide direct support for NIEM JSON

The SSGT and other NIEM tools have provided support for a single data model - the NIEM reference data model. This application provides multi-model NIEM support. This will allow published message models from the community to be included in searches and subsets, and will give users access to the same functionality that will be used to support NIEM model management.

Most support for the data model is currently implemented.

• Stewards
• Models
• Versions
• Namespaces
• Properties
• Types
• Subproperties
• Facets

The following model features are not yet implemented:

• Type unions
• Namespace local terminology
• NIEM 1.0 - 2.1 reference properties
• NIEM 2.0 - 2.1 augmentations
• Special EXT namespace support, including
  • Facets on datatype classes (complex types with simple content)
  • Class restriction
  • choice blocks

The API currently only supports read access to NIEM data models. The ability to create, update, and delete models and their contents will be added in the future:

• Read model content
• Create new model content
• Update existing model content
• Delete model content

Search models stored in the application's database, which includes the NIEM reference model and may include published and contributed IEPDs.

• Properties
• Types
• Codes

Property search features include:

The application leverages the CMF tool to transform supported representations of NIEM models to available output formats. Current support:

Inputs:

• CMF 1.0-beta.1
• NIEM XML Schemas (XSD), beginning with NDR version 3.0
• SSGT wantlist (to support users migrating from the SSGT, especially for NIEM 1.0 - 2.1)

Outputs:

• CMF
• NIEM XML Schemas
• NIEM JSON Schema
• Draft OWL representation
• Lite UML representation, such as PlantUML class diagram
• CSVs
• Documentation spreadsheet
• Model stats
• Legacy NIEM XML schemas (NIEM 1.0 - 2.1)

• Migrates a NIEM subset represent in CMF from one version to any subsequent version (multi-step support).
• Generates a migration report to track changes and issues.
• Migrate a model that includes a NIEM or other supported subset, plus extensions.

Note that if a component cannot be migrated, there are two possible reasons:

• The component does not have a counterpart in a later version.
• The component does have a counterpart, but the migration rule has not been added so there is no link between the two.

Migration issues will need to be resolved manually.

• XML - Validate a XML file against provided XML schemas.

• XSD - Validate a set of XML Schemas.

• CMF - Validate a CMF XML file (v1.0-beta.1) against the CMF schemas.

• XML catalog - Validate a XML catalog against the OASIS eXML catalog schema.

• IEPD or message catalog - Validate a NIEM 3.0 MPD catalog or a NIEM 5.0 IEPD catalog XML file against their schemas.

• NDR conformance - Validate NIEM XML schemas against NDR REF and EXT Schematron rules.

  [!NOTE] There are 5 NDR 3.0 rules that were set to always throw errors to encourage user evaluation. These rules were subsequently changed to text rules in NDR 4.0, no longer throwing automatic errors. These 3.0 rules have been disabled here for more consistent rule handling.

  • Rule 4-3: Schema is CTAS-conformant
  • Rule 7-1: Document is an XML document
  • Rule 7-2: Document uses XML namespaces properly
  • Rule 7-3: Document is a schema document
  • Rule 9-83: Target namespace is absolute URI

  NDR 6.0 Status

  [!WARNING] NDR 6.0 conformance validation currently uses an older set of draft rules based on the 5.0 rule set but updated to 6.0 namespaces and rule numbers. Support for the latest 6.0 NDR PSD01 rules requires additional work and is still pending. See issue #74 for the issue status.

• JSON - Validate a JSON instance document against its provided JSON schema.

• JSON schema - Validate a JSON schema document against the JSON schema specification.

• CMF QA - Check a CMF model for general QA issues.

• Property QA - Check a property for NDR conformance issues.

• Type QA - Check a type for NDR conformance issues.

Type

This application uses Type to encompass what the NDR now refers to as classes and datatypes.

Subproperty

This application uses Subproperty (based on XML Schema terminology sub-elements and sub-attributes). The NDR now refers to these as Child Property Associations.

Build jars and run tests:

./gradlew build

Run the application:

./gradlew bootRun

Build a new version of the application:

• Update the version number in field project.ext.draft of file build.gradle
• Run tests
• Run checkstyle
• Build JavaDocs
• Note: Ignore JavaDoc warnings for use of default constructor, which does not provide a comment when the class itself is documented
• Build OpenAPI JSON file
• Build the application
• Deploy the application
• Update the search index
• Test endpoints

API documentation files:

• OpenAPI JSON available at https://api.niemopen.org/v2/api-docs or in the project repo under /docs/openapi.json.
• Swagger HTML available at https://api.niemopen.org/v2/swagger-ui/index.html.

Build documentation:

./gradlew generateOpenApiDocs

Known issues:

• JavaDoc definitions for overridden methods

  OpenAPI schema components are not picking up JavaDoc definitions for methods that are overridden, either in the parent or the child. This is why definitions are being repeated in the @Schema annotations.

• Project properties not expanded during generateOpenApiDocs gradle task.

The task to build the OpenAPI JSON file, which is generated and included in the docs/ folder, is not pulling environment variables used by the Spring profiles from the .env file.

Create a openapi profile to set the database URL, username and password, or change the active profile argument in build.gradle's openApi task custom boot run settings.

• Request body parameters.

  OpenAPI annotation @RequestParam should be able to be used for request body parameters for endpoints that consume multipart form data. These instead are being generated as query parameters in the OpenAPI documentation.

  @RequestPart can be used to document request body parameters, but has the following drawbacks when compared to @RequestParam:

  • Allowable values are not listed in the OpenAPI documentation for params with an enum type. These parameters are simply marked as strings.
  • Default values are not listed.
  • Example values are not listed.
  • Type conversion in the controllers for parameters types besides Strings or multipart files is not automatically handled.

  To simplify the code, the @RequestParam annotation is being used despite the incorrect marking of request body parameters as query parameters. Additional documentation has been added to each of the parameters as the simplest workaround.

This project uses lombok to reduce boilerplate code. See the Install section of their website to add support for your IDE.

Note: When reviewing Javadoc warnings, correct the original src file, not the generated one under build/generated/sources/delombok.

You can create file .env to define postgresql url, username and password values.

The variables declared in this file will be imported into application.yaml if available via the spring.config.import property (optional import).

There or other ways to include these variables, such as via system or user environment variables and via CI/CD settings.

NDR validation is currently run against XML Schemas by applying NDR Schematron rules converted to Schematron Validation Report Language (SVRL) XML Stylesheets (XSL).

To generate new XSL files:

• Download the NDR Schematron rules (.sch files) and the associated ndr-functions.xsl file from https://github.com/niemopen/niem-naming-design-rules.

• Apply the iso_svrl_for_xslt2.xsl stylesheet in this projects /src/main/resources/validation/ndr/ directory to convert Schematron to a stylesheet that generates SVRL.

  See more, including additional stylesheets to assemble included files, at https://github.com/Schematron/stf/tree/master/iso-schematron-xslt2.

Apply stylesheet in Oxygen

• Open the Schematron rule file
• Go to Document / Transformation / Configure Transformation Scenario
• Create a new scenario:
  • Select XML transformation with XSLT
  • Name the scenario, e.g., "Schematron to SVRL XSL"
  • In the XSLT tab, XSL URL field, select the iso_svrl_for_xslt2.xsl file.
  • In the Output tab, change Output file to Prompt for file.
• Apply the new scenario to the .sch file.

Apply stylesheet via the command line using Saxon jars

java net.sf.saxon.Transform -s:source -xsl:stylesheet -o:output

• Source: Schematron file
• Stylesheet: iso_svrl_for_xslt2.xsl
• Output: Path and filename for results, e.g., niem-ndr-rules-5.0-ref.xsl

Post transform

• Add the following line to the XSL results to include the ndr-functions-#.#.xsl file, with the appropriate NDR version number manually added to the ndr-functions filename to support multiple versions:

  <xsl:include xmlns:sch="http://purl.oclc.org/dsdl/schematron" href="ndr-functions-#.#.xsl"/>

Notes

As mentioned above under the Features > Validation > Conformance Validation section, the following adjustments have been made:

• NDR 3.0: Removed Schematron testing for rules that cannot be evaluated in Schematron and always throw errors.
• NDR 6.0: Older draft rule set with updated URIs and rule numbers is being used until additional support needed for the latest rule set can be added.

A separate database schema (test) is used for testing purposes.

• Run git submodule update --remote --recursive.
• Update app.cmftool properties in application.yaml.

If the version of CMF has changed during an upgrade to the CMF Tool:

• Update CMF schemas for the CMF validation endpoint under src/main/resources/validation/cmf.
• Update the path to the CMF schemas in NiemValidationService method validateCmf().
• Update app.cmf and app.cmftool properties in application.yaml.
• Update CMF files used in src/test/resources.

Adjustments

Comment out the following dependency in lib-cmf/build.gradle and lib-util/build.gradle:

testImplementation libs.junit.jupiter