Updated for Version 4.16

Overview

Larger organizations (e.g., the Department of Transportation) may submit the same form type (e.g., Construction Stormwater Permit, monthly reports, etc.) in a regular and repetitive manner. To help better support these situations and to minimize data entry requirements, a feature was established in the nFORM system which allows these organizations to upload data for a form via a form-compliant XML document.

The system dynamically defines an XML Schema (i.e., defines valid XML document structure/data) based on the established form design. The regulated entity will establish an XML document from their data management system(s) that is compliant with the form design and XML Schema structure. The regulated entity will then upload this document within the system.

The uploaded document is validated against a system generated XML Schema (which mimics the forms structure) and if valid, a new submission will be established (when needed) and data on the submission is populated. For existing submissions, the data on the submission will be replaced. The regulated entity will then have the ability to review the submission, address any issues and submit their submission to the agency. If the XML document is considered invalid, the user will be notified of an error and will need to resolve this issue before proceeding.

It is worth noting that as a forms design changes, so does the supporting XML Schema/structure needed to support the form design. For this reason, this feature should be used sparingly and forms that support the upload process should not be changed without careful consideration as to the potential impacts on the upload process and the regulated community.

Introduction

The nFORM tool supports the ability for a reporting entity to upload structured data into a submission via a form-compliant XML document. This feature helps to automate form submissions for those organizations that have the need to submit many submissions of the same form type. These organizations will now have the ability to generate compliant XML documents, representing their submissions, and upload them into the nFORM system to minimize data entry requirements.

This document provides guidance on how to establish and import submission data into a form.

Please note that this is an advanced topic and most content is intended to be utilized by advanced technical staff which have knowledge of XML and XML Schemas.

XML Document Upload Process

This section of the document describes how a user will upload and import data using XML.

Prerequisites

Before data can be uploaded and imported into a form, some prerequisites must be met, including:

• The XML Upload feature must be enabled for the form in question. See the Feature Enablement topic for additional details in this regard.

• The data to be imported must be stored in a compliant XML document for the form in question. See the XML Document Development topic for additional details in this regard.

XML Document Upload Process Walkthrough

Users will have the ability to upload and import data into a form via a form compliant XML Document that represents the submission and then to subsequently review the submission, address any issues and submit their submission to the agency.

The regulated entity will generate compliant XML Document (with submission data for submission) from their source system(s) that is compliant with the form’s defined XML Schema. See the XML Document Development topic for additional details in this regard.

For new submissions, the regulated entity will click on the Upload [XML Document Name] button from the nFORM Form View page and select the file to upload from the Windows Open file dialog to initiate the upload process.

For existing submissions, the regulated entity will open the submission in draft mode and will select the Upload [XML Document Name] (XML) option found within the cog menu in the header area of the form in the Submission Wizard and select the relevant XML document from the Windows Open file dialog to initiate the upload process.

Alternatively, for existing submissions, the regulated entity can open the Submission Overview page for the Submission (which is NOT in a draft state), select the down arrow in the Revise Submission button and select Revise with new [XML Document Name] (upload) option.

The user will select the relevant XML Document from the Windows Open file dialog to initiate the upload process. Using this option will start a new version of the submission, will navigate the user to the Submission Wizard and will pre-fill the existing submission with the data selected. Note: any editable attributes on the form, will be replaced with the data provided.

The system will receive the selected XML Document and begin processing. The system will validate that the XML Document is compliant with the defined XML Schema. This is a test of whether the data complies to the appropriate structure, size and data type. If the XML Document is invalid, the error(s) returned from the XML validator will be presented to the user. The upload will be halted (no data will be changed in the system) and the user will need to address the reported issues and regenerate/reupload the XML document. If the XML is valid, the system will begin the data upload process.

If the submission does not exist, a new submission will be established. nFORM will establish the new submission and will prepopulate this submission with data based on the form configuration, the provided pre-population context (e.g., site, permit, etc.) and then the uploaded submission data (from XML) will be populated in the submission. If a draft version of the submission doesn’t already exist, a revision will be established. The uploaded submission data (from the XML Document) will be replaced in the submission, following normal form design rules (e.g., form controls that were prepopulated or attributes that are marked as read-only will not be changed/overwritten on the form).

The user will have the opportunity to review the submission and address/review form/control level validation issues, review populated data for accuracy and completeness and to submit the form.

If and when appropriate, the user will have the ability to initiate the signature (single or multiple) process for the submission.

XML Document Development

Process Overview

When a regulated entity wishes to establish an automated process to generate an XML Document that represents their submission, they will need to design/develop a custom process to generate repeatedly these XML Documents using their data system(s). When approved, the agency will provide users with the XML Schema (see the XML Schema Generation/Access section) that dictates the valid structure of the data and any supplemental documentation that may be helpful (e.g., sample submission files, how to documents, XML Schema, etc.).

The regulated entity will request techniques and methods for upload of a particular form from the agency. 

Agency staff will navigate to the form designer and open the form in question. The system will generate the XML Schema for the current form and return it to the current user. 

Agency staff package this XML Schema with any other supporting documentation and will provide this to the regulated entity that requested the upload materials.

Using the form’s XML Schema and any other supporting materials, the regulated entity will establish the ability to repeatedly generate compliant XML from their source system(s). See the XML Document Development section of this document for additional details on the expected structure of the XML document. Additionally, see the Testing/Validating XML Document section of this document for details on how to test the upload process.

Development Components

In order to upload an XML Document to a form submission, the XML Document must be compliant with the XML XSD Schema for the form in question.

This section describes the following topics:

1. XML Schema: The structure for which the XML Document must be compliant.

2. XML Document: The elements of the XML Document.

3. Testing/Validating XML Document: Recommendations for how to test a generated XML Document.

XML Schema

In order to upload an XML Document to a form submission, the XML Document must be compliant with the XML XSD Schema for the form in question.

Per W3C, an XML Schema is a language for expressing constraints about XML documents and is used:

• to provide a list of elements and attributes in a vocabulary;

• to associate types, such as integer, string, etc. with values found in documents;

• to constrain where elements and attributes can appear, and what can appear inside those elements;

• to provide documentation that is both human-readable and machine-processable;

• to give a formal description of one or more documents.

The schema language utilized is W3C XSD (XML Schema Definitions).

Standard XML editing tools (e.g., XMLSpy, OxygenXML, etc.) can be used to check/validate an XML Document against an XML Schema. Such a check/validation will be performed by the nFORM system upon every XML document upload. As such, XML Documents should be validated against the XML Schema in the development process to ensure compliance with the expected structure. The XML Schema can be received from agency staff. There are a significant amount of resources available on the web with regards to XML XSD Schemas so we won’t go into great deal on that matter here, but we will provide details on how nFORM utilizes this toolset. See XML Schema Generation/Access for additional details on location/availability of the XML Schema.

The XML Schema will mimic the form structural elements and utilizes similar datatypes.

XML Document

This section provides details as to the expected elements and structure of the XML Document.

Example Submission XML Document

To ease the process of XML development, a sample XML Document can be generated from within the system. To generate this document for a given submission, enter the following URL in your browser window (while signed into the system as a user with edit access to the selected submission). To do this assemble the URL in the following format:

[root application URL]/SubmissionVersionXml/[SubmissionVersionID]

For example, in Windsor’s QA environment, for a specific submission, the URL would look like the following:

https://qaweb.windsorsolutions.biz/nForm/SubmissionVersionXml/42d9f636-f31a-40bb-aa13-d132932c960b

We have found that a fully populated submission of the form type in question, will provide a great example of the expected XML Document format.

Important Note: if the form includes controls that are configured to be read-only (not read-only if pre-filled), these properties will be included in this export, but will not be allowed to be imported. Any read-only properties should be removed from the XML Document, before upload, for full compliance.

See Appendix A – Example XML Document section for a visual example of what the XML Document could look like.

XML Document Structure

Below is a summary of how elements are represented within the XML document.

Header

No specific header information is anticipated or expected.

Base Structure

The table below represents the expected structure of the payload within the document.

The following properties will be dynamically assigned from form-to-form:

• Schema properties:

  • Local XML Schema File Location: The local workstation path to the form’s XML Schema. This is used to allow the XML editing tool to validate the XML file against the schema.

• Form properties: These form properties are included for reference and troubleshooting purposes.

  • Form Name: The name of the form which will be populated from this document.

  • Form Tag: The Tag name of the form which will be populated from this document.

  • Form Major Version Number: The major version # of the form which will be populated from this document. This is used to ensure the structure of the XML document is aligned with the current version of the form.

  • Form Minor Version Number: The minor version # of the form which will be populated from this document. This is used to ensure the structure of the XML document is aligned with the current version of the form.

• Form/Submission Sections and Controls properties:

  • Form/Submission Sections and Controls: This is where the main submission/form content will reside. This portion of the XML document will vary from form-to-form, based on the configuration of the specific form.

For example:

Form/Submission Sections and Controls

The structure of the form/submission sections and controls section will be dependent upon the structure and configuration of the form’s sections and controls.

Note: Section references must utilize the Tag name assigned to the section and Control references must utilize the Tag name assigned to the control

The following table provides an example of the structure expected within the form/submission sections and controls area:

In the above example, the forms includes three sections: Company, Site Location and Contacts.

The first section, Company, includes the Company Name, Site # and Facility Type. The second section, Site Location, includes Location Name, Location Coordinates and Location Address. The third section, Contacts is a repeating section and includes multiple contacts, including Contact Type and Contact Person details (e.g., name, title, email, address and phone).

The following sections describes how different types of dynamic attributes can be included/utilized within the XML document.

Sections

Two section types exist: 1) Discrete and Repeating. Discrete sections are sections that stand alone and are not repeated. Repeating sections are those where all the attributes within the section can be repeated, as appropriate.

Discrete Section

A discrete section will be represented as a complex type, utilizing the Section Tags as the name for the complex type.

For example:

Repeating Section

A repeating section will be represented as multiple complex types, one representing the grouping of repeated elements and one representing each set of repeated elements within the section. The grouping of repeated elements complex type is identified as the Section Tag name plus “_REPEATER” and the complex type representing each set of repeated elements is identified as the Section Tag name.

A repeating section will be represented as depicted in the example below:

Controls

Simple Text Controls

Simple text controls are values stored in controls such as Short Text, Paragraph and SSN.

Simple text control values are represented within a simple type assigning the control Tag name as the name of the simple type.

A simple text control will follow this format:

Simple text control values will be represented as depicted in the example below:

Note: the max length will be enforced based on the max length configured for the control.

Select Controls

Select controls are values stored in the Single and Multiple Select controls.

Select control values are represented within a complex type assigning the control Tag name as the name of the complex type and the value(s) within Value tags.

Single Select controls support the selection of zero or one values.

A Single Select control will follow this format:

For example:

Multiple Select controls support the selection of zero or more values.

A Multiple Select control will follow this format:

For example:

Formatted Controls

Formatted controls are values stored in controls such as Number, Date, Time, Email, URL and Phone.

Formatted control values are represented within a simple type assigning the control Tag name as the name of the simple type in similar fashioned to the simple text controls. The only difference is that the data provided in may be validated against basic datatypes (e.g., decimal, date, time, string, etc.).

Formatted controls will follow this format:

Formatted control values will be represented as depicted in the six examples below:

Note: the max length will be enforced based on the max length configured for the control.

Name Control

The Name control values are represented as two simple types within a complex type assigning the control Tag name as the name of the complex type. Within the complex type are two attributes, Name and Title.

The Name control will follow this format:

The Name control values will be represented as depicted in the example below:

Location Control

The Location control values are represented as two simple types within a complex type assigning the control Tag name as the name of the complex type. Within the complex type are two attributes, Lat and Long. The Latitude and longitude values can be negative positive decimal values.

The Location control will follow this format:

The Location control values will be represented as depicted in the example below:

Address Control

The Address control values are represented as multiple simple types within a complex type assigning the control Tag name as the name of the complex type. Within the complex type there are multiple attributes, such as AddressLine1, AddressLine2, LocationDescription, State, Country, County, City and Zip. These attributes are string values.

It is worth noting that some properties such as LocationDescription, Country and County may not be available/needed for a given form, based on form configuration.

The Address control will follow this format:

The Address control values will be represented as depicted in the example below:

Contact Control

The Contact control values are represented as multiple simple types within a complex type assigning the control Tag name as the name of the complex type. Within the complex type there are multiple attributes, such as FirstName, LastName, Title, Email, AddressLine1, AddressLine2, LocationDescription, State, Country, County, City, Zip, Phones, Phone Number, etc. These attributes are string values and some (e.g., phone/email are valid for basic formatting compliance).

It is worth noting that some properties may not be available/needed for a given form, based on form configuration or existence of data.

Whether the form is configured for a single phone number or multiple phone numbers will impact the structure of the XML. For this reason, multiple examples were provided below.

Single Phone Number

The Contact control with support for a single phone number will follow this format:

The Contact control values with support for a single phone number will be represented as depicted in the example below:

Support for multiple phone numbers

The Contact control with support for a multiple phone numbers will follow this format:

The Contact control values with support for multiple phone numbers will be represented as depicted in the example below:

Advanced Table Control

The Advanced Table control values are represented as multiple control types (simple and/or complex) within a complex type. The Advanced Table Tag will be assigned to the Advanced Table control complex type. Each row within the table will be represented with the ITEM tag. Each individual control will be Tagged with the individual control Tag name as configured within the table. Note: The configuration of the individual controls will follow the same rules/patterns as discussed in prior topics within this document.

The Advanced Table control will follow this format:

The Advanced Table control values will be represented as depicted in the example below:

[Bulk Upload] Table Control

The Bulk Upload Table control value is represented as a single simple type. The value is stored as a tab delimited (for columns) and carriage return delimited (for rows) dataset

The Bulk Upload Table control will follow this format:

The Bulk Upload Table control values will be represented as depicted in the example below:

Testing/Validating XML Document

To effectively test your XML Document, it is recommend that the following tests be performed, in order of need:

1. Using an XML editor, visually inspect the XML file for structural errors/issues.

2. Using an XML editor, validate that the generated XML Document is a valid XML document (using an XML editor).

3. Using an XML editor, validate that the generated XML Document is valid against the expected XML Schema.

4. Attempt to upload the XML Document to the form in question within the nFORM system and ensure no errors are received.

Feature Activation

In order to provide the agency with the appropriate level of control of this feature, form designers will have the ability enable/disable the upload process on a form-by-form basis. This will ensure that the agency explicitly determines that this feature will be useful on the given form and that the agency considers the impacts of enabling the feature (e.g., future changes will need to be careful considered).

To enable the XML Upload feature, the form designer will select the XML Upload checkbox within the XML Upload section of the Pre-fill tab in the Form Designer.

Once activated, the following attributes can be specified:

• A Reference name for submission data will be specified. This attribute will define the short name that will be used to reference the XML document.

• Help Text can be entered to provide additional instructions to users who utilize this function. These instructions will be presented in the areas where the upload process is available.

• The form designer can force the import process to be used for all submission (if appropriate) by selecting the XML Upload is required to begin submission.

• The XML Schema which enforces the data structure and content for this form can be downloaded using the Download XML Schema button.

XML Schema Generation/Access

In order to support a dynamic (and sometimes evolving) XML structure for upload, the system provides a means for defining and testing the XML Document structure, based on the design of a given form. The XML Document structure will be defined and tested using an XML Schema Definition (XSD) for each form. An XML Schema is used to describe and validate the structure and the content of a XML Document, enforcing the structure of the data, data types (when possible), data max lengths, field requirements, etc.

A XML Schema defines the expected elements, attributes, and data types in a XML Document, in similar fashion to how a database schema describes the data elements in a database.

A form designer opens a form, enables the upload process, and publishes the updated form. In this publishing process, the system will automatically establish an XML Schema that represents the required XML Document structure representing the form and related controls. The XML Schema will be saved with the form and can be accessed by clicking on the Download XML Schema button within the XML Upload section of the Pre-fill tab in the Form Designer for any form where this feature is enabled.

Appendix

Appendix A – Example XML Document