Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Updated for Version 4.16

Overview

Larger organizations (e.g., the Department of Transportation) may submit the same form type (e.g., Construction Stormwater Permit, monthly reports, etc.) in a regular and repetitive manner. To help better support these situations and to minimize data entry requirements, a feature was established in the nFORM system which allows these organizations to upload data for a form via a form-compliant XML document.

...

Anchor
Introduction
Introduction
Introduction

The nFORM tool supports the ability for a reporting entity to upload structured data into a submission via a form-compliant XML document. This feature helps to automate form submissions for those organizations that have the need to submit many submissions of the same form type. These organizations will now have the ability to generate compliant XML documents, representing their submissions, and upload them into the nFORM system to minimize data entry requirements.

...

Please note that this is an advanced topic and most content is intended to be utilized by advanced technical staff which have knowledge of XML and XML Schemas.

Anchor
XMLDocumentUploadProcess
XMLDocumentUploadProcess
XML Document Upload Process

This section of the document describes how a user will upload and import data using XML.

Anchor
Prerequisites
Prerequisites
Prerequisites

Before data can be uploaded and imported into a form, some prerequisites must be met, including:

  • The XML Upload feature must be enabled for the form in question. See the Feature Enablement topic for additional details in this regard.

  • The data to be imported must be stored in a compliant XML document for the form in question. See the XML Document Development topic for additional details in this regard.

Anchor
XMLDocumentUploadProcessWalkthrough
XMLDocumentUploadProcessWalkthrough
XML Document Upload Process Walkthrough

Users will have the ability to upload and import data into a form via a form compliant XML Document that represents the submission and then to subsequently review the submission, address any issues and submit their submission to the agency.

...

If and when appropriate, the user will have the ability to initiate the signature (single or multiple) process for the submission.

Anchor
XMLDocumentDevelopment
XMLDocumentDevelopment
XML Document Development

Anchor
ProcessOverview
ProcessOverview
Process Overview

When a regulated entity wishes to establish an automated process to generate an XML Document that represents their submission, they will need to design/develop a custom process to generate repeatedly these XML Documents using their data system(s). When approved, the agency will provide users with the XML Schema (see the XML Schema Generation/Access section) that dictates the valid structure of the data and any supplemental documentation that may be helpful (e.g., sample submission files, how to documents, XML Schema, etc.).

...

Using the form’s XML Schema and any other supporting materials, the regulated entity will establish the ability to repeatedly generate compliant XML from their source system(s). See the XML Document Development section of this document for additional details on the expected structure of the XML document. Additionally, see the Testing/Validating XML Document section of this document for details on how to test the upload process.

Anchor
DevelopmentComponents
DevelopmentComponents
Development Components

In order to upload an XML Document to a form submission, the XML Document must be compliant with the XML XSD Schema for the form in question.

...

  1. XML Schema: The structure for which the XML Document must be compliant.

  2. XML Document: The elements of the XML Document.

  3. Testing/Validating XML Document: Recommendations for how to test a generated XML Document.

XML Schema

In order to upload an XML Document to a form submission, the XML Document must be compliant with the XML XSD Schema for the form in question.

...

The XML Schema will mimic the form structural elements and utilizes similar datatypes.

XML Document

This section provides details as to the expected elements and structure of the XML Document.

Example Submission XML Document

To ease the process of XML development, a sample XML Document can be generated from within the system. To generate this document for a given submission, enter the following URL in your browser window (while signed into the system as a user with edit access to the selected submission). To do this assemble the URL in the following format:

...

See Appendix A – Example XML Document section for a visual example of what the XML Document could look like.

XML Document Structure

Below is a summary of how elements are represented within the XML document.

Header

No specific header information is anticipated or expected.

Base Structure

The table below represents the expected structure of the payload within the document.

...

<?xml version=”1.0” encoding="utf-8"?>

<PAYLOAD xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="C:\Users\miscuser\Desktop\XMLImportDemo Schema.xsd">

<SUBMISSION>

<FormMetaData>

<FormName>XML Import Demo</FormName>

<FormTag>XMLImportDemo</FormTag>

<FormVersion>

<MajorVersion>1</MajorVersion>

<MinorVersion>0</MinorVersion>

</FormVersion>

</FormMetaData>

<SECTION1>

<CONTROL1>Control1Value</CONTROL1>

<CONTROL2>Control2Value</CONTROL2>

<CONTROL3>Control3Value</CONTROL3>

</SECTION1>

<SECTION2>

<CONTROL4>Control4Value</CONTROL4>

<CONTROL5>Control5Value</CONTROL5>

<CONTROL6>Control6Value</CONTROL6>

</SECTION2>

<SECTION3>

<CONTROL7>Control7Value</CONTROL7>

<CONTROL8>Control8Value</CONTROL8>

<CONTROL9>Control9Value</CONTROL9>

</SECTION3>

</SUBMISSION>

</PAYLOAD>

Form/Submission Sections and Controls

The structure of the form/submission sections and controls section will be dependent upon the structure and configuration of the form’s sections and controls.

...

The following sections describes how different types of dynamic attributes can be included/utilized within the XML document.

Sections

Two section types exist: 1) Discrete and Repeating. Discrete sections are sections that stand alone and are not repeated. Repeating sections are those where all the attributes within the section can be repeated, as appropriate.

Discrete Section

A discrete section will be represented as a complex type, utilizing the Section Tags as the name for the complex type.

...

<DISCRETESECTION>

<CONTROL1>ABC</CONTROL1>

<CONTROL2>XYZ</CONTROL2>

</DISCRETESECTION>

Repeating Section

A repeating section will be represented as multiple complex types, one representing the grouping of repeated elements and one representing each set of repeated elements within the section. The grouping of repeated elements complex type is identified as the Section Tag name plus “_REPEATER” and the complex type representing each set of repeated elements is identified as the Section Tag name.

...

<REPEATINGSECTION_REPEATER>

<REPEATINGSECTION>

<CONTROL1>ABC</CONTROL1>

<CONTROL2>XYZ</CONTROL2>

</REPEATINGSECTION>

<REPEATINGSECTION>

<CONTROL1>BCD</CONTROL1>

<CONTROL2>WXY</CONTROL2>

</REPEATINGSECTION>

<REPEATINGSECTION>

<CONTROL1>CDE</CONTROL1>

<CONTROL2>VWX</CONTROL2>

</REPEATINGSECTION>

</REPEATINGSECTION_REPEATER>

Controls

Simple Text Controls

Simple text controls are values stored in controls such as Short Text, Paragraph and SSN.

...

Note: the max length will be enforced based on the max length configured for the control.

Select Controls

Select controls are values stored in the Single and Multiple Select controls.

...

<FACILITYTYPE>

<Value>Incinerator</Value>

<Value>Storage Facility</Value>

</FACILITYTYPE>

Formatted Controls

Formatted controls are values stored in controls such as Number, Date, Time, Email, URL and Phone.

...

Note: the max length will be enforced based on the max length configured for the control.

Name Control

The Name control values are represented as two simple types within a complex type assigning the control Tag name as the name of the complex type. Within the complex type are two attributes, Name and Title.

...

<OWNER>

<Name>Michael Jordon</Name>

<Title>Flight Leader</Title>

</OWNER>

Location Control

The Location control values are represented as two simple types within a complex type assigning the control Tag name as the name of the complex type. Within the complex type are two attributes, Lat and Long. The Latitude and longitude values can be negative positive decimal values.

...

< LOCATIONCOORDINATES >

<Lat>34.740768385792386</Lat>

<Long>-92.31431908835937</Long>

</LOCATIONCOORDINATES>

Address Control

The Address control values are represented as multiple simple types within a complex type assigning the control Tag name as the name of the complex type. Within the complex type there are multiple attributes, such as AddressLine1, AddressLine2, LocationDescription, State, Country, County, City and Zip. These attributes are string values.

...

< SITEADDRESS >

<AddressLine1>111 SW 1st Ave</AddressLine1>

<AddressLine2>Suite 101</AddressLine2>

<LocationDescription>Milepoint 24</LocationDescription>

<State>OR</State>

<Country>United States</Country>

<County>Multnomah</County>

<City>Portland</City>

<Zip>97223</Zip>

</SITEADDRESS>

Contact Control

The Contact control values are represented as multiple simple types within a complex type assigning the control Tag name as the name of the complex type. Within the complex type there are multiple attributes, such as FirstName, LastName, Title, Email, AddressLine1, AddressLine2, LocationDescription, State, Country, County, City, Zip, Phones, Phone Number, etc. These attributes are string values and some (e.g., phone/email are valid for basic formatting compliance).

...

< SITECONTACT >

<Salutation>Dr.</Salutation >

<FirstName>Jim</FirstName>

<MiddleName>Merle</MiddleName>

<LastName>Johnson</LastName>

<Title>Project Manager</Title>

<Email>test@test.com</Email>

<AddressLine1>111 SW 1st Ave</AddressLine1>

<AddressLine2>Suite 101</AddressLine2>

<LocationDescription>Milepoint 24</LocationDescription>

<State>OR</State>

<Country>United States</Country>

<County>Multnomah</County>

<City>Portland</City>

<Zip>97223</Zip>

<Phones>

<BUSINESS>

<PhoneNumber>555-555-5555</PhoneNumber >

<PhoneExtension>555</PhoneExtension >

</BUSINESS>

<MAIN>

<PhoneNumber>444-444-4444</PhoneNumber >

<PhoneExtension>444</PhoneExtension >

</MAIN>

<HOME>

<PhoneNumber>222-222-2222</PhoneNumber >

</HOME>

<MOBILE>

<PhoneNumber>111-111-1111</PhoneNumber >

</MOBILE>

<OTHER>

<PhoneNumber>777-777-7777</PhoneNumber >

<PhoneExtension>777</PhoneExtension >

</OTHER>

</Phones >

</SITECONTACT>

Advanced Table Control

The Advanced Table control values are represented as multiple control types (simple and/or complex) within a complex type. The Advanced Table Tag will be assigned to the Advanced Table control complex type. Each row within the table will be represented with the ITEM tag. Each individual control will be Tagged with the individual control Tag name as configured within the table. Note: The configuration of the individual controls will follow the same rules/patterns as discussed in prior topics within this document.

...

<ANIMALS >

<ITEM>

<ANIMALTYPE>Frog</ANIMALTYPE>

<ANIMALCOUNT>43</ANIMALCOUNT>

<WHERELOCATED>

<VALUE>In Stream</VALUE>

</WHERELOCATED>

</ITEM>

<ITEM>

<ANIMALTYPE>Salmon</ANIMALTYPE>

<ANIMALCOUNT>3</ANIMALCOUNT>

<WHERELOCATED>

<VALUE>In Stream</VALUE>

</WHERELOCATED>

</ITEM>

<ITEM>

<ANIMALTYPE>Wolf</ANIMALTYPE>

<ANIMALCOUNT>1</ANIMALCOUNT>

<WHERELOCATED>

<VALUE>Plains</VALUE>

</WHERELOCATED>

</ITEM>

</ANIMALS>

[Bulk Upload] Table Control

The Bulk Upload Table control value is represented as a single simple type. The value is stored as a tab delimited (for columns) and carriage return delimited (for rows) dataset

...

< UPLOADTABLE> Column 1 Column 2 Column 3

A 1 9

B 2 8

C 3 7

D 4 6 </UPLOADTABLE>

Testing/Validating XML Document

To effectively test your XML Document, it is recommend that the following tests be performed, in order of need:

  1. Using an XML editor, visually inspect the XML file for structural errors/issues.

  2. Using an XML editor, validate that the generated XML Document is a valid XML document (using an XML editor).

  3. Using an XML editor, validate that the generated XML Document is valid against the expected XML Schema.

  4. Attempt to upload the XML Document to the form in question within the nFORM system and ensure no errors are received.

Anchor
FeatureActivation
FeatureActivation
Feature Activation

In order to provide the agency with the appropriate level of control of this feature, form designers will have the ability enable/disable the upload process on a form-by-form basis. This will ensure that the agency explicitly determines that this feature will be useful on the given form and that the agency considers the impacts of enabling the feature (e.g., future changes will need to be careful considered).

...

  • A Reference name for submission data will be specified. This attribute will define the short name that will be used to reference the XML document.

  • Help Text can be entered to provide additional instructions to users who utilize this function. These instructions will be presented in the areas where the upload process is available.

  • The form designer can force the import process to be used for all submission (if appropriate) by selecting the XML Upload is required to begin submission.

  • The XML Schema which enforces the data structure and content for this form can be downloaded using the Download XML Schema button.

Anchor
XMLSchemaGenerationAccess
XMLSchemaGenerationAccess
XML Schema Generation/Access

In order to support a dynamic (and sometimes evolving) XML structure for upload, the system provides a means for defining and testing the XML Document structure, based on the design of a given form. The XML Document structure will be defined and tested using an XML Schema Definition (XSD) for each form. An XML Schema is used to describe and validate the structure and the content of a XML Document, enforcing the structure of the data, data types (when possible), data max lengths, field requirements, etc.

...

A form designer opens a form, enables the upload process, and publishes the updated form. In this publishing process, the system will automatically establish an XML Schema that represents the required XML Document structure representing the form and related controls. The XML Schema will be saved with the form and can be accessed by clicking on the Download XML Schema button within the XML Upload section of the Pre-fill tab in the Form Designer for any form where this feature is enabled.

...

Anchor
Appendix
Appendix
Appendix

Anchor
AppendixExampleXMLDocument
AppendixExampleXMLDocument
Appendix A – Example XML Document

<?xml version="1.0" encoding="utf-8"?>

<PAYLOAD>

<SUBMISSION>

<FormMetaData>

<FormName>XML Import Demo (Do Not Delete)</FormName>

<FormTag>XMLImportDemo</FormTag>

<FormVersion>

<MajorVersion>1</MajorVersion>

<MinorVersion>1</MinorVersion>

</FormVersion>

</FormMetaData>

<SIMPLECONTROLSSECTION>

<REPEAT_NAME_SEC_1>Repeater Record 1</REPEAT_NAME_SEC_1>

<SHORT_TEXT_SEC_1>Short Text</SHORT_TEXT_SEC_1>

<PARAGRAPH_SEC_1>sample paragraph</PARAGRAPH_SEC_1>

<SINGLE_SELECT_SEC_1>

<Value>1- State of Hawaii</Value>

</SINGLE_SELECT_SEC_1>

<MULTIPLESELECT_SEC_1>

<Value>Mailed letter</Value>

<Value>Text Message</Value>

</MULTIPLESELECT_SEC_1>

<SSN_SEC_1>111111111</SSN_SEC_1>

</SIMPLECONTROLSSECTION>

<FORMATTEDCONTROLSSECTION>

<NUMBER_SEC_2>1.0000</NUMBER_SEC_2>

<DATE_SEC_2>2021-07-21</DATE_SEC_2>

<TIME_SEC_2>12:22:00</TIME_SEC_2>

<EMAIL_SEC_2>test@test.com</EMAIL_SEC_2>

<URL_SEC_2>http://cnn.com</URL_SEC_2>

<PHONE_SEC_2>555-555-5555</PHONE_SEC_2>

</FORMATTEDCONTROLSSECTION>

<ADVANCEDCONTROLSSECTION>

<NAME_SEC_3>

<Name>Sally</Name>

<Title>Wing</Title>

</NAME_SEC_3>

<LOCATION_SEC_3>

<Lat>34.740768385792386</Lat>

<Long>-92.31431908835937</Long>

</LOCATION_SEC_3>

<ADDRESS_SEC_3>

<AddressLine1>111 SW 1st St</AddressLine1>

<AddressLine2>address line 2 text</AddressLine2>

<LocationDescription>location description text</LocationDescription>

<State>OR</State>

<Country>United States</Country>

<County>Multnomah</County>

<City>Tigard</City>

<Zip>97223</Zip>

</ADDRESS_SEC_3>

<TABLE_SEC_3>Column 1 Column 2 Column 3

A 1 9

B 2 8

C 3 7

D 4 6</TABLE_SEC_3>

<CONTACT_SEC_3>

<Salutation>Dr.</Salutation>

<FirstName>Jim</FirstName>

<MiddleName>Merle</MiddleName>

<LastName>Johnson</LastName>

<Title>Project Manager</Title>

<Email>test@test.com</Email>

<AddressLine1>222 NW 2nd Ave</AddressLine1>

<AddressLine2>address line 2 text</AddressLine2>

<LocationDescription>location description text</LocationDescription>

<State>OR</State>

<Country>United States</Country>

<County>Multnomah</County>

<City>Tigard</City>

<Zip>97223</Zip>

<Phones>

<BUSINESS>

<PhoneNumber>222-222-2222</PhoneNumber>

<PhoneExtension>212</PhoneExtension>

</BUSINESS>

<MAIN>

<PhoneNumber>555-555-5555</PhoneNumber>

<PhoneExtension>111</PhoneExtension>

</MAIN>

<HOME>

<PhoneNumber>222-222-2222</PhoneNumber>

</HOME>

<MOBILE>

<PhoneNumber>333-333-3333</PhoneNumber>

</MOBILE>

<OTHER>

<PhoneNumber>444-444-4444</PhoneNumber>

<PhoneExtension>444</PhoneExtension>

</OTHER>

</Phones>

</CONTACT_SEC_3>

</ADVANCEDCONTROLSSECTION>

<ADVANCEDTABLESECTION>

<ADV_TABLE_SEC_4>

<ITEM>

<SHORT_AT>Jumpy</SHORT_AT>

<SINGLE_AT>

<Value>frog</Value>

</SINGLE_AT>

<NUMBER_AT>8</NUMBER_AT>

</ITEM>

<ITEM>

<SHORT_AT>High</SHORT_AT>

<SINGLE_AT>

<Value>bird</Value>

</SINGLE_AT>

<NUMBER_AT>4</NUMBER_AT>

</ITEM>

<ITEM>

<SHORT_AT>Loud</SHORT_AT>

<SINGLE_AT>

<Value>dog</Value>

</SINGLE_AT>

<NUMBER_AT>2</NUMBER_AT>

</ITEM>

</ADV_TABLE_SEC_4>

</ADVANCEDTABLESECTION>

<REPEATINGSECTION_REPEATER>

<REPEATINGSECTION>

<SHORTTEXTINREPEATER>Test text section 1</SHORTTEXTINREPEATER>

</REPEATINGSECTION>

<REPEATINGSECTION>

<SHORTTEXTINREPEATER>Test text section 2</SHORTTEXTINREPEATER>

</REPEATINGSECTION>

<REPEATINGSECTION>

<SHORTTEXTINREPEATER>Test text section 3</SHORTTEXTINREPEATER>

</REPEATINGSECTION>

</REPEATINGSECTION_REPEATER>

</SUBMISSION>

</PAYLOAD>

...