SDD proposal: Free-form text data elements

TDWG working group: Structure of Descriptive Data (SDD)

Introduction

Occasionally it has been argued that free-form text is unnecessary for the new SDD standard. Although free-form text may be problematic or irrelevant in many analysis situations (including phylogenetic analysis and interactive identifications) it is a valuable feature in many other situations. This document attempts to provide arguments why free-form text should be supported and analyzes which free-form data elements are required. For a quick look at the results of the analysis please go to "Proposal" at the end.

Note: This discussion is only concerned with free-form text that may be present in the object descriptions. Many additional label and wording elements are available in the section concerned with the definition of terminology.

Free-form text in descriptions

Traditionally, all organism descriptions in biology have been prepared as free-form text. Most authors have realized the importance of consistent and well defined descriptive terminology and have strived to create consistent and complete descriptions in free-form text. This is, however, difficult to achieve. Unsurprisingly, attempts to convert existing descriptions into structured databases usually reveal many shortcomings of current biodiversity descriptions.

Numerical taxonomy, including distance clustering and phylogenetic methods (e. g. parsimony analysis) were important attempts to improve objectivity and consistency of biodiversity descriptions. The great achievement of the DELTA format was to combine categorical information (nominal and ordinal scale) and numerical information (cardinal and interval scale) with free-format text elements. This made DELTA usable for general purpose descriptions as well as for identification and analysis applications.

Some free-form text elements present in DELTA should be reconsidered in the new SDD standard. To improve interactive identification, data analysis, and translation into multiple languages elements like frequency modifiers or comments expressing certainty should be moved from free-form text to structured data. However, the general arguments for the necessity of free-form text remain valid and such elements should be included in the new standard.

Textual data elements in descriptions:

• Free-form text as the only content of a character ("text characters", DELTA character type "TE")
• Free-form text as a user-definable extension to defined categories in a categorical character (Example: shape: 'round', 'oval', 'cylindrical' as defined categories, 'other [please specify]' as extension mechanism during data entry.)
• Free-form text as annotations relating to a specific character state observation in an object (reportable and internal notes)
• Text as lists or collections of text elements (e. g. geographical names, host taxa, person names)

Each of these textual data elements requires a separate discussion.

Free-form text as the only content of a character

The need for free-form text characters arises in the following situations:

• Free-form text is currently considered the only practical way to express some information.
  • The nature of the descriptive information is such that categorical or quantitative characters seem fundamentally inappropriate. This is, e. g., the case if complex arrangements are difficult to atomize and no terminology exists yet. The introduction of ad-hoc terminology is sometimes undesirable, especially if it would apply to very few taxa. An example could be the description of the relative positions of the inner organs of an animal.
  • The terminology is not yet fully defined in the application and perhaps particularly difficult to define. Such text characters may or may not later be converted into categorical, terminology-based characters.
  • Current limitations of the exchange format or the application used offer no better alternative. In DELTA it was usually impractical to code host-plants or geographical areas as categorical characters, so they are often treated as lists in free-form text characters. Other examples are molecular sequence or pattern data in the absence or appropriate data types.
• Existing data are migrated into a fully structured, terminology-based format and free-form text is needed to store parts that could not yet be migrated.
  • For discussion: compare the "NaturalLanguageDescription" element defined in the SDD working group session in March 2002. This element contains full natural language descriptions and allows partial markup. This element is intended to store the original and unchanged free-form text, but incrementally add markup. Migration is a different strategy since it will usually change the text and does not allow one to analyze the parts that are not yet migrated. Question: Can or should these these different strategies be combined?

Although only in relatively few situations free-form text characters are ultimately the best choice, the presence of free-form text elements greatly enhances the flexibility of applications. The acceptance of a standard requiring a rigorous and laborious definition of terminology by any content author is greatly enhanced if free-form text characters (and also user-definable state extension, see below) allow the gradual introduction of a strict terminology, rather than forcing the content author to solve terminological problems before starting to enter data.

The text in a text-character may be:

• language-neutral (requiring no translation) or
• language-specific (requires translation into each language), and
• plain unformatted text or
• basic formatted text (italic, sub-/superscript, but not paragraph or block level formatting)

More information about the proposal to introduce basic formatting markup into the SDD standard interchange format is available in the document "Formatted free-form text".

Free-form text as user-definable extensions to the defined categories

It is often difficult to capture all categories of a property (e. g. "shape") in the terminology before starting to work on the descriptions, because state categories may be rare ("hamate shape") or applicable only to few organisms. Many existing DELTA-based descriptive data applications (e. g. LucID, Pankey/Pandora, the CSIRO DELTA programs, or DeltaAccess) allow the dynamic addition of character states while working on object descriptions. This enables the content and terminology author to identify a problem, refactor the terminology, and continue data entry work.

This approach, however, will often be impossible in collaborative working situations. A document-based system requires that only a single person is working on a project at the time of state-reorganization. A database-based system may allow reorganizations of the terminology as long as all users work on the same central database. Still, if distributed copies or database replication are used, changing the terminology becomes increasingly difficult. Furthermore, in large collaborative projects, it may be desirable to centralize the responsibility for changing the terminology, or changes must be agreed upon after a discussion process.

Consequently, a free-form text "category" as a catch-all remainder is an important feature. Such a category ("Other") is frequently encountered in sociological or market research questionnaires and serves to supply rare or unexpected information:
   Profession:
   Farmer
   Teacher
   ...
   Nuclear Physicist
   Other (please specify): [___________________]

Such a feature can act as an important feed-back mechanism about categories that may have been overlooked during the definition of the terminology. Information present in the "Other" free-form text category can be analyzed and migrated to new categories if the terminology is expanded.

This mixture is inherently (and not well documented) present in the DELTA format. Any categorical (UM/OM) or numerical (IN/RN) character always can be used as a text character. If character 2 is TE (text) and character 3 is UM (nominal categorical), the following can be coded:

Free-form text as annotation of a object specific character state observation

In this case the basic information is already captured in a structured format and only additions or annotations are added as free-form text.

When creating natural language reports, the "wording" or "phrasing" defined in the general terminology and the state-specific annotations within the description are combined to produce the report. An important issue is, how closely these two are expected to work together.

The BioLink application proposes free-form text elements that can be inserted before or after the state code (as "textBefore" and "textAfter" attributes)1. These elements have no semantic definition, other than that they work hand-in-hand together with the wording defined in the terminology for characters and states. This produces several problems. Firstly, it makes the creation of multilingual data sets very difficult, because data can not be published without translating every single piece of annotation. This is not the case if the annotations are output in a separate place, where it would be acceptable to have a readable Spanish description, with occasional English annotations in brackets. Secondly, and most importantly, the task of expanding and improving the character terminology is made almost impossible. Any change in the labeling or wording of characters and states requires a reassessment and revision of every object description using this information in combination with a "textBefore"/"textAfter" annotation.

This is not the case if annotations and terminology are decoupled. Annotations should be considered independent data items that have to be understandable without the immediate context of the wording choosen in the terminology. "textBefore"/ "textAfter" elements should not be supported for this reason.

¹ The BioLink document "Enhanced Item Descriptions" (publ. 2001) states that "the restrictions on the placement of comments imposed by DELTA will be relaxed so that character values and comments can be intermixed as required". However, the preferred model is currently only slightly relaxed, using fixed "textBefore" and "textAfter" attributes (S. Shattuck, pers. comm. 2002).

No need for language-unspecific state annotations could be identified. It is therefore proposed that state annotations are always language specific.

Furthermore, content authors frequently request basic text formatting in annotations. It is proposed that state annotations provide limited text formatting, see the separate document "Basic text formatting". It may be desirable to allow terminology designers to limit this ability in user interfaces. This could be defined on the project, character, or state level. Currently such an option is not proposed by SDD.

In contrast to DELTA and especially the DELTA-2 proposal, the following text features should be rejected:

• Reportable comments anywhere in the document.
  This does not exclude xml-comments ("<!--comment-->") from being used in the document. However, the semantics of such comments are not part of the standard and no interoperability between applications is supported. Any SDD conformant application reading descriptive documents is free to discard xml-comments when writing an updated version of the document. In general, the use of xml-comments should be reserved for application internal (esp. debugging) purposes.
• Management of textual enumerations ("List characters") like taxon names, taxon authors, host plant names, specimen lists, references, etc. within an SDD document. Such lists should be obtained from separately managed external data providers (see next chapter Text as list elements).
  List information may be included in the SDD document as cached data, provided an update from the external source is later possible again (i. e. sufficient information about providers access and list item ID must be passed on).

Text as list elements

This topic has not yet been sufficiently explored in the discussions. It strongly depends on the availability to validate or pick list information from external data sources, e. g. lists of host names, pathogens, etc. Please do raise the subject in the discussions!

Structural comparison of text elements in DELTA and SDD

Text elements in DELTA / DELTA-2 descriptions:

Note that an explicit free-form text category for "Other" is not available. In categorical characters it can be simulated through an "other" category, but this would be treated as analytically meaningful by DELTA applications (e. g. user could be asked in keys: "is it 'other' or not?").

Text elements in SDD descriptions:

Any element (perhaps with the exception of coding status (= "missing data indicators", = "special states") like "unknown", "not applicable", etc.; this needs to be discussed!) may have an optional reported note. Any element (including coding status values) may have an optional internal note. "-" indicates that the column is not applicable to the state.

Proposal

1. A single "text state" containing only free-form text (but having no other analytical properties) can be added to the definition of any categorical or numerical character.
2. The structure of "text states" should be analogous to coding status values (expressing that a character is inapplicable or unknown).
3. A "text state" may have a label and wording defined in the terminology section (for all descriptions together).
   • This is desirable if text is combined with categorical or numerical data. The label (for reporting or editing forms) for this data item should be definable in the terminology. A label could simply be "Other, please specify:".
   • By default the wording is empty and prevents natural language output
   • The label always has a default (e. g. "Text:") and cannot be empty. An editing application can however, ignore the standard label if no other states are defined in the character.
4. A "text character" is a character containing only a text state (but no categorical, numerical, or statistical states). Special states expressing "unknown", "not applicable", etc. may optionally be present.
5. Each categorical or statistical state used in an object description may have an annotation that is publicly visible (ReportedNote).
6. Each state used in an object description may have an an additional free-form annotation that is visible only to the authors and editors of the data set (InternalNote).
7. Text and annotations may contain limited xhtml markup as specified in "Formatted free-form text".

The following constraints are considered desirable to simplify the structural model:

1. Two or more text states per character are not supported.
2. A text state provides no annotation (ReportedNote) in addition to the text. The annotation and the text are identical; all free-form reportable text in object description states resides in the same data element.
3. Annotations applicable to an entire character within a description are not supported. All object description annotations occur on the state level. If a reported or internal note is required for the entire character rather than for specific states, a "text state" would have to be added to the character (to avoid a possible misunderstanding: this does not refer to "character notes" defined in the terminology!).
4. Special states may or may not support annotations, this needs further discussion!

This proposal is designed to handle free-form text as a) single child of a character, b) sister or other child elements of a character, and c) child element of states with a minimum of structural complexity.

Request for discussion

Please send your criticism or suggestions to the SDD mailing list or to the author.

Gregor Hagedorn; Vers. 2; 10. March 2003
Earlier versions: Version 1