NEESGrid logo

Data Turbine: Design Patterns

Paul Hubbard
Argonne National Lab
hubbard@mcs.anl.gov

Chuck Severance and Terry Weymouth
University of Michigan
weymouth@umich.edu
csev@umich.edu

Revision History

1.0 May 4, 2004
1.1 May 6, 2004 - Clarify some NTCP details 1.2 May 6, 2004 - Clean up the notation section a bit

Introduction

This document is an attempt to write up the results of an informal design discussion between the authors that happened on 4/30/2004. The design and code are still evolving, but it seems worthwhile to document the current state of affairs and solicit feedback.

Design Pattern: NEES turbine component

A standard NEESgrid component that talks to the data turbine has several requirements:
  1. A source, for data and metadata
  2. A source, to send logging, debug and informational messages
  3. A sink, to receive commands
  4. A source, to send command responses
Given the naming conventions below and a standard design pattern, we can provide code to, for example, 'Iterate over all video sources and generate QCIF thumbnails at 1 frame per second.'

Pictorially, it looks like this:

main design pattern


Note the arrows - they denote source/sink.

One question pops up right away: "OK, what if the process starts listening for commands (its sink) but the corresponding source hasn't started yet?"
Interestingly, it is valid for a process to start up, and start subscribing to a source that doesn't exist yet. The process will block when it does a subscribe, and all will be well. This means that we don't have a forced ordering on startup, which is beneficial.

Examples (or instances) of the pattern include DAQ, video, still image cameras, and audio.

For end-user applications, we will have different designs depending on the goals. For example, a Chef server might want to connect to the DAQ, enumerate channels and metadata, and subscribe to several streaming data channels while also displaying high-resolution still images. A different application might want to display multiple video channels with audio. Given the broader nature of end applications, it seems less appropriate at this early stage to proscribe their patterns.

Naming Conventions

In the following the asterisk (*) stands for an arbitrary string of non-blank characters, intended to be a
short title or description (e.g. anl_camera1), which identifies the specific source. The list of name templates is:
<type>_<name>_<detail>
where
Example:
    video_baseCamera1_data_source/video.jpg
    video_baseCamera1_debug_log_soruce/log.txt
    video_baseCamera1_command_source/command.txt
    video_baseCamera1_command_sink

NTCP design pattern

Given the turbine design pattern, how can we integrate NTCP into this picture? For control systems using the current LabVIEW (ASCII) plugin, we can easily add the turbine as a transport mechanism.


Here's the current system, using the ASCII (LabVIEW) plugin:

Current design

If we want to use the turbine for transport instead of ASCII over TCP, we end up with this:

ntcp with turbine

The blue-green boxes are new code that handle the translations required.

Serdes NTCP plugin

The 'serdes plugin' is named from an EE device called a 'serializer/de-serializer'. In this case, the serdes plugin is a modified version of the existing 'LabVIEW plugin' that talks to the turbine. It simply takes the ASCII strings consumed and produced from the existing plugin, and transports them over the turbine using the proposed design pattern.

Transport Translator

The 'transport translator' has a similar job, but does not require an OGSA/NTCP container. All it does it take messages from one channel and put them on the other - commands to the control system, and replies from it. This should be less than a hundred lines of code.

For both, the contents on the turbine are the same ASCII strings currently used; we simply add an intermediate transport and a bit of overhead.

Benefits of this are mainly on the ends: the existing NTCP clients do not have to change, we still have full Grid security, and the control systems using the ASCII protocol are unaffected. Note that control systems using other plugins are not addressed.

Master Control Program

We are considering a 'master control program' to run on the NEES-POP to handle launch, starting and stopping of various components:
This is still in the discussion stage, so I don't have any details on it yet.

Metadata specification

We're working on a list of standard metadata to include the pattern. Metadata, in the turbine environment, is attached to a source. In our pattern, that will be the data source. For example, a DAQ system could include
This is also being discussed; if you have suggestions please send them to discuss@neesgrid.org. As a second example, video channels could include

Open questions

One interesting question is that of 'who knows what.' In other words, each component has specific things it needs to know to work: The transport translator needs to know the TCP/IP address of the turbine and control host, for example. By using the naming convention, some things can become automatic - the thumbnailer can connect to the turbine, and query it to learn video channels and thumbnail parameters for each. However, each component still has information it needs at startup, so we are thinking about the best way to store, retrieve and manage those.

Navigation links

  • Back to NEESGrid at Argonne
  • Back to home page
    •

    Support

    This work was supported primarily by the George E. Brown, Jr. Network for Earthquake Engineering Simulation (NEES) Program of the National Science Foundation under Award Number CMS-0117853.