Technical Report NEESgrid-2004-35
www.neesgrid.org
Draft Whitepaper Version:
1.1.3
Last modified: August 23, 2004
DAQ data format Final
Specification
Paul Hubbard1
1 Argonne National Laboratory
Feedback on this document should be directed to hubbard@mcs.anl.gov
1. Summary...................................................................................................................... 2
2. Revision History.......................................................................................................... 2
3. Introduction.................................................................................................................. 4
4. Terminology................................................................................................................. 4
5. Code Access, Mailing
List and Bugzilla Archive.......................................................... 4
6. Protocol Validation and
Testing................................................................................... 4
7. Rough Design Overview.............................................................................................. 6
8. Communications Overview........................................................................................... 6
9. Connections and Ports................................................................................................. 7
10. Timestamps
and Timezones..................................................................................... 7
11. The Role of
the Driver.............................................................................................. 7
12. Streaming
Data Channel Protocol and Format......................................................... 7
13. Data File
Format...................................................................................................... 8
14. Control
Channel Protocol and Format...................................................................... 9
15. Error
Handling and Syntax.................................................................................... 10
16. The Role of
the Server Daemon............................................................................. 11
17. The Role of
the Library Code................................................................................. 12
18. Post-experiment
FTP and Samba uploads.............................................................. 13
19. Testing,
Benchmarking and Profiling..................................................................... 13
20. Where to
Begin?.................................................................................................... 14
1. Summary
This document explains the interactions between the NSDS, driver, and DAQ for the purposes of debugging, replacing the driver or DAQ code, or customizing the DAQ code. In addition to describing in detail the various components that constitute this system, the document also explains protocols and formats associated with data and control channels and data files, as well as error handling and syntax, and provides examples for each. Tools for testing components that have been debugged or replaced are also discussed.
2. Revision History
This document provides a minor correction to 1.1, which in turn superseded NEESgrid Technical Report 2003-04, version 1.0. New information in 1.1, which was released 8/18/03, included:
- Optional open-ports and close-ports commands
- Defined syntax of error messages for open-port{s} and unknown commands
- Information about how to use the DNDTester, and where to get it
- Miscellaneous descriptive text
The 1.1.1 version, released 8/28/03, includes corrections to “Section 15: Error Handling and Syntax.”
The 1.1.2 version, released 5/3/04, documents the list-units verb, which had erroneously been left out, and a correction in the final section about CVS package naming.
The 1.1.3 version corresponds to the NEESGrid 3.0 software release. Many changes, corresponding to the deprecation of the NSDS and the addition of the Data Turbine. Name changed to reflect change from NSDS to RBNB.
3. Introduction
This document is for people who want to understand the details of the NEESGrid specification for data acquisition (DAQ) and streaming. It covers the design, the DAQ protocol, and other details sufficient to rewrite sections for local sites.
It is assumed that the reader is familiar with the NEESgrid project and its goals. If not, information can be found at http://www.neesgrid.org/.
In releases previous to 3.0. NEESGrid used the NEESgrid Streaming Data Server, or NSDS. This has now been replaced with the Creare Data Turbine, and the NSDS has been deprecated. However, since the DAQ interface is defined by a protocol, we have been able to preserve backward compatibility with existing systems.
4. Terminology
The Data Turbine, sometimes referred to as ‘RBNB’ for Ring-Buffered Network Bus, is a commercial Java-based program that handles data streaming, archiving, synchronization, plugins, and many other features. It’s free for non-commercial use, and we’ve negotiated and OEM redistribution license and some enhancements as well. It’s a replacement for the NSDS in v3.0.
The NEESgrid reference platform is LabVIEW from National Instruments. We provide example code in CVS that implements the functionality in this document.
In previous versions, ‘driver’ meant the NSDS driver. This code is now obsolete and no longer useful with the RBNB. For this document, driver means its replacement, a Java program called DaqToRbnb.
5. Code Access, Mailing List and Bugzilla Archive
CVS access to the NEESgrid code archive is documented at http://www.mcs.anl.gov/neesgrid/cvs.html. A Bugzilla archive for logging and tracking bugs and feature requests is available at http://bugzilla.ncsa.uiuc.edu/neesgrid. There is also a mailing list; instructions can be found at http://www.mcs.anl.gov/neesgrid/
More information about our use of the Data Turbine can be found at http://www.mcs.anl.gov/neesgrid/turbine/ and http://www.mcs.anl.gov/neesgrid/dtpatterns/
6. Protocol Validation and Testing
As of July 2003, the DNDTester application, found at http://www.mcs.anl.gov/neesgrid/dndtester, will test and validate any DAQ implementation for correctness and completeness. Its use is strongly recommended.
7. Rough Design Overview
Figure 1: System diagram.
Depending how your site is configured, you may need to rewrite the driver, DAQ code, server daemon or library code. The rest of the document should clarify which portions need reworking.
8. Communications Overview
The main goals of this system were simplicity, portability and transparency. From that, the following choices were made:
- All data (numeric) and commands will be sent in ASCII
- All numeric data should use exponential notation to preserve all significant digits.
- We will assume that the network between the DAQ and driver is secure
- We will use two TCP ports per logical connection, in the interests of keeping data and control distinct.
- We will add a driver, so that sites not using LabVIEW can use their DAQ system with minimal effort.
- DAQ should not know or care if the network is present, missing or unreliable. Data comes first; we stream it out if possible but strive not to interfere with DAQ.
More design information can be found in the NEESgrid System Architecture white paper.
9. Connections and Ports
The communications between the driver and DAQ utilize two TCP ports. Currently, the RBNB listens on 3333 and the DAQ listens on 55055/55056. However, these are changeable – the driver via the command line, and in the example LabVIEW code, it’s a front panel control in the ‘Server Daemon’ program.
10. Timestamps and Timezones
It is of note that this code uses ISO 8601 timestamps for all time markers. We use the UTC encoding, with fractional seconds for subsecond-sampled data. See nsds-util.c for C code to generate these; there is LabVIEW code in the subroutines library as well.
For reasons best described as baroque, LabVIEW cannot generate ISO8601 timezone abbreviations or determine UTC as of version 6.1. Because of this, the driver has functionality to convert from local time to UTC. By default, the driver assumes that it and the DAQ are in the same timezone, and uses that UTC conversion. You can change this via the –o argument.
11. The Role of the Driver
The driver serves to bridge the gap between the ASCII DAQ protocol and Java API used by the RBNB. It opens the DAQ connections, lists the available channels, and the subscribes to them all. Data is streamed into the RBNB until an error occurs. Since the DAQ protocol from versions 1 and 2 of the software was sufficiently flexible, we simply wrote Java code that maintained compatibility and talked to the RBNB instead of the NSDS. As noted above, it also handles timezone conversions, subscriptions and ASCII->native data conversion.
We are working on adding more features to the driver and downstream programs; e.g. units in the plot program, so expect that the driver code will continue to evolve.
12. Streaming Data Channel Protocol and Format
This is as simple as possible. The format is columnar ASCII, one line per timestamp, with a new line to terminate the record. Each channel is appended as
Channel name TAB value
For example:
2004-08-23T14:44:34.23600 ATL1 1.1919982731E-1 ATT1 3.3599853516E-2
is a two-channel datum.
That’s all there is to it, really. If your data is not simply converted to ASCII (e.g., image data), you will need to work out a data format with the NEESgrid data repository team at NCSA.
Note that we repeatedly send the channel names because the contents of the data stream will change over time as channels are subscribed or released.
13. Data File Format
The data file has a very similar format, with a couple of changes. It’s also tab-delimited ASCII, one line per timestamp, but there’s a ‘metadata header’ as defined by the metadata folks from NCSA, and there’s no need to replicate the channel names, since the format is fixed. Here’s an example to illustrate the format:
Active channels: ATL1,ATT1,ATL3,ATT3
Sample rate: 200.000000
Channel units: g,g,in,kip
Time ATL1 ATT1 ATL3 ATT3
2002-11-13T15:48:55.26499 2.71828E0 3.1415E0 0.0000E0 6.6600E2
2002-11-13T15:48:55.41499 8.67531E6 6.0200E23 1.0000E0 4.2000E1
Note the addition of a column header line that serves as a key to the data, and the metadata header, defining what channels are present and their engineering units.
14. Control Channel Protocol and Format
Similarly, the control channel is ASCII, one command or response per line, and newline delimited. All command are synchronous, in that the command is completed before a response is returned.
Command listing:
|
daq-status |
Error, Offline, Unknown, Running, or Stopped |
|
list-channels |
Active channel listing, comma delimited. |
|
list-units |
Active channel unit list, comma delimited |
|
open-port{channel
name} |
See below for exact syntax |
|
close-port{channel
name} |
Stop subscription to given channel; see below |
|
open-ports
{comma-separated list} |
Optional – open N ports at once |
|
close-ports
{comma-separated list} |
Optional – close N ports at once |
|
daq-stop |
Unimplemented command |
|
daq-start |
Ditto |
Sample exchange:
Driver->DAQ:
daq-status
DAQ->driver:
Running
list-channels
DAQ->driver:
ATL1,ATT1,ATV1,ATV2,Temp,RH
Driver->DAQ:
list-units
DAQ->driver:
mm,mm,mm,mm,C,percent
Driver-DAQ:
open-port ATT1
DAQ->driver:
Streaming data on data
channel from port ATT1
(The data for sensor ATT1 will be added to the data stream on the data channel, usually by the next data point)
Driver-DAQ:
open-ports ATT1,ATT3
DAQ->driver:
Streaming data on
data channel from port ATT1,ATT3
Note that this is an optional command, not yet present in the reference implementations.
Driver->-DAQ:
close-port ATT1
DAQ->driver:
Stopping data on
data channel from port ATT1
Driver->-DAQ:
close-ports
ATT1,ATT3
DAQ->driver:
Stopping data on
data channel from port ATT1,ATT3
Note that this is an optional command, not yet present in the reference implementations.
15. Error Handling and Syntax
As of the release of the DNDTester protocol validation tool, we have added error handling and clarified to remove ambiguities.
In the event of an unrecognized command, the DAQ should respond back with the following:
Unknown command ‘%s’
Where %s is the receieved command, e.g.
NSDSàdriver->DAQ:
orken-port Temp
DAQ->driver:
Unknown command
‘orken-port Temp’
Code for this can be found in fake_daq, Dataq-194, and the ADXL202 streamer, as well
as the server daemon.
In the event of an unknown or invalid channel ID on an open/close port{s} command, the
DAQ should return the following string:
Invalid port ‘%s’
Where %s is the string sent from the NSDS, e.g.
Driver->DAQ:
open-port Borked
DAQ->driver:
Invalid port ‘Borked’
16. The Role of the Server Daemon
This is a standalone LabVIEW program that is running in the background. Its job is to set up the TCP connections, handle the control channel, and set global variables for the library code. For example, when a subscribe (‘open-port’) request comes in, the daemon adds the channel name to a global list of subscribed channels. The next time the DAQ sends data out, the library code will note the new subscription and add it to the outgoing datum.
Note that the server daemon and/or TCP connections are intentionally independent of the DAQ code. The global variable to mediate may seem peculiar, but it allows the daemon and DAQ to decouple nicely. DAQ can run whether or not the server is up, or if the network fails.
Sites replacing the server daemon should have little problem re-writing it in C/C++, Java or Perl – it’s quite simple.
17. The Role of the Library Code
The library code has supporting routines for the server daemon and the DAQ code. There are subroutines for converting data into ASCII, saving to disk, and streaming via TCP. Most of this is not of interest unless you want to see how these are done with LabVIEW.
Two routines are of special interest ‘Data array to NSDS stream’ and ‘Data array to datafile stream.’ They perform similar function but have one key difference: Both take a vector of real numbers and a channel listing, but differ in their resulting ASCII. The NSDS version looks up each channel in the global list of subscribed channels, and only adds the sensor if it is needed. The datafile version always saves every sensor; the data file always contains all of the data captured.
18. Post-experiment FTP and Samba uploads
There are two methods of remote data access, streaming and batch. So far, we’ve looked at the mechanisms for delivering streaming data. However, if you look at the DAQ examples, you will find the code to do post-experiment data uploads. We use the National Instruments Internet Toolkit, specifically the FTP transfer routines. Post-experiment, we FTP the data file to the NEES-POP or other specified destination. Once that’s complete, we write a semaphore file named {datafile}.written. This triggers its transfer into the metadata handling system. The repository upload is handled by the NFMS upload agent.
19. Testing, Benchmarking and Profiling
Once you have assimilated the design and replaced portions of it, you will need to test your components. There are several programs in the archive to assist with this.
a) fake_daq.c A simplest-case data source, useful for testing the current driver and NSDS. Useful as an end-to-end test, and possibly for benchmarking. You can set the sample rate from the command line, and the number of simulated channels is a compile-time constant. Also useful for C/C++ programmers as a codebase when interfacing your DAQ to the NEESGrid.
b) NSDS Simulator.vi LabVIEW code to emulate a normal set of operations – query DAQ status, list channels, subscribe to a requested channel, and plot the streaming data as it arrives. A good test of end-to-end functionality, it also has the side benefit of plotting the data, which is often very telling.
c) NSDS Stress Tester.vi LabVIEW code that is created for benchmarking and stress testing. It subscribes to all listed channels, which is useful.
d) Stress test Fake DAQ.vi Generates as many channels of simulated data as you request.
e) DNDTester will exercise any DAQ implementation and test its protocol correctness and compliance.
f) RBNB Admin is a good way to view what channels are present in the Turbine, show listeners, and so forth.
g) RBNB Plot can also show live server statistics
Note that the Server Daemon and NSDS simulator programs display the commands and responses on their front panels; this is useful for checking and response command syntax.
Unfortunately, the Chef portion of the system does not yet have decomposition and testing tools made available. Until those are present, I recommend you use the above list to test your new components.
Profiling can take several forms: network utilization, CPU usage, sampling rate, etc. Different tools are appropriate for each; here are some I’ve found useful so far.
a) LabVIEW has an excellent profiler (Menu is Tools/Advanced/Profile VIs) that is quite useful in locating bottlenecks.
b) If you have Windows 2k, NT or XP, the Task Manager is useful for checking CPU, memory and network utilization.
c) On the NEES-POP, standard Unix tools such as top and lsof are invaluable for monitoring the driver and NSDS.
20. Where to Begin?
Having read this far, you are probably a bit adrift in new acronyms and design flotsam. Check out the fake_daq package from CVS, and browse through fake_daq.c. It implements much of what your system will need, and is extensively commented. This code is documented in the html subdirectory, with auto-generated Doxygen pages.
Note that you will also need to check out and compile the ‘flog’ message library and ‘nsds-driver’ before you can compile fake_daq. (nsds-driver has utility routines used by fake_daq)
Best of luck!