NEESit logo

Background and Intro

As of release 3.0, the NEESGrid code internals changed significantly with the introduction of the Creare Data Turbine code to replace the NSDS.  The data turbine is a flexible network ring buffer, useful to streaming, synchronization, aggregation, scalability and other data tasks.

As we've learned how to utilize the turbine, we've written a set of applications that utilize it. This page documents what they are and how to run them at your site.

Revision History

  • V1.0 August 24 2004 Initial posting.
  • V1.1 September 23 2004 - Notes on DataVideoGather
  • V1.2 November 19, 2004 - Small edits for post to it.nees.org
  • V1.3 February 16, 2005 - New programs, edits for move to sdsc.edu, lots of changes.
  • V1.4 February 17, 2005 - Continuing to add new code
  • V1.5 February 23, 2005 - Corrected classpath
  • V1.6 March 29, 2005 - Added RandomWalk, PictureSource, JpgSaverSource, other edits
  • V1.7 March 30, 2005 - Added NumberSubSample, spellcheck, script section, DVG reformat
  • V1.7.1 April 4, 2005 - Corrected DAQ protocol URL
  • V1.7.2 May 10, 2005 - Added new params to DaqToRbnb
  • V1.7.3 July 27, 2005 - Added version flag to common arguments
    •

    Changes and Implications: NSDS to Turbine

    The Data Turbine is considerably more flexible and capable than the NSDS it replaces, meaning that its well worth reading the documentation included in the distribution. Things such as plugins, subscribe vs monitor, etc require study to use and understand.

    On the minus side, we currently require that desktop users download the Creare distribution to use the client applications; we're still working on embedding the video or plot programs in a Java web applet. We do have a JNLP (Java WebStart) wrapper written that works pretty well, and Oregon State has written a web applet, so there's progress on this front.

    Compiling versus Running the code

    Depending on your needs, you may not need to compile the code; we certainly hope that most people will simply need to run the programs as distributed.

    Environment variables

    As distributed in the 3.0 NEES-POP tarball, the compiled java applications are installed into 
     /usr/local/nees/opt/rbnb-nees/turbine.jar
    and the data turbine goes into 
     /usr/local/nees/opt/rbnb/

    To run the code, you need to have a few environment variables set correctly. I'll show you how to do so for C shell; you Bash or Zsh fans can modify to suit.

    Here's the section of my .cshrc from a NEES-POP; 

    setenv NEES /usr/local/nees

    source ${NEES}/etc/csh.cshrc
    setenv OGSA_ROOT $NEES/opt/grid
    setenv JMFHOME /usr/local/JMF-2.1.1e
    setenv CLASSPATH $NEES/opt/rbnb-nees/commons-cli-1.0.jar:$NEES/opt/rbnb-nees/turbine.jar:$RBNB_HOME/lib/rbnb.jar:${JMFHOME}/lib/jmf.jar:.
    To be honest, I doubt this is optimal/minimal; please email me if you have a better configuration.

    The trailing dot, meaning 'current working directory', is to get the log4j.properties file that some programs use. I don't like the fact that classpath is being used to read a properties file, but such is the state of things.

    If you want to use code from CVS, simply change the CLASSPATH to point to that instead of $NEES. For example:

    setenv CLASSPATH $HOME/code/turbine/build/classes:$RBNB_HOME/bin/rbnb.jar:$JMFHOME/lib/jmf.jar:.

    Test: running an application

    Now, you should be able to just run the fully-named application, and Java will find it via the classpath. To test this, run any program with the -h (help) argument. Let's try DaqToRbnb:
    [hubbard@cee-rszs ~]$ java org.nees.rbnb.DaqToRbnb -h                           
    Default time: Time Zone offset: 6.0
    Default time: Daylight Savings Time offset (in hours): -1.0
    Set new parameter (in DaqToRbnb) name = ServerHostName
    Set new parameter (in DaqToRbnb) name = ServerPortNumber
    Set new parameter (in DaqToRbnb) name = DAQServer
    Set new parameter (in DaqToRbnb) name = DAQControlPort
    Set new parameter (in DaqToRbnb) name = DAQDataPort
    Set new parameter (in DaqToRbnb) name = SourceName
    Set new parameter (in DaqToRbnb) name = CacheSize
    DaqToRbnb: usage is...
    DaqToRbnb 
    [-s Server Hostname *localhost] 
    [-p Server Port Number *3333] 
    [-q DAQ Server *localhost] 
    [-c DAQ Control Port *55055] 
    [-d DAQ Data Port *55056] 
    [-n RBNB Source Name *FromDAQ] 
    [-z cache size *100]
    [-o time offset, floating point, hours to GMT *5.0]
    [-D flag (t/f) to debug print data time stamps *f]
    It worked!

    Compiling from CVS

    Please see this page for instruction on downloading and compiling the code from CVS.

    Note that the default Ant target post-3.0 is to generate a jarfile, so you can run 

     ant deploy
    to copy turbine.jar into $OGSA_ROOT/lib. There's also an undeploy target to remove it. The section above on environment variables explains the changes you'll need to make in the CLASSPATH.

    Compiling from the NEES-POP distribution

    While similar to compiling from CVS, this is complicated by the fact that the source and build directories are owned by root and therefore not writable by other users. You'll need to copy the source code from 
     /usr/local/src/neesgrid-{version}/components/rbnb-nees/
    to your home directory or other writable location, and work from there. Modify the CLASSPATH to point to your new location so Java will run your code and not the NEES versions.

    Programs, most common first

    Common arguments

    All of the turbine programs have a few arguments in common; I'll put them here rather than repeat them for every program.

    Important note on command-line help    : All of apps have a -h flag, as shown here, to print their arguments out. Concise help, very useful, and I highly recommend its use to you.

    Common arguments are

  • -s {hostname} IP or hostname of the machine running the Data Turbine
  • -p {port} Port number of the data turbine
  • -h Command help; prints arguments and default values for each.
  • -v Version. Prints version, source file name and build time.
  • -z Cache size How many items to keep in memory
  • -Z archive size How many items to keep on disk

    • Note that cache and archive look the same to the clients - it's all data in the turbine. Cache is faster, but archive can be very large if necessary. When testing, I tend to keep lots of DAQ data in memory, about 10,000 points. Video, however, can chew up memory much faster, so be careful in asking for the moon.

    org.nees.rbnb.DaqToRbnb

    This program is the NSDS driver replacement. It speaks DAQ protocol to the DAQ, converts the timestamps from ISO8601 into Java times, subscribes to all data channels, and stuffs the data into the Turbine.

    Arguments are:

  • -q {hostname} IP or hostname of the DAQ system
  • -c {port} Port on DAQ for control channel
  • -d {port} Port on DAQ for data channel
  • -n {Name} RBNB source name. This will show up in the Admin and plot tools, so it's worth naming your DAQs by function or location, e.g. 'DaqFloor3NW' or 'Shake table DAQ 1'
  • -o {Float} Time offset. Due to LabVIEW limitations, the DAQ timestamps do not include timezone information. DaqToRbnb knows how to convert from local to UTC time; by default it uses the local timezone. You only need to change this if the DAQ and turbine are in different timezones and you are using the LabVIEW DAQ distribution. Note: If you are running fake_daq, use -o 0.0. Fake daq generates timestamps already corrected to UTC, and thus requires no offset.
  • -D {0/1} Debug flag. -D 1 turns on a raft of printouts relating to timestamp conversion. Not generally useful.

    • More options related to buffers and frames

    For release 3.3, we've added code to buffer data into 'frames' before sending it. Previously, we'd send each datum seperately, which was a bottleneck. You can now buffer data either by number of samples, or by time. In other words, you can say 'Send every 100 samples' or 'send every 500 milliseconds'.
  • -K {No args} Use count mode (default is time)
  • -T {No args} Use time mode (default)
  • -k {integer} In count mode, how many samples to buffer into a frame
  • -t {integer} Time, in millseconds, to buffer before sending a frame to the turbine. This is a speed optimization, in general the turbine likes frames to arrive at 50Hz or less.
    •

    org.nees.rbnb.AxisSource

    This generates a video feed from an Axis camera or source, e.g. 2401, 2420, 2400+, etc. By default, it feeds video at 704x480, 30 frames per second.

    Arguments are:

  • -A {hostname} IP or hostname of the Axis unit
  • -S {string} Source name. This is how they are named and retrieved, so as with the DAQ its useful to give them descriptive names. For example, 'NW2ndFloor'.
  • -C {string} Channel name. Default is video.jpg, and I'd advise leaving this at the default.
  • -U username. These are only required if you've configured your Axis to require passwords.
  • -P password
  • -H This selects high resolution (new default), 704x480
  • -L This selects low resolution, 352x240
  • -f {integer} This selects frames per second to capture, default is 30

    • org.nees.rbnb.FlexTpsSource

    Similar to AxisSource, this takes a FlexTPS video stream and puts it into the data turbine. The arguments are slightly different.

    Arguments are:

  • -A {hostname} Hostname of TPS source
  • -S {string} Source name. This is how they are named and retrieved, so as with the DAQ its useful to give them descriptive names. For example, 'NW2ndFloor'.
  • -T {public|private} TPS feed type (required). Also known as the TPS feed name.
  • -U username Only required if usernames and passwords are mandatory
  • -P Password Must match username if used
  • -V version {0.4.x|*1.x.x) Defaults to newer version, you can override if you have an older server
  • -C "string" Channel name. Default is video.jpg, and I'd advise leaving this at the default.

    • org.nees.rbnb.DataVideoGather

    This is one of the coolest of the new features; this class combines DAQ data and video to produce a quicktime movie where every frame is sychronized to a DAQ sample. See this page for some examples of the output.

    To use this, you need

    1. Synchonized clocks on all applicable machines. This is harder than it seems. I recommend a local NTP server, such as this local GPS clock with NTP server software and NTP clients on all machines.
    2. Video feed. Any source that dumps JPEGs into the turbine is fine. We usually use the AxisSource but jcamera works too.
    3. DAQ feed. This is the source of the sync; the code takes frames closest to the DAQ samples.
    DVG, as its sometimes called, requires a bit of explanation to understand. Roughly, what it's doing is building a movie, where every frame is selected based on a DAQ sample. In effect, we are using a DAQ channel as a timebase or sample clock. Whichever channel(s) you select via -C will control which frames are chosen. For this to work, you want the DAQ channel to have a sample rate less than or equal to the video frames-per-second, otherwise you'll get duplicate frames in the movie.

    DVG also writes to disk the data samples that it used to generate the movie, so you can load the data+movie into Chef or other analysis tool for further processing.

    Here are the arguments:
     
    usage: org.nees.rbnb.DataVideoGather
     -C    Data Channels list - comma seperated list of Paths
     -v    Video Source path - no path means no movie
     -c    Data Channels Path Pattern - perl-like pattern to match
     -D    Data output file name *Data.txt
     -M    Movie output file name *Movie.mov
     -W    duration in seconds *1800
     -Z    Flag for 'zero for missing data'; otherzise last value repeats
     -d    Archive directory root *.
     -f    Frame Rate of final movie *1.0
     -h    Print help
     -i    Items to skip in data for each movie frame *0
     -k    Sink Name *DataVideoGather
     -p    Server Port Number *3333
     -s    Server Hostname *localhost
     -t    end time; detaults to 'now'
     -x    Flag to include Hidden channels
    Note: start time can either be yyyy-mm-dd:hh:mm:ss.nnn or an arbitraty floating point number

    Examples of DataVideoGather in action

    Note that these have the older -w argument. This is no longer needed and should be ignored.
    As used at 13WCEE: 
      java org/nees/rbnb/DataVideoGather -c "FromDAQ/.*" -v "axis46/video.jpg" -w $steps -W $steps -d $directory -f 15
    And here's another one I wrote for a recent demo, as csh script: 
    #!/bin/tcsh

    set interval = 600
    set framerate = 5

    echo Movie from Axis 1
    java org.nees.rbnb.DataVideoGather -s cee-zzle.cee.uiuc.edu -c "MainDAQ/.*"  -v "axis1/video.jpg" -M axis1.mov -w $interval -W $interval -f $framerate

    echo Movie from Axis 2
    java org.nees.rbnb.DataVideoGather -s cee-zzle.cee.uiuc.edu -c "MainDAQ/.*"  -v "axis2/video.jpg" -M axis2.mov -w $interval -W $interval -f $framerate

    org.nees.rbnb.FileToRbnb

    It's a program to take an DAQ-format ASCII file and upload it into the data turbine. This is useful for reloading data into the turbine, e.g. on a multi-day test or on a laptop away from the net.

    Arguments are

  • -n {string} source name. As with all of these, a descriptive name helps.
  • -c {string} channel name.
  • -d {directory} Directory to find the data file
  • -D {filename} Data file name
  • -z Cache size. Choose a buffer large enough to hold all the data.

    • org.nees.rbnb.GrabDataMultipleSink

    This is the complement of FileToRbnb; it downloads all specified channels to local disk. This is probably going to be replaced.

    Arguments are

  • -k {string} Sink name. Not sure why you need this.
  • -d {directory} Directory to write data into.
  • -x {true|false} Include hidden. No, I don't understand this either.
  • -a {time} start time
  • -z {time} end time

    • Times can either be yyyy-mm-dd;hh:mm:ss.nnn or an arbitrary floating point number.

    org.nees.rbnb.JpgSaverSink

    This is new program to save images from a given channel to disk. Unlike DataVideoGather, this writes individual files in a multi-level directory hierarchy, suitable for later reloading with JpgLoaderSource.

    The idea is to save the data from the turbine in an unmodified format, which can be processed offline, backed up, or reloaded via JpgLoaderSource.

    Arguments are

    org.nees.JpgLoaderSource

    This is the complement of JpgSaverSink; it will load images into a turbine from disk. The images are loaded in order, from within a specified time range. The directory structure expected is that created by JpgSaverSink.

    Arguments are
      TimeStamp format is yyyy-mm-dd:hh:mm:ss.nnn

    org.nees.rbnb.PictureSource

    This is a debugging tool; it simulates a normal video source like AxisSource or tpsSource. It repeatedly posts a series of images to the turbine at a configurable interval. Very useful for code testing and such.

    It assumes that the directory for the images contains a series of images ending with a common, numbered, suffix between 00 and 99. E.g.:
    	./img/test00.jpg
    	./img/test01.jpg
    etc.

    Arguments are

    org.nees.rbnb.WalkerSource

    A test RBNB source, akin to PictureSource, for numeric data. Generates numeric data in a bounded random walk, for single or multiple channels. The sample/generation rate is configurable. For a GUI source, see the source.jar that ships with the RBNB distribution; this is similar. Note that, unlike the source.jar, this one generates units.

    Arguments are

    org.nees.RbnbToFile

    Coming soon, not yet written. This will save numeric channels to ASCII on disk.

    Plugin-type programs

    These are programs that operate like a plugin, which is similar to a Unix pipe. Each takes a turbine source as and input, and produces a turbine source as an output. This is very useful, as you can see from these examples. The turbine also has a plugin API, however these programs are (for now) coded as plain turbine applications.

    org.nees.rbnb.JPEGThumbnailer

    This is a very cool program, originally written by Jason Hanley at SUNY, that takes an image channel and generates thumbnails. (Reduced size images). Why? Well, it's excellent for reducing the bandwidth required to see a video channel. Think outreach, slow net connections and demos.

    Arguments are

    org.nees.rbnb.NumberSubSample

    This is really a decimation program. It takes a numeric channel, decimates it and sends out 1 out of every N samples as a new numeric channel. For example, you could use this to decimate a 1kHz sample down to 1Hz for low-bandwidth remote monitoring.

    Arguments are

    NumberSubSample usage example:

    This is from one of our workflow tests, where we take the output of the LVDT signal and decimate it down. The result is a channel called 'NumberSubSampleSource/SubSample’:
    	java org.nees.rbnb.NumberSubSample -P FromDAQ/LVDT

    Debugging and diagnostic programs

    These are not generally useful, but are included for testing and debugging. See also PictureSource and RandomWalk above.

    org.nees.rbnb.ChannelList

    A simple program to print out the contents of a turbine - sources, sinks and channels.

    Arguments are just the usual server and port.

    org.nees.rbnb.ChannelListSwing

    A graphical version of ChannelList, this puts up a GUI with the same information, auto-updated.

    org.nees.rbnb.StringSink

    Simply prints out all strings sent to the named channel. Useful for monitoring chats and units.

    Arguments are

    org.nees.rbnb.NumberSink

    Similarly to StringSink, this prints out all the data that arrive on the given channel.

    Arguments are

    Increasing the amount of memory for the turbine

    By default, Java gives 256MB of memory to the virtual machine running the turbine. While more than enough for most java programs, its often useful to increase this to allow for more channels and longer time histories. To do so, edit 
     /usr/local/nees/etc/rc.d/init.d/rbnb
    and change 
      java -jar lib/rbnb.jar &
    to 
      java -Xmx1024M -jar lib/rbnb.jar &
    Or whatever you want; this example allocates 1024MB (1GB) to the Turbine. We've used up to 1500MB on a 2GB machine with no problems.

    You'll have to restart the turbine for the change to take effect.

    Putting it all together

    The following is the script we use to launch the turbine sources used in our test and demo turbine. It doesn't show the sinks or movie generation, but might be useful as an example.

    Notes:

    1. The paths are all hardcoded
    2. The logfile are all appended to, so they have to exist before you run the script.
    3. Framerates, passwords (changed from their real versions, of course) and such are all defined as variables at the top.
    4. My usual smart-alec comments
    #!/bin/tcsh
    setenv FPS 2
    setenv CACHE_SIZE 10000
    setenv CACHE_JPGS 5000
    setenv RBNB localhost
    setenv AX_USER username
    setenv AX_PASS password

    echo Going home. Take that, Dorothy.
    cd $HOME

    echo Starting fake DAQ
    nohup code/fake_daq/fake_daq >>& logs/fake-daq-src.log &

    echo Starting fake DAQ feed...
    nohup java org.nees.rbnb.DaqToRbnb -o 0.0 -n "FakeDAQ" -z $CACHE_SIZE >>& ~/logs/fake-daq.log &

    echo Starting DAQ feed for MiniMOST...
    nohup java org.nees.rbnb.DaqToRbnb -q neesdaq.sdsc.edu -s $RBNB -n "M1 DAQ" -z $CACHE_SIZE >>& ~/logs/neesdaq.log &

    echo Starting Axis source for camera 1 of MiniMOST
    nohup java org.nees.rbnb.AxisSource -s $RBNB -U $AX_USER -P $AX_PASS -S "M1 cam1" -A neescam1.sdsc.edu -n 1 -z $CACHE_JPGS -f $FPS >>& ~/logs/m1cam1.log &

    echo Starting Axis source for camera 2 of MiniMOST
    nohup java org.nees.rbnb.AxisSource -s $RBNB -U $AX_USER -P $AX_PASS -S "M1 cam2" -A neescam1.sdsc.edu -n 2 -z $CACHE_JPGS -f $FPS >>& ~/logs/m1cam2.log &

    echo Starting DAQ feed for shaketable...
    nohup java org.nees.rbnb.DaqToRbnb -q shaketable.sdsc.edu -s $RBNB -n "Shake table DAQ" -z $CACHE_SIZE >>& ~/logs/stdaq.log &

    echo Starting Axis source for shaketable
    nohup java org.nees.rbnb.AxisSource -s $RBNB -U $AX_USER -P $AX_PASS -S "Shake table camera" -A neescam2.sdsc.edu -z $CACHE_JPGS -f $FPS >>& ~/logs/stcam.log &

    echo Done.

    Code documentation

    I've started adding Doxygen to the source code, for internal documentation. This has a long way to go, but may be useful for those wanting to look more at the internals.

    The latest version is included with the code, and can be generated by running 

     
      ant doxygen
    I've uploaded a snapshot generated on Aug 24 2004 so you can see what it looks like.

    Navigation links

  • More detail on the RBNB time API
  • Back to NEESit page
  • Back to home page
    •

    Support

    This work was supported primarily by the George E. Brown, Jr. Network for Earthquake Engineering Simulation (NEES) Program of the National Science Foundation under Award Number CMS-0117853.