NEESgrid logo

An introduction to jcamera


Paul Hubbard
San Diego Supercomputer Center
hubbard@scsd.edu


Revision history

1.0 April 19, 2004 - Initial posting
1.1 May 25, 2004 - Update based on new code from Scott Gose.
1.2 June 9 2004 - Added link to PTP support page/list.
1.3 Dec 2004 - Prep for posting to NEESit
1.4 Jan 13 2005 - Added a couple of new cameras, USB hotplug script, 8x00 notes, changed MCS references to SDSC

Notation and conventions

Bold monospace type
denotes commands typed into the computer and
Monospace type
denotes output from a command.

Introduction

jcamera is a NEESGrid project to remotely control a digital camera over the network, using the NTCP (NEESGrid teleoperations control protocol). Here's a diagram we'll be referring to quite a bit:

Diagram of jcamera system

To get this working, we have to go by layer, starting at the bottom.

Assumptions and requirements

I assume you are trying to replicate all of this with a RedHat box; in particular this was written with Enterprise Linux-Workstation v3.0. You'll also need Java J2SDK version 1.4.2 or later.

Nikon and PTP

We'll start with the camera itself. This software is intended for use with consumer-grade digital still cameras. It will not work with DV or similar camcorders! For it to work, the camera must have a USB connection, and support for a protocol known as PTP, or Picture Transfer Protocol. (PIMA/ISO 15740)

More information about PTP can be found at ptp.sourceforge.net.

For a list of supported cameras, see Digital Camera Support for Unix. PTP support in cameras varies enormously, and should not be assumed, even if you are buying the latest and greatest. We have done all of our testing on a Nikon Coolpix 5700. This camera has worked well for us; note that it ships with PTP disabled; you have to re-enable it as a menu option. You'll also want the AC adapter, as leaving the camera running drains batteries quickly.

Other Tested Cameras

  1. Sony DSC-V1: While it has partial PTP support for transferring images, we were unable to control the shutter. It closes the lens as soon as USB is connected, which makes it no good for this application.
  2. Nikon 4300: No shutter control
  3. Sony DSC-T11: No shutter control
  4. Nikon 8800: Testing in progress, having USB2 issues, looks like it may work.

Since we've tested a small number of cameras, I would be interested in hearing your results with other units.

How do I know when I've got it connected and configured correctly?

Run
/sbin/lsusb
and look at the output. Somewhere in there you should see this (or similar if you have a different camera):
Bus 001 Device 004: ID 04b0:010d Nikon Corp. 
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 1.10
bDeviceClass 0 Interface
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 8
idVendor 0x04b0 Nikon Corp.
idProduct 0x010d
bcdDevice 1.00
iManufacturer 1 NIKON
iProduct 2 Nikon Digital Camera E5700_PTP
iSerial 3 000003115457
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 39
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xc0
Self Powered
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 3
bInterfaceClass 6 Imaging
bInterfaceSubClass 1 Still Image Capture
bInterfaceProtocol 1 Picture Transfer Protocol (PIMA 15470)
iInterface 0
(more of listing removed)

Note the 'E5700_PTP'; that's how you tell that PTP mode is enabled.

jusb

OK, you've got the camera in place and talking to the USB bus. The next component is the hardest one.

jusb (jusb.sourceforge.net) is the place to start. Unfortunately for us, the project has not been updated in a few years, and the included makefile does not work with newer Linux and Java. There are two solutions:
  1. Download jusb 0.4.4 from sourceforge, and then apply this patch to the Makefile
  2. Download the modified copy of 0.4.4 with the patch already in place from this link (MCS)
(Note that I sent the patch to the authors on 4/19/2004; if they update jusb I will update this page as well)

Building jusb

Unpacking the modified tarball will produce a 'jusb' directory. cd into it and run
make all
twice. The first time, it will report an error, and then the second time will succeed. Why? Good question!

If it worked, the directory should contain jusb.jar and libjusb.so; my copies look like this:
-rw-rw-r--    1 hubbard  hubbard     96722 Apr 19 15:02 jusb.jar
-rwxrwxr-x 1 hubbard hubbard 49263 Apr 19 15:02 libjusb.so*

Installing jusb

Once you've gotten it to build, you have two pieces to install.
  1. The native library, libjusb.so, which has to go into
     $JAVA_HOME/jre/lib/i386. 
    On RHEL-WS v3.0, with J2SDK 1.4.2_04, that ends up being
     /usr/java/j2sdk1.4.2_04/jre/lib/i386/libjusb.so
  2. The jar file, jusb.jar, goes into
      $JAVA_HOME/jre/lib/ext
You need to be root for both of these installs.

Testing the jusb install

make show
from the jusb directory should dump out an XML-formatted bus listing:
<!-- Linux usbfs -->
<host busses='1'>
<!-- Bus #1 -->
<?addresses portid='usb-f8883000' busaddr='1'?>
<hub usb='1.1'
vendorId='0x0' productId='0x0' device-version='0.0'
product='USB OHCI Root Hub'
serial='f8883000'
class='9' subclass='0' protocol='0'
ep0-maxpacket='8' configurations='1'>
<language id='0x0'/>
<config value='1' total-length='25' attributes='40'
max-power='0 mA.' interfaces='1'>
<!-- interface 0 is claimed by: hub driver -->
<hub number='0' alt='0' endpoints='1'
class='9' subclass='0' protocol='0'>
<endpoint addr='1' direction='in' type='interrupt'
attributes='3' maxpacket='2' interval='255'/>
</hub>
</config>
Root Hub, 4 ports
overcurrent protection: global
power switching: ganged

<!-- Port 2 -->
<?addresses portid='usb-f8883000-2' busaddr='4'?>
<device usb='1.1'
vendorId='0x4b0' productId='0x10d' device-version='1.0'
manufacturer='NIKON'
product='Nikon Digital Camera E5700_PTP'
serial='000003115457'
class='0' subclass='0' protocol='0'
ep0-maxpacket='8' configurations='1'>
<language id='0x409' locale='en_US'/>
<config value='1' total-length='39' attributes='c0'
max-power='0 mA.' interfaces='1'>
<!-- interface 0 is unclaimed -->
<unknown-6 number='0' alt='0' endpoints='3'
class='6' subclass='1' protocol='1'>
<endpoint addr='4' direction='out' type='bulk'
attributes='2' maxpacket='64' interval='0'/>
<endpoint addr='3' direction='in' type='bulk'
attributes='2' maxpacket='64' interval='0'/>
<endpoint addr='1' direction='in' type='interrupt'
attributes='3' maxpacket='8' interval='10'/>
</unknown-6>
</config>
</device>
</hub>
</host>
Note that this does not test the version intalled into $JAVA_HOME. That has to wait until jphoto.

USB device permissions

If you get permissions errors, you may need to change the permissions on the USB device filesystem; see 'README.linux' in the jusb directory. The magic bit is adding "devmode=0666" in your usbdevfs entry in /etc/fstab. One some systems, usbdevfs is mounted in /etc/rc.sysinit and doesn’t have an entry in /etc/fstab, so you need to add a line similar to the following:
none                    /proc/bus/usb           usbdevfs   defaults        0 0
to /etc/fstab. This makes all devices world-writeable on USB, which is a rather crude solution. Finer control requires a bit more work. When you plug a USB device in, the kernel looks up the device ID in a file and then (optionally) runs what's known as a hot-plug script apppropriate for the device. This is useful for auto-mounting memory sticks, for example. For cameras, the hotplug script is
/etc/hotplug/usb/usbcam
Modify the third-from-end line from
chmod 0600 "${DEVICE}
to
chmod 0666 ${DEVICE}
This means that all digital cameras will be world-writeable, which is probably OK. Secondly, we need to add the camera to the USB ID table, so that the kernel knows to run the usbcam script. To do that, edit the /etc/hotplug/usb.usermap file, adding this entry:
# Nikon Coolpix 5700 (PTP mode)
usbcam 0x0003 0x04b0 0x010d 0x0000 0x0000 0x00 0x00 0x00 0x00 0x00 0x00 0x00000000
Of course, if you have a different camera, you'll have to change the above to match its ID. See http://jphoto.sourceforge.net/?selected=sync for more information.

jphoto enters the picture

The next layer is where it starts to get interesting. JPhoto (jphoto.sourceforge.net) is a collection of command-line tools for Java and PTP, with the most useful being jphoto itself. As with jusb, we had to update the jphoto code to accomodate newer Java and Linux. While doing so, we added some logging, code cleanup, and functionality changes.

Unfortunately, the JPhoto project on sourceforge is no longer actively maintained so we have committed our changes to the NEES CVS. Anonymous checkouts are available through CVS:
$ export CVSROOT=:pserver:anonymous@cvs.nees.org:/disks/cvs/neesgrid
$ cvs co jphoto

Building jphoto

To build JPhoto, check out the code using the above instructions and run 'ant' in the jphoto directory.
$ cd jphoto
$ ant

Testing the jphoto build

To test JPhoto, use the script jphoto.sh as follows:
./jphoto.sh status
This should produce something like:
PTP device at usb-f8883000-2
Manufacturer: Nikon Corporation
Model: E5700
Device Version: E5700v1.1
Serial Number: 000003115457
Extensions (0xa): Nikon PTP Extensions
Operations: 15
Modes: Pull
Object count: 4
Storage Type: Removable RAM
You can take a test picture with
./jphoto.sh capture
./jphoto.sh images
which produces an image in the 'images' subdirectory.

Installing jphoto

Jphoto has one file you need, jphoto.jar. Since we're calling jphoto as a Java class, we don't need the shell script jphoto.sh; if you want you can put that in your path as well.

After building, install lib/jphoto.jar file into your JRE's classpath:
$ cp lib/jphoto.jar $JAVA_HOME/jre/lib/ext
You will probably need to be root to do so.

jcamera - we're getting close now

The next layer is the last one that has to run on the same machine as the camera; jcamera. This is code that bridges the camera into the world of NTCP. To do so, it uses the JPhoto library under the command of an NTCP connection.

In the interests of code re-use and expediency, jcamera is written to communicate with the NTCP serialization plugin, a.k.a ASCII plugin

Note that the labview plugin initiates the TCP connection to jcamera, so we do not have to configure the network in jcamera - it just listens until a connection arrives. There's no such thing as a free lunch, of course, so we do have to
  1. Set up an NTCP server
  2. Tell it to use the labview plugin as  backend
  3. Tell the plugin the host and TCP port running jcamera
All of this is covered in the NTCP client writeup, so I will not duplicate that material here.

Multiple Camera Support

jcamera can support multiple cameras in a variety of ways. When jcamera first starts up, it queries the USB bus for PTP enabled devices. As it finds them, it assigns them the name, "cameraX" where X is a positive integer starting at 1. So if you have 3 cameras hooked up, when you start jcamera, you should see it find three cameras and name them: camera1, camera2, and camera3. These names are what you use when in jclient (described below) to tell jcamera which camera to take a picture with.

Additionally, you can invoke jclient below with the generic device name "camera" and it will simultaneously take a picture on all connected devices, saving them to local disk with a suffix name of "_cameraX" where X is the camera that took the picture. More information is provided below in jclient.

jcamera has Data Turbine support enabled which allows it to upload the pictures it takes to a data turbine. If it cannot find a data turbine, it will only provide a warning message before continuing on as normal. Regardless of whether you have a data turbine setup, all images are stored to the local hard drive in the "image/" directory.

Downloading jcamera

The JCamera project lives in the NEESGrid CVS archive. To download it, you need to check it out using CVS as detailed below:
$ export CVSROOT=:pserver:anonymous@cvs.nees.org:/disks/cvs/neesgrid
$ cvs co jcamera

Building jcamera

cd into jcamera directory, and run ant as follows:
$ cd jcamera
$ ant
If you see 'BUILD SUCCESSFUL', congratulate yourself, 'cause this has been a tough one.

Running jcamera

From the jcamera directory, run
$ ./jcamera.sh
You should see some print statements which will let you know that JCamera started ok. It will list what cameras it finds and assign them names starting with camera1, camera2, etc. depending on how many cameras you have attached to your system. After it starts up, it will sit and wait for an NTCP connection to arrive and get the party started.

NTCP and the serialization plugin

On the same or other network-reachable machine, you need an NTCP server with the labview plugin. Complete instructions for this are in the NTCP writeup, and needn't be duplicated here.

Once you are ready to run the NTCP container, edit server-config.wsdd and make sure that
lvHost and lvPort 
are set to the machine running jcamera. That way, when an NTCP connection arrives, the lvplugin will open communications with jcamera, and all will be good. In my test setup, I have this section of server-config.wsdd:
  <parameter name="lvHost" value="neestpm.sdsc.edu"/>
<parameter name="lvPort" value="44000"/>
Note that port 44000 is also set in the jcamera code, so if you change it here you'll need to change it there as well.

Run the NTCP container

I use port 8090 for my NTCP testing; your will probably differ based on local issues. I use
ant startContainer -Dservice.port=8090

jclient: the final piece

jclient is a very basic NTCP client, which knows how to take pictures. Here are the magic bits of information:
For example, here are several I took while writing this document:
[hubbard@neestpm ~/code/jcamera] ls -l image/
total 3880
-rw-rw-r-- 1 hubbard hubbard 1005837 Apr 19 15:58 TR1048792482.jpg
-rw-rw-r-- 1 hubbard hubbard 982054 Apr 19 15:58 TR1815875301.jpg
-rw-rw-r-- 1 hubbard hubbard 978661 Apr 19 16:09 TR234623115.jpg
-rw-rw-r-- 1 hubbard hubbard 983473 Apr 19 16:09 TR281657422.jpg

Getting and building jclient

JClient also lives in the NEESGrid CVS and can be downloaded as follows:
$ export CVSROOT=:pserver:anonymous@cvs.nees.org:/disks/cvs/neesgrid
$ cvs co jclient
Before running jclient, you need to modify the source file:
src/org/nees/ntcp/jclient/JClient.java
Look for the "serverURL" variable and change it to point to the address of your NTCP server.

Next, we'll compile and run the client.

Running jclient

Note: The format of the output for jclient and jcamera has changed. The gist of it well represented though in the output below.

From the same directory, run the following command to compile and run jclient:
ant run
On the machine running jclient, you should see something like this:
[hubbard@wdhcp-1 ~/code/jclient] ant run
Buildfile: build.xml

checkOGSA:

prepare:
[mkdir] Created dir: /Users/hubbard/code/jclient/build
[mkdir] Created dir: /Users/hubbard/code/jclient/build/classes
[mkdir] Created dir: /Users/hubbard/code/jclient/build/lib
[mkdir] Created dir: /Users/hubbard/code/jclient/docs

compile:
[javac] Compiling 1 source file to /Users/hubbard/code/jclient/build/classes

jar:
[jar] Building jar: /Users/hubbard/code/jclient/build/lib/jclient.jar

run:
[java] Opening connection to NTCP server at http://neespop.mcs.anl.gov:8090/ogsa/services/nees/ntcp/ in non-secure mode
[java] Connection opened OK, now sending openSession
[java] - Logging is working
[java] - Date: Mon Apr 19 17:57:26 CDT 2004
[java] - Version: Apache-XML-Security-J 1.0.4
[java] Proposing transaction #63921 id: TR2066439084
[java] Transaction proposal accepted. Excellent.
[java] Executing, may take 60-90 seconds
[java] Execute completed; checking status
[java] Move incomplete - transaction not terminated!!
[java] Proposing transaction #23507 id: TR304099882
[java] Transaction proposal accepted. Excellent.
[java] Executing, may take 60-90 seconds
[java] Execute completed; checking status
[java] Move incomplete - transaction not terminated!!
[java] Closing NTCP session
[java] Done.

BUILD SUCCESSFUL
Total time: 50 seconds
On the machine running NTCP:
[hubbard@neespop ~/code/ogsa-3.0.2] ant startContainer -Dservice.port=8090
Buildfile: build.xml

startContainer:

startContainer:
[java] [04/19/2004 17:57:09:889 ] org.globus.ogsa.server.ServiceContainer [run:569] INFO: Starting SOAP server at: http://127.0.0.1:8090/ogsa/services/
[java] With the following persistent services:
[java]
[java] http://127.0.0.1:8090/ogsa/services/nees/ntcp/NTCPServer
[java] http://127.0.0.1:8090/ogsa/services/core/admin/AdminService
[java] http://127.0.0.1:8090/ogsa/services/core/management/OgsiManagementService
[java] http://127.0.0.1:8090/ogsa/services/core/registry/ContainerRegistryService
[java] http://127.0.0.1:8090/ogsa/services/core/jmsadapter/JMSAdapterFactoryService
[java] http://127.0.0.1:8090/ogsa/services/core/logging/OgsiLoggingService
[java] http://127.0.0.1:8090/ogsa/services/core/notification/httpg/NotificationSubscriptionFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/registry/VORegistryService
[java] http://127.0.0.1:8090/ogsa/services/samples/counter/secure/CounterFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/counter/notification/CounterFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/counter/encoded/CounterFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/counter/persistent/CounterFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/counter/basic/CounterFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/counter/delegation/CounterFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/counter/routable/MasterRedirectedCounter
[java] http://127.0.0.1:8090/ogsa/services/samples/counter/routable/LocalCounterFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/counter/generate/CounterFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/counter/deactivation/CounterFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/counter/soap-secure/CounterFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/counter/logging/CounterFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/exception/ExceptionFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/notification/SinkListenerFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/complex/NestedArrayFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/complex/NestedFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/weather/WeatherFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/serialization/SerializationService
[java] http://127.0.0.1:8090/ogsa/services/samples/google/GoogleSearchFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/any/AnyFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/servicedata/ServiceDataService
[java] http://127.0.0.1:8090/ogsa/services/samples/array/ArraySampleFactoryService
[java] http://127.0.0.1:8090/ogsa/services/samples/chat/ChatFactoryService
[java] http://127.0.0.1:8090/ogsa/services/ogsi/NotificationSubscriptionFactoryService
[java] http://127.0.0.1:8090/ogsa/services/ogsi/HandleResolverService
[java] http://127.0.0.1:8090/ogsa/services/gsi/AuthenticationService
[java] http://127.0.0.1:8090/ogsa/services/gsi/SecureNotificationSubscriptionFactoryService

[java] sending open-session dummyOpenSession

[java] OK 0 dummyOpenSession
[java] neestpm.mcs.anl.gov return dummyOpenSession I expect: dummyOpenSession
[java] In propose**********************
[java] sending propose TR2066439084 camera z displacement 3500.0

[java] OK 0 TR2066439084
[java] first token is: OK
[java] str =:TR2066439084:should be:TR2066439084:
[java] leaving propose() with true
[java] in execute*********************
[java] sending execute TR2066439084

[java] OK 0 TR2066439084
[java] got OK on execute
[java] In getControlPoint
[java] sending get-control-point dummy camera
[java] in getControlPoint for loop
[java] OK 0 dummy camera x rotation 0 y rotation 0 z displacement 3500
[java] first token is: OK
[java] str =:dummy:should be:dummy:
[java] leaving getControlPoint() *****************************
[java] In propose**********************
[java] sending propose TR304099882 camera z displacement 28000.0

[java] OK 0 TR304099882
[java] first token is: OK
[java] str =:TR304099882:should be:TR304099882:
[java] leaving propose() with true
[java] in execute*********************
[java] sending execute TR304099882

[java] OK 0 TR304099882
[java] got OK on execute
[java] In getControlPoint
[java] sending get-control-point dummy camera
[java] in getControlPoint for loop
[java] OK 0 dummy camera x rotation 0 y rotation 0 z displacement 28000
[java] first token is: OK
[java] str =:dummy:should be:dummy:
[java] leaving getControlPoint() *****************************
[java] sending close-session dummy

[java] OK 0 dummy
One the machine running jcamera:
[hubbard@neestpm ~/code/jcamera] ant run
Buildfile: build.xml

prepare:

compile:

run:
[java] PTP device at usb-f8883000-2
[java] PTP device at usb-f8883000-2
[java] focalLenght = 28000
[java] Incoming command: open-session dummyOpenSession
[java] Response is: OK 0 dummyOpenSession
[java] Incoming command: propose TR2066439084 camera z displacement 3500.0
[java] Response is: OK 0 TR2066439084
[java] Incoming command: execute TR2066439084
[java] PTP device at usb-f8883000-2
[java] Original value
[java] we did not find code = 0
[java] returning 0
[java] vaue is { data OUT; len 16; 0; xid 0}
[java] Type: u32, Value: 3500
[java] PTP device at usb-f8883000-2
[java] PTP device at usb-f8883000-2
[java] image saved, size 971963
[java] PTP device at usb-f8883000-2
[java] deleting
[java] Response is: OK 0 TR2066439084
[java] Incoming command: get-control-point dummy camera
[java] Response is: OK 0 dummy camera x rotation 0y rotation 0 z displacement 3500
[java] Incoming command: propose TR304099882 camera z displacement 28000.0
[java] Response is: OK 0 TR304099882
[java] Incoming command: execute TR304099882
[java] PTP device at usb-f8883000-2
[java] Original value
[java] we did not find code = 0
[java] returning 0
[java] vaue is { data OUT; len 16; 0; xid 0}
[java] Type: u32, Value: 28000
[java] PTP device at usb-f8883000-2
[java] PTP device at usb-f8883000-2
[java] image saved, size 983853
[java] PTP device at usb-f8883000-2
[java] deleting
[java] Response is: OK 0 TR304099882
[java] Incoming command: get-control-point dummy camera
[java] Response is: OK 0 dummy camera x rotation 0y rotation 0 z displacement 28000
[java] Incoming command: close-session dummy
[java] Response is: OK 0 dummy

Data Turbine: Viewing the data

Although the turbine is not required, pictures are sent to it if its present. You can find more information on the turbine on
the NEESGrid turbine page. Concisely put, you can view images with the bundled RBNB JPEG viewer:
cd {directory where you installed the turbine}
cd bin
java -jar rbnbjview.jar -a {turbine IP address or name} -c "camera1/Video.jpg,camera2/Video.jpg" &
You may need to allocate more memory for the Java virtual machine, for that you use an additional command line argument:
java -Xmx128M ...
Note that the -c argument can take one or more channels; given the high-res nature of these images, I'd suggest only viewing one at a time. Work on the turbine is ongoing, so expect improvements to arrive rapidly.

Conclusions and comments

As of 5/21/04, Scott Gose has added multiple camera support and data turbine integration. We have tested with 3 cameras on the same machine, taking pictures at the same time and putting the results into the turbine. I'll update this page next week with more details.
Please send feedback on this document to hubbard at sdsc.edu

Navigation links

  • Back to NEESGrid at SDSC
  • Back to home page
  • Support

    This work was supported primarily by the George E. Brown, Jr. Network for Earthquake Engineering Simulation (NEES) Program of the National Science Foundation under Award Number CMS-0117853.