I got a ticket from a user reporting that the following header files were missing from the version of Thread Building Blocks that is bundled with the Intel commpilers we provide on SDSC Trestles:

After a bit of Googling around, I discovered that the problem arose because concurrent_priority_queue.h only became available in Intel TBB version 3.0 update 4 and newer. Our version of the Intel compiler, 11.1.072, was bundled with an older version, TBB 2.2:

$ grep TBB_VERSION /opt/intel/Compiler/11.1/072/tbb/include/tbb/tbb_stddef.h

However Intel has released Thread Building Blocks under the GNU GPL license so building a version newer than the one bundled with Intel's compilers should be straightforward. These instructions are pretty generic and can be applied to any SDSC resource; Trestles is just suffering from having a significantly older TBB version.

Compiling TBB on Trestles

1. Downloading the source

The most current version of TBB at the time of this writing is 4.1u2, so that's what I installed. The TBB source package can be downloaded from Intel's official open-source TBB site and transferred to Trestles, or downloaded directly to Trestles using wget:

$ wget
--2013-03-18 12:21:55--
Connecting to||:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2461689 (2.3M) [application/octet-stream]
Saving to: `tbb41_20130116oss_src.tgz'
100%[======================================>] 2,461,689   2.12M/s   in 1.1s    
2013-03-18 12:21:57 (2.13 MB/s) - `tbb41_20130116oss_src.tgz' saved [2461689/2461689]

2. Unpacking

Unpack the tarball to whatever location you wish to install TBB. The source distribution does not use autoconf in its build process and installs itself in-place. I like to install my Trestles software in ~/trestles, so

$ cp tbb41_20130116oss_src.tgz ~/trestles/
$ cd ~/trestles
$ tar zxvf tbb41_20130116oss_src.tgz

3. Compiling

As I said, building TBB is not a simple ./configure && make && make install, and the install documentation (which is all in html, which is nice unless you are installing from the command line) is not terribly comprehensive. By default it builds with g++, and it wasn't immediately clear how to set a different compiler since the build process isn't the familiar autoconf process. As it turns out, you have to pass compiler=icc to make. So,

$ cd tbb41_20130116oss_src
$ module purge
$ module load intel
$ compiler=icc make all
Created ./build/linux_intel64_icc_cc4.1.2_libc2.5_kernel2.6.18_release and ..._debug directories
make -C "./build/linux_intel64_icc_cc4.1.2_libc2.5_kernel2.6.18_debug"  -r -f ../../build/Makefile.tbb cfg=debug tbb_root=../..
make[1]: Entering directory `/home/glock/tbb41_20130116oss/build/linux_intel64_icc_cc4.1.2_libc2.5_kernel2.6.18_debug'
../../build/Makefile.tbb:39: CONFIG: cfg=debug arch=intel64 compiler=icc os=linux runtime=cc4.1.2_libc2.5_kernel2.6.18

This will take a while, and once it completes and tests the build, Thread Building Blocks is installed. There is no separate make install step.

Linking against TBB

One of the files that was created during the build step is which initializes the environment variables necessary to use this new installation of Thread Building Blocks. Assuming TBB was built and installed in ~/trestles/tbb41_20130116oss_src, this file will be in somewhere like


and to load it, you should issue

$ source ~/trestles/tbb41_20130116oss/build/linux_intel64_icc_cc4.1.2_libc2.5_kernel2.6.18_release/

You will have to do this every time you want to build or run a TBB application, and you must issue this command after loading the intel module. If you do this out of order, the intel module's older TBB will supercede yours.

Once you've loaded this file, you can now link against this new TBB by using -ltbb instead of the -tbb switch you probably used to use. For example, here's a simple hello world program called hello.cpp that uses TBB:

#include <iostream>
#include <tbb/parallel_for.h>
using namespace tbb;
using namespace std;
class hello {
   void operator()(int x) const {
       cout << "Hello world!  TBB version is " << TBB_VERSION_MAJOR <<
           "." << TBB_VERSION_MINOR << ", interface " <<
           (float)TBB_INTERFACE_VERSION/1000.0 << "\n";
int main() {
   parallel_for( 0, 4, 1, hello() );
   return 0;

To compile this with our new installation of TBB, make sure your intel module is loaded, you've loaded and then compile:

$ icpc hello.cpp -o hello -ltbb
$ ./hello
Hello world!  TBB version is 4.1, interface 6.102
Hello world!  TBB version is 4.1, interface 6.102
Hello world!  TBB version is 4.1, interface 6.102
Hello world!  TBB version is 4.1, interface 6.102

If you get an error like this,

$ icpc hello.cpp -o hello -ltbb
ld: cannot find -ltbb

it means that

When running your TBB applicaton on a compute node, you may get an error like this:

./hello: error while loading shared libraries: cannot open
shared object file: No such file or directory

This happens because your queue script doesn't have the correct environment variables set up. Add the following lines to your queue script:

source /etc/profile.d/
module load intel
source ~/trestles/tbb41_20130116oss/build/linux_intel64_icc_cc4.1.2_libc2.5_kernel2.6.18_release/