BoostStlportUpgrade2009

Summary

This page is for planning and tracking the progress of upgrading the following Fennel thirdparty dependencies:

• Boost 1.33 -> 1.40
• STLport 5.0.2 (or 4.6.2 for Windows) -> 5.1.6

We have deferred upgrades on these for far too long, making it harder and harder to maintain builds and patchsets with each new g++ release.

To facilitate this work, we ended up overhauling the Fennel build system to use cmake instead of autotools (and MSVC instead of mingw g++ on Windows). For details, see FennelBuildSystem.

Branching

The 1.38 upgrade existed for some time and has been integrated up to //open/dev and to other branches. The upgrade to 1.40 currently exists only on the //open/dt/dev branch.

The upgrade to 1.38 is checked into the branch as eigenchange 11811. The upgrade to 1.40 is checked into the branch as eigenchange 13228 (currenly only on //open/dt/dev) and there is a fix to the patch at 13230.

Testing

We would like to get the upgrade tested with as many configurations as possible before integrating it to //open/dev. If you are testing a configuration, please record your results here so that we know what has been covered:

• 64-bit Jaunty Jackalope (GCC) gcc version 4.1.3 20080623 (prerelease) (Ubuntu 4.1.2-ubuntu1) all Fennel and Farrago checkin acceptance tests are passing for HCP
  • optimized build and tests work as of eigenchange 13230
• Windows: definitely broken at the moment; JVS is working on it
• 64-bit CentOS (GCC) gcc version 4.1.2 20070925 (Red Hat 4.1.2-33)
• 32-bit CentOS (GCC) gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)
• 64-bit CentOS (GCC) gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)

Boost Patch

One Boost patch is still needed:

diff -ur boost-orig/boost/test/unit_test_suite_impl.hpp boost/boost/test/unit_test_suite_impl.hpp
--- boost-orig/boost/test/unit_test_suite_impl.hpp	2008-10-13 01:20:26.000000000 -0700
+++ boost/boost/test/unit_test_suite_impl.hpp	2009-04-19 00:48:10.000000000 -0700
@@ -72,7 +72,7 @@
     void                                increase_exp_fail( unsigned num );
 
 protected:
-    ~test_unit();
+    virtual ~test_unit();
 
 private:
     // Data members

See this mailing list thread for (so-far-unresolved) discussion.

These patches were added in this integration:

--- boost-new/boost/exception/detail/exception_ptr_base.hpp 2009-05-20 12:41:20.000000000 -0700 +++ boost/boost/exception/detail/exception_ptr_base.hpp 2009-12-12 17:44:16.000000000 -0800 @@ -16,7 +16,7 @@

                exception_ptr_base
                        {
                        public:

- + virtual ~exception_ptr_base() { }

                        virtual void _rethrow() const=0;
                        virtual bool _empty() const=0;
                        };

diff -Naur boost-new/boost/memory_order.hpp boost/boost/memory_order.hpp --- boost-new/boost/memory_order.hpp 2008-07-12 05:21:19.000000000 -0700 +++ boost/boost/memory_order.hpp 2009-12-12 17:42:54.000000000 -0800 @@ -27,6 +27,7 @@

    memory_order_acquire = 1,
    memory_order_release = 2,
    memory_order_acq_rel = 3, // acquire | release

+ memory_order_consume = 4,

    memory_order_seq_cst = 7  // acq_rel | 4
};

Boost Trimming

The Boost source tarball size has grown by a huge amount from 1.33 (11MB) to 1.38 (29MB). Consequently, the checked in source tarball has been put on a diet, bringing it down to 13MB. This was done by deleting the following directories:

doc
libs/{math,spirit,gil,mpl,serialization,fusion,python,asio,iterator,wave}

The documentation is all available online at www.boost.org (and you can download the full tarball from there if you need it). The deleted libraries are unlikely to be used by Fennel; if we end up needing them, we can update the tarball.

The checked-in tarball is named boost_1_40_0-slimfast.tar.bz2 and is 15MB even after the trimming.

Besides the tarball, one further reduction happens in fennel/build/buildBoostLibs.sh: after "installing" Boost, a redundant copy of the header files are created under boost/include, but we don't need these eating disk space, and their presence is confusing, so we delete them. (Is there a way to suppress the copy in the first place?)

Boost Configuration

The same configuration options from 1.33 are carried over (debug+release builds, dynamic link, multithreaded, STLport support). One new Boost library ("system") needs to be built.

In addition, the new option --layout=system is used so that library filenames come out like this:

libboost_date_time-mt-gdp.so

instead of like this

libboost_date_time-gcc41-mt-gdp-1_38.so

We don't want the version info (particularly the compiler version) in the library name, since that way Fennel doesn't have to figure out what to link with. --layout=tagged also gets rid of all of those symlinks for which we had no use.

STLport Patch

Some STLport patches remain necessary even after the upgrade:

jvs@bagheera:~/open/tpu/thirdparty$ cat STLport-fennel.patch 
# patch STLport for backwards compatibility with a Fennel dependency
# $Id: //open/users/jvs/tpu/thirdparty/STLport-fennel.patch#2 $
diff -ur stlport-orig/stlport/stl/_threads.h stlport/stlport/stl/_threads.h 
--- stlport-orig/stlport/stl/_threads.h	2008-10-06 23:44:37.000000000 -0700
+++ stlport/stlport/stl/_threads.h	2008-11-01 20:01:08.000000000 -0700
@@ -57,7 +57,7 @@
  * does *__target = *__target + __val and returns the old *__target value
  */
 
-#if defined (_STLP_WIN32) || defined (__sgi) || defined (_STLP_SPARC_SOLARIS_THREADS)
+#if defined (_STLP_WIN32) || defined (__sgi) || defined (_STLP_SPARC_SOLARIS_THREADS) || defined(__GNUC__)
 typedef long __stl_atomic_t;
 #else
 /* Don't import whole namespace!!!! - ptr */
@@ -458,6 +458,8 @@
   // The data member _M_ref_count
 #if defined (__DMC__)
 public:
+#else
+protected:
 #endif
   _STLP_VOLATILE __stl_atomic_t _M_ref_count;

diff -ur stlport-orig/stlport/stl/debug/_string.h stlport/stlport/stl/debug/_string.h
--- stlport-orig/stlport/stl/debug/_string.h	2008-10-06 23:44:37.000000000 -0700
+++ stlport/stlport/stl/debug/_string.h	2009-04-20 15:45:40.000000000 -0700
@@ -347,7 +347,6 @@
   _Self& assign(const _CharT* __s, size_type __n) {
     _STLP_FIX_LITERAL_BUG(__s)
     _STLP_VERBOSE_ASSERT((__s != 0), _StlMsg_INVALID_ARGUMENT)
-    _M_check_assign((min) (_Traits::length(__s), __n));
     _M_non_dbg_impl.assign(__s, __s + __n);
     return *this;
   }

Long term, we need to stop doing the protected hack, but Boost still hasn't added the necessary atomic counter support. (There is a boost::detail::atomic_counter internal implementation, but it does not provide a way to reset the counter.) There was an atomic package added in this patch to support lockfree data structures. This library is very new and not used by Fennel itself. After a time, we should consider using this boost.atomic library when its included in boost itself.

The other changes are

• avoid "type punning" warnings from the -fstrict-aliasing optimization
• fix for STLport bug 2495252, about which the STLport developers seem to be in denial the last time I checked

On Windows, stlport-msvc9-r3503.patch is also needed for MSVC build support.

Boost Threads Breaking Changes

See http://www.boost.org/doc/libs/1_36_0/doc/html/thread/changes.html for some breaking changes we are dealing with as part of this upgrade.

typedef LocalCondition now maps to boost::condition_variable. It used to be boost::condition, but that now maps to boost::condition_variable_any, which is less efficient.

The "Try" variants of mutexes and guards are no longer distinguished from the normal variants; instead the try specification is on the lock attempt where it always should have been.

Where we would formerly have written this to take a lock on a try-mutex and wait for it (instead of returning immediately):

    StrictMutexTryGuard pageGuard(page.mutex,true);

We now use a normal guard:

    StrictMutexGuard pageGuard(page.mutex);

Where we would formerly have written this to use the "try" semantics for the lock:

    StrictMutexTryGuard pageGuard(page.mutex);

We now write this:

    StrictMutexGuard pageGuard(page.mutex, boost::try_to_lock);

Finally, where we would formerly have written this to check whether a guard's lock has been taken:

     if (!pageGuard.locked()) {

We now write this:

     if (!pageGuard.owns_lock()) {

Note that these breaking changes mean the Fennel code can no longer be compatible with the old Boost version. So if we end up stuck on that for Windows, some conditional compilation will need to be added.

Besides Fennel, extension projects with thread dependencies will also have to make corresponding changes before we can integrate the Boost upgrade to //open/dev.

Miscellaneous Changes

A few other changes were required to build Fennel with the upgrade:

• #include <stdexcept> in CommonPreamble.h; previously it must have been getting included implicitly by some other header
• in configure.in, use the presence of $stlport_location/stlport/stl/_vector.h as the AC_CHECK_FILE for the existence of STLport; we used to use _config.h, but it is gone now
• also in configure.in, match the new library filenames for STLport
• when passing integer arguments to math functions like log10, have to explicitly cast to double since they are overloaded now
  • --Jvs 18:10, 26 October 2008 (PDT): for the ones under disruptivetech, I have integrated these back to //open/dev already so that moved files don't cause complications in EigenbaseCodeReorg2008

For details, see eigenchange 11811.

Enhancements

There have been many enhancements in boost. We can start taking advantage of these on the tpu branch, but that should not be a dependency for the integration; we can do these after integration (or never) if we choose.

• the thread library finally provides a portable means of obtaining the current thread's ID
• thread library now uses datetime library for timeouts
• we can get rid of the warning suppression in configure.in (the one where we kept adding new compiler versions to the list)