How to build and include stxxl in a Qt project on Windows?

Multimodcrafter

Hi everybody

I'm developing a qt desktop application, in which I need the stxxl library for big containers (http://stxxl.sourceforge.net/). Now unfortunately I'm a complete noob when it comes to programming with libraries that are not part of an installation. And Everything I found so far on the topic was either Linux related or not specific enough. I would be very thankful if somebody could explain to me step by step what I need to do to include the library in my qt desktop application on Windows (I use the Desktop Qt 5.5.1 MinGW 32bit Kit)

Thank you in advance

SGaist

Hi and welcome to devnet,

Do you mean add it as a dependency of your project or build a release package to distribute that includes that dependency ?

Multimodcrafter

Hi

I'd rather include it in the build since I would like to ship my application without the user needing to install any 3rd party software.

SGaist

Again, that's two different things:

Building against the stxxl library
Ship it as part of your application package

Those are two different steps taken at different time of your application development cycle.

If you are using Qt Creator, here there's an explanation on how to add an external library with it.

As for deployment, here you have the guide.

Multimodcrafter

Ok, yes I understand! Thank you!
I guess I just really have to find out what steps I need to perform so the stxxl library is recognized as such.
The QT creator asks for a .lib file which stxxl does not provide me (Or I just can't find it at least). Does anybody know how I can get that?

SGaist

Did you already build stxxl ?

If not, there's the INSTALL file that describes the steps

Multimodcrafter

Yes I did build it with CMake (The windows version of course) once via command line and once via the gui just to make sure there are no differences. However the installation instructions say that after that I need to run make (or in my case I guess mingw32-make) but neither the command line nor the gui process created a makefile (at least none that's called that way). And like I already said, in both cases I couldn't find a .lib file (If that's even necessary which I think because the QT wizard demands one).Is the make step only necessary on Linux platforms?

Multimodcrafter

Ah wait! I finally get it: I have to use visual studio to build the LIBRARY and not the projects! Ok, so I got the .lib file now. But I'm struggling with which path I should choose as the include path.

Multimodcrafter

So I think I figured it out: I have to select the include folder in the sources folder of stxxl but now I get a bunch of errors of the following kind:

D:\Coding\c++\stxxl-1.4.1\include\stxxl\bits\config.h:28: Error: invalid preprocessing directive #cmakedefine
#cmakedefine STXXL_VERSION_GIT_REFSPEC "${STXXL_VERSION_GIT_REFSPEC}"
^
D:\Coding\c++\stxxl-1.4.1\include\stxxl\bits\common\mutex.h:31: Error: #error "Thread implementation not detected."
#error "Thread implementation not detected."
^

SGaist

Did you call make install and point to that folder/folder set ?

Multimodcrafter

No, that would be the case if I were on linux. Since I'm on windows, I had to open the visualstudio project that was created by cmake and run the build process from there. That way I got the .lib file I needed and I went through the steps of the QT creator wizard. I chose external library, then selected the .lib file and the include folder with the necessary header files. When I then copy pasted the sample test-program provided by stxxl, I was able to confirm that the includes worked as expected. But when I try to build it (The QT application) I get the errors mentioned above. Inside the config.h there are a lot of those #cmakedefine macros which don't get recognized by QT; therefore throw an error. And the other error is in connection with all header files that have something to do with multi threading (at least as far as I can tell).

this is the config.h file:

// -*- mode: c++ -*-
/***************************************************************************
 *  include/stxxl/bits/config.h.in
 *
 *  Template file processed by cmake to set all define switches for this build
 *  according to the cmake build options.
 *
 *  Part of the STXXL. See http://stxxl.sourceforge.net
 *
 *  Copyright (C) 2012-2013 Timo Bingmann <tb@panthema.net>
 *
 *  Distributed under the Boost Software License, Version 1.0.
 *  (See accompanying file LICENSE_1_0.txt or copy at
 *  http://www.boost.org/LICENSE_1_0.txt)
 **************************************************************************/

#ifndef STXXL_CONFIG_HEADER
#define STXXL_CONFIG_HEADER

// the STXXL library version variables
#define STXXL_VERSION_MAJOR ${STXXL_VERSION_MAJOR}
#define STXXL_VERSION_MINOR ${STXXL_VERSION_MINOR}
#define STXXL_VERSION_PATCH ${STXXL_VERSION_PATCH}
#define STXXL_VERSION_STRING "${STXXL_VERSION_STRING}"
#define STXXL_VERSION_PHASE "${STXXL_VERSION_PHASE}"

// if this is a git repository, add the refspec and commit sha
#cmakedefine STXXL_VERSION_GIT_REFSPEC "${STXXL_VERSION_GIT_REFSPEC}"
#cmakedefine STXXL_VERSION_GIT_SHA1 "${STXXL_VERSION_GIT_SHA1}"

#cmakedefine STXXL_DIRECT_IO_OFF ${STXXL_DIRECT_IO_OFF}
// default: 0/1 (platform dependent)
// cmake:   detection of platform and flag
// effect:  disables use of O_DIRECT flag on unsupported platforms

#cmakedefine STXXL_HAVE_MMAP_FILE ${STXXL_HAVE_MMAP_FILE}
// default: 0/1 (platform dependent)
// used in: io/mmap_file.h/cpp
// effect:  enables/disables memory mapped file implementation

#cmakedefine STXXL_HAVE_LINUXAIO_FILE ${STXXL_HAVE_LINUXAIO_FILE}
// default: 0/1 (platform dependent)
// used in: io/linuxaio_file.h/cpp
// effect:  enables/disables Linux AIO file implementation

#cmakedefine STXXL_POSIX_THREADS ${STXXL_POSIX_THREADS}
// default: off
// cmake:   detection of pthreads by cmake
// effect:  uses POSIX thread and mutexes on linux

#cmakedefine STXXL_STD_THREADS ${STXXL_STD_THREADS}
// default: off
// cmake:   detection of <thread> and <mutex>
// effect:  uses std thread and mutex on windows or (forced on linux)

#cmakedefine STXXL_WINDOWS ${STXXL_WINDOWS}
// default: off
// cmake:   detection of ms windows platform (32- or 64-bit)
// effect:  enables windows-specific api calls (mingw or msvc)

#cmakedefine STXXL_MSVC ${STXXL_MSVC}
// default: off
// cmake:   detection of ms visual c++ via CMake (contains version number)
// effect:  enables msvc-specific headers and macros

#cmakedefine STXXL_HAVE_CXX11_RANGE_FOR_LOOP ${STXXL_HAVE_CXX11_RANGE_FOR_LOOP}
// default: off
// run-time: detection C++11 support for "for (auto i : obj) { }"
// effect:  enables some C++11 construct (currently only allowed in examples)

#cmakedefine STXXL_HAVE_SYNC_ADD_AND_FETCH ${STXXL_HAVE_SYNC_ADD_AND_FETCH}
// default: off
// cmake:   detection of __sync_add_and_fetch() intrinsic
// effect:  enables use of atomics in counting_ptr

#cmakedefine STXXL_PARALLEL_MODE_EXPLICIT ${STXXL_PARALLEL_MODE_EXPLICIT}
// default: off
// cmake:   -DUSE_GNU_PARALLEL=ON
// effect:  explicitly enables use of __gnu_parallel algorithms

#cmakedefine STXXL_BOOST_CONFIG ${STXXL_BOOST_CONFIG}
#cmakedefine STXXL_BOOST_FILESYSTEM ${STXXL_BOOST_FILESYSTEM}
#cmakedefine STXXL_BOOST_RANDOM ${STXXL_BOOST_RANDOM}
#cmakedefine STXXL_BOOST_THREADS ${STXXL_BOOST_THREADS}
#cmakedefine STXXL_BOOST_TIMESTAMP ${STXXL_BOOST_TIMESTAMP}
// default: off
// cmake:   -DUSE_BOOST=ON
// effect:  enables use of boost libraries in different parts of STXXL.

#if STXXL_BOOST_CONFIG
  #include <boost/config.hpp>
#endif

#cmakedefine STXXL_STD_RANDOM ${STXXL_STD_RANDOM}
// default: off
// cmake:   detection of <random>
// effect:  uses std random generator on windows or (forced on linux)

#cmakedefine STXXL_HAVE_MALLINFO_PROTO ${STXXL_HAVE_MALLINFO_PROTO}
// default: off
// cmake:   detection of mallinfo() function in <malloc.h>
// effect:  used by stxxl_tool/mallinfo for malloc stats

#cmakedefine STXXL_HAVE_MLOCK_PROTO ${STXXL_HAVE_MLOCK_PROTO}
// default: off
// cmake:   detection of mlock() function in <sys/mman.h>
// effect:  used by stxxl_tool/mlock for locking physical pages

#cmakedefine STXXL_WITH_VALGRIND ${STXXL_WITH_VALGRIND}
// default: off
// cmake:   option USE_VALGRIND=ON
// effect:  run all tests with valgrind and pre-initialize some memory buffers

#endif // !STXXL_CONFIG_HEADER

It was originally called config.h.in but when it was like that, I got the error that it could not find a file called config.h so I changed the name (maybe that was a mistake?)

This is one of the files that throws the "Thread implementation not detected error":

/***************************************************************************
 *  include/stxxl/bits/common/mutex.h
 *
 *  Part of the STXXL. See http://stxxl.sourceforge.net
 *
 *  Copyright (C) 2002 Roman Dementiev <dementiev@mpi-sb.mpg.de>
 *  Copyright (C) 2008 Andreas Beckmann <beckmann@cs.uni-frankfurt.de>
 *  Copyright (C) 2013 Timo Bingmann <tb@panthema.net>
 *
 *  Distributed under the Boost Software License, Version 1.0.
 *  (See accompanying file LICENSE_1_0.txt or copy at
 *  http://www.boost.org/LICENSE_1_0.txt)
 **************************************************************************/

#ifndef STXXL_COMMON_MUTEX_HEADER
#define STXXL_COMMON_MUTEX_HEADER

#include <stxxl/bits/config.h>
#include <stxxl/bits/namespace.h>

#if STXXL_STD_THREADS
 #include <mutex>
#elif STXXL_BOOST_THREADS
 #include <boost/thread/mutex.hpp>
#elif STXXL_POSIX_THREADS
 #include <pthread.h>

 #include <stxxl/bits/noncopyable.h>
 #include <stxxl/bits/common/error_handling.h>
#else
 #error "Thread implementation not detected."
#endif

STXXL_BEGIN_NAMESPACE

#if STXXL_STD_THREADS

typedef std::mutex mutex;

#elif STXXL_BOOST_THREADS

typedef boost::mutex mutex;

#elif STXXL_POSIX_THREADS

class mutex : private noncopyable
{
    //! mutex handle
    pthread_mutex_t m_mutex;

public:
    //! construct unlocked mutex
    mutex()
    {
        STXXL_CHECK_PTHREAD_CALL(pthread_mutex_init(&m_mutex, NULL));
    }
    //! destroy mutex handle
    ~mutex()
    {
        // try simple delete first
        int res = pthread_mutex_destroy(&m_mutex);
        if (res == 0) return;

        // try to lock and unlock mutex
        res = pthread_mutex_trylock(&m_mutex);

        if (res == 0 || res == EBUSY) {
            STXXL_CHECK_PTHREAD_CALL(pthread_mutex_unlock(&m_mutex));
        } else {
            STXXL_THROW_ERRNO2(resource_error, "pthread_mutex_trylock() failed", res);
        }

        STXXL_CHECK_PTHREAD_CALL(pthread_mutex_destroy(&m_mutex));
    }
    //! lock mutex, may block
    void lock()
    {
        STXXL_CHECK_PTHREAD_CALL(pthread_mutex_lock(&m_mutex));
    }
    //! unlock mutex
    void unlock()
    {
        STXXL_CHECK_PTHREAD_CALL(pthread_mutex_unlock(&m_mutex));
    }
    //! return platform specific handle
    pthread_mutex_t & native_handle()
    {
        return m_mutex;
    }
};

#endif

#if STXXL_STD_THREADS

typedef std::unique_lock<std::mutex> scoped_mutex_lock;

#elif STXXL_BOOST_THREADS

typedef boost::mutex::scoped_lock scoped_mutex_lock;

#else

//! Aquire a lock that's valid until the end of scope.
class scoped_mutex_lock
{
    //! mutex reference
    mutex& m_mutex;

    //! marker if already unlocked by this thread (needs no synchronization)
    bool is_locked;

public:
    //! lock mutex
    scoped_mutex_lock(mutex& m)
        : m_mutex(m), is_locked(true)
    {
        m_mutex.lock();
    }
    //! unlock mutex hold when object goes out of scope.
    ~scoped_mutex_lock()
    {
        unlock();
    }
    //! unlock mutex hold prematurely
    void unlock()
    {
        if (is_locked) {
            is_locked = false;
            m_mutex.unlock();
        }
    }
    //! return platform specific handle
    pthread_mutex_t & native_handle()
    {
        return m_mutex.native_handle();
    }
};

#endif

STXXL_END_NAMESPACE

#endif // !STXXL_COMMON_MUTEX_HEADER

And just for reference here is the sample test program I'm trying to compile:

#include "mainwindow.h"
#include <QApplication>

/*int main(int argc, char *argv[])
{
    QApplication a(argc, argv);
    MainWindow w;
    w.show();

    return a.exec();

}*/

#include <iostream>
#include <limits>

#include <stxxl/vector>
#include <stxxl/random>
#include <stxxl/sort>

struct my_less_int : std::less<int>
{
    int min_value() const { return std::numeric_limits<int>::min(); }
    int max_value() const { return std::numeric_limits<int>::max(); }
};

int main()
{
    // create vector
    stxxl::VECTOR_GENERATOR<int>::result vector;

    // fill vector with random integers
    {
        stxxl::scoped_print_timer
            timer("write random numbers", 100 * 1024 * 1024 * sizeof(int));

        stxxl::random_number32 random;

        for (size_t i = 0; i < 100 * 1024 * 1024; ++i) {
            vector.push_back(random());
        }
    }

    // sort vector using 16 MiB RAM
    {
        stxxl::scoped_print_timer
            timer("sorting random numbers", 100 * 1024 * 1024 * sizeof(int));

        stxxl::sort(vector.begin(), vector.end(), my_less_int(), 16 * 1024 * 1024);
    }

    // output first and last items:
    std::cout << vector.size() << " items sorted ranging from "
              << vector.front() << " to " << vector.back() << std::endl;

    return 0;
}

Oh and maybe it's useful to know how my .pro file looks:

#-------------------------------------------------
#
# Project created by QtCreator 2016-02-11T18:45:57
#
#-------------------------------------------------

QT       += core gui

greaterThan(QT_MAJOR_VERSION, 4): QT += widgets

TARGET = TestProject
TEMPLATE = app


SOURCES += main.cpp\
        mainwindow.cpp

HEADERS  += mainwindow.h

FORMS    += mainwindow.ui

win32:CONFIG(release, debug|release): LIBS += -L$$PWD/../../stxxl-1.4.1/build/lib/release/ -lstxxl
else:win32:CONFIG(debug, debug|release): LIBS += -L$$PWD/../../stxxl-1.4.1/build/lib/debug/ -lstxxl
else:unix: LIBS += -L$$PWD/../../stxxl-1.4.1/build/lib/ -lstxxl

INCLUDEPATH += $$PWD/../../stxxl-1.4.1/include
DEPENDPATH += $$PWD/../../stxxl-1.4.1/include

SGaist

Yes that's a mistake, config.h.in is a template that will be processed by CMake to generate config.h that is likely somewhere in the build folder of stxxl

Multimodcrafter

Ah that makes sense! I found the actual configfile and the errors are reduced down to two instances of:

D:\Coding\c++\stxxl-1.4.1\include\stxxl\bits\common\counting_ptr.h:417: Error: '_InterlockedIncrement' was not declared in this scope
_InterlockedIncrement(&m_reference_count);
^

They appear in the file "counting_ptr.h" (I only pasted the relevant part):

public:
    //! Call whenever setting a pointer to the object
    void inc_reference() const
    {
#if STXXL_MSVC
        _InterlockedIncrement(&m_reference_count);
#else
        __sync_add_and_fetch(&m_reference_count, +1);
#endif
    }

    //! Call whenever resetting (i.e. overwriting) a pointer to the object.
    //! IMPORTANT: In case of self-assignment, call AFTER inc_reference().
    //! \return if the object has to be deleted (i.e. if it's reference count dropped to zero)
    bool dec_reference() const
    {
#if STXXL_MSVC
        return (_InterlockedDecrement(&m_reference_count) == 0);
#else
        return (__sync_add_and_fetch(&m_reference_count, -1) == 0);
#endif
    }

SGaist

That's when using your MinGW Qt build ?

Multimodcrafter

Yes! Desktop QT 5.5.1 MinGW 32bit

SGaist

You can't mix and match C++ libraries built with different compilers on Windows. Either build stxxl with MinGW or install Qt for the same Visual Studio version you used to build stxxl.

Note that it's not specific to MinGW vs Visual Studio nor Qt. Visual Studio compilers are not compatible one with the other so you have to build everything with the same compiler.

Multimodcrafter

Aww... * sadface * gotta figure out how to build it with mingw then...

SGaist

Is using MinGW mandatory ? If not and since you already have stxxl built with Visual Studio, why not use it ? You don't have to use the IDE at all.

Multimodcrafter

You mean I can use Visual Studios compiler inside the QT creator?
Because I really don't want to use visual studios IDE if avoidable since IntelliSense is awfully slow!

SGaist

That's the whole goal of Qt Creator: use it on all supported platform. Qt Creator is an IDE and will use the compiler matching the Qt you installed (provided you installed said compiler)