Trolltech Home | Qt-interest Home | Recent Threads | All Threads | Author | Date
All threads index page 3

Qt-interest Archive, April 2008
QTextStream/QString threads issues

Message 1 in thread

I am getting random errors in a multithreaded application related to 
runtime logging. Logging is done through QTextStream instances:

QTextStream qcerr (stderr, QIODevice::WriteOnly |QIODevice::Unbuffered);

The location where the error occurs looks "random" (different place 
everytime), but the backtrace always starts from places in my code that 
output to the qcerr using the << operator. The backtrace looks as follows:

> (gdb) bt
> #0  0x00007f9d47de5095 in raise () from /lib/libc.so.6
> #1  0x00007f9d47de6af0 in abort () from /lib/libc.so.6
> #2  0x00007f9d47e1fa7b in ?? () from /lib/libc.so.6
> #3  0x00007f9d47e28e02 in ?? () from /lib/libc.so.6
> #4  0x00007f9d47e2aea0 in realloc () from /lib/libc.so.6
> #5  0x00007f9d4962107d in qRealloc (ptr=0x7f9d40000e10, size=256) at global/qmalloc.cpp:67
> #6  0x00007f9d496787cb in QString::realloc (this=0x661808, alloc=112) at tools/qstring.cpp:1060
> #7  0x00007f9d4967a231 in QString::append (this=0x661808, str=@0x422a90d0) at tools/qstring.cpp:1276
> #8  0x00007f9d49620a0b in QString::operator+= (this=0x661808, s=@0x422a90d0) at ../../include/QtCore/../../src/corelib/tools/qstring.h:261
> #9  0x00007f9d496d5bb0 in QTextStreamPrivate::write (this=0x661780, data=@0x422a90d0) at io/qtextstream.cpp:830
> #10 0x00007f9d496d5f9d in QTextStreamPrivate::putString (this=0x661780, s=@0x422a9160, number=true) at io/qtextstream.cpp:909
> #11 0x00007f9d496d0dd2 in QTextStreamPrivate::putNumber (this=0x661780, number=11581, negative=false) at io/qtextstream.cpp:2214
> #12 0x00007f9d496d124b in QTextStream::operator<< (this=0x64c860, i=11581) at io/qtextstream.cpp:2291
> #13 0x000000000042c5f3 in DSTWSThread::addSymbol (this=0x776400, fullname=@0x7f9d40000a90) at DSTWSThread.cpp:199

DSTWSThread is code from my application, and the actual code is:

   LOGDEBUG << "got addSymbol on tid " << gettid () << " " << fullname 
<< endl;

LOGDEBUG is a simple macro that is the qcerr QTextStream as shown 
earlier in this message (and a noop when debugging is turned off).

The actual error message output from glibc is:

*** glibc detected *** /home/marius/p/tickbt/ibbot/ibbot: realloc(): 
invalid next size: 0x00007f9d40000e10 ***

The code uses qPrintable some places to generate C strings from QString 
instances, but I haven't looked into that macro how it locates it's 
buffers and similar to see if that has any threads issues to it.

If I turn debugging off for the DSTWSThread instance (a QThread 
inherited threads class) the application runs just fine.

Any ideas before I start digging?

Thanks,

Marius K.

--
 [ signature omitted ]
Message 2 in thread

On Sunday 13 April 2008 22:06:08 Marius Kjeldahl wrote:
> I am getting random errors in a multithreaded application related to
> runtime logging. Logging is done through QTextStream instances:
>
> QTextStream qcerr (stderr, QIODevice::WriteOnly |QIODevice::Unbuffered);
>
> The location where the error occurs looks "random" (different place
> everytime), but the backtrace always starts from places in my code that
>
> output to the qcerr using the << operator. The backtrace looks as follows:
> > (gdb) bt
> > #0  0x00007f9d47de5095 in raise () from /lib/libc.so.6
> > #1  0x00007f9d47de6af0 in abort () from /lib/libc.so.6
> > #2  0x00007f9d47e1fa7b in ?? () from /lib/libc.so.6
> > #3  0x00007f9d47e28e02 in ?? () from /lib/libc.so.6
> > #4  0x00007f9d47e2aea0 in realloc () from /lib/libc.so.6
> > #5  0x00007f9d4962107d in qRealloc (ptr=0x7f9d40000e10, size=256) at
> > global/qmalloc.cpp:67 #6  0x00007f9d496787cb in QString::realloc
> > (this=0x661808, alloc=112) at tools/qstring.cpp:1060 #7 
> > 0x00007f9d4967a231 in QString::append (this=0x661808, str=@0x422a90d0) at
> > tools/qstring.cpp:1276 #8  0x00007f9d49620a0b in QString::operator+=
    ^^^^^^
> > (this=0x661808, s=@0x422a90d0) at
> > ../../include/QtCore/../../src/corelib/tools/qstring.h:261 #9 
> > 0x00007f9d496d5bb0 in QTextStreamPrivate::write (this=0x661780,
> > data=@0x422a90d0) at io/qtextstream.cpp:830 #10 0x00007f9d496d5f9d in
> > QTextStreamPrivate::putString (this=0x661780, s=@0x422a9160, number=true)
> > at io/qtextstream.cpp:909 #11 0x00007f9d496d0dd2 in
> > QTextStreamPrivate::putNumber (this=0x661780, number=11581,
> > negative=false) at io/qtextstream.cpp:2214 #12 0x00007f9d496d124b in
> > QTextStream::operator<< (this=0x64c860, i=11581) at
> > io/qtextstream.cpp:2291 #13 0x000000000042c5f3 in DSTWSThread::addSymbol
> > (this=0x776400, fullname=@0x7f9d40000a90) at DSTWSThread.cpp:199

It looks very much looks like QTextStream is using a QString buffer, which is 
occasionally resized. That is not going to be threadsafe, you'll need to use 
some kind of lock to get around this. Or use a threadsafe stream.

Remember, in threads *every memory location that is ever modified in any 
thread and accessed in any other thread needs to be synchronized*. A common 
stream definitely falls under this rule. This is why multithread programming 
is harder than multiprocess, where the OS protects you from many of these 
errors.

I do wonder that this is the used method. I'd think, perhaps naively, that a 
fragmented approach would be more efficient that resizing. No doubt, the 
trolls have their reasons. Would make it more resistant to race conditions, 
but by no means immune.

>
> Any ideas before I start digging?

That was my idea. Valgrind has a thread debugging tool, if that helps. Only 
linux, of course.

-- 
 [ signature omitted ]
Message 3 in thread

Esben Mose Hansen wrote:
> It looks very much looks like QTextStream is using a QString buffer, which is 
> occasionally resized. That is not going to be threadsafe, you'll need to use 
> some kind of lock to get around this. Or use a threadsafe stream.

Yes, looks like you are correct. This is what the threads documentation 
in Qt says:

"Most Qt classes are reentrant and not thread-safe, to avoid the 
overhead of repeatedly locking and unlocking a QMutex. For example, 
QString is reentrant, meaning that you can use it in different threads, 
but you can't access the same QString object from different threads 
simultaneously (unless you protect it with a mutex yourself). A few 
classes and functions are thread-safe; these are mainly thread-related 
classes such as QMutex, or fundamental functions such as 
QCoreApplication::postEvent()."

Assuming that Qt does not have any thread-safe stream classes, I'll add 
a mutex to my logging macros.

Thanks,

Marius K.

--
 [ signature omitted ]
Message 4 in thread

Marius Kjeldahl wrote:
>I am getting random errors in a multithreaded application related to
>runtime logging. Logging is done through QTextStream instances:
[snip]
>Any ideas before I start digging?

If you look at the documentation of QTextStream 
(http://doc.trolltech.com/4.3/qtextstream.html), it says:

"Note: All the functions in this class are reentrant."

That means QTextStream is not thread-safe.

Unless you're willing to stop using threads, your only option is to 
implement a locking mechanism using QMutex/QMutexLocker. One option is to 
replace QTextStream with your own locking class.

Another is to protect each use of qcerr with a mutex locking. In fact, you 
can probably use your LOGDEBUG macro for that.

Something like:

class DebugLocker
{
    QMutexLocker locker;
public:
    QTextStream &stream;
    DebugLocker(QTextStream &s, QMutex *mutex)
        : locker(mutex), stream(s)
    { }
};

#define LOGDEBUG    DebugLocker(qcerr, qcerr_mutex).stream

Since that creates a DebugLocker temporary instance, it'll be deleted at 
the next statement, which means that the mutex is locked at the beginning 
and unlocked at the end.

You'll have to be careful with what you call in the << calls though. 
They'll be called with a locked mutex.

Another alternative is to use some template magic:

class LockingTextStream
{
    QMutex *mutex;
    QTextStream &stream;
public:
    LockingTextStream(QTextStream &s, QMutex *m)
        : mutex(m), stream(s)
    { }

    template<typename T> inline
    LockingTextStream& operator<<(const T &t)
    {
        mutex->lock();
        stream << t;
        mutex->unlock();
    }
};
#define LOGDEBUG   LockingTextStream(qcerr, qcerr_mutex)
-- 
 [ signature omitted ]

Attachment: signature.asc
Description: This is a digitally signed message part.

Message 5 in thread

Thiago Macieira wrote:

> If you look at the documentation of QTextStream
> (http://doc.trolltech.com/4.3/qtextstream.html), it says:
> 
> "Note: All the functions in this class are reentrant."
> 
> That means QTextStream is not thread-safe.

IMHO it means QTextStream is completely thread safe.
See: http://en.wikipedia.org/wiki/Reentrant

Uwe

--
 [ signature omitted ]
Message 6 in thread

Uwe Rathmann wrote:
> IMHO it means QTextStream is completely thread safe.
> See: http://en.wikipedia.org/wiki/Reentrant
>   


There is debate on the exact meaning of the terms "thread safe" and 
"reentrant." For that reason, the Trolls carefully worded their 
documentation:

From http://doc.trolltech.com/4.3/threads.html#reentrancy-and-thread-safety

 * A reentrant function can be called simultaneously by multiple threads 
provided that each invocation of the function references unique data.
 * A thread-safe function can be called simultaneously by multiple 
threads when each invocation references shared data. All access to the 
shared data is serialized.


According to this definition, a Qt class which is reentrant is not 
necessarily thread safe.

--Dave

--
 [ signature omitted ]
Message 7 in thread

Dave Smith wrote:

> According to this definition, a Qt class which is reentrant is not
> necessarily thread safe.

I see, but at least the conclusion, that a class is not thread safe, because
it is reentrant, is not valid. 

Or is this a hidden message of the docs, when a class is described as
reentrant, but not explicitely as thread safe ?

Uwe

--
 [ signature omitted ]
Message 8 in thread

Am Montag, 14. April 2008 schrieb Uwe Rathmann:
> Dave Smith wrote:
> > According to this definition, a Qt class which is reentrant is not
> > necessarily thread safe.
> I see, but at least the conclusion, that a class is not thread safe,
> because it is reentrant, is not valid.
> Or is this a hidden message of the docs, when a class is described as
> reentrant, but not explicitely as thread safe ?

That is exactly what the trolls mean by their definition of reentrant vs. 
thread safe in 
http://doc.trolltech.com/4.3/threads.html#reentrancy-and-thread-safety

Arnold
-- 
 [ signature omitted ]

Attachment: signature.asc
Description: This is a digitally signed message part.

Message 9 in thread

Uwe Rathmann wrote:
>Dave Smith wrote:
>> According to this definition, a Qt class which is reentrant is not
>> necessarily thread safe.
>
>I see, but at least the conclusion, that a class is not thread safe,
> because it is reentrant, is not valid.
>
>Or is this a hidden message of the docs, when a class is described as
>reentrant, but not explicitely as thread safe ?

You're right, from the definition of the terms, one thing doesn't exclude 
the other.

But, from our documentation guidelines, it does. You'll never find a class 
that claims to be reentrant and thread-safe. So if it says it is 
reentrant, it is not thread-safe for the same, shared data.

-- 
 [ signature omitted ]

Attachment: signature.asc
Description: This is a digitally signed message part.

Message 10 in thread

Thanks Thiago,

Your template magic version seems to fit the bill just fine. You forgot 
the return value from the operator<< overload however, so I'm reposting 
your code with it here in case anybody else wants to try it:

class LockingTextStream
{
     QMutex *mutex;
     QTextStream &stream;
public:
     LockingTextStream(QTextStream &s, QMutex *m)
         : mutex(m), stream(s)
     { }

     template<typename T> inline
     LockingTextStream& operator<<(const T &t)
     {
         mutex->lock();
         stream << t;
         mutex->unlock();
         return *this;
     }
};

Regards,

Marius K.

Thiago Macieira wrote:
> Marius Kjeldahl wrote:
>> I am getting random errors in a multithreaded application related to
>> runtime logging. Logging is done through QTextStream instances:
> [snip]
>> Any ideas before I start digging?
> 
> If you look at the documentation of QTextStream 
> (http://doc.trolltech.com/4.3/qtextstream.html), it says:
> 
> "Note: All the functions in this class are reentrant."
> 
> That means QTextStream is not thread-safe.
> 
> Unless you're willing to stop using threads, your only option is to 
> implement a locking mechanism using QMutex/QMutexLocker. One option is to 
> replace QTextStream with your own locking class.
> 
> Another is to protect each use of qcerr with a mutex locking. In fact, you 
> can probably use your LOGDEBUG macro for that.
> 
> Something like:
> 
> class DebugLocker
> {
>     QMutexLocker locker;
> public:
>     QTextStream &stream;
>     DebugLocker(QTextStream &s, QMutex *mutex)
>         : locker(mutex), stream(s)
>     { }
> };
> 
> #define LOGDEBUG    DebugLocker(qcerr, qcerr_mutex).stream
> 
> Since that creates a DebugLocker temporary instance, it'll be deleted at 
> the next statement, which means that the mutex is locked at the beginning 
> and unlocked at the end.
> 
> You'll have to be careful with what you call in the << calls though. 
> They'll be called with a locked mutex.
> 
> Another alternative is to use some template magic:
> 
> class LockingTextStream
> {
>     QMutex *mutex;
>     QTextStream &stream;
> public:
>     LockingTextStream(QTextStream &s, QMutex *m)
>         : mutex(m), stream(s)
>     { }
> 
>     template<typename T> inline
>     LockingTextStream& operator<<(const T &t)
>     {
>         mutex->lock();
>         stream << t;
>         mutex->unlock();
>     }
> };
> #define LOGDEBUG   LockingTextStream(qcerr, qcerr_mutex)

--
 [ signature omitted ]