| Trolltech Home | Qt-interest Home | Recent Threads | All Threads | Author | Date | |
| All threads index page 1 | |
Hi! I uses QXmlStreamWriter, and storing huge files. I store them in a QByteArray which in the stream writer uses QBuffer internally. The problem is that the QByteArray doesnt manage to grow big enough. This is only reported by a small qDebug message in qbuffer.cpp. How I'm I supposed to know there has been an error? If I use a QFile instead I can check the state of the QFile object later on, but for the QBuffer there is no such methods. Another question, what is the size restricions on QByteArray? Regards Knut -- [ signature omitted ]
Knut Finstad wrote: >Another question, what is the size restricions on QByteArray? Available memory or 2 GB, whichever comes first. (QByteArray stores the sizes in a signed 32-bit integer) -- [ signature omitted ]
Attachment:
signature.asc
Description: This is a digitally signed message part.
Hm, I get problem after ca 130 MB. (On a new laptop). Strange... On 3/4/08, Thiago Macieira <thiago.macieira@xxxxxxxxxxxxx> wrote: > Knut Finstad wrote: > >Another question, what is the size restricions on QByteArray? > > > Available memory or 2 GB, whichever comes first. (QByteArray stores the > sizes in a signed 32-bit integer) > > > -- > Thiago José Macieira - thiago.macieira AT trolltech.com > Trolltech ASA - Sandakerveien 116, NO-0402 Oslo, Norway > > -- [ signature omitted ]
Hi, Which "small qDebug message in qbuffer.cpp" are you referring to? > Hm, I get problem after ca 130 MB. (On a new laptop). Strange... After reading the first 130 MB of the file or after QByteArray is 130MB large? -- [ signature omitted ]
Hi!
> Which "small qDebug message in qbuffer.cpp" are you referring to?
>
Sorry, its a qWarning message: See qbuffer.cpp, line 411:
qWarning("QBuffer::writeData: Memory allocation error");
>
> > Hm, I get problem after ca 130 MB. (On a new laptop). Strange...
>
>
> After reading the first 130 MB of the file or after QByteArray is 130MB large?
When the qbuffer tryes to resize is buffer to about 130 MB
(134225172). It's not reading a file, but storing an xml file (usin
QXmlStreamWriter) temporary before saving it in a big chunk throug
QtIOCompressor to a QFile.
There is a (reported) bug in QtIOCompressor when writing empty
strings, so I had problem using it directrly. Now when i have found
the bug in QtIOCompressor I will patch it and use it directrly. Still
I find the size of QBuffer (QByteArray) restricting.
Regards Knut
>
> --
> Dimitri
>
>
> --
> To unsubscribe - send a mail to qt-interest-request@xxxxxxxxxxxxx with "unsubscribe" in the subject or the body.
> List archive and information: http://lists.trolltech.com/qt-interest/
>
>
--
[ signature omitted ]
On Thursday 06 March 2008 10:07:37 Knut Finstad wrote:
> Sorry, its a qWarning message: See qbuffer.cpp, line 411:
> qWarning("QBuffer::writeData: Memory allocation error");
>
> > > Hm, I get problem after ca 130 MB. (On a new laptop). Strange...
> >
> >
> > After reading the first 130 MB of the file or after QByteArray is 130MB
> > large?
>
> When the qbuffer tryes to resize is buffer to about 130 MB
> (134225172). It's not reading a file, but storing an xml file (usin
> QXmlStreamWriter) temporary before saving it in a big chunk throug
> QtIOCompressor to a QFile.
>
> There is a (reported) bug in QtIOCompressor when writing empty
> strings, so I had problem using it directrly. Now when i have found
> the bug in QtIOCompressor I will patch it and use it directrly. Still
> I find the size of QBuffer (QByteArray) restricting.
Like I said, the restriction is 2 GB. You're probably running out of memory.
Notice that you may need in some situations twice the size of your buffer, due
to reallocating and copying. That is to say, QBuffer would be extremely slow
for that operation.
I recommend writing directly to the QIODevice, so that it's compressed and
written to disk as soon as possible, without taking memory.
--
[ signature omitted ]
Attachment:
signature.asc
Description: This is a digitally signed message part.
Yes I do that now as I have removed the bug in QtIOCompressor.
But still the issue of knowing from inside the program that an io
operation has gone wrong whne working through a QIODevice is still no
solved.
regards Knut
2008/3/6, Thiago Macieira <thiago.macieira@xxxxxxxxxxxxx>:
> On Thursday 06 March 2008 10:07:37 Knut Finstad wrote:
> > Sorry, its a qWarning message: See qbuffer.cpp, line 411:
> > qWarning("QBuffer::writeData: Memory allocation error");
> >
> > > > Hm, I get problem after ca 130 MB. (On a new laptop). Strange...
> > >
> > >
> > > After reading the first 130 MB of the file or after QByteArray is 130MB
> > > large?
> >
> > When the qbuffer tryes to resize is buffer to about 130 MB
> > (134225172). It's not reading a file, but storing an xml file (usin
> > QXmlStreamWriter) temporary before saving it in a big chunk throug
> > QtIOCompressor to a QFile.
> >
> > There is a (reported) bug in QtIOCompressor when writing empty
> > strings, so I had problem using it directrly. Now when i have found
> > the bug in QtIOCompressor I will patch it and use it directrly. Still
> > I find the size of QBuffer (QByteArray) restricting.
>
> Like I said, the restriction is 2 GB. You're probably running out of memory.
>
> Notice that you may need in some situations twice the size of your buffer,
> due
> to reallocating and copying. That is to say, QBuffer would be extremely slow
> for that operation.
>
> I recommend writing directly to the QIODevice, so that it's compressed and
> written to disk as soon as possible, without taking memory.
>
> --
> Thiago José Macieira - thiago.macieira AT trolltech.com
> Trolltech ASA - Sandakerveien 116, NO-0402 Oslo, Norway
>
--
[ signature omitted ]
Hi,
> Sorry, its a qWarning message: See qbuffer.cpp, line 411:
> qWarning("QBuffer::writeData: Memory allocation error");
Ah, I see. It's in QBuffer::writeDate() which returns -1 if an error occurred:
http://doc.trolltech.com/4.3/qiodevice.html#writeData
Which exact function of the Qt API are you calling from your code? Can you
show some sample code?
--
[ signature omitted ]
Hi!
I call it (wery) indirectly through QXmlStreamWrtier::writeXxx
functions. There is no way the -1 return may be read by caller of an
QXmlStreamWritre method(?)
Regards Knut
On 3/6/08, Dimitri <dimitri@xxxxxxxxxxxxx> wrote:
> Hi,
>
>
> > Sorry, its a qWarning message: See qbuffer.cpp, line 411:
> > qWarning("QBuffer::writeData: Memory allocation error");
>
>
> Ah, I see. It's in QBuffer::writeDate() which returns -1 if an error occurred:
> http://doc.trolltech.com/4.3/qiodevice.html#writeData
>
> Which exact function of the Qt API are you calling from your code? Can you
> show some sample code?
>
>
> --
> Dimitri
>
> --
> To unsubscribe - send a mail to qt-interest-request@xxxxxxxxxxxxx with "unsubscribe" in the subject or the body.
> List archive and information: http://lists.trolltech.com/qt-interest/
>
>
--
[ signature omitted ]
Hi, > I call it (wery) indirectly through QXmlStreamWrtier::writeXxx > functions. There is no way the -1 return may be read by caller of an > QXmlStreamWritre method(?) I see. Indeed QXmlStreamWriter::write* methods return void. Unfortunately this cannot be changed in Qt 4. There are other solutions, such as adding a QXmlStreamWriter method that could return an error status in Qt 4.5. I suggest you file a request to add a way to recover the error status: http://trolltech.com/bugreport-form In the meantime you can access the underlying device using QXmlStreamWriter::device(): http://doc.trolltech.com/4.3/qxmlstreamwriter.html#device Then you might inherit QBuffer and override writeDate() so that it logs errors. Also QIODevice::errorString() might return something useful - but I haven't actually tried: http://doc.trolltech.com/4.3/qiodevice.html#errorString -- [ signature omitted ]
Inheriting QBuffer is a good suggestion, even better if you could provide a QIODevice method for asking if ny error has occured. I tried the errorString method, and it returns Unknown error by default also when there is no error, and there is no way to know if there has been any... I will submit a feature request. Thanks for all help! Regards Knut 2008/3/8, Dimitri <dimitri@xxxxxxxxxxxxx>: > Hi, > > > I call it (wery) indirectly through QXmlStreamWrtier::writeXxx > > functions. There is no way the -1 return may be read by caller of an > > QXmlStreamWritre method(?) > > I see. Indeed QXmlStreamWriter::write* methods return void. Unfortunately > this > cannot be changed in Qt 4. There are other solutions, such as adding a > QXmlStreamWriter method that could return an error status in Qt 4.5. I > suggest > you file a request to add a way to recover the error status: > http://trolltech.com/bugreport-form > > In the meantime you can access the underlying device using > QXmlStreamWriter::device(): > http://doc.trolltech.com/4.3/qxmlstreamwriter.html#device > > Then you might inherit QBuffer and override writeDate() so that it logs > errors. > > Also QIODevice::errorString() might return something useful - but I haven't > actually tried: > http://doc.trolltech.com/4.3/qiodevice.html#errorString > > > -- > Dimitri > > -- > To unsubscribe - send a mail to qt-interest-request@xxxxxxxxxxxxx with > "unsubscribe" in the subject or the body. > List archive and information: http://lists.trolltech.com/qt-interest/ > > -- [ signature omitted ]