Qt-interest Archive, June 2007
random access to QDataStream data
Message 1 in thread
Hello!
I need to store huge matrixes with qint32 values.
It is impossible to load them into memory.
So i'm using memory mapped files.
Values to files i store using QDataStream.
Then i map these files to memory and need to get values with random
positions.
Currently i'm doing so:
char * p - pointer to memory of mapped file
row, col - coordinates of matrix cell, i need value for
m_ncols - number of columns in matrix
p += (row * m_ncols + col) * 4;
QByteArray ba = QByteArray::fromRawData(p,4);
QDataStream ds(&ba, QIODevice::ReadOnly);
qint32 result;
ds >> result;
return result;
but it works slowly because i need to create QDataStream and
QByteArray each time, i get value.
I tried to create QDataStream once and read all values sequentially.
It works much much more faster! But as i understand i can't move
position of QDataStream backward. So fo random access i can't create
QDataStream only once for entire matrix. Is it possible somehow to
optimize algorithm, which i use now?
--
[ signature omitted ]
Message 2 in thread
Why use a datastream here at all? why not just access the data pointed
to, cast to qint32 and use it?
result = *reinterpret_cast<qint32>(p);
Cheers,
Peter
> -----Original Message-----
>
> p += (row * m_ncols + col) * 4;
> QByteArray ba = QByteArray::fromRawData(p,4);
> QDataStream ds(&ba, QIODevice::ReadOnly);
> qint32 result;
> ds >> result;
> return result;
>
> but it works slowly because i need to create QDataStream and
> QByteArray each time, i get value.
>
> --
> Sergey mailto:mb0@xxxxx
--
[ signature omitted ]
Message 3 in thread
On 6/11/07, Peter Prade <prade@xxxxxxxxxxx> wrote:
>
> Why use a datastream here at all? why not just access the data pointed
> to, cast to qint32 and use it?
>
> result = *reinterpret_cast<qint32>(p);
>
This is not portable and does not handle endianness. Besides, when storing
data with QDataStream the data is encoded and to make sure you read it
correctly you should read it with QDataStream as well, otherwise you start
depending on implementation details. IMHO, if one wants to read file
manually, it should write it manually by himself, knowing the exact file
format.
--
[ signature omitted ]
Message 4 in thread
> I need to store huge matrixes with qint32 values.
> It is impossible to load them into memory.
> So i'm using memory mapped files.
> Values to files i store using QDataStream.
> Then i map these files to memory and need to get values with random
> positions.
Sounds like an application for a lightweight database for me. SQLite
should come in handy here.
create table Matrix (
row integer,
col integer,
value integer,
primary key(row, col)
);
Should give you all the access and performance you need.
Martin
--
[ signature omitted ]
Message 5 in thread
> Hello!
>
> I need to store huge matrixes with qint32 values.
> It is impossible to load them into memory.
> So i'm using memory mapped files.
> Values to files i store using QDataStream.
> Then i map these files to memory and need to get values with random
> positions.
>
> Currently i'm doing so:
>
> char * p - pointer to memory of mapped file
> row, col - coordinates of matrix cell, i need value for
> m_ncols - number of columns in matrix
>
> p += (row * m_ncols + col) * 4;
> QByteArray ba = QByteArray::fromRawData(p,4);
> QDataStream ds(&ba, QIODevice::ReadOnly);
> qint32 result;
> ds >> result;
> return result;
>
> but it works slowly because i need to create QDataStream and
> QByteArray each time, i get value.
>
> I tried to create QDataStream once and read all values sequentially.
> It works much much more faster! But as i understand i can't move
> position of QDataStream backward. So fo random access i can't create
> QDataStream only once for entire matrix. Is it possible somehow to
> optimize algorithm, which i use now?
>
> --
> Sergey mailto:mb0@xxxxx
>
What is the sparseness of the matrix? There are many compression
algorithms out there for sparse matrices that work quite well when the
sparcity I sless then 40%
Scott
--
[ signature omitted ]
