| Trolltech Home | Qt-interest Home | Recent Threads | All Threads | Author | Date | |
| All threads index page 1 | |
Dear all, For those who have read my "Parse a huge XML file" thread, this is in the same trend. Currently, I have a QHttp that gets bz2 zipped data from a server. It feeds a QProcess running bzip2 to uncompress the data, and then parse it through a QXmlStreamReader, that populates a sql database. It worked well until I tried it with a faster internet connection. Now, QHttp is downloading data faster than bzip2 or QXmlStringReader are capable of processing the data (I do not know which one is the bottleneck). This causes most of the data to accumulate in a memory buffer. As my objective is to do this with huge files, (4GB bzip2, 60GB xml) this will cause some memory issues... Is there a way to make QHttp wait the data is processed before continuing the download? I haven't found anything in the doc. Thanks, Etienne
I doubt you can do that in a good way. If you somehow limit the speed QHttp reads data, system's TCP/IP stack will overflow and new arriving data will be throwed away until there's enough space in buffers. Consequently that'll cause remote computer to re-send lost data. And bigger is the difference of data receiving and processing speeds, bigger part of it will need to be re-transferred... In the end you'll overload your network and screw entire connection. So, IMHO, you should search a way to slow down *remote* application so it doesn't send more than you can process. On Mon, 04 Feb 2008 17:24:19 +0300, Etienne Sandré <etienne.sandre@xxxxxxxxxxxxxxxxx> wrote: > Dear all, > > For those who have read my "Parse a huge XML file" thread, this is in the > same trend. > > Currently, I have a QHttp that gets bz2 zipped data from a server. It > feeds > a QProcess running bzip2 to uncompress the data, and then parse it > through a > QXmlStreamReader, that populates a sql database. > > It worked well until I tried it with a faster internet connection. Now, > QHttp is downloading data faster than bzip2 or QXmlStringReader are > capable > of processing the data (I do not know which one is the bottleneck). This > causes most of the data to accumulate in a memory buffer. > As my objective is to do this with huge files, (4GB bzip2, 60GB xml) this > will cause some memory issues... > > Is there a way to make QHttp wait the data is processed before continuing > the download? > I haven't found anything in the doc. > > Thanks, > > Etienne -- [ signature omitted ]
Constantin Makshin wrote: > I doubt you can do that in a good way. > > If you somehow limit the speed QHttp reads data, system's TCP/IP stack > will overflow and new arriving data will be throwed away until there's > enough space in buffers. Consequently that'll cause remote computer to > re-send lost data. And bigger is the difference of data receiving and > processing speeds, bigger part of it will need to be re-transferred... > In the end you'll overload your network and screw entire connection. This is entirely incorrect - TCP takes care of flow control for you, so you need not worry about the transmitter blindly sending data that will be discarded. -- [ signature omitted ]
Hi, > Constantin Makshin wrote: > > > > If you somehow limit the speed QHttp reads data, system's TCP/IP stack > > will overflow and new arriving data will be throwed away until there's > > enough space in buffers. Consequently that'll cause remote computer to > > re-send lost data. And bigger is the difference of data receiving and > > processing speeds, bigger part of it will need to be re-transferred... > > In the end you'll overload your network and screw entire connection. > > This is entirely incorrect - TCP takes care of flow control for > you, so you need not worry about the transmitter blindly sending > data that will be discarded. > I second that - If you're interested, google "TCP Sliding window" for more information on how TCP does it's flow control. A random link on the first page: http://www.rhyshaden.com/tcp.htm Look at point 6. Cheers, -- [ signature omitted ]
May be you're right. I should better learnt network programming to prevent giving stupid answers. :) Then Etienne shouldn't worry that data processing speed is lower than receiving one. On Mon, 04 Feb 2008 19:19:33 +0300, Stephen Collyer <scollyer@xxxxxxxxxxxxxxxx> wrote: > Constantin Makshin wrote: >> I doubt you can do that in a good way. >> >> If you somehow limit the speed QHttp reads data, system's TCP/IP stack >> will overflow and new arriving data will be throwed away until there's >> enough space in buffers. Consequently that'll cause remote computer to >> re-send lost data. And bigger is the difference of data receiving and >> processing speeds, bigger part of it will need to be re-transferred... >> In the end you'll overload your network and screw entire connection. > > This is entirely incorrect - TCP takes care of flow control for > you, so you need not worry about the transmitter blindly sending > data that will be discarded. -- [ signature omitted ]
Constantin Makshin wrote: > May be you're right. I should better learnt network programming to > prevent giving stupid answers. :) Take a look at "TCP/IP Illustrated, Volume 1" by Rich Stevens for all the details. It's a very well written book. > Then Etienne shouldn't worry that data processing speed is lower than > receiving one. It sounds as if QHttp is reading data asynchronously, so it may be that he has to worry about it. -- [ signature omitted ]
On Mon, Feb 04, 2008 at 03:24:19PM +0100, Etienne Sandrà wrote: > Dear all, > > For those who have read my "Parse a huge XML file" thread, this is in the > same trend. > > Currently, I have a QHttp that gets bz2 zipped data from a server. It feeds > a QProcess running bzip2 to uncompress the data, and then parse it through a > QXmlStreamReader, that populates a sql database. > > It worked well until I tried it with a faster internet connection. Now, > QHttp is downloading data faster than bzip2 or QXmlStringReader are capable > of processing the data (I do not know which one is the bottleneck). This > causes most of the data to accumulate in a memory buffer. > As my objective is to do this with huge files, (4GB bzip2, 60GB xml) this > will cause some memory issues... > You can create QTcpSocket and set it's read buffer (setReadBufferSize()). After this you can pass this socket to QHttp (setSocket() method). TCP/IP stack will done all other stuff for you. -- [ signature omitted ]
Attachment:
signature.asc
Description: Digital signature
Dmitry Nezhevenko wrote: >You can create QTcpSocket and set it's read buffer > (setReadBufferSize()). After this you can pass this socket to QHttp > (setSocket() method). TCP/IP stack will done all other stuff for you. Qt 4.4's API is simpler for that. QNetworkReply has a setReadBufferSize() and its internals should automatically throttle the sending. If it doesn't, it's a bug and you should let me know (via support). -- [ signature omitted ]
Attachment:
signature.asc
Description: This is a digitally signed message part.