Unsolved Which Container is best suited for holding 10 millions of records (String, String) ?
-
WOW 4 responses within a minute. Never saw this before.
-
This post is deleted! -
@ksranjith786
Hi
You should really test with SQlite if your use case is to select amount 10 millions lines and display the subset. -
@ksranjith786 said in Which Container is best suited for holding 10 millions of records (String, String) ?:
Around 5 msec for lookup
You probably need a proper database (not just SQLite but maybe PostgreSQL), index it properly and maybe even give it standalone-resources (i.e. run it on a dedicated machine) to achieve that performance
-
@VRonin said in Which Container is best suited for holding 10 millions of records (String, String) ?:
You probably need a proper database (not just SQLite but maybe PostgreSQL), index it properly and maybe even give it standalone-resources (i.e. run it on a dedicated machine) to achieve that performance
Even this may not be viable. In a typical 10/100 network you'd get about 1ms of latency from the TCP/IP go-around, which shrinks that 5ms window considerably.
@ksranjith786
How are you going to use that dataset?Our use case is that, our application need to fetch offers associated for an item during item scan.
Elaborate on that, break it step by step for us and do say what are "offers" and "items" in this context, and most importantly what's an "item scan".
-
My db in MySQL with 3000 of records, takes at least 1.5 seconds to response.
-
That's simply too long. You should inspect your database and how you use it.
@VRonin
It just occurred to me that this problem is a prime candidate for usage and testing of your big hash lib. :) -
@kshegunov Lol, thanks but 5msec is not achievable even in my wildest dreams. I also suspect 10m QString as key (that are not dumped on the hard drive, only values are) are enough to blow most memory
-
oh, I think Redis may help you!
-
@joeQ You need to have spent quite a bit of money on RAM for your PC for that to be an option