Skip to content
  • 0 Votes
    9 Posts
    4k Views
    kshegunovK

    @stormracer
    Hi,

    my development pc has this amount of memory. But for other pc i can't rely on this and i will get larger data files later. Therefor i search for a solution for managing large amount of data. My idea for sql was that database systems are made for efficient data storing and searching. I choose sqlite because it doesn't need a server in the background.

    Yes, somewhat, for general purpose data. Seeing you're searching for a set of particles in space (2D space?), and assuming you're wanting to mostly search and extract from the dataset not modifying it, I'd consider some sort of HDD swapped (or database backed) k-d tree. From my experience SQL databases behave rather poorly for scientific data, especially when the datasets are big. Also you could try getting a stripped down b-tree implementation (basically a database without the bells and whistles) that you can use for storage of the data and do the indexing/searching by yourself. Or the most basic approach is to work with the data as you'd do normally, but swap the dataset instead of holding it in memory (e.g. QCache comes to mind here).

    PS.
    Depending on how exactly you want to build the histogram you could also simply discretize the space (if I'm correct that you're working with floating point numbers for coordinates).