QPlainTextEdit for very large files 4+ GB.

Mr. Utri

I am working on a Python program to read very large log files. Because I do not want to load the file into memory, I have created a thread that will parse out the seek points for each line and am using appendPlainText(Line) to append them to the editor window. Although it is not implemented right now i do plan on expanding this feature to grab other meta data as well.

The parsing and creating a table of seek points for each line is working great. I have created a method that allows me to grab a line by providing the line number (see below). I also have the edit box being initially loaded with data from the beginning of the file (see below).

The only part I am missing now is the ability to override the controls of the edit box. I am not quite sure of the best way to change the contents of the edit box, vased off of what part of the file they are looking at. the behavior I want is to allow the edit box to show a small window of the actual file without using lot of memory. For example, when they scroll either by key press, mouse wheel, etc, I would like to reload the contents of the editbox to match the part of the file they are looking at.

Some things to note:

I am dealing with huge files 4+ Gigabytes
I want the UI to be snappy and responsive.
I have a thread running in the background collecting meta data to make finding things quicker and more seamless.
I do have plans on using some of the syntax highlighting features in the future.

  def ReadLine(self, LineNumber):
    Line = ""

    if LineNumber <= len(self.LineOffsetList):
      self.FileMutex.lock()
      # Get the Current position so we can restor it later.
      CurrentPos = self.File.tell()
      self.File.seek(self.LineOffsetList[LineNumber])
      Line = self.File.readline()

      # Restor the position after we have done our read.
      self.File.seek(CurrentPos)
      self.FileMutex.unlock()

    #print("%s>> %s" %(str(LineNumber), str(Line)))
    return Line

  # @todo Farm this off to a completer to make the UI more responsive.
  def LoadFilteredResults(self):
    DataFilter = None
    CurLine = 0
    Index = 0

    self.Edit.clear()
    NumLines = int(self.Edit.height() / (self.FontMetrics.xHeight() + 4))

    #print("Current: %d,  Lines: %d" %(self.CurrentLine, NumLines))

    while CurLine < NumLines:
      Line = self.Handler.ReadLine(Index).strip()
      Index += 1

      if not Line:
        break
      elif self.DataSelection:
        # Create the Filter list.
        DataFilter = '|'.join(self.DataSelection)

        if re.search(""str("<[ ]*(" + DataFilter + ")[ ]*:[a-zA-z ]+\>")"", Line):
          self.Edit.appendPlainText(Line)
          CurLine += 1
      else:
        self.Edit.appendPlainText(Line)
        CurLine += 1

Thanks for your help in advance.