There is frequently a need to scan system or application log files for notifications of particular events and the like. There are some subtle points about reading from a file that another process is actively writing to:

• When you reach the current end of the file, some more content may be appended a few seconds or a few minutes later. Thus, you need to wait a while and then try to resume reading from the current position. This requires clearing the EOF indication without closing and reopening the file.
• Many systems implement logfile rollover, where periodically the current log file is closed and renamed, and a new log file opened with the previous name. This is to keep the log files from growing without bounds.

Some Python code that handles both these issues is as follows:

LogFileName = "/var/log/messages" # or whatever log file you're wanting to monitor
MonitorInterval = 60 # number of seconds between checks for new content
LogFile = None # to begin with
while True :
    if LogFile == None :
        LogFile = open(LogFileName, "r")
    #end if
    while True :
        Line = LogFile.readline()
        if Line == "" :
            break # end of file reached
        #end if
        ... do whatever processing of Line is necessary ...
    #end while
    if \
            os.path.realpath("/proc/self/fd/%d" % LogFile.fileno()) \
        != \
            os.path.realpath(LogFileName) \
    :
        # rollover has started, might still be in progress
        LogFile.close()
        LogFile = None
    else :
        LogFile.seek(0, 1) # clear EOF state
    #end if
    time.sleep(MonitorInterval)
      # wait for more content, also gives time for rollover to complete
#end while

Note that the check whether the log file still has the same name is reliant on the functionality of procfs under Linux—will this work under any other systems?