class PEROBS::FlatFileBlobHeader

The FlatFile blob header has the following structure:

1 Byte: Flags byte.

Bit 0: 0 deleted entry, 1 valid entry
Bit 1: 0 reserved, must be 0
Bit 2: 0 uncompressed data, 1 compressed data
Bit 3: 0 current entry, 1 outdated entry
Bit 4 - 7: reserved, must be 0

8 bytes: Length of the data blob in bytes 8 bytes: ID of the value in the data blob 4 bytes: CRC32 checksum of the data blob

If the bit 0 of the flags byte is 0, only the length is valid. The blob is empty. Only of bit 0 is set then entry is valid.

Constants

COMPRESSED_FLAG_BIT
FORMAT

The ‘pack()’ format of the header.

LENGTH

The length of the header in bytes.

OUTDATED_FLAG_BIT
VALID_FLAG_BIT

Attributes

addr[R]
corruption_start[RW]
crc[R]
flags[R]
id[R]
length[R]

Public Class Methods

new(file, addr, flags, length, id, crc) click to toggle source

Create a new FlatFileBlobHeader with the given flags, length, id and crc. @param file [File] the FlatFile that contains the header @param addr [Integer] the offset address of the header in the file @param flags [Integer] 8 bit number, see above @param length [Integer] length of the header in bytes @param id [Integer] ID of the blob entry @param crc [Integer] CRC32 checksum of the blob entry

# File lib/perobs/FlatFileBlobHeader.rb, line 66
def initialize(file, addr, flags, length, id, crc)
  @file = file
  @addr = addr
  @flags = flags
  @length = length
  @id = id
  @crc = crc
  # This is only set if the header is preceded by a corrupted blob.
  @corruption_start = nil
end
read(file, addr = nil, id = nil) click to toggle source

Read the header from the given File. @param file [File] @param addr [Integer] address in the file to start reading. If no

address is specified use the current position in the file.

@param id [Integer] Optional ID that the header should have. If no id is

specified there is no check against the actual ID done.

@return FlatFileBlobHeader or nil if there are no more blobs to read in

the file.
# File lib/perobs/FlatFileBlobHeader.rb, line 85
def FlatFileBlobHeader::read(file, addr = nil, id = nil)
  # If an address was specified we expect the read to always succeed. If
  # no address is specified and we can't read the header we generate an
  # error message but it is not fatal.
  errors_are_fatal = !addr.nil?

  mode = :searching_next_header
  addr = file.pos unless addr
  buf = nil
  corruption_start = nil

  loop do
    buf_with_crc = nil
    begin
      file.seek(addr)
      buf_with_crc = file.read(LENGTH)
    rescue IOError => e
      if errors_are_fatal
        PEROBS.log.fatal "Cannot read blob header in flat file DB at " +
          "address #{addr}: #{e.message}"
      else
        PEROBS.log.error "Cannot read blob header in flat file DB: " +
          e.message
        return nil
      end
    end

    # Did we read anything?
    if buf_with_crc.nil?
      if errors_are_fatal
        PEROBS.log.fatal "Cannot read blob header " +
          "#{id ? "for ID #{id} " : ''}at address #{addr}"
      else
        if corruption_start
          PEROBS.log.error "Corruption found at end of blob file at " +
            "address #{addr}"
        end
        # We have reached the end of the file.
        return nil
      end
    end

    # Did we get the full header?
    if buf_with_crc.length != LENGTH
      msg = "Incomplete FlatFileBlobHeader: Only " +
        "#{buf_with_crc.length} " +
        "bytes of #{LENGTH} could be read "
      "#{id ? "for ID #{id} " : ''}at address #{addr}"
      if errors_are_fatal
        PEROBS.log.fatal msg
      else
        PEROBS.log.error msg
      end
      return nil
    end

    # Check the CRC of the header
    buf = buf_with_crc[0..-5]
    crc = buf_with_crc[-4..-1].unpack('L')[0]

    if (read_crc = Zlib.crc32(buf, 0)) == crc
      # We have found a valid header.
      if corruption_start
        PEROBS.log.error "FlatFile corruption ends at #{addr}. " +
          "#{addr - corruption_start} bytes skipped. Some data may " +
          "not be recoverable."
      end
      break
    else
      if errors_are_fatal
        PEROBS.log.fatal "FlatFile Header CRC mismatch at address " +
          "#{addr}. Header CRC is #{'%08x' % read_crc} but should be " +
          "#{'%08x' % crc}."
      else
        if corruption_start.nil?
          if errors_are_fatal
            PEROBS.log.fatal "FlatFile corruption found. The FlatFile " +
              "Header CRC mismatch at address #{addr}. Header CRC is " +
              "#{'%08x' % read_crc} but should be #{'%08x' % crc}."
          else
            PEROBS.log.error "FlatFile corruption found. The FlatFile " +
              "Header CRC mismatch at address #{addr}. Header CRC is " +
              "#{'%08x' % read_crc} but should be #{'%08x' % crc}. " +
              "Trying to find the next header."
          end
          corruption_start = addr
        end
        # The blob file is corrupted. There is no valid header at the
        # current position in the file. We now try to find the next valid
        # header by iterating over the remainder of the file advanding one
        # byte with each step until we hit the end of the file or find the
        # next valid header.
        addr += 1
      end
    end
  end

  header = FlatFileBlobHeader.new(file, addr, *buf.unpack(FORMAT))
  if corruption_start
    header.corruption_start = corruption_start
  end

  if id && header.id != id
    PEROBS.log.fatal "Mismatch between FlatFile index and blob file " +
      "found. FlatFile has entry with ID #{header.id} at address " +
      "#{addr}. Index has ID #{id} for this address."
  end

  return header
end

Public Instance Methods

clear_flags() click to toggle source

Reset all the flags bit to 0. This marks the blob as invalid.

# File lib/perobs/FlatFileBlobHeader.rb, line 210
def clear_flags
  @flags = 0
  write
end
is_compressed?() click to toggle source

Return true if the blob contains compressed data.

# File lib/perobs/FlatFileBlobHeader.rb, line 221
def is_compressed?
  bit_set?(COMPRESSED_FLAG_BIT)
end
is_outdated?() click to toggle source

Return true if the blob contains outdated data.

# File lib/perobs/FlatFileBlobHeader.rb, line 233
def is_outdated?
  bit_set?(OUTDATED_FLAG_BIT)
end
is_valid?() click to toggle source

Return true if the header is for a non-empty blob.

# File lib/perobs/FlatFileBlobHeader.rb, line 216
def is_valid?
  bit_set?(VALID_FLAG_BIT)
end
set_outdated_flag() click to toggle source

Set the outdated bit. The entry will be invalid as soon as the current transaction has been completed.

# File lib/perobs/FlatFileBlobHeader.rb, line 227
def set_outdated_flag
  set_flag(OUTDATED_FLAG_BIT)
  write
end
write() click to toggle source

Write the header to a given File.

# File lib/perobs/FlatFileBlobHeader.rb, line 197
def write
  begin
    buf = [ @flags, @length, @id, @crc].pack(FORMAT)
    crc = Zlib.crc32(buf, 0)
    @file.seek(@addr)
    @file.write(buf + [ crc ].pack('L'))
  rescue IOError => e
    PEROBS.log.fatal "Cannot write blob header into flat file DB: " +
      e.message
  end
end

Private Instance Methods

bit_set?(n) click to toggle source
# File lib/perobs/FlatFileBlobHeader.rb, line 239
def bit_set?(n)
  mask = 1 << n
  @flags & mask == mask
end
clear_flag(n) click to toggle source
# File lib/perobs/FlatFileBlobHeader.rb, line 248
def clear_flag(n)
  @flags &= ~(1 << n) & 0xFF
end
set_flag(n) click to toggle source
# File lib/perobs/FlatFileBlobHeader.rb, line 244
def set_flag(n)
  @flags |= (1 << n)
end