# changelog

author: Joe Grund <joe.grund@intel.com> - 1.0.2-1
changelog:
  - Fix missing keys to be option types.
  - Add rules for scsi ids
  - Add keys on change|add so we can `udevadm trigger` after install
  - Trigger udevadm change event after install
  - Read new state into scanner after install


author: Joe Grund <joe.grund@intel.com> - 1.1.0-1
changelog:
  - Exclude unneeded devices.
  - Get device ro status.
  - Remove manual udev parsing.
  - Remove socat as dep, device-scanner will listen to change events directly.


author: Tom Nabarro <tom.nabarro@intel.com> - 2.0.0-1
changelog:
  - Add mount detection to device-scanner
  - Integrate device-aggregator
  - Move device munging inside aggregator


author: Will Johnson <william.c.johnson@intel.com> - 2.3.0-1
changelog:
  - Add page for managed zfs filesystem creation
  - Add page for creating offline repos
  - Small fix-up for installing repos on client nodes


author: Joe Grund <joe.grund@intel.com> 2.0.1-1
changelog:
  - Add yum plugin to iml-diagnostics.
  - Ensure no log tailing is performed for chroma log collection.


author: Brian J. Murrell <brian.murrell@intel.com> 1-2
changelog:
  - Add post to work around LU-9745 by removing the autoinstalled
    lustre module and re-installing it


author: Tom Nabarro <tom.nabarro@intel.com> 1.4.3-1
changelog:
  - Remove use of modprobe to test presence of kernel modules.
  - Remove unused FirewallControl classes and relevant tests.


author: Pavel Raiskup <praiskup@redhat.com> - 2.7.5-1
changelog:
  - rebase to latest upstream release, per release notes:
    http://initd.org/psycopg/articles/2018/06/17/psycopg-275-released/


author: Pavel Raiskup <praiskup@redhat.com> - 2.7.5-4
changelog:
  - drop python2* on f30+ (rhbz#1634973)
  - use proper compiler/linker flags (rhbz#1631713)
  - correct the (build)requires


author: Pavel Raiskup <praiskup@redhat.com> - 2.7.5-5
changelog:
  - prepare --without=debugrpms option (rhbz#1635166)
  - get the python2 packages back for a while (rhbz#1634973)


author: Brian J. Murrell <brian.murrell@intel.com> - 0.1.5-2
changelog:
  - Need to override pyp2rpm's Requires: python2-urllib3 >= 1.8 as
    urllib3 has not been migrated to python{2,3} yet.


author: Thomas Leibovici <thomas.leibovici@cea.fr> 2.4.0
changelog:
  - [feature] rbh-du and rbh-find: "du" and "find" clones querying robinhood's database
  - [report] new directory stats, top directories per dirent count, per avg file size
  - [report] file size profiling: global, per user, per group, per fileclass...
  - [report] sorting user/groups by size range (eg. percentage of files < 1G)
  - [webUI] "size" section in webUI to display file size profiles
  - [feature] Partial scans to update only a subset of the filesystem (also allows distributed scans)
  - [packaging] rpm name 'robinhood-tmp_fs_mgr' changed to 'robinhood-tmpfs'
  - [packaging] 'rbh-config' command moved to new RPM 'robinhood-adm'
  - [report] refurbished rbh-report output format
  - [policies] new criteria on file creation time
  - [database] use innodb by default for MySQL engine
  - [system] ability to detect "fake mtime" (mtime != actual modification time)
  - [system] improved filesystem detection,
             using fsname or devid as FS identifier (config driven)
  - [scan] can trigger external completion command when a scan ends
  - [misc.] can use short config name instead of full path
            (eg. "-f <name>" instead of "-f /full/path/to/name.conf")
  - [backup] directory and symlink recovery
  - [lustre] port to Lustre 2.2 and 2.3
  - [lustre] support for new Changelog record struct (lu-1331)
  - [fix] max_rm_count=0 resulted in no rm (instead of unlimited)
  - [fix] segfault in realpath() on Ubuntu
  - [fix] unsigned arithmetic issue with MySQL 5.5


author: Thomas Leibovici <thomas.leibovici@cea.fr> 2.4.1
changelog:
  - [lustre] better file size change detection using CLOSE events
    from MDT ChangeLog (requires Lustre 2.2 or +)
  - [scan] optimization: using fstatat and getdents
  - [rbh-find] added -atime/-amin options
  - [logs] add'l information in logs (DB operations, HSM_rm details)
  - [logs] log to stderr if opening of the log file fails
  - [fix] scan blocked if final DB operation failed
  - [fix] avoid DB lock exhaustion for huge requests
  - [backup] manage cross device rename in backend
  - [backup] rebind an entry in backend after fid change (e.g. restripe)


author: Thomas Leibovici <thomas.leibovici@cea.fr> 2.4.2
changelog:
  - [general] immediate exit on ctrl+C: don't process all queued operations, just finish current.
  - [general] LSB compliance if daemon is already started.
  - [DB] validation with MariaDB (replacement for MySQL in Fedora19).
  - [config] can set default config file using RBH_CFG_DEFAULT environment variable.
  - [config] more precise message if no config file is found.
  - [rbh-find] added -not/-! option to rbh-find.
  - [lustre] fix for 16 chars pool names.
  - [bugfix] fixed memleak in rbh-find.
  - [bugfix] fixed segfault if checking scan deadline occured exactly when scan ended.
  - [logs] display bandwidth and rate stats during migration run.
  - [logs] fix: DB get operations were counted twice in stats.
  - [cosmetic] removed "connection failed" warning for one shot commands.
  - [cosmetic] fix typos in logs.
  - [devel] port to automake 1.12 (since Fedora18).
  - [shook] port to shook 1.3.5


author: Thomas Leibovici <thomas.leibovici@cea.fr> 2.4.3
changelog:
  - [lustre] support of Lustre 2.4
      - DNE not fully supported yet: if running multiple MDS,
        run 1 instance of changelog reader per MDT.
      - Detect file layout changes (new changelog record CL_LAYOUT).
  - [lustre] added statistics about changelog processing speed.
  - [policies] new parameter 'recheck_ignored_classes' to allow/avoid
      rematching entries from ignored classes in migration and purge policies.
  - [web ui] security patch to prevent from SQL injection.
  - [lustre] fix stack overflow when handling files with wide stripes.
  - [DB] better handling of ER_QUERY_INTERRUPTED MySQL error.
  - [DB] fixed DB connection leaks.
  - Backup & HSM modes:
     - [doc] added config tutorial for backup mode
     - [fix] fix segfault in import command when uid/gid can't be resolved.
     - [rbh-report] fix bad display of total volume with -u or -g.
  - Migration policy features and optimizations:
     - [feature] new parameter 'lru_sort_attr' to select LRU sort criteria for policy application.
          Previously based on last modification time, it can now be one of:
          creation, last_archive, last_mod, last_access.
     - [feature] special meaning for condition 'last_archive == 0':
          matches entries that have never been archived.
     - [feature] suspend migration if copy error rate exceed a threshold.
          This is controled by 'suspend_error_pct' and 'suspend_error_min' parameters.
     - [stats] migration stats while migration is running: added skipped and error counters.
     - [optim] avoid rechecking ignored entries at each pass
     - [optim] smoother feeding of migration workers queue
  - Code & environment:
     - [build] can specify a path to alternative lustre source tree in ./configure
     - [tests] allow specifying an alternative path to lfs command


author: Thomas Leibovici <thomas.leibovici@cea.fr> 2.5.0-1
changelog:
  Summary:
  - filesystem disaster recovery features
  - new namespace management (new DB schema to properly handle hardlinks, renames...)
  - scanning and changelog processing optimizations
  - database optimizations (requests batching)
  - many other changes, improvements and code cleaning...

  Details:
  - rbh-diff:
      * new command to detect differences between the filesystem and the information
        in robinhood database.
      * option "--apply=fs" for disaster recovery purpose: restore the filesystem
        metadata from robinhood DB.
      * makes it possible to rebuild a Lustre MDT from scratch, or from a LVM snapshot
        (see "Robinhood Lustre disaster recovery guide" for more details).
  - database:
      * new namespace implementation in database with new NAMES table (Cray contribution)
          - fixes/improves hardlink support
          - fixes/improves Lustre ChangeLog hardlink/rename/unlink support
          - saves DB storage space
      * database request batching: significantly increase database ingest rate.
        No longer needs innodb_flush_log_at_tx_commit != 1 to speed up DB operations.
      * additional information in DB that can help for disaster recovery:
        symlink info, access rights, stripe object indexes, stripe order, nlink...
      * set default commit behavior to transaction (prevent from DB inconsistencies)
      * optimized multi-table requests
      * optimization: minimized attribute set in DB update operations
        (don't update attributes that didn't change)
      * Fix: deal with mysql case insensitivity for string matching
      * triggers and stored procedures versioning mechanism
      * prevent from overflows for large INSERT requests, wide stripes...
      * prevent from DB deadlocks
  - scanning:
      * --partial-scan option is deprecated and replaced by an optional argument to --scan (e.g. --scan=/fs/subdir).
      * better management of partial scans:
          - better detection of removed entries vs. entries moved from a directory to another.
          - partial scans can be used for initial DB population (even if the DB is initially empty).
      * dealing with dead/deactivated OSTs: don't remove entries from DB if stat() returns ESHUTDOWN.
      * garbage collection of removed entries in DB is a long operation when terminating a scan (and even more
        when terminating a partial scan). Added --no-gc option to skip it (recommanded for partial scans).
      * automatically enabling --no-gc if the DB is initially empty (eg. for initial scan).
      * optimization: use *at() functions (openat, fstatat) and readdir by chunk (using getdents) instead of POSIX lstat() and readdir_r().
      * optimization: use NOATIME flag to access entries as much as possible
      * optimizations of get_stripe and get_fid operations.
      * new --diff option for robinhood --scan and --readlog: output detected changes in a diff-like format.
  - Lustre changelogs:
      * changelog batching (Cray contribution): to speed up changelog processing,
        robinhood retains changelog records in memory a short time,
        to aggregate similar/redundant Changelog records on the same entry before
        updating its database.
      * support multiple changelog readers (for DNE) as multiple threads (default)
        or as multiple processes, possibly on different hosts, by giving a MDT index
        to --readlog option.
      * resilience to filesystem umount/mount.
  - rbh-report:
      * new option --entry-info to get all the stored information about an entry
      * option --dump-ost can now list multiple OSTs and support ranges notation (e.g. 3,5-8,12-23).
      * --dump-ost output indicates if a file has data on a given OST (could be striped on the OST but have no data on it).
  - rbh-find:
      * new option -crtime to filter entries on creation time.
      * output ordering closer to find output
      * added missing info in 'rbh-find -ls' output (nlink, mode, symlink info...)
  - robinhood-backup:
      * by default, use a built-in copy function to avoid the cost of forking copy commands.
      * rbh-backup-rebind: tool to rebind an entry in the backend if its fid changed in the filesystem
        for any reason (file copied to a new one to change its stripe, etc...)
      * rbh-backup-recov new features and options:
          --list (list information about entries to be recovered)
          --ost <ost_set> to only recover entries for a given set of OSTs (support range notation):
              the basic use-case is OST disaster recovery.
          --since <time> to only recover entries modified since a given date:
              the basic use case is after restoring an OST snapshot.
      * symlinks archiving to backend made optional (new parameter 'archive_symlinks')
        as they can now be restored using robinhood database information.
      * new parameter: sync_archive_data: force sync'ing data to disk to
        finalize the copy.
  - configuration:
      * can specify environment variables in config file (e.g. fs_path = $ROOT_DIR ;)
      * prevent from using a wrong config file (Cray contribution):
          - only check files in /etc/robinhood.d/<purpose>, no longer in the current directory
          - fails if to many config files are available.
  - documentation:
      * added man pages for robinhood daemon, rbh-report, rbh-find.


author: Thomas Leibovici <thomas.leibovici@cea.fr> 2.5.1
changelog:
  - entry processing (major fix): fixed deadlock when the pipeline is full
    and an entry with an unknown parent is encountered.
  - purge (enhancement): start purging data from the most used OSTs.
  - rbh-find (features): new options: -pool, -exec, -print, -nouser, -nogroup, -lsost
  - rbh-find (optimization): automatically switch to bulk DB request mode when
    command argument is filesystem root (+new option -nobulk to disable it).
  - logging (enhancement): new config parameters to control log header format
  - backup (feature): allow compressing data in archive.
  - backup (fix): wrong path in archive when robinhood root directory != mount point.
  - backup (fix): fix segfault when importing a single file with a FID-ending name.


author: Thomas Leibovici <thomas.leibovici@cea.fr> 2.5.2
changelog:
  - rbh-du: fixed major performance regression (since v2.5.0).
  - rbh-find: fixed occasional crash.
  - HSM and backup modes: fixed a risk of removing an existing entry from the backend (in some situations of hardlink/rename+unlink).
  - backup mode: optimized sendfile()-based copy (Linux kernel >= 2.6.33).
  - logs: avoid flood of log messages in case of DB connection error.
  - alerts: added host name to alert mail title.
  - rbh-config empty_db/repair_db: also manage/fix stored procedures.
  - cosmetic: fix wrong display of purged blocks for count-based triggers.
  - cosmetic: fix migration counter display.
  - init script: check that 'ulimit -s' is reasonable.
  - fixed build dependancies on Fedora19 and Fedora20.
  - code sanity: fixed many 'coverity' warnings + a couple of minor memleaks.
  - doc: details about RPM installation locations.
  - doc: detail of 'backend' paramaters for backup mode.


author: Thomas Leibovici <thomas.leibovici@cea.fr> 2.5.3
changelog:
  - [bugfix] custom purge_command: fixed vulnerability to malicious file names.
  - [bugfix] changelog processing: fixed errors 'Entry has incomplete path in DB' in some case of rm/create patterns.
  - [bugfix] migration policy (fix): don't trigger copy of files that no longer exist.
  - [feature] rbh-config: new option 'reset_acct' to rebuild accounting info.
  - [feature] changelog reader: new parameter 'dump_file' to dump all incoming changelog records to a file.
  - [compat] port to Lustre 2.6.0.


author: Thomas Leibovici <thomas.leibovici@cea.fr> 2.5.4
changelog:
  - [lustre] Lustre 2.4+: detect all stripe changes and update the DB accordingly.
  - [scan] Prevent from dropping entries from DB when opendir or stat fail.
  - [DB] Optimization to batch more DB requests.
  - [DB] Allow using any MySQL engine (new config parameter: listmanager::mysql::engine).
  - [rbh-config] Avoid backup_db to lock the whole tables with innodb.
  - [bugfix] Improve robustness to corrupted mtimes.
  - [bugfix] Fix possible crash in db_exec_sql.
  - [bugfix] Fix possible overflow when executing a custom archive command.
  - [backup mode] Clean all non-printable characters in backend path.
  - [pkg] libattr-devel is now mandatory to build robinhood.


author: Thomas Leibovici <thomas.leibovici@cea.fr> 2.5.5
changelog:
  - [lustre] Support Lustre versions up to 2.7.
  - [DB performance] Allow both batching and parallelizing DB operations when accounting is OFF. Benchmarks show a x3~x4 speedup of DB ingest rate in this case.
  - [DB performance] Reduce lock contention on indexes of STRIPE_ITEMS table.
  - [acct] Avoid accounting stats inconsistency when this feature is enabled/disabled/re-enabled.
  - [policy performance] Disable ignored class re-matching by default.
  - [policy performance] Disable sort by mtime by default.
  - [policy performance] Allow disabling sort by atime for purge policies.
  - [fix] Fix segfault when log rotation occurs under heavy logging.
  - [fix] When processing a RENAME changelog record, match the new path in filesystem instead of building it from DB (which caused 'incomplete path' errors).
  - [fix] Fix sendfile copy operation for backup mode.
  - [fix] Fix various requests about directories: --topdirs with --filter-path option, --toprmdir, ...
  - [fix] Fix possible duplicate insert error when updated entry attributes are only in ANNEX_INFO table (rare).
  - [fix] Some purge parameters may not be taken into account.
  - [packaging] clean permissions in distribution tarball.
  - [report performance] Improve performance of path matching when running directory reports with --filter-path option.


