1 XrdClRecorder Plugin
This XRootD Client Plugin can be used to record all user's actions on XrdCl::File object and store them into a csv file. Afterwards, using the xrdreplay utility the actions can be replayed preserving the original timing. The output file can be provided either using the XRD_RECORDERPATH environment variable or the output key in the plug-in configuration file (the enviroment variable takes precedence). If neither is provided the recorded actions will be stored at a default location: /tmp/xrdrecord.csv.
Config file format:
recorder.conf:
url = *
lib = /usr/lib64/libXrdClRecorder-5.so
enable = true
output = /tmp/out.csv # optional
In order to replay either do:
or
cat /tmp/out.csv | xrdreplay
2 xrdreplay tool
The xrdreplay application provides the following operation modes:
- print mode (-p) : display runtime and IO statistics for a record file
- verify mode (-v) : verify the existence of the required input files for a record file
- creation mode (-c,-t) : create the required input data using file creation and write the minimal required size (-c) or truncate files to the minimal required size (-t)
- playback mode (default) : replay a given record file
2.1 print mode (-p)
To display the IO statistics from a recorded file without playback do:
xrdreplay -p recording.csv
# =============================================
# IO Summary (print mode)
# =============================================
# Sampled Runtime : 5.485724 s
# Playback Speed : 1.00
# IO Volume (R) : 536.87 MB [ std:536.87 MB vec:0 B page:0 B ]
# IO Volume (W) : 536.87 MB [ std:536.87 MB vec:0 B page:0 B ]
# IOPS (R) : 64 [ std:64 vec:0 page:0 ]
# IOPS (W) : 64 [ std:64 vec:0 page:0 ]
# Files (R) : 1
# Files (W) : 1
# Datasize (R) : 536.87 MB
# Datasize (W) : 536.87 MB
# ---------------------------------------------
# Quality Estimation
# ---------------------------------------------
# Synchronicity(R) : 4.55%
# Synchronicity(W) : 100.00%
To further inspect details of the recording file (which files are accessed and IO per file) use the long format (-l) and optionally the summary option (-s):
xrdreplay -p recording.csv
# -----------------------------------------------------------------
# File: root://cmsserver//store/cms/higgs.root
# Sync: 4.55%
# Errs: 0
# -----------------------------------------------------------------
# close::texec : 0.00 s
# close::tnomi : 0.00 s
# open::texec : 0.00 s
# open::tnomi : 0.13 s
# read::texec : 0.04 s
# read::tnomi : 17.02 s
# stat::texec : 0.00 s
# stat::tnomi : 0.00 s
# close::n : 1
# open::n : 1
# openr::n : 1
# read::b : 536870912
# read::n : 64
# read::o : 536870912
# stat::n : 1
# -----------------------------------------------------------------
...
All tags used in the output format are explained in the tags section.
2.2 verify mode (-v)
To verify the availability of all input files one uses:
xrdreplay -v recording.cvs
...
# ---------------------------------------------
# Verifying Dataset ...
# .............................................
# file: root://cmsserver//store/cms/higgs.root
# size: 536.87 MB [ 0 B out of 536.87 MB ] ( 0.00% )
# ---> info: file exists and has sufficient size
On success the shell returns 0, if there was a missing, too small or inaccessible file it returns -5 (251).
Warning: xrdreplay considers a file only as an input file if it has no bytes written.
2.3 creation mode (-c)
In creation mode xrdreplay will create the required input files. In this context it is worthwhile to explain the –replace option, which allows to modify the input and output path used by xrdreplay.
2.3.1 using the –replace option
You can use the –replace option (multiple times) to rewrite the URLs of input and output data e.g.:
xrdreplay --replace root://cmsserver//store/cms/:=root://mycluster//mypath/ --replace file:/data/:=file:/gpfs/data/ -v
The option works in combination with all modes of xrdreplay e.g. you can create the required input files in a different location than given in the record file.
There are two ways to create input data:
- -c create files and write well defined patterns into the files to the minimum required offset given by the recorded pattern
- -t create files and truncate files to the minum required offset given by the recorded pattern (files will contain 0)
The truncate option might be not good to produce useful results when the storage systems supports sparse files and/or compression.
Her is an example to create input data in create mode in a modified storage endpoint:
xrdreplay --replace root://cmsserver//store/cms/:=root://mycluster//mypath/ -c
2.4 playback mode (default)
xrdreplay without print, verify or creation option will playback a recorded pattern file. By default xrdreplay will replay the IO trying to keep the original timing of each request. This might not be possible if responses are slower than in the original recording. It is possible to modify the playback speed using the -x speedval option. A value of 2 means to try to run the recorded pattern with double speed. A value of 0.5 means to replay the pattern at half speed. Increasing the playback speed can increase memory requirements significantly.
The playback mode creates some additional output lines:
xrdreplay recording.cvs
# =============================================
# IO Summary
# =============================================
# Total Runtime : 5.488581 s
# Sampled Runtime : 5.485724 s
# Playback Speed : 1.00
# IO Volume (R) : 536.87 MB [ std:536.87 MB vec:0 B page:0 B ]
# IO Volume (W) : 536.87 MB [ std:536.87 MB vec:0 B page:0 B ]
# IOPS (R) : 64 [ std:64 vec:0 page:0 ]
# IOPS (W) : 64 [ std:64 vec:0 page:0 ]
# Files (R) : 1
# Files (W) : 1
# Datasize (R) : 536.87 MB
# Datasize (W) : 536.87 MB
# IO BW (R) : 97.82 MB/s
# IO BW (W) : 97.82 MB/s
# ---------------------------------------------
# Quality Estimation
# ---------------------------------------------
# Performance Mark : 99.95%
# Gain Mark(R) : 86.69%
# Gain Mark(W) : 98.29%
# Synchronicity(R) : 4.55%
# Synchronicity(W) : 100.00%
# ---------------------------------------------
# Response Errors : 0
# =============================================
Most of the output fields are self-explaining. Performance Mark puts the original run-time to the achieved run-time into relation. The <Gain Marks> indicates if the IO could potentially be run faster than given by the recording (when > 100%). The Synchronicity measures the amount of IO requests within a given file are overlapping between request and response. A value of 100% indicates synchronous IO, a value towards 0 indicates asynchronous IO. This value does not measure parallelism between files.
In case of IO errors you will see a response error counter != 0 and a shell return code of -5 (251).
# ---------------------------------------------
# Response Errors : 67
# =============================================
2.4.1 using the force (error suppression) mode (-f)
By default xrdreplay will reject to replay a recording file with error responses. By using the -f flag you can force the player to run. In this case unsuccessful IO events will be skipped in the replay.
2.5 json output
The print and playback modes support json output of results.
{
"iosummary": {
"player::runtime": 7.8835,
"player::speed": 1,
"sampled::runtime": 5.48572,
"volume::totalread": 536870912,
"volume::totalwrite": 536870912,
"volume::read": 536870912,
"volume::write": 536870912,
"volume::pgread": 0,
"volume::pgwrite": 0,
"volume::vectorread": 0,
"volume::vectorwrite": 0,
"iops::read": 64,
"iops::write": 64,
"iops::pgread": 0,
"iops::pgwrite": 0,
"iops::vectorread": 0,
"iops::vectorwrite": 0,
"files::read": 1,
"files::write": 1,
"bandwidth::MB::read": 68.1005,
"bandwdith::MB::write": 68.1005,
"performancemark": 69.5849,
"gain::read":9.94212,
"gain::write":42.2262
"synchronicity::read":4.54545,
"synchronicity::write":100,
"response::error:":0
}
}
Also the -l and -s options support json output.
2.6 command line usage
usage: xrdreplay [-p|--print] [-c|--create-data] [t|--truncate-data] [-l|--long] [-s|--summary] [-h|--help] [-r|--replace <arg>:=<newarg>] [-f|--suppress] [-v|--verify] [-x|--speed <value] p<recordfilename>]
-h | --help : show this help
-f | --suppress : force to run all IO with all successful result status - suppress all others
- by default the player won't run with an unsuccessful recorded IO
-p | --print : print only mode - shows all the IO for the given replay file without actually running any IO
-s | --summary : print summary - shows all the aggregated IO counter summed for all files
-l | --long : print long - show all file IO counter for each individual file
-v | --verify : verify the existence of all input files
-x | --speed <x> : change playback speed by factor <x> [ <x> > 0.0 ]
-r | --replace <a>:=<b> : replace in the argument list the string <a> with <b>
- option is usable several times e.g. to change storage prefixes or filenames
[recordfilename] : if a file is given, it will be used as record input otherwise STDIN is used to read records!
example: ... --replace file:://localhost:=root://xrootd.eu/ : redirect local file to remote
2.7 xrdreplay output tags
Per file (-l) or aggregated statistics (-s):
tag | description |
*::texec | playback execution time of * in sec |
*::tnomi | recorded execution time of * in sec |
*::tloss | time running after recording in sec |
*::tgain | time gained vs recording in sec |
*::n | number of IO calls to * |
*::b | sum of bytes for IO calls to * |
*::o | highest file offset accessed |
When * is listed, this can be any allowed operation like open,close,read,write,truncate,stat,sync,pgread,pgwrite,vectoread,vectorwrite.
General output:
tag | description | json tag |
Total Runtime | runtime of the replayed recording | player::runtime |
Sampled Runtime | original run time of the recording | sampled::runtime |
Playback Speed | timestretch factor for the playback | player::speed |
IO Volume (R) | total data INGRESS (by read func) | volume::totalread,read,pgread,vectoread |
IO Volume (W) | total data EGRESS (by write func) | volume::totalwrite,write,pgwrite,vectorwrite |
IOPS (R) | total read IO ops (by read func) | iops::read,pgread,vectoread |
IOPS (W) | total write IO ops (by write func) | iops::write,pgwrite,vectorwrite |
Files (R) | sum files with reads | files::read |
Files (W) | sum files with writes | files::write |
Datasize (R) | sum of max file offsets in reads | datasetsize::read |
Datasize (W) | sum of max file offsets in writes | datasetsize::write |
IO BW (R) | bandwidth in MB/s from runtime for reading | bandwidth::mb::read |
IO BW (W) | bandwidth in MB/s from runtimg for writing | bandwidth::mb::write |
Performance Mark | 100 * (sampled runtime) / (total runtime) | performancemark |
Gain Mark (R) | 100 * (sampled read times) / (total rtime) | gain::read |
Gain Mark (W) | 100 * (sampled write times) / (total wtime) | gain::write |
Synchronicity (R) | 100=sync 1=async IO for reads | synchronicity::read |
Syncrhonicity (W) | 100=sync 1=async IO for writes | synchronicity::write |
Response Errors | number of IOs which were not successfull | response::error |
2.8 Memory consumption
It is possible to limit the memory consumption of xrdreplay by setting the XRD_MAXBUFFERSIZE environment variable, following sufixes are supported: kb, mb, gb (case insensitive).