120 lines
4.2 KiB
Markdown
120 lines
4.2 KiB
Markdown
# `bench_compare`
|
|
This tool helps with comparing times different YAML parsers take to parse the same input.
|
|
|
|
## Synopsis
|
|
```
|
|
bench_compare time_parse
|
|
bench_compare run_bench
|
|
```
|
|
|
|
This will run either `time_parse` or `run_bench` (described below) with the given set of parsers from the configuration file.
|
|
|
|
## Parsers requirements
|
|
Parsers are expected to be event-based. In order to be fair to this crate's benchmark implementation, parsers should:
|
|
|
|
* Load the file into memory (a string, `mmap`, ...) **prior** to starting the clock
|
|
* Initialize the parser, if needed
|
|
* **Start the clock**
|
|
* Read events from the parser while the parser has not finished parsing
|
|
* Discard events as they are received (dropping them, `free`ing them or anything similar) so as to not grow their memory consumption too high, and allowing the parser to reuse event structures
|
|
* **Stop the clock**
|
|
* Destroy the resources, if needed/wanted (parser, file buffer, ...). The kernel will reap after the process exits.
|
|
|
|
|
|
## Parsers required binaries
|
|
This tool recognizes 2 binaries: `time_parse` and `run_bench`.
|
|
|
|
### `time_parse`
|
|
Synopsis:
|
|
```
|
|
time_parse file.yaml [--short]
|
|
```
|
|
|
|
The binary must run the aforementioned steps and display on its output the time the parser took to parse the given file.
|
|
With the `--short` option, the binary must only output the benchmark time in nanoseconds.
|
|
|
|
```sh
|
|
# This is meant to be human-readable.
|
|
# The example below is what this crate implements.
|
|
$> time_parse file.yaml
|
|
Loaded 200MiB in 1.74389s.
|
|
|
|
# This will be read by this tool.
|
|
# This must output ONLY the time, in nanoseconds.
|
|
$> time_parse file.yaml --short
|
|
1743892394
|
|
```
|
|
|
|
This tool will always provide the `--short` option.
|
|
|
|
### `run_bench`
|
|
Synopsis:
|
|
```
|
|
run_bench file.yaml <iterations> [--output-yaml]
|
|
```
|
|
|
|
The binary is expected to run `<iteration>` runs of the aforementioned steps and display on its output relevant information.
|
|
The `--output-yaml` instructs the binary to output details about its runs in YAML on its standard output.
|
|
The binary may optionally perform some warmup runs prior to running the benchmark. The time it took the binary to run will not be evaluated.
|
|
|
|
```sh
|
|
# This is meant to be human-readable.
|
|
# The example below is what this crate implements.
|
|
$> run_bench file.yaml 100
|
|
Average: 1.589485s
|
|
Min : 1.583078s
|
|
Max : 1.597028s
|
|
95% : 1.593219s
|
|
|
|
# This will be read by this tool.
|
|
# This must output a YAML as described below.
|
|
$> run_bench ../file.yaml 10 --output-yaml
|
|
parser: yaml-rust2
|
|
input: ../file.yaml
|
|
average: 1620303590
|
|
min: 1611632108
|
|
max: 1636401896
|
|
percentile95: 1636401896
|
|
iterations: 10
|
|
times:
|
|
- 1636401896
|
|
- 1623914538
|
|
- 1611632108
|
|
- 1612973608
|
|
- 1617748930
|
|
- 1615419514
|
|
- 1612172250
|
|
- 1620791346
|
|
- 1629339306
|
|
- 1622642412
|
|
```
|
|
|
|
The expected fields are (all times in nanoseconds):
|
|
|
|
* `parser`: The name of the parser (in case of a mistake renaming files)
|
|
* `input`: The path to the input file as given to the binary arguments
|
|
* `average`: The average time it took to run the parser
|
|
* `min`: The shortest time it took to run the parser
|
|
* `max`: The longest time it took to run the parser
|
|
* `percentile95`: The 95th percentile time of the runs
|
|
* `iterations`: The number of times the parser was run (`<iterations>`)
|
|
* `times`: An array of `iterations` times, one for each run, in the order they were run (first run first)
|
|
|
|
## Configuration
|
|
`bench_compare` is configured through a `bench_compare.toml` file. This file must be located in the current directory.
|
|
As of now, default values are unsupported and all fields must be set. The following fields are required:
|
|
```toml
|
|
yaml_input_dir = "bench_yaml" # The path to the directory containing the input yaml files
|
|
iterations = 10 # The number of iterations, if using `run_bench`
|
|
yaml_output_dir = "yaml_output" # The directory in which `run_bench`'s yamls are saved
|
|
csv_output = "benchmark.csv" # The CSV output aggregating times for each parser and file
|
|
|
|
[[parsers]] # A parser, can be repeated as many times as there are parsers
|
|
name = "yaml-rust2" # The name of the parser (used for logging)
|
|
path = "target/release/" # The path in which the parsers' `run_bench` and `time_parse` are
|
|
|
|
# If there is another parser, another block can be added
|
|
# [[parsers]]
|
|
# name = "libfyaml"
|
|
# path = "../libfyaml/build"
|
|
```
|