Reference

MediaWiki::DumpFile::Benchmarks(3pm)

Documentation on parsing speeds

NAME

MediaWiki::DumpFile::Benchmarks - Documentation on parsing speeds

ENVIRONMENT

The tests were conducted on Debian/squeeze using the vendor supplied Perl version 5.10.1. All modules were installed from CPAN and not from the Debian package archives.

The host hardware is a 4 core Intel i7 at 2.8ghz with 16 GiB of RAM and disks that can read at 100 MiB/sec.

SOFTWARE

The benchmark software is included in the MediaWiki::DumpFile distribution tarball in the speed_test/ directory.

The tested version of MediaWiki::DumpFile is version 0.2.0

Parse::MediaWikiDump version 1.0.6 is included for comparison against the original implementation.

RESULTS

All times are expressed in seconds. See the SPEED section of the MediaWiki::DumpFile documentation for an explanation of the various parsing modes.

With XML::CompactTree::XS

English Wikipedia

  markup_density: 0.171869387735729
  size: 25247483017
  tests:
    - MiB/sec: 35.0121798854383
      name: suite//MediaWiki-DumpFile-FastPages.t
      percentage: 100
      runtimes:
        system: 25.04
        total: 687.7
        user: 662.66
    - MiB/sec: 30.0620222578669
      name: suite//MediaWiki-DumpFile-Pages_fastmode.t
      percentage: 116
      runtimes:
        system: 26.52
        total: 800.94
        user: 774.42
    - MiB/sec: 23.3355715754023
      name: suite//MediaWiki-DumpFile-Compat_fastmode.t
      percentage: 150
      runtimes:
        system: 28.1
        total: 1031.81
        user: 1003.71
    - MiB/sec: 9.00969754502099
      name: suite//MediaWiki-DumpFile-Pages.t
      percentage: 388
      runtimes:
        system: 31.56
        total: 2672.44
        user: 2640.88
    - MiB/sec: 7.83820750529512
      name: suite//MediaWiki-DumpFile-Compat.t
      percentage: 446
      runtimes:
        system: 31.28
        total: 3071.86
        user: 3040.58
    - MiB/sec: 5.19432459350304
      name: suite//Parse-MediaWikiDump.t
      percentage: 674
      runtimes:
        system: 23.22
        total: 4635.42
        user: 4612.2

Simple English Wikipedia

  markup_density: 0.202659609191331
  size: 227681797
  tests:
    - MiB/sec: 27.9092907599128
      name: suite//MediaWiki-DumpFile-FastPages.t
      percentage: 100
      runtimes:
        system: 0.17
        total: 7.78
        user: 7.61
    - MiB/sec: 26.19231388566
      name: suite//MediaWiki-DumpFile-Pages_fastmode.t
      percentage: 106
      runtimes:
        system: 0.15
        total: 8.29
        user: 8.14
    - MiB/sec: 19.632394404351
      name: suite//MediaWiki-DumpFile-Compat_fastmode.t
      percentage: 142
      runtimes:
        system: 0.19
        total: 11.06
        user: 10.87
    - MiB/sec: 7.20180040172874
      name: suite//MediaWiki-DumpFile-Pages.t
      percentage: 387
      runtimes:
        system: 0.27
        total: 30.15
        user: 29.88
    - MiB/sec: 6.39382456160546
      name: suite//MediaWiki-DumpFile-Compat.t
      percentage: 436
      runtimes:
        system: 0.26
        total: 33.96
        user: 33.7
    - MiB/sec: 4.14457495919301
      name: suite//Parse-MediaWikiDump.t
      percentage: 673
      runtimes:
        system: 0.12
        total: 52.39
        user: 52.27

Chuvash Wikipedia

  markup_density: 0.18934898819024
  size: 39436366
  tests:
    - MiB/sec: 25.7598968401347
      name: suite//MediaWiki-DumpFile-FastPages.t
      percentage: 100
      runtimes:
        system: 0.03
        total: 1.46
        user: 1.43
    - MiB/sec: 22.6562948112028
      name: suite//MediaWiki-DumpFile-Pages_fastmode.t
      percentage: 113
      runtimes:
        system: 0.06
        total: 1.66
        user: 1.6
    - MiB/sec: 17.9949518596156
      name: suite//MediaWiki-DumpFile-Compat_fastmode.t
      percentage: 143
      runtimes:
        system: 0.04
        total: 2.09
        user: 2.05
    - MiB/sec: 7.46219233861045
      name: suite//MediaWiki-DumpFile-Pages.t
      percentage: 345
      runtimes:
        system: 0.04
        total: 5.04
        user: 5
    - MiB/sec: 6.45102047797542
      name: suite//MediaWiki-DumpFile-Compat.t
      percentage: 399
      runtimes:
        system: 0.05
        total: 5.83
        user: 5.78
    - MiB/sec: 4.15574026371234
      name: suite//Parse-MediaWikiDump.t
      percentage: 619
      runtimes:
        system: 0.02
        total: 9.05
        user: 9.03

With out XML::CompactTree::XS

Simple English Wikipedia

  markup_density: 0.202659609191331
  size: 227681797
  tests:
    - MiB/sec: 30.0740002925376
      name: suite//MediaWiki-DumpFile-FastPages.t
      percentage: 100
      runtimes:
        system: 0.18
        total: 7.22
        user: 7.04
    - MiB/sec: 24.3697286321124
      name: suite//MediaWiki-DumpFile-Pages_fastmode.t
      percentage: 123
      runtimes:
        system: 0.23
        total: 8.91
        user: 8.68
    - MiB/sec: 20.2550636298621
      name: suite//MediaWiki-DumpFile-Compat_fastmode.t
      percentage: 148
      runtimes:
        system: 0.19
        total: 10.72
        user: 10.53
    - MiB/sec: 4.21456292919491
      name: suite//Parse-MediaWikiDump.t
      percentage: 713
      runtimes:
        system: 0.12
        total: 51.52
        user: 51.4
    - MiB/sec: 4.06770854462573
      name: suite//MediaWiki-DumpFile-Pages.t
      percentage: 739
      runtimes:
        system: 0.24
        total: 53.38
        user: 53.14
    - MiB/sec: 3.79871032386497
      name: suite//MediaWiki-DumpFile-Compat.t
      percentage: 791
      runtimes:
        system: 0.28
        total: 57.16
        user: 56.88

Chuvash Wikipedia

  markup_density: 0.18934898819024
  size: 39436366
  tests:
    - MiB/sec: 25.5846594466644
      name: suite//MediaWiki-DumpFile-FastPages.t
      percentage: 100
      runtimes:
        system: 0.04
        total: 1.47
        user: 1.43
    - MiB/sec: 20.4399181448895
      name: suite//MediaWiki-DumpFile-Pages_fastmode.t
      percentage: 125
      runtimes:
        system: 0.04
        total: 1.84
        user: 1.8
    - MiB/sec: 14.5210229291879
      name: suite//MediaWiki-DumpFile-Compat_fastmode.t
      percentage: 176
      runtimes:
        system: 0.03
        total: 2.59
        user: 2.56
    - MiB/sec: 4.11481940772393
      name: suite//Parse-MediaWikiDump.t
      percentage: 621
      runtimes:
        system: 0.03
        total: 9.14
        user: 9.11
    - MiB/sec: 3.87726282336048
      name: suite//MediaWiki-DumpFile-Pages.t
      percentage: 659
      runtimes:
        system: 0.03
        total: 9.7
        user: 9.67
    - MiB/sec: 3.53140369827199
      name: suite//MediaWiki-DumpFile-Compat.t
      percentage: 724
      runtimes:
        system: 0.06
        total: 10.65
        user: 10.59

History