If you’re into parsing XML or HTML with ruby then chances are you’re familiar with the various gems out there for getting the job done.  Lately, there have been a lot of things flying around about which is the fastest and to settle it, Aaron Patterson (author of Nokogiri and Mechanize) wrote a test suite.

After it’s release, RubyInside posted about how the tests showed how fast Nokogiri was compared to Hpricot in this article here: Ruby XML Performance Shootout: Nokogiri vs LibXML vs Hpricot vs REXML.  Later in the day, I saw Why’s posting about the release of Hpricot here: hpricot 0.7 and decided to modify Aaron’s tests to use Hpricot.XML and here are the results:

Tests were run at N=5 to get a clearer picture of the differences between the various gems.  At N=2, tests were pretty close, which indicated that a larger sample was needed.

test_IO_parsing(XmlTruth::DOM::XML::LargeDocumentParsingTest) N=5
user     system      total        real   kBps
null          0.690000   0.070000   0.760000 (  0.768641) 46343.68
nokogiri      2.790000   0.130000   2.920000 (  3.015303) 11813.62
libxml-ruby   2.970000   0.140000   3.110000 (  3.130175) 11380.08
hpricot      13.660000   0.370000  14.030000 ( 14.088780) 2528.37
.
test_in_memory_parsing(XmlTruth::DOM::XML::LargeDocumentParsingTest) N=5
user     system      total        real   kBps
null          1.240000   0.010000   1.250000 (  1.260841) 28252.30
nokogiri      4.360000   0.060000   4.420000 (  4.444468) 8014.83
libxml-ruby   4.570000   0.050000   4.620000 (  4.641338) 7674.87
hpricot      13.750000   0.210000  13.960000 ( 14.045647) 2536.13
.
test_simple_xpath(XmlTruth::DOM::XML::LargeDocumentXPathSearchTest) N=5
user     system      total        real   kBps
nokogiri     44.430000   0.300000  44.730000 ( 44.972003) 792.09
libxml-ruby  40.950000   0.210000  41.160000 ( 41.300780) 862.49
hpricot      18.410000   0.090000  18.500000 ( 18.540239) 1921.32
.
test_IO_parsing(XmlTruth::DOM::XML::SmallDocumentParsingTest) N=1944
user     system      total        real   kBps
null          8.150000   0.130000   8.280000 (  8.326070) 4278.17
nokogiri     17.850000   0.100000  17.950000 ( 17.950534) 1984.36
libxml-ruby  19.010000   0.260000  19.270000 ( 19.370769) 1838.87
hpricot      25.320000   0.460000  25.780000 ( 25.827516) 1379.16
.
test_in_memory_parsing(XmlTruth::DOM::XML::SmallDocumentParsingTest) N=1944
user     system      total        real   kBps
null          3.960000   0.030000   3.990000 (  4.005522) 8892.82
nokogiri     18.140000   0.200000  18.340000 ( 18.403396) 1935.53
libxml-ruby  19.760000   0.230000  19.990000 ( 19.999905) 1781.03
hpricot      15.980000   0.150000  16.130000 ( 16.133157) 2207.90
.
Finished in 426.233021 seconds.

5 tests, 0 assertions, 0 failures, 0 errors

You can find my fork of the test suite on github here: Patrick Tulskie’s Fork of XMLTruth

From this small sample of tests, it appears as though Nokogiri and libxml-ruby are similar in performance for most items. This makes sense though since Nokogiri utilizes the native libxml of the current operating environment. Nokogiri clearly excels at parsing larger documents while Hpricot appears to handle smaller, in-memory documents rather quickly.

In real-world scenarios, one might expect Nokogiri to be the ideal solution to parsing large XML or HTML documents from the disk into a database, whereas Hpricot might be a more ideal gem for use in a web crawler where it is rare that a page’s DOM is more than a 1MB.

Please post any other thoughts you might have in the comments.

Naturally I was a little freaked out.

After retracing my steps over the past hour I remembered I had updated my ruby gems with a good ol “sudo gem update.”  I do it all the time so I didn’t see the cause for concern.  I went and looked at the newly installed gems and saw that there was libxml-ruby-1.0.0.  I browsed inside the gem and saw that it had a bin directory that had a ruby executable in it.  Cute.  Whoever the person is who released that needs to pay super close attention to what they are doing in the future.

Anyhow, I uninstalled the gem and when it asked if I wanted to remove the ruby executable I said yes.  This of course trashed the ruby executable in my /usr/bin.  Luckily I was able to retrieve it from Jay Amster and all was well.  If I was to do things over I’d say not to trash the executable and just delete the gem and all of its files.

Having that broken ruby executable in my path devastated my system though.  Half of my Textmate scripts no longer worked, none of my rails apps would execute, etc.  It was awful.  Thankfully I was able to figure it out quickly and hopefully if you run a search for “you need to write me” then you’ll stumble upon this post and know what to do to fix your machine.

UPDATE:

It would appear as though this problem is now resolved.  Maybe I got a bad install of the gem?  Maybe it was just a fluke?  Who knows?  It appears safe to install the latest libxml-ruby now though.