Here’s how you can make the change, on the fly, inside of your rake task:

task :swimming_pool => :environment do
  ActiveRecord::Base.connection_pool.instance_variable_set('@size', 15)
  ActiveRecord::Base.connection_pool.instance_variable_set('@timeout', 10)
  # Do awesome threaded stuff here
end

The first line of the rake task increases your pool size and the second line changes your timeout when waiting for a connection from the pool. You can adjust those values to whatever you need. That just happened to be what I need for this particular task.

If you’re into parsing XML or HTML with ruby then chances are you’re familiar with the various gems out there for getting the job done.  Lately, there have been a lot of things flying around about which is the fastest and to settle it, Aaron Patterson (author of Nokogiri and Mechanize) wrote a test suite.

After it’s release, RubyInside posted about how the tests showed how fast Nokogiri was compared to Hpricot in this article here: Ruby XML Performance Shootout: Nokogiri vs LibXML vs Hpricot vs REXML.  Later in the day, I saw Why’s posting about the release of Hpricot here: hpricot 0.7 and decided to modify Aaron’s tests to use Hpricot.XML and here are the results:

Tests were run at N=5 to get a clearer picture of the differences between the various gems.  At N=2, tests were pretty close, which indicated that a larger sample was needed.

test_IO_parsing(XmlTruth::DOM::XML::LargeDocumentParsingTest) N=5
user     system      total        real   kBps
null          0.690000   0.070000   0.760000 (  0.768641) 46343.68
nokogiri      2.790000   0.130000   2.920000 (  3.015303) 11813.62
libxml-ruby   2.970000   0.140000   3.110000 (  3.130175) 11380.08
hpricot      13.660000   0.370000  14.030000 ( 14.088780) 2528.37
.
test_in_memory_parsing(XmlTruth::DOM::XML::LargeDocumentParsingTest) N=5
user     system      total        real   kBps
null          1.240000   0.010000   1.250000 (  1.260841) 28252.30
nokogiri      4.360000   0.060000   4.420000 (  4.444468) 8014.83
libxml-ruby   4.570000   0.050000   4.620000 (  4.641338) 7674.87
hpricot      13.750000   0.210000  13.960000 ( 14.045647) 2536.13
.
test_simple_xpath(XmlTruth::DOM::XML::LargeDocumentXPathSearchTest) N=5
user     system      total        real   kBps
nokogiri     44.430000   0.300000  44.730000 ( 44.972003) 792.09
libxml-ruby  40.950000   0.210000  41.160000 ( 41.300780) 862.49
hpricot      18.410000   0.090000  18.500000 ( 18.540239) 1921.32
.
test_IO_parsing(XmlTruth::DOM::XML::SmallDocumentParsingTest) N=1944
user     system      total        real   kBps
null          8.150000   0.130000   8.280000 (  8.326070) 4278.17
nokogiri     17.850000   0.100000  17.950000 ( 17.950534) 1984.36
libxml-ruby  19.010000   0.260000  19.270000 ( 19.370769) 1838.87
hpricot      25.320000   0.460000  25.780000 ( 25.827516) 1379.16
.
test_in_memory_parsing(XmlTruth::DOM::XML::SmallDocumentParsingTest) N=1944
user     system      total        real   kBps
null          3.960000   0.030000   3.990000 (  4.005522) 8892.82
nokogiri     18.140000   0.200000  18.340000 ( 18.403396) 1935.53
libxml-ruby  19.760000   0.230000  19.990000 ( 19.999905) 1781.03
hpricot      15.980000   0.150000  16.130000 ( 16.133157) 2207.90
.
Finished in 426.233021 seconds.

5 tests, 0 assertions, 0 failures, 0 errors

You can find my fork of the test suite on github here: Patrick Tulskie’s Fork of XMLTruth

From this small sample of tests, it appears as though Nokogiri and libxml-ruby are similar in performance for most items. This makes sense though since Nokogiri utilizes the native libxml of the current operating environment. Nokogiri clearly excels at parsing larger documents while Hpricot appears to handle smaller, in-memory documents rather quickly.

In real-world scenarios, one might expect Nokogiri to be the ideal solution to parsing large XML or HTML documents from the disk into a database, whereas Hpricot might be a more ideal gem for use in a web crawler where it is rare that a page’s DOM is more than a 1MB.

Please post any other thoughts you might have in the comments.

Naturally I was a little freaked out.

After retracing my steps over the past hour I remembered I had updated my ruby gems with a good ol “sudo gem update.”  I do it all the time so I didn’t see the cause for concern.  I went and looked at the newly installed gems and saw that there was libxml-ruby-1.0.0.  I browsed inside the gem and saw that it had a bin directory that had a ruby executable in it.  Cute.  Whoever the person is who released that needs to pay super close attention to what they are doing in the future.

Anyhow, I uninstalled the gem and when it asked if I wanted to remove the ruby executable I said yes.  This of course trashed the ruby executable in my /usr/bin.  Luckily I was able to retrieve it from Jay Amster and all was well.  If I was to do things over I’d say not to trash the executable and just delete the gem and all of its files.

Having that broken ruby executable in my path devastated my system though.  Half of my Textmate scripts no longer worked, none of my rails apps would execute, etc.  It was awful.  Thankfully I was able to figure it out quickly and hopefully if you run a search for “you need to write me” then you’ll stumble upon this post and know what to do to fix your machine.

UPDATE:

It would appear as though this problem is now resolved.  Maybe I got a bad install of the gem?  Maybe it was just a fluke?  Who knows?  It appears safe to install the latest libxml-ruby now though.

The only real “feature” is that it will now handle full names just fine. Otherwise, the rest of the stuff was mostly performance related.  Here it is:

class String
  # Extension of the string class to properly handle camel names
  def nameize
    if self.match(/ /)
      # If the name has a space in it, we gotta run the parts through the nameizer.
      name = self.split(' ').each { |part| part.nameize! }.join(' ')
      return name
    elsif self.match(/^[A-Z]/)
      # If they took the time to capitalize their name then let's just jump out.
      return self
    else
      # If there are no spaces and there is no prior
      # capitalization then let's downcase the whole thing.
      name = self.downcase
    end
    # Let's now assume that they were lazy...
    return case
    when name.match(/^mac/)
      name.gsub(/^mac/, "").capitalize.insert(0, "Mac")
    when name.match(/^mc/)
      name.gsub(/^mc/, "").capitalize.insert(0, "Mc")
    when name.match(/^o\'/)
      name.split("'").each{ |piece| piece.capitalize! }.join("'")
    else
      name.capitalize # name is a first name or it's not Irish then capitalize it.
    end
  end

  def nameize!
    replace nameize # BANG!
  end

end

As always – question, comments, suggestions – shoot me an email, leave a comment, or hit me on Twitter (@PatrickTulskie).

Yeah it’s great, except this Friday it didn’t happen.  Thanksgiving weekend happened instead so we pushed it to Monday.  Being the silly goose I am, I decided to get a new Macbook on Black Friday.  I restored my stuff from Time Machine, installed my Ruby Gems, and thought all was well.  Monday morning, my turn to present code came up and there was a problem with screen sharing.  Crap.  All of my code is in a git branch that is not pushed to a server yet and the time it would have taken to get to a state where we could present it from another machine would have been too much so we postponed my review until my screen sharing would work.  This was most displeasing to me.

How To Handle It Next Time.

As of late, there has been a lot of talk about bonjour in the ruby world.  There are a lot of really awesome applications that use the technology.  Take for example, gitjour.  It takes an existing git repository and makes it available to machines on the local network through bonjour.  All they would need is the gem and to type “gitjour list” in a terminal window to see what’s available.  That’s badass.

So if we had to rewind to Monday morning, what we should have done is take a functioning machine on the local network and had that person do a clone of my repository to a new working directory and then we could have screen shared that.  It would have taken a fraction of the time that it would have taken to push it to the server with my own local branch, tell someone to get a copy of my branch from the server and what branch to take.  Gitjour would have cut out all of the steps and it would have been quicker since it’s on the local network.

This thought alone makes me want to dive deeper into bonjour and figure out how to use it for various things.  If you have any good resources, post them in the comments or hit me up on Twitter. (http://twitter.com/patricktulskie)

What Was The Screen Sharing Problem?

Turns out iChat 4.0.5 doesn’t know how to view the screen of someone with iChat 4.0.6.  The new Macbook comes with 4.0.6 by default and there is no easy way to upgrade to it.  I downgraded to 4.0.2 from my Blackbook and now all is well.

WARNING: You’re using the Ruby-based MySQL library that ships with Rails. This library is not suited for production. Please install the C-based MySQL library instead (gem install mysql).

Normally I don’t care, but I figured since I was doing some cleanup today and getting things ready to move on to a longer term it might be good to have a properly working MySQL gem.  I like to run with a system that is close to what we run production.  The closer you get, the less surprises you have when you push it live.

More Fixing.  Less Talking.

Damn you’re so pushy sometimes.  Anyhow.  I did a sudo gem install mysql and got another damn error.

Building native extensions.  This could take a while...
ERROR:  Error installing mysql:
ERROR: Failed to build gem native extension.

/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby extconf.rb install mysql
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lm... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lz... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lsocket... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lnsl... no
checking for mysql_query() in -lmysqlclient... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Oh good.  Another error.  Perfect.  I searched around the interwebs and someone suggested using “–with-mysql-lib=/usr/local/mysql/lib” in the options since that’s where the libraries are located.  It still barfed on me with that.

A Solution Please.

A few more minutes of hunting and pecking and I found the golden command:

sudo gem install mysql -- --with-mysql-config
Building native extensions.  This could take a while...
Successfully installed mysql-2.7
1 gem installed

Now I don’t have any errors when starting my rails applications that use MySQL and I’m closer to what I have in production.

class String
  # Extension of the string class to properly handle camel names
  # Should be used on pieces of names, not full names.
  def nameize
    # If they took the time to capitalize their name then let's just jump out
    if self.match(/\A[A-Z]/)
      return self
    else
      name = self.downcase
    end
    # Let's now assume that they were lazy...
    return case
    when name.match(/^mac/)
      name.capitalize.gsub(/Mac/, "").capitalize.insert(0, "Mac")
    when name.match(/^mc/)
      name.capitalize.gsub(/Mc/, "").capitalize.insert(0, "Mc")
    when name.match(/^o\'/)
      name.split("'").each{ |piece| piece.capitalize! }.join("'")
    else
      name.capitalize
    end
  end
end

I know it’s not perfect, but it should handle most oddly capitalized names.  Let me know what you think in the comments or on Twitter or whatever and definitely give me any changes you think should be in there.

…

So what if I started off my post with a pun?  Look you’re just going to have to deal with it.  Let’s move on.  No stop it with the face, it’s my blog and I can do what I want.

Anyhow – REST is one of those things I hear a lot of people talk about as if it is this big mystery.  Then, when you finally find someone to explain it, it turns out they have never actually used it and only understand the theory.  It’s so frustrating to the beginning Ruby on Rails developer because not everyone has a mini David Hansson in their pocket and it really is one of those things you need to see to understand.

Ok Patrick.  I’ll bite.  Let’s hear more about this.

It’s a really simple concept that I will explain abstractly at first and then I want you to forget what I said until I compare it to what you’ve probably already seen.  REST stands for Representational State Transfer.  Technically speaking, REST should be RST but since Roy Fielding coined the term in the year 2000 and he’s kind of a big deal, people tend to stick with his acronym.

The theory behind it is that your destination |object| is nothing more than a representation of that object at the time you are viewing it.  This plays extremely well into a MVC framework like Ruby on Rails since your controllers and models handle the object and the view is just how you see the object at a given point in time.  Then in the URL, you would just be using nouns to describe what you’re getting.  Sorta like this:

http://yourdomain.com/noun1/…/your_object_name

That makes no sense.

I know.  Look, it’s actually really simple.

Ok so say I have a web application that you would like to pull data from in order to handle however you’d like with a separate application you are creating.  An example might be a news site – they provide for you an RSS feed that.  The data is all the same server side but how your web browser sees it is different from how your RSS feed reader sees it so the object (the data) needs to change its form to be readable to the current application.  With me so far?

If I wrote my web application to be RESTful then your application could request the data in RSS or XML or whatever form you prefer to parse, all while maintaining a normal user’s view in their web browser.  Ah ha!  So we’re no longer web page builders – we’re data providers.  You’re now actually building page templates and writing more code to handle your data.

I’m still a little unsure about this Patrick.

I was too, so I sat down and tried it.

On my side project – www.Twexaminer.com – I made sure that my controller was pulling data from twitter and storing it into objects instead of just “building a page.”  From there I created 2 views.  One is the default HTML view that you see when you view the page from a web browser and the other is a special .xml request that gives you the output of all of the data that the application has collected.

Ex:
http://www.twexaminer.com/examiner/result/patricktulskie
-vs-
http://www.twexaminer.com/examiner/result/patricktulskie.xml

Basically I’m going to my examiner and saying “hey, give me a result” and then clarifying that it should be PatrickTulskie.  In the second example there I tell the examiner to give me the result of PatrickTulskie in XML format.

I could further expand this to make more views or representations for the same data, but for the time being I just have the 2.  In this way, I am now passing on the content using standard looking URLs that a common person can understand.  In addition, if you want to parse the data easily, there is a very intuitive way to get the data without the formatting fluff.  I am effectively passing data to your app, without the use of SOAP or any special hacks, in a format it understands just with a simple GET.  Ah ha!  Simple!

Ok now if I wanted to get fancier then I could add something to my examiner controller to handle updating of a cached result so it would look something like this… in pseudo code…

pt_object = new RestResource(http://www.twexaminer.com/examiner/result/patricktulskie)
pt_object.update

…and then pull off data from that.  This would all be handled through HTTP requests though without needing any additional layers to handle transferring XML to and from client and server.  RestResource is a hypothetical object that one could create that has a simple XML parser to make an easy to use hash out of the data.  XML parsing and generation is easy in Ruby so this just makes sense.

Ah so this is pretty sweet.

Yeah. I know.  It makes things much simpler when you’re creating something that uses my data now doesn’t it?  Go, play with it in your project and let me know what you think in the comments or on Twitter, or through email, or through facebook… or whatever.  Just make sure I get the “comment” you’re trying to convey through some medium.