website screen capture in ruby - HowTo

For one of the projects that I am working on, I needed a website capture utility, that will dump output into a png file. I couldn't find any one-stop-shop for getting it done. There are many websites that have this service (see references towards the end in the post), however I needed something where I have access to the source code, so that I could tweak it to my needs Also something in ruby was prefereable, so that I can integrate it with other stuff I am working on. Google, google and google and finally I hit upon an implementation in ruby called moz-snapshooter.rb, which has got a good screen capture facility that uses 'gtkmozembed' , which is an embeddable Gecko rendering widget that GTK applications can embed. This was pretty much it. All that was needed was the ruby binding for gnome2 libraries etc. They can be found here.

So here is the final deal (on ubuntu Dapper) -

1. Get the ruby-gnome2-all package from here

2. Get the firefox-dev package using apt-get. (Note this is required for building the gtkmozembed.)

3. Get the moz-snapshooter.rb

4. Edit it to your needs

5. Fire ruby moz-snapshooter.rb

Most of the things just work out of the box.

Now comes a little tricky part. All of this requires an X server running. So what if one wants to run it on a headless X server? The answer is Xvfb (which is a virtual frame buffer based X server.) Continuing further.

6. Install Xvfb using apt-get.

7. Finally run the 'ruby moz-snapshooter.rb' inside 'xvfb-run' as follows
xvfb-run -s "-screen 800x600x16" ruby moz-snapshooter.rb

Note to pass -s "-screen 800x600x16" option, since most of the websites work well in that resolution.

Caveats:

1. The libgtk-mozembed-ruby package that comes for 'ubuntu' uses the gtkembed.so library provided by mozilla 1.7.X (which is not compiled with pango renderer. It is better to use firefox-dev library and hence build by hand the ruby-gnome2-all package.)

2. The final output resolution image should be of smaller resolution than the one passed to xvfb-run.

3. This is not 'perfect' yet, but a good starting point.

Here are some samples.

1. Google in 150x100 resolution -



2. paahijen.com in 320x200 resolution -



3. Marathi wikipedia in 640x480 resolution -




References :

1. A pygtk implementation by Andrew McCall
2. moz-snapshooter.rb by Mirko Maischberger
3. Ideum screenshot capture prototype

A nifty Ruby regexp trick

Just hit upon this while working on a rails project. Thought it might be worth sharing with others in case someone is interested.
The idea is as follows ->
suppose one defines a simple regular expression as follows

r = '(a)|(b)|(c)'

re = Regexp.new(r)

This re can be used with either String.match. The String.match returns a MatchData object. Often one is interested in an array. which can be obtained by String.scan instead of String.match. However this array is actually an array of Arrays, with "nil" at places where the string doesn't match..
eg.

"abhijit".scan(re)

will return

[["a", nil, nil], [nil, "b", nil], [nil, nil, "h"]]

If one just wants the array as ["a", "b", "c"], one could use following trick.

"abhijit".scan(re).flatten.compact

The flatten flattens out the array and compact gets rid of all nil
pretty useful .. One can then iterate over this array. Somehow this convenience is not offered by MatchData or atleast I don't know of it.

Understanding blocks in Ruby...

Blocks in Ruby.

I am trying to learn Ruby, because I am working on a Rails project. If one browses around for Ruby literature on the web, one often finds a common thing - 'Blocks are the best part of Ruby'. I have often used them while trying to do something.find { do stuff } . But never really appreciated what they were, till I decided to solve that mystery. If one googles 'ruby blocks', one finds typical examples illustrating how they are used, but not necessarily why and how they should be used. This sort of confused me to a greater degree. Basically, after spending a good couple of hours, I realized what blocks are how they are used and what is similarity difference between ruby blocks and python lambda?


1. Blocks are typically executed with an iterator -

Basically, a block is a piece of code which is enclosed between a '{' and '}' and follows immediately following a method call. Typically one would find

myarray.each {|x| p x;}


'each' is basically an 'iterator' over an Array object. Ok so what is an 'iterator' - is a method that invokes block of code repeatedly. Thus first use of 'blocks' is - they can be used to pass random code to an 'iterator' and that gets executed. But then what is the big deal about it? Well basically at first sight, one doesn't quite appreciate this. But consider -
- Find the square of each number in the array
- Find a list of all numbers in an array that is divisible by three

Now to solve the two problems ordinarily one would require two methods 'do_square' and 'is_divisible_by_3', but 'blocks' make it much simpler. Just pass a different code block to the 'each' methord.

2. Blocks can be used for 'transactional use'.

A very common example of this is code like

File.open('foo.txt', 'r') do |afile|
  while ( s = afile.gets)
    p s
  end
end

At first sight, this might appear like another use of 'iterator' an iterator oer lines of a file. But this is not strictly true.

the 'open' function of a File class (note not Object) 'yield's the block above. Thus actually the whole of while (s = ...) code is getting executed in the File.open function itself. Sounds a bit un-intuitive. Ok.. So what's the big deal again? Well this takes care of Opening and Closing the file when done automatically in case the user forgets to do so.

3. Blocks can be used to create 'Proc' objects

This use of ruby blocks is like 'python lambda'. Though ruby blocks are far more flexible than python lambda as python lambda allows you only a single line of code.
consider this

def foo(x)
  return proc { |y| x+y;}
end

p = foo(2)

p.call(2) = 4
p.call(2000) = 2002

I haven't really used this form of blocks yet. So can't comment about where this can be useful.

Actually a lot of what I have understood can be found on this page of ruby book.