X.Y.Z

Inspect Ruby Process From gdb

2011-08-11T09:55:00.000-07:00

If you have been developing in Ruby and Ruby on Rails for any length of time, it is very likely that you have encountered a 'Segmentation fault' from Ruby without any information about where the error is coming from.

Normally, a gem written in C extension is to blame. If pure Ruby code meets an error, a nice stacktrace with a line number will be there to point you in the right direction, but not for code in C extension. You won't find more information about the error unless you fire up gdb.

Assuming the code that crashes is bug.rb, instead of doing 'ruby bug.rb', just do:
$gdb ruby
(gdb) run bug.rb

Once the the crash is reproduced, type:
(gdb) bt

'bt' means 'backtrace'. A C callstack will show up. And you will find some valuable information there. If 'bt' does not give you function name and line# you want, it normally means it can not locate the source code, and you can use the 'dir' command to help gdb locate the source code (Ruby's source code is freely available, just download the one that matches the binary):
(gdb) dir /root/downloaded/ruby-1.8.6-p111

If the program that crashed is a rake task, just do some thing similar. Rake itself is a Ruby script:
$gdb ruby
(gdb) run /usr/bin/rake -T

If you have a daemon written in Ruby, sometimes you might see that it hangs (does nothing) or uses too many resources (100% cpu usage). To inspect a live process with process id 12345, do this:
$gdb ruby
(gdb) attach 12345
(gdb) bt

For Rails applications, a bug may only happen in the production environment. Before you enter 'run', use this to setup the environment:
(gdb) set environment RAILS_ENV = production

Here are some tips to get more from a C stacktrace.
You can call a C function with a 'call' command under gdb shell -- yep, that makes C feel like an interpreted language. For example, you have a stack trace entry like this:
rb_call0 (klass=47110398570320, recv=47110462241640, id=22977,
oid=, argc=0, argv=0x7fffee0453c8, body=0x2ad8be3946c8,
flags=) at eval.c:5998
You can find out what 'klass' is by using its memory address:
(gdb) call rb_class2name(47110398570320)
Or you can look up a Ruby function's name:
(gdb) call rb_id2name(22977)

This GDB QUICK REFERENCE and Extending Ruby chapter of 'Programming Ruby' book are great references if you want to know more!

How to debug or test cron script

2009-01-12T13:31:00.000-08:00

If your simple cron script (/etc/cron.daily, etc/cron.hourly etc) does not run as expected, do the following:

1. be sure to have '#!/bin/sh' as the first line
2. make sure the script's name does not contains '.'. for example, you should rename 'yourscript.sh' to 'yourscript'.
3. use absolute path if you need to read/write files in the script
4. try run the script in the shell to make sure there is no obvious problem
5. 'run-parts --test /etc/cron.hourly/' will tell you what scripts will run, you should see your script in the list
6. next try to run it like cron does, for example:
cd / && run-parts --report /etc/cron.hourly
7. cron uses syslog for logging. Check your syslog config file ( /etc/syslog.conf) to see where the log is, and check the log for errors.

Ref:
[1] Why are cron.hourly files not running? SOLVED
https://lists.ubuntu.com/archives/ubuntu-users/2007-July/118973.html

Google Summer of Code with Ruby Central

2008-03-24T18:40:00.000-07:00

Ruby Central will once again participate the Google Summer of Code in 2008 as a mentoring organization:
http://code.google.com/soc/2008/ruby/about.html

Pat Eyler has put together an idea page, which is a great reference. While every organization may have different policy, Ruby Central has been happy to see students' own ideas in the past, and I believe this year is not an exception. Feel free to submit your own exciting ideas so that the mentors can pick the best ones.

Today is the first day for Google to accept student applications. For student who are interested in participating, my advise is to apply as early as possible: normally students who apply early has a slight advantage because mentors will have more time to read their applications.

And another advise for future GSOC students is: once accepted, please make sure you are well prepared for the program. Google and mentors normally assume you to work near fulltime in the summer, but unfortunately every year we saw 'disappearing' students during the program. If for some reasons you can not make it after being accepted, please communicate with your mentor organization early so that they can make better adjustment.

XRuby license changed to BSD

2008-03-24T17:57:00.001-07:00

XRuby had been licensed under GNU General Public License v2. For people who want to create solutions based on XRuby, this means they had to adopt GPL as well, which is not always possible. So since 0.3.3, we changed the license to BSD. Hopefully this will help XRuby gain more acceptance, especially from the corporate world.

XRuby 0.3.1 released

2007-09-24T07:45:00.001-07:00

XRuby 0.3.1 is released today and you can download the latest version here:
http://code.google.com/p/xruby/downloads/list

The major changes in this version are:

1. Ruby standard libraries are now pre-compiled and the compilation results (java bytecode) are shipped as part of the xruby.jar. This makes deployment really easy, as there is one jar file to deal with. Pre-compiled libraries means it isn't necessary to compile these code every time by the user which is helpful to performance.

The size of the final jar may cause some concerns: while the number of ruby stdlib supported by the current version of XRuby is still limited, the size of the compiled stdlib has already reached 2,000 KB (the entire xruby-0.3.1.jar is over 4,000 KB). Maybe we will ship ruby stdlib in a separated jar files in the future.

2. Use annotation and code generation to bind Java level method to Ruby level method. We started to make this change in 0.3.0 and finished it in 0.3.1. It helped us to get rid of lots of redundant code. For more information about this change, please read our last announcement and dreamhead's articles(1, 2) on his blogs (in Chinese).

Thank everyone who has contributed to this release.

xruby 0.3.0 released

2007-08-07T22:05:00.000-07:00

I am pleased to announce that XRuby 0.3.0 is released:
http://code.google.com/p/xruby/downloads/list

We have fixed lots of bugs and made significant improvement in the code.

Changes from 0.2.1 to 0.3.0:
1. Use annotation and code generation to bind Java level method to Ruby level method (I will talk more about this later).

2. More unit tests passed. We have not eliminated all test failures in test/ruby. But as most of the failures are caused by the implementation of builtin libraries, we will be able to fixed them soon in 0.4.0.

Changes from 0.2.0 to 0.2.1:
1) Dreamhead optimized method/block calls for methods with zero/one
arguments. It makes our performance even better.

2) ZhangYu improved Java integration significantly, he also created a wiki page with lots of good examples:
http://code.google.com/p/xruby/wiki/JavaIntegration

3) Mechiland and I made more ruby unit tests pass.

The most significant change of 0.3.0 is the using of annotation and code generation to bind Java level method to ruby level method. The idea was inspired by the discussions about Java 5 on jruby's maillist, and dreamhead turned it into reality quickly.

As we know, a Ruby method does a little bit more than a Java method. So if we have a method like this in Java:

public class RubyString {
   public RubyFloat to_f() {
       ...
   }
}

To turn it (RubyString.to_f) into a Ruby level method, we have to add a few more code to 'wrap' into a class (extends RubyMethod) and 'register' it (defineMethod), e.g:

public class String_to_f extends RubyNoArgMethod {
   protected RubyValue run(RubyValue receiver, RubyBlock block) {
       return ((RubyString)receiver).to_f();
   }
}
...
RubyRuntime.StringClass.defineMethod("to_f", new String_to_f());

For every method, we need to write similar code and it is not fun to repeat yourself. In 0.3.0, we no longer have to do this anymore. As as long as you add annotation like this:

@RubyLevelClass(name="String")
public class RubyString {
   @RubyLevelMethod(name="to_f")
   public RubyFloat to_f() {
       ...
   }
}

XRuby will turn it into a Ruby level method automatically (using ASM to generate Java bytecode).

I have not used Java 5's annotation feature before, but this looks like an very elegant solution.

Thank everyone who has contributed to this release.

ruby -y

2007-06-12T08:15:00.001-07:00

For people who are interested in how ruby's yacc parser works, there is an undocumented command line option("-y") that may be helpful. It will display a trace of the parser's operations.

To use it, you need to clear your RUBYOPT environment variable to NOT use "rubygems" (this will break some ruby applications), otherwise it will make too much noise.

Here is an example:


$ruby -y -e "a=1"
Starting parse
Entering state 0
Reducing stack by rule 1 (line 328), -> @1
Stack now 0
Entering state 2
Reading a token: Next token is token tIDENTIFIER ()
Shifting token tIDENTIFIER, Entering state 34
Reading a token: Next token is token '=' ()
Reducing stack by rule 418 (line 2146), tIDENTIFIER -> variable
Stack now 0 2
Entering state 90
Next token is token '=' ()
Reducing stack by rule 83 (line 827), variable -> lhs
Stack now 0 2
Entering state 73
Next token is token '=' ()
Shifting token '=', Entering state 315
Reading a token: Next token is token tINTEGER ()
Shifting token tINTEGER, Entering state 40
Reducing stack by rule 414 (line 2134), tINTEGER -> numeric
Stack now 0 2 73 315
Entering state 89
Reducing stack by rule 376 (line 1899), numeric -> literal
Stack now 0 2 73 315
Entering state 79
Reducing stack by rule 267 (line 1421), literal -> primary
Stack now 0 2 73 315
Entering state 75
Reading a token: Next token is token '\n' ()
Reducing stack by rule 217 (line 1199), primary -> arg
Stack now 0 2 73 315
Entering state 488
Next token is token '\n' ()
Reducing stack by rule 173 (line 953), lhs '=' arg -> arg
Stack now 0 2
Entering state 74
Next token is token '\n' ()
Reducing stack by rule 40 (line 616), arg -> expr
Stack now 0 2
Entering state 64
Next token is token '\n' ()
Reducing stack by rule 34 (line 596), expr -> stmt
Stack now 0 2
Entering state 63
Next token is token '\n' ()
Reducing stack by rule 6 (line 381), stmt -> stmts
Stack now 0 2
Entering state 62
Next token is token '\n' ()
Shifting token '\n', Entering state 216
Reducing stack by rule 496 (line 2429), '\n' -> term
Stack now 0 2 62
Entering state 220
Reducing stack by rule 497 (line 2432), term -> terms
Stack now 0 2 62
Entering state 300
Reading a token: Now at end of input.
Reducing stack by rule 489 (line 2416), terms -> opt_terms
Stack now 0 2 62
Entering state 299
Reducing stack by rule 4 (line 373), stmts opt_terms -> compstmt
Stack now 0 2
Entering state 61
Reducing stack by rule 2 (line 328), @1 compstmt -> program
Stack now 0
Entering state 1
Now at end of input.

XRuby's ANTLR parser does not have this option. But as ANRLE produces human readable code, you can just read the generated code or step through it in a debugger to learn how it works.

XRuby 0.2.0 released

2007-05-15T08:18:00.001-07:00

I am glad to announce that XRuby 0.2.0 is now available for download at:
http://code.google.com/p/xruby/downloads/list

Here is a summary of major changes in this release:

1. Beanworms started to work on the debugging support and we now have a basic debugger in trunk.
2. Zhang Yu and Haofei Wang added/fixed lots of builtin methods
3. Dreamhead rewrote RubySymbol as the old one was plain wrong
4. As test::unit works on XRuby, we imported unit tests from c ruby (test\ruby) to test our implementation. The result was not very impressive so far, less than half of the tests passed. We are planning to make all tests pass in the coming weeks.
5. Meanwhile, Femto is leading our efforts to create an ANTLR v3 based ruby parser in another branch. Upgrading our parser from ANTLR v2 to v3 is indeed challenging, but he has make very impressive progress.

Thank you everybody who contributed to this release!

InfoQ article on XRuby

2007-04-17T04:16:00.001-07:00

Werner Schuster from InfoQ has a very nice article about XRuby, you can read it here.

UPDATE 04/27/2007: InfoQ China had an interview with our core developer Dreamhead. You can read it here (in Chinese). And congratulations to Dreamhead for joining Thoughtworks.

XRuby 0.1.4 released

2007-04-10T09:15:00.000-07:00

I am glad to announce that XRuby 0.1.4 is now available for download at: http://code.google.com/p/xruby/downloads/list

The most significant change in this release is: one of our team member, dreamhead, single-handly created a new runtime (the core of the system that manages ruby types, methods...) to make it very close to the classic C ruby, and therefore fixed several compatibility issues with C ruby. As a translator of the Chinese version Ruby Hacking Guide, dreamhead has very good knowledge of the ruby internal and gives a great lift on the XRuby's development.

As we have learned rewriting is very difficult, we learned it again in a hard way. Dreamhead wrote most of code last year, several months before 0.1.0 release. First we tried to make a wholesale replacement of the runtime, but problems were here and there and lots of unit tests broke. Since we do not want to have a broken system in the SVN trunk, we decided to take 'refactoring' approach and merge changes step by step. It is still very hard, but dreamhead made lots of efforts and finished the merge yesterday.

Actually it may take less time if we insist on following the 'rewrite' approach (replace all the old code and fix broken tests), but the benefit of 'refactoring' is significant: it gives us a stable code base so that we made several releases during this big change.

There are more exciting news behind the scene now: Beanworms is working on the debugger support, and Femto and Yuesefa have showed interest to migrate our ruby parser to ANTLR 3.0.

Thank you for people who helps us on making this release. It has been a great learning experience working with the project.

Ruby builtin in pure Ruby

2007-03-11T20:15:00.000-07:00

One of the best things I love about rubinius project is: their developers try to keep the dependency on system language (C in their case) minimal, and they take it seriously. Take a look of http://code.fallingsnow.net/svn/rubinius/trunk/kernel/core/, you will find out lots of buitlin libraries are implemented in pure ruby, even lots of string functions.

Pure ruby builtin has some drawbacks, though. One is performance penalty, the other is potential side effects. For example, basically Integer#times can be implemented as simple as this in pure ruby:


class Integer
 def times  
  i = 0  
  while i < self
   yield i
   i += 1;  
  end 
  self
 end
end

But this version of Integer#times does not work exactly the same as Ruby 1.8.5. If an user want to override Fixnum#+ (this may never happen in real life):


class Fixnum
  def + x
    return 9999
  end
end

Our Integer#times's behavior will change, while Ruby 1.8.5 won't. That is because Ruby 1.8.5's implementation (int_dotimes() in Numeric.c) optimizes for Fixnum: it does not call '+' method dynamically for Fixnum, instead, it just increases the integer directly. If you want to implement this method as same as Ruby 1.8.5 then you have to write code in system language.

This kind of optimization is all over the place in c ruby. I am not clear about its motivation, but I guess performance is one of the reasons. For example, "30000000.times {|x|}" is about twice faster than "i = 0; while i < 30000000; i +=1; end" in Ruby 1.8.5.

Difference people may have different opinions on what the 'right' behavior should be. As for me, I like the behavior of the pure ruby implementation better.

XRuby is faster than Ruby 1.8.5 in most benchmarks

2007-03-06T19:18:00.001-08:00

Two weeks ago, Antonio Cangiano compared the performance of different ruby implementations using Ruby 1.9 (YARV)'s benchmark suite. His numbers got me thinking: all alternative implementations performed badly -- most are even way slower than ruby 1.8.5. Does it signal that JVM and .NET are bad platform for Ruby language?

With this doubt I tried the benchmark with XRuby. XRuby is a ruby compiler. Unlike other implementations, it generates Java bytecode that run directly on JVM. But at first the numbers are not impressive: the 0.1.2 version is still slower than Ruby 1.8.5 in most of the cases.

Maybe I should mention that the XRuby team had done virtually nothing for performance before, and we would avoid optimization as long as possible if it makes our code complicated. But after doing some measurements, it turns out our bad performance are largely caused by a logic 'error': as we know Ruby Fixnum can not have singleton methods, but in 0.1.2 it still lookup an empty method table. And along with some bad code practices (iterating an empty ArrayList without checking if it is empty first etc), it makes method lookup much slower than it should be.

I fixed the problem by adding about 10 lines of code, and got great result: In most benchmarks, XRuby 0.1.3 is faster than Ruby 1.8.5. For some, faster in a significant way. There are still some tests in which we are slower, but it looks like caused by poorly implemented builtin.

The following table shows the benchmark result for XRuby 0.1.3. The best part is: we did it without a method cache. YARV is still faster than XRuby, but we have lots of room to improve too.

>java -Xmx512m -jar xruby-0.1.3.jar benchmark\run.rb

Test	Ruby 1.8.5	XRuby 0.1.3
bm_app_answer.rb	fail	fail
bm_app_factorial.rb	fail	fail
bm_app_fib.rb	20.02	12.29
bm_app_mandelbrot.rb	7.099	8.252
bm_app_pentomino.rb	289.8	538.5
bm_app_raise.rb	4.846	3.986
bm_app_strconcat.rb	5.898	3.234
bm_app_tak.rb	26.14	22.12
bm_app_tarai.rb	20.89	18.35
bm_loop_times.rb	14.28	19.30
bm_loop_whileloop.rb	26.03	19.27
bm_loop_whileloop2.rb	5.257	4.786
bm_so_ackermann.rb	fail	fail
bm_so_array.rb	19.17	46.84
bm_so_concatenate.rb	5.727	9.684
bm_so_count_words.rb	2.944	45.50
bm_so_exception.rb	9.793	7.399
bm_so_lists.rb	3.666	24.59
bm_so_matrix.rb	6.249	8.452
bm_so_nested_loop.rb	15.17	13.45
bm_so_object.rb	21.49	7.991
bm_so_random.rb	6.169	5.888
bm_so_sieve.rb	2.042	2.753
bm_vm1_block.rb	64.57	38.69
bm_vm1_const.rb	47.47	25.57
bm_vm1_ensure.rb	45.54	20.01
bm_vm1_length.rb	55.50	40.89
bm_vm1_rescue.rb	39.61	20.64
bm_vm1_simplereturn.rb	56.02	29.06
bm_vm1_swap.rb	76.35	30.52
bm_vm2_array.rb	19.34	8.532
bm_vm2_method.rb	33.72	19.63
bm_vm2_poly_method.rb	45.23	20.62
bm_vm2_poly_method_ov.rb	12.64	8.261
bm_vm2_proc.rb	21.08	17.86
bm_vm2_regexp.rb	13.09	30.87
bm_vm2_send.rb	11.71	15.75
bm_vm2_super.rb	13.92	7.510
bm_vm2_unif1.rb	11.30	8.292
bm_vm2_zsuper.rb	15.71	7.740
bm_vm3_thread_create_join.rb	0.110	1.331

* The test environment is Intel Pentium M 1G CPU, 1G Memory, Windows XP SP2, Java 1.5.0_09.

XRuby 0.1.2 released

2007-02-28T19:13:00.001-08:00

XRuby 0.1.2 is now available for download from the project download page: http://code.google.com/p/xruby/downloads/list

This latest release contains more built-in libraries and a few bug fixes for the compiler. We expect to release a new stable version every month.

Last week, Antonio Cangiano did a very interesting job to compare the performance of different ruby implementations using YARV's benchmark suite. XRuby is not included in his test as Antonio may not know our project exists. For people who are interested, here is the XRuby's performance data (due to my limited resource, only ruby 1.8.5 interpreter was compared):

>java -Xmx512m -jar xruby-0.1.2.jar benchmark\run.rb

Test	Ruby 1.8.5	XRuby 0.1.2
bm_app_answer.rb	fail	fail
bm_app_factorial.rb	fail	fail
bm_app_fib.rb	20.42	33.84
bm_app_mandelbrot.rb	7.270	15.33
bm_app_pentomino.rb	294.2	956.9
bm_app_raise.rb	4.906	4.597
bm_app_strconcat.rb	5.917	5.858
bm_app_tak.rb	26.46	38.95
bm_app_tarai.rb	21.08	35.51
bm_loop_times.rb	14.83	93.55
bm_loop_whileloop.rb	26.26	76.5
bm_loop_whileloop2.rb	5.327	16.15
bm_so_ackermann.rb	fail	fail
bm_so_array.rb	19.35	300.2
bm_so_concatenate.rb	5.868	26.58
bm_so_count_words.rb	2.253	46.43
bm_so_exception.rb	9.893	9.973
bm_so_lists.rb	3.696	41.04
bm_so_matrix.rb	6.328	21.29
bm_so_nested_loop.rb	15.38	68.20
bm_so_object.rb	21.97	20.77
bm_so_random.rb	6.269	15.69
bm_so_sieve.rb	1.992	5.476
bm_vm1_block.rb	65.67	98.91
bm_vm1_const.rb	48.05	82.31
bm_vm1_ensure.rb	46.19	75.38
bm_vm1_length.rb	55.62	105.6
bm_vm1_rescue.rb	36.47	80.91
bm_vm1_simplereturn.rb	57.12	88.16
bm_vm1_swap.rb	76.21	85.73
bm_vm2_array.rb	18.64	19.65
bm_vm2_method.rb	33.41	35.32
bm_vm2_poly_method.rb	45.42	47.78
bm_vm2_poly_method_ov.rb	12.81	25.98
bm_vm2_proc.rb	21.54	31.23
bm_vm2_regexp.rb	13.40	52.35
bm_vm2_send.rb	11.87	31.60
bm_vm2_super.rb	13.98	20.25
bm_vm2_unif1.rb	11.46	19.95
bm_vm2_zsuper.rb	15.84	21.92
bm_vm3_thread_create_join.rb	0.120	1.402

The test environment is Intel Pentium M 1G CPU, 1G Memory, Windows XP SP2, Java 1.5.0_09. It may be worth to mention we have done almost nothing for optimization, not even a simple method cache. I think it is quite possible for a JVM based ruby compiler to beat the classic Ruby interpreter on performance, while YARV's number is going to be very hard for us to reach.

On-Ruby interview

2007-02-18T19:33:00.001-08:00

Pat Eyler recently interviewed XRuby team, the first episode is now posted on his popular On-Ruby blog: http://on-ruby.blogspot.com/2007/02/serial-xruby-interview-episode-i.html

XRuby 0.1.0 released

2007-01-29T14:16:00.000-08:00

[Link of this article: http://xruby.blogspot.com/2007/01/xruby-010-released.html]

Today I am glad to announce the release XRuby 0.1.0 , a ruby compiler which compiles ruby script (.rb) to java bytecode (.class). You can download it from here: http://code.google.com/p/xruby/downloads/list

XRuby 0.1.0 is the first release of XRuby. It is able to pass all tests in samples/test.rb - a decent test suite that comes with ruby installation.

As a compiler, XRuby compiles ruby source code so that your program can run on top of Java Virtual Machine. For example, if you have a ruby script like this:

class MyClass
  def say_hello_three_times
    3.times {puts "hello"}
  end
end

MyClass.new.say_hello_three_times

You can compile it with XRuby (assuming its name is test.rb):

>java -jar xruby-0.1.0.jar -c test.rb

The compiler will generate a test.jar. It can be launched as a regular java application, with the following command:

>java -jar test.jar
hello
hello
hello

For your convenience, you can also run the script directly (without -c option), just like the classic ruby interpreter. Under the hood it compiles the script then run the bytecode:

>java -jar xruby-0.1.0.jar test.rb
hello
hello
hello

While little known, this project has been under active development for one year and eight months. I started the project around mid 2005, and it took me about eight months to write a ruby parser with Antlr, and then one year to implement the compiler and runtime. Some people joined me during the second half of 2006 and made great contribution. It is just the beginning of an exciting journey.

We still have a long way to go before claiming XRuby is a very competent alternative implementation of ruby. And comparing with our peers we lag behind in implementing ruby's built-in libraries and do not have Ruby-Java bridge. Our next focus is to improve this situation and make major ruby libraries and framework work under XRuby. And we hope by end of 2007 you can use XRuby to compile your Ruby On Rails application to a .war file and run it directly on a J2EE server.

We appreciate your feedback. Our development maillist is at:
http://groups.google.com/group/xruby-devel

I also setup a new blog to be used as XRuby's team blog: http://xruby.blogspot.com . While new articles will be cross posted for quite a while, I encourage you to subscribe http://feeds.feedburner.com/xruby if have not done so.

Thank you and have fun.

XRuby 0.1.0 will be released in January, 2007

2007-01-02T08:45:00.000-08:00

Well, the release date of xruby 0.1.0 (our first release) has to be pushed back a little due to the lack of time of our projects members.

The goal of the xruby 0.1.0 is to pass all the tests in ruby's sample\test.rb, and we are very close to the goal. Right now we still have a few failed tests to be fixed, the majority of them are serialization (Array#pack, Marshal) and eval related. Most technical obstacles of compiling ruby have been solved, it just takes time to do the implementation (especially for the large amount of builtin libraries).

2007 is going to be an exciting year:)

XRuby Hacking Guide on wiki

2006-10-24T11:05:00.000-07:00

Update 12/20/2006: As Google Code now provides Wiki by itself, I just move the project wiki over there as it is better integrated.

Update 12/21/2006: Oops, now I regret what I did...Google Code only allow project members and owners add and edit wiki pages, which make it almost useless.

Right now the only document we have is XRuby Hacking Guide. The goal of this guide is to help users/developers understand XRuby's implementation.

XRuby project is now hosted on Google Code

2006-09-10T10:51:00.000-07:00

XRuby is an open source Ruby compiler which compiles Ruby source code (.rb) to Java bytecode (.class). On last Friday I moved the source code from my local Subversion server to Google Code(code.google.com/p/xruby/), so the project is now public available (GPL license).

Although I have never announced this project, I have been working on it for more than one year. This February I made the parser public available, now the main effort is on the compiler backend (java bytecode generation). I expect to release the first stable version at the end of 2006.

So far I am not very satisfied with my progress. I have a busy day job which has nothing related to Ruby and Java, so can only manage to work on XRuby about one hour a day(except weekends and holidays). When I started the project I thought XRuby is going to be the first ruby compiler in the world. But now Gardens Point Ruby.NET has released their first alpha version, and I expect JRuby will start to do their compiler in a few months (since they have been hired by Sun to work on JRuby full time).

Anyway, the project gives me lots of fun and the implementation starts to gain momentum. It looks very promising to see the first stable release around this new year, stay tuned:)

Operator overloading in ruby

2006-08-09T10:37:00.000-07:00

Most operators in ruby (e.g. +, -, *, /, % etc) are actually method calls, and they can be overloaded just like other method calls. But unlike C++, Ruby does not have operators like +=, -=, *=, /= etc, so a += 1 is translated directly to a = a + 1.

This involves some performance penalty in some cases, for example:


a1 = [1, 2, 3]
a2 = [4, 5, 6]
a1 += a2

Behind the scenes, a new array is created (as a1 += a2 is equivalent to a1 = a1 + a2, operator + will create a new array, then a1 is assigned to hold the reference to the new array, and the old a1 will be garbage collected). If you just want to append a2 to a1, the following code is a little bit more efficient:


a1.concat(a2)

concat method will directly update the content of a1 without creating a new array.

And it is surprising to know that not all ruby's operators are method calls, so you can not overload !, not, &&, and, ||, or.

UPDATE 09/12/2006: Chris pointed out an error in my original article(I shouldn't use << operator as the alternative of +=), see comments for details. Thanks, Chris!

Reference:
[1]Are +, -, * ... operators?

Rubyfront 0.2.0 released

2006-04-30T12:15:00.000-07:00

This new version of Rubyfront fixed a bug found when parsing Ruby on Rails 1.1.2. You can download the source code from here.

In rubyfront 0.2.0, I changed the way it handles expression substitution. The older versions of rubyfront ignores the content in expression substitution. For example, if the input is:


"hello, #{name}"

The lexer will first start to scan the input as a double quote string. After it sees "#{", it will go find the first '}' and think that is the end of the expression substitution. Unfortunately, expression substitution may contain '}' itself. For example:


"begin #{1.times {"block"}} end"

The lexer can not match the above input as one string. Expression substitution can contain any legal ruby program (compoundStatement), and only the parser understands the structure.

To correctly handle this situation, the new lexer will suspend its state when the start of expression substitution is seen (only 'begin ' is matched so far). After the parser finishes its processing of expression substitution, it will notify the lexer and the lexer will resume its state and matches the rest of the string (' end').

Btw, when you run parser smoke test against Ruby on Rails 1.1.2, you will see the following failures:


C:\Documents and Settings\xxx\workspace\rubyfront>build parsersmoketest
<...>
parsersmoketest:
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\activerecord-1.14.2\test\connections\native_sqlite3\in_memory_connection.rb: line 18:76: expecting EOF, found ')'
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\environments\environment.rb: line 8:1: unexpected token: <
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\controller\templates\controller.rb: line 1:7: unexpected token: <
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\controller\templates\functional_test.rb: line 1:41: unexpected token: ..
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\controller\templates\helper.rb: line 1:8: unexpected token: <
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\integration_test\templates\integration_test.rb: line 3:7: unexpected token: <
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\mailer\templates\mailer.rb: line 1:7: unexpected token: <
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\mailer\templates\unit_test.rb: line 4:7: unexpected token: <
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\migration\templates\migration.rb: line 1:7: unexpected token: <
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\model\templates\migration.rb: line 1:7: unexpected token: <
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\model\templates\model.rb: line 1:7: unexpected token: <
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\model\templates\unit_test.rb: line 1:41: unexpected token: ..
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\plugin\templates\generator.rb: line 1:7: unexpected token: <
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\plugin\templates\unit_test.rb: line 3:7: unexpected token: <
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\scaffold\templates\controller.rb: line 1:7: unexpected token: <
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\scaffold\templates\functional_test.rb: line 5:7: unexpected token: <
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\scaffold\templates\helper.rb: line 1:8: unexpected token: <
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\session_migration\templates\migration.rb: line 1:7: unexpected token: <
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\web_service\templates\api_definition.rb: line 1:7: unexpected token: <
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\web_service\templates\controller.rb: line 1:7: unexpected token: <
     [java] parser exception for C:\ruby\lib\ruby\gems\1.8\gems\rails-1.1.2\lib\rails_generator\generators\components\web_service\templates\functional_test.rb: line 1:41: unexpected token: ..
     [java] 2428 ruby programs have been parsed, 21 failed.

Those errors are expected as those files are in .rhtml format (not valid ruby, ROR requires erb to preprocess them).

Compile-time type inference for ruby has its limit

2006-04-17T18:49:00.000-07:00

I have been thinking about doing type inference at compiler time for ruby. Whiles lots of the cases can be handled in a straightforward way, type inference won't work in all situations.

One of the problems is, ruby supports multiple return types:


def f
  if rand(2) == 1
       return 1
  else
       return "hello"
  end
end

In the above code, method 'f' will randomly return 1 (Fixnum) or "hello" (String). At compiler time there is no way for a compiler to know what type to return. All it can do is to insert meta data so that typing checking can be done at runtime.

And multiple return types is not uncommon in real code. For example, Ruby' s array or hash can hold objects of different types:


  a = [1, 1.5, "hello"]

In the above code, Array 'a' contains three different types of objects: Fixnum, Float and String. And element reference("[]") is an example of method that has multiple return types.

First release of Rubyfront

2006-02-16T20:55:00.000-08:00

UPDATE 02/27/2006: a new version of rubyfront was released, and now it is capable to parse ruby's stable snapshot and Ruby on Rails. Thanks Mauricio Fernandez for reporting the bugs.

I am glad to announce the first release of rubyfront (a ruby parser powered by antlr), you can download it here.

Overview
The goal of this project is to create an error checking tool for ruby. Currently we are using antlr as the parser generator and java as the implementation language.

Current Status
Even though Rubyfront is in its very early stage, the current parser can accept the entire ruby standard library. So far the parser only do error checking at syntax level, no semantic checking is implemented.

What's next
We will focus on improving the parser to make it closer to C ruby's yacc counterpart.

Requirement
J2SE 5.0 and up.

Demonstration
1. Open build.xml and change "rubycode.dir" to the directory of your ruby source code.
2. Run the smoke test. On windows, you can simply type "build parsersmoketest".

For example, if you have Ruby 1.8.4's source code installed on "c:\ruby", you are going to see the following:


parsersmoketest:
 [java] 1574 ruby programs have been parsed, 0 failed.

The above result shows that all 1574 ruby programs are accepted by the parser.

By the way, if you run smoke test against Ruby's windows One-Click installer, you are going to see a few failures from fxruby 1.2.6 and others. The good news is: they are known to be broken (You can use "ruby -c" to verify). If you have Ruby on Rails installed, you are going to see more failures. But all of the them are expected, since those are actually .rhtml files.

Ruby interpreter crashes with invalid input

2006-01-09T08:51:00.000-08:00

Last week I found a bug in ruby's parser, NULL pointer reference will happen when the following program is parsed:


C:\ruby-1.8.4-i386-mswin32\bin>ruby -e 'def a=.a=;end'
-e:1: identifier a= is not valid
-e:1: [BUG] Segmentation fault ruby 1.8.4 (2005-12-24) [i386-mswin32]
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

Now it was fixed in CVS.

This is not a serious security problem. I tried to find if it will result in a remote DOS vulnerability in Ruby on Rails, but find nothing.

Why "undef" is keyword in ruby

2006-01-06T09:58:00.000-08:00

undef is a keyword in ruby. But its syntax:
"undef method-name (, method-name)*", looks just like a normal command call (function call without parameter, e.g. print "hello"). Then why not just implement it as a built-in method? As we know, "private", "protected" and "public", which are keywords in lots of other language, are just implemented as plain method in ruby.

What makes "undef" special is in attribute methods. In ruby we can define method like this:


class A
  def a=(value)  
    @value = value  
  end
end

And later, we can undefine "a=" with "undef":


undef a=

"a=" is not a valid identifier(because of the '=') unless the parser is expecting a method name, for example, after "alias", "def" or "undef".

Note ":a=" is valid symbol. "private", "protected" and "public" are normal methods since they only accept symbols:


class MyClass

   def method1
   end

   public :method1
  #This is invalid: public method1
end

Since "undef" is followed by method name, it has to be implemented as a keyword, so the parser/lexer knows when to change its state to accept the extra '='.

MiniRails

2005-12-16T10:48:00.000-08:00

Lots of people are using web technologies in their desktop applications, and it is especially common to find a windows application which embeds WebBrowser control to render great GUI elements. There are some bumps, for example, you can not do disk/socket IO with javascript(even java applet), which most applications have to reply on.

To overcome this, we can either use ActiveX control(I am using it myself and am satisfied with it), or launch a web server on the background. The second has an additional advantage: the server and client are decoupled, so in the future you can easily migrate to a pure web environment without touching too much code. While this approach looks attractive, however, we have to face a reality: most web application servers can be too big for most desktop applications to distribute. Note I am taking web application servers here, not simple web servers. It is easy to find a small footprint web server or even write one by yourself, but your development style will have to go back to the 80s (CGI in C). Even though Tomcat Embedded 5.5.12 can be downloaded as a 3.1 MB zip file, your customers may not have JVM installed in the first place. Cassini suffers the same problem, Microsoft has not pushed hard enough to make .NET universal.

How about the Ruby on Rails? It turns out to be a great choice. While a complete distribution takes lots of disk space, it is easy to customize so that we only ship what we need.

Here is how I build my minimum ruby distribution that rail can run on(use windows as example):

1. Have ruby installed to C:\ruby. Later we need to copy files from it.

2. Download rails stand-alone packages, uncompress it to C:\rails, and empty doc/ folder to save space.

3. Copy ruby.exe and msvcrt-ruby18.dll from C:\ruby\bin to C:\rails.

4. Copy the following OS independent libraries (implemented in pure ruby) from C:\ruby\lib\ruby\1.8 to C:\rails\lib\ruby\1.8:


 base64.rb
 benchmark.rb
 cgi.rb and cgi/
 date.rb
 date/format.rb
 delegate.rb
 drb.rb and drb/
 e2mmap.rb
 erb.rb
 English.rb
 fileutils.rb
 ipaddr.rb
 irb.rb and irb/
 kconv.rb
 logger.rb
 mutex_m.rb
 net/http.rb
 net/protocol.rb
 net/smtp.rb
 observer.rb
 optparse.rb and optparse/
 parsedate.rb
 pathname.rb
 pstore.rb
 rational.rb
 rexml.rb and rexml/
 set.rb
 singleton.rb
 soap.rb and soap/
 tempfile.rb
 thread.rb
 time.rb
 timeout.rb
 tmpdir.rb
 uri.rb and uri/
 webrick.rb and webrick/
 xmlrpc/
 xsd/
 yaml.rb and yaml/

5. Copy the following OS dependent libraries from C:\ruby\lib\ruby\1.8\i386-mswin32 to C:\rails\lib\ruby\1.8\i386-mswin32:


 digest.so
 digest/md5.so
 digest/sha1.so 
 fcntl.so
 nkf.so
 rbconfig.rb
 socket.so
 strscan.so
 stringio.so
 syck.so

6. Now we may still need a database (optional). Here is how I get sqlite3 installed:
a) Download precompiled binaries. Copy sqlite3.exe and sqlite3.dll into C:\rails folder.
b) Download SQLite-Ruby. Copy sqlite3_api.so to C:\rails\lib\ruby\site_ruby\1.8\i386-msvcrt\, copy sqlite3.rb and sqlite3/ to C:\rails\lib\ruby\site_ruby\1.8
That's it! You can start WEBrick now:


C:\rails>ruby script\server
=> Booting WEBrick...
=> Rails application started on http://0.0.0.0:3000
=> Ctrl-C to shutdown server; call with --help for options
[2005-12-16 15:23:09] INFO  WEBrick 1.3.1
[2005-12-16 15:23:09] INFO  ruby 1.8.2 (2004-12-25) [i386-mswin32]
[2005-12-16 15:23:09] INFO  WEBrick::HTTPServer#start: pid=440 port=3000

And do whatever rails developement as you want.

This distribution takes 6.28M on disk. If you zip it, it shrinks to a 2.3M file.

Good experience with boost memory pool

2005-12-13T15:55:00.000-08:00

With increasing number of source code to compile, the compiler I maintain becomes unacceptable slow. After profiling the code with AQTime, I found about 30% percent CPU is used by malloc(), and half of them are for tree node allocation(fixed size).

After testing the performance of boost memory pool, I simply replaced malloc() with boost::object_pool::malloc(). The result is great, it almost eliminated the cost of tree node allocation. The total amount of time used for malloc is down to 13%. Still plenty of room for improvement, but the performance is at least acceptable.

There is one thing to remember: boost::object_pool is designed for "allocate many objects and then destroy all of them" type of scenario, so freeing memory from the pool can be very slow. For example, at the beginning I also replaced free() with boost::object_pool::free(), which make the performance even worse: boost::simple_segregated_storage::find_prev() and boost::simple_segregated_storage::nextof() start to take over all the CPU. So I do not do free anymore and simply let ~object_pool() to release the memory. This works perfect for my compiler, but may not be good for other applications.

Scope of a local variable in ruby

2005-12-12T18:55:00.000-08:00

In ruby, only class, module and method can introduce new scope (see Ruby FAQ). Even if ruby has "if .. end", "while ... end" constructs, unlike c/c++/java/c#, they won't create new scope.

For example:


if true
  x = "hello"
end

# we can access x outside of "if". This will print "hello".
puts x

While there might be other motivations for this design, I think it is most likely driven by a technical issue: as I wrote before, ruby's lexer need to distinguish local variable and method call(command). It is easy for it to check "module", "class" and "def" keyword to recognize a new scope. However, as for "if", "while" etc (may followed by a complex expression), it is too hard for the lexer to handle (requires serious parsing capability).

Compare performance of malloc and boost::pool

2005-12-07T15:23:00.000-08:00

Andrei Gorine and Konstantin Knizhnik wrote a great article on Dr. Dobb's Journal titled Memory Management & Embedded Databases. In this article, they examine several memory-allocation strategies, explained the advantages/disadvantages based on their experience creating the eXtremeDB in-memory database.

As we know it is normal for malloc/free(new/delete) to take 10-20 percent of the CPU time of an application. Sometime it is nice to have a memory pool to handle some fixed size allocation, especially when you have Boost Pool Library for free.

I wrote a small program, which creates a small and simple data structure on heap a million times to benchmark malloc and boost::object_pool, and the result shows boost::object_pool is a clear winner. I also tried to compare with new operator, even though there is no constructor for the dummy data structure. The difference between malloc and new in this case is insignificant.

The result also shows something interesting which have been mentioned in Raymond Chen's "5 Things Every Win32 Developer Should Know" PDC 2005 talk. If boost::object_pool is used as a pointer, the performance hurts a lot, even through still way faster than malloc.

Here is the result on my laptop (Windows XP, VC 2003 release build, P4 2.4G, 512M):


alloc_with_new_array:   26 microseconds
alloc_with_malloc_array:        11 microseconds
alloc_with_new: 365120 microseconds
alloc_with_malloc:      351586 microseconds
alloc_with_boost_pool_construct:        7852 microseconds
alloc_with_boost_pool_malloc:   3371 microseconds
alloc_with_boost_pool_malloc_use_pointer:       11751 microseconds
alloc_with_boost_pool_malloc_use_reference:     3383 microseconds

Here is code:


#include <boost/pool/object_pool.hpp>

struct dummy_data
{
 int a;
 int b;
};

inline void alloc_with_new(size_t size)
{
 dummy_data* p;
 for (size_t i = 0; i < size; ++i)
 {
  p = new dummy_data;
  delete p;
 }
}

inline void alloc_with_malloc(size_t size)
{
 dummy_data* p;
 for (size_t i = 0; i < size; ++i)
 {
  p = (dummy_data*)malloc(sizeof(dummy_data));
  free(p);
 }
}

inline void alloc_with_boost_pool_construct(size_t size)
{
 boost::object_pool<dummy_data> pool;
 dummy_data* p;
 for (size_t i = 0; i < size; ++i)
 {
  p = pool.construct();
  pool.destroy(p);
 }
}



inline void alloc_with_boost_pool_malloc(size_t size)
{
 boost::object_pool<dummy_data> pool;
 dummy_data* p;
 for (size_t i = 0; i < size; ++i)
 {
  p = pool.malloc();
  pool.free(p);
 }
}

inline void alloc_with_boost_pool_malloc_use_pointer(size_t size)
{
 boost::object_pool<dummy_data>* pool = new boost::object_pool<dummy_data>;
 dummy_data* p;
 for (size_t i = 0; i < size; ++i)
 {
  p = pool->malloc();
  pool->free(p);
 }

 delete pool;
}

inline void alloc_with_boost_pool_malloc_use_reference(size_t size)
{
 boost::object_pool<dummy_data> _pool;
 boost::object_pool<dummy_data>& pool = _pool;
 dummy_data* p;
 for (size_t i = 0; i < size; ++i)
 {
  p = pool.malloc();
  pool.free(p);
 }
}

inline void alloc_with_new_array(size_t size)
{
 dummy_data* p = new dummy_data[size];
 delete[] p;
}

inline void alloc_with_malloc_array(size_t size)
{
 dummy_data* p = (dummy_data*)malloc(sizeof(dummy_data) * size);
 free(p);
}

int main (int, char**)
{
 LARGE_INTEGER freq, t0, t1;
 QueryPerformanceFrequency(&freq);

 size_t number = 1000000;

 QueryPerformanceCounter(&t0);
 alloc_with_new_array(number);
 QueryPerformanceCounter(&t1);
 printf("alloc_with_new_array: %d microseconds\n",
  ((t1.QuadPart-t0.QuadPart)*1000000)/freq.QuadPart);

 QueryPerformanceCounter(&t0);
 alloc_with_malloc_array(number);
 QueryPerformanceCounter(&t1);
 printf("alloc_with_malloc_array: %d microseconds\n",
  ((t1.QuadPart-t0.QuadPart)*1000000)/freq.QuadPart);

 QueryPerformanceCounter(&t0);
 alloc_with_new(number);
 QueryPerformanceCounter(&t1);
 printf("alloc_with_new: %d microseconds\n",
  ((t1.QuadPart-t0.QuadPart)*1000000)/freq.QuadPart);

 QueryPerformanceCounter(&t0);
 alloc_with_malloc(number);
 QueryPerformanceCounter(&t1);
 printf("alloc_with_malloc: %d microseconds\n",
  ((t1.QuadPart-t0.QuadPart)*1000000)/freq.QuadPart);

 QueryPerformanceCounter(&t0);
 alloc_with_boost_pool_construct(number);
 QueryPerformanceCounter(&t1);
 printf("alloc_with_boost_pool_construct: %d microseconds\n",
  ((t1.QuadPart-t0.QuadPart)*1000000)/freq.QuadPart);

 QueryPerformanceCounter(&t0);
 alloc_with_boost_pool_malloc(number);
 QueryPerformanceCounter(&t1);
 printf("alloc_with_boost_pool_malloc: %d microseconds\n",
  ((t1.QuadPart-t0.QuadPart)*1000000)/freq.QuadPart);

 QueryPerformanceCounter(&t0);
 alloc_with_boost_pool_malloc_use_pointer(number);
 QueryPerformanceCounter(&t1);
 printf("alloc_with_boost_pool_malloc_use_pointer: %d microseconds\n",
  ((t1.QuadPart-t0.QuadPart)*1000000)/freq.QuadPart);

 QueryPerformanceCounter(&t0);
 alloc_with_boost_pool_malloc_use_reference(number);
 QueryPerformanceCounter(&t1);
 printf("alloc_with_boost_pool_malloc_use_reference: %d microseconds\n",
  ((t1.QuadPart-t0.QuadPart)*1000000)/freq.QuadPart);

 return 0;
}

Programming and playing chess

2005-12-01T12:24:00.000-08:00

There are lots of things in common between leaning how to program and leaning how to play chess:

First, you need to leaning the rules
For programming, the rules are the grammar of your language of choice. You can not hit the road without knowing them, but this is the easiest part.

Nothing can replace realworld experience
That's why some gurus claim building software is a craftsmanship rather than engineering. While there are tons of tools, smart IDEs to assist you, software design masters reach their level by years of practicing.

Pattern is shortcut
Then how do we improve ourselves quicker? Lots of people know how to play chess and keep playing for years, but have very little improvement on their skills. Well, if you do not know it yet, patterns can be the shortcut. If you study how chess masters deal with common scenarios, you will improve much faster than your friends how do not.

So far, it is all about individuals. Multiplayer game is much more complicated, and so is professional software developement in corporate.

More on leftshift and heredoc

2005-11-29T21:11:00.000-08:00

A few weeks ago I wrote something on distinguishing between leftshift and heredoc in ruby lexer. Even though I knew how things work in theory, during the implementation I found out quite a lot complicated cases, and to handle all of them requires a complete symbol table, which is something I try to avoid right now.

Today I spent a few hours on trying examples and reading ruby's source code, and finally decoded the rules (it turns out to be much simpler than I thought).

Now let me show you how it works. Suppose the lexer is going to tokenize the following line:


x<<blahblah

Obviously, "x" is an IDENTIFIER, but what about "<<"? Is it a SHIFT operator or the start of HEREDOC? Well, sometime you can do it at syntax level. In the above case, since there is no whitespace between "x" and "<<", ruby always treated "<<" as shift operator.

Then let's move on to another example:


x  <<blahblah

As you can see the only difference is that we put space between "x" and "<<". Now comes rule 2: if "x" is a variable, "<<" becomes SHIFT operator, if "x" is a method, "<<" and the following will be treated as HEREDOC. As you can see, ruby will choose the way that makes most sense. But how do we know whether "x" is a method or not? Well, sometime it is easy: if "x" looks like "a.b.c" (notice the DOT), then it is always a method. Otherwise, we need a symbol table to lookup.

And there is a trick when implementing the symbol table: you can not put only methods in it and later check if a simple IDENTIFIER is indeed a method. It does not work for two reasons: 1) The method you called may not be defined yet; 2) The method may be defined in an included module and you do not want to parse them here. So how do we do it? Simply store the local variables in the symbol table: if a simple IDENTIFIER is not a local variable, then it must be a method.

The above rules make lexer implementation easy. Originally I thought I need to check the number of parameters of a method to get more accurate predict, but that is too much for a lexer.

Entertainment Reward Scam

2005-11-28T16:01:00.000-08:00

Purchasing a ticket on TicketMaster in last November made me a member of so called "Entertainment Reward" program, and they started to charge $9 from my credit card each month. They deliberately make the process unnoticable during the purchase. If my coworker did not break the news, I might no even know I was in the trap.

A quick search on google shows there are lots of people who are scamed, and now there is a website for sharing the stoies: www.entertainmentrewardscam.com. So be careful, better don't do business with those unethical companies.

Shame on TicketMaster and Entertainment Reward!

Does pure OO matter?

2005-11-09T13:59:00.000-08:00

Ruby is a pure OO language, in other words, it has no functions. Even if a "def" statement outside of a class definition looks and behaves like a function, it is actually a private methods of class Object.

You may ask, is this just an implementation details? Well, most of the time it doesn't matter, people can ignore the difference and use those methods as functions -- no problem at all. It only matters when you use introspect(reflection). For example, the following code:


def my_global_function
end

puts Object.private_methods

will print:


...
my_global_function
exit
putc
system
...

Reference:
[0] Ruby FAQ: How Does Ruby Compare With Python?

Linear approximate lookahead

2005-11-03T21:10:00.000-08:00

Even though we talk antlr as a LL(k) parser generator for simplicity, it actually does linear approximate lookahead instead of full LL(k).
Most of the time this is a good thing, because it reduces the complexity of storing and testing lookahead (from exponential to linear reduction). But occasionally the weaker lookahead is insufficient and results in nondeterministic parser/lexer.
Here is an example to illustrate the problem:


RULE1
   : "ab" | "ba"
   ;

RULE2
   : "ac" | "ca"
   ;

The above is prefect LL(2) grammar, but antlr will give you a warning:


[antlr] test.g: warning:lexical nondeterminism between rules RULE1 and RULE2 upon
[antlr] test.g:     k==1:'a'
[antlr] test.g:     k==2:'a'

If you look at the generated code, soon you will realized this is because of linear approximate lookahead:


else if ((LA(1)=='a'||LA(1)=='b') && (LA(2)=='a'||LA(2)=='b')) {
   ...//match RULE1
}
else if ((LA(1)=='a'||LA(1)=='c') && (LA(2)=='a'||LA(2)=='c')) {
   ...//match RULE2
}

If the input is "aa", even though it does not match any of the two rules, both tests will be true, and that is what the nondeterminism warning trying to tell you.

A full LL(k) parser generator should generate the following code instead:


else if ((LA(1)=='a'&&LA(2)=='b') || (LA(1)=='b'&&LA(2)=='a')) {
   ...//match RULE1
}
else if ((LA(1)=='a'&&LA(2)=='c') || (LA(1)=='c'&&LA(2)=='a')) {
   ...//match RULE2
}

Distinguish leftshift and heredoc

2005-11-01T15:02:00.000-08:00

I am working on a ruby parser with antlr. Last weekend I tested my ruby lexer on lots of realworld ruby scripts, and found an interesting problem. That is, how to distinguish left shift operator and "here document(heredoc)"?

As we know, "<<" is left shift operator in lots of computer languages. In perl and ruby, it is also the start of heredoc. For those of you who do not have unix shell script/perl/ruby background, heredoc acts like a string, but with flexibility. Here is an example in ruby:


str = <<EOF
hello, world!
EOF

Since perl/ruby choose not to introduce a new symbol for the start of heredoc, their parser developers have to deal with the ambiguity. I tried to find out a simple solution, but later found out the ambiguity can not be solved at syntax level.

For example, in the following code, "<<" should be parsed as the start of heredoc:


def x(var)
  puts var
end

x <<1
test
1

But in another case, "<<" should be parsed as left shift operator:


x = 1

x <<1
test
1

The only difference is: in the first case, x is a method, while in the second case x is a variable. So there is no other choice, we have to look up symbol table to decide lexer state.

Today I checked the source code of ruby 1.8.3. No surprise, it uses the same approach.

ngen

2005-10-27T21:13:00.000-07:00

As mentioned in Reid Wilkes's article on MSDN Magazine (April 2005), Microsoft has always used NGen on its own core Framework assemblies at .NET Framework installation time.
Here is screen capture during my installtion of .NET 2.0 (the installer spent quite a lot time doing ngen):

Do not disable "DCOM Server Process Launcher" service

2005-10-27T18:52:00.000-07:00

DCOM Server Process Launcher is a new service introduced in windows XP SP2. It plays a very important role on the system, and will start other "manual" services when necessary. Since it is relatively new and the documentation about it is poor, some people believe this service is useless and disabled it.

I have experienced several weird problems when the service is not started:

1. Watir scripts won't run, Ruby reports "Failed to create WIN32OLE object from `InternetExplorer.Application' (WIN32OLERuntimeError)
HRESULT error code:0x800706ba The RPC server is unavailable.".

2. Symantec Antivirus LiveUpdate fails with LU1803 error. All the workarounds on the web do not help.

3. Most installers won't run with "RPC server is unavailable" error message. Although you can fix that by starting "Windows Installer" service manually.

If you have the same problems, first check if its startup type is "Automatic".

Let watir do the job

2005-10-16T20:54:00.000-07:00

AutoLinking is a very convenience feature for almost every blog writer. For example, when you are writing about antlr, you may want to give a link to www.antlr.org, and a smart blog authoring tool should be able to do that for you. But as Wesner Moise mentioned on his blog , currently no blog-posting tool provides that service.

So I wrote a watir script(Watir is a great web application testing framework in Ruby) to help me. First it uses regular expression to do autolinking, then go to the website and publish the post (so I do not have to click all the links and wait for a reply).

Here is the code:



require 'watir'
include Watir 

#Start your writing here
title = "Let watir do the job"
content = <<EOF
I love watir !
EOF

#Polish and Autolinking etc
content = content.gsub(/\n/, ' ') #replace linebreak with whitespace 
content = content.gsub(/\bwatir\b/, '<a href="http://wtr.rubyforge.org">watir</a>')

#Go to the website
ie = IE.new
ie.goto("http://www.blogger.com/start") 

#Login
ie.text_field(:name, "username").set("yourusrename")
ie.text_field(:name, "password").set("yourpassword")
ie.link(:text, "Sign in").click

#Open blog
ie.link(:text, "your blog title").click

#Create new post
ie.link(:text, "Create new post").click

#Publish it ("value=" is faster then "set")
ie.text_field(:name, "title").value=(title)
ie.text_field(:name, "postBody").value=(content) 
ie.button(:value, "Publish Post").click

A simple port scanner using ACE proactor

2005-10-14T09:03:00.000-07:00

Today I wrote a simple port scanner using ACE proactor. Even though the code itself is useless, the process helped me to know ACE proctor's implementation details much better.

Proactor is a very good model for high performance IO, but you can not implement an efficient port scanner with it.

First, since most platforms do not have native capability for asynchronously connecting sockets, ACE may choos to emulate asynchronous connections with an instance of ACE_Select_reactor managed privately by the framework (for example, on windows). Btw, the description on C++NP v2 page 283, sidebar 57 is inaccurate: even though Windows XP/2003 has that capability(via ConnectEX() API), ACE is not using it now (to support older platform like windows 2000). You can check out the source code of ACE_WIN32_Asynch_Connect::connect() to see yourself. So in this case you are using a Reactor in deed, and since the framework need to spawn thread to do the emulation, it is less efficient than using Reactor directly.

Second, since each ACE_Handler can only be bundled with one handle/socket, you need to create one ACE_Handler for each connection, which definitely has a few more extra overhead than just creating a plain old socket handle.

Here is the source:


#include <ace/Log_Msg.h>
#include <ace/INET_Addr.h>
#include <ace/OS.h>
#include <ace/Proactor.h>
#include <ace/WIN32_Proactor.h>

class PortScanner : public ACE_Handler
{
public:

 PortScanner()
 {
 }

 int open(ACE_Proactor& proactor, const ACE_INET_Addr& remote_addr)
 {
  this->proactor(&proactor);
  this->remote_addr_.set(remote_addr);

  if (connect_.open(*this, ACE_INVALID_HANDLE, 0, this->proactor()) < 0)
  {
   return -1;
  }

  return connect_.connect(ACE_INVALID_HANDLE, remote_addr_, ACE_Addr::sap_any, 1);
 }

 void handle_connect(const ACE_Asynch_Connect::Result &result)
 {
  //This handler will be called if connection is established or RST is received,
  //You may no get any response in a timely manner even if there is no firewall.

  ACE_ASSERT(result.connect_handle () != ACE_INVALID_HANDLE);

  if (result.success())
  {
   //Connection is established
   ACE_DEBUG((LM_NOTICE, "%s:%d is open\n",
        remote_addr_.get_host_addr(),
        remote_addr_.get_port_number()));
  }
  else
  {
   //Got a RST
  }
 }

private:
 ACE_INET_Addr remote_addr_;
 ACE_Asynch_Connect connect_;
};

int ACE_TMAIN(int /*argc*/, ACE_TCHAR** /*argv*/)
{
 ACE_Proactor proactor;
 ACE_INET_Addr scan_target("127.0.0.1:0");

 //Asynchronous connection are simulated on Both Windows and Unix
 //(C++NP v2 page 283, sidebar 57 is inaccurate).
 //Since ACE_Select_reactor is used internally, the handles limit
 //applies. You can either increase the limit(1024 by default), or
 //simply do not register too much handles simultaneously.
 unsigned short start_port  = 1;
 unsigned short end_port = 1024;

 ACE_ASSERT(end_port > start_port);
 const size_t size = end_port - start_port + 1;
 //'size' is not const expression so we can not create the array on stack
 PortScanner* portscanners = new PortScanner[size];

 for (unsigned short i = start_port; i < end_port; ++i)
 {
  scan_target.set_port_number(i);
  portscanners[i].open(proactor, scan_target);
 }

 //Run event loop for 10 seconds
 ACE_DEBUG((LM_NOTICE, ACE_TEXT("Portscan started\n")));
 ACE_Time_Value timeout(30);
 proactor.proactor_run_event_loop(timeout);

 delete[] portscanners;

 return 0;
}

Access Control for Ruby Class

2005-10-12T20:51:00.000-07:00

The following words are from Programming Ruby: The Pragmatic Programmers' Guide:

"Ruby differs from other OO languages in another important way. Access control is determined dynamically, as the program runs, not statically. You will get an access violation only when the code attempts to execute the restricted method."

That's because those "public"/"protected"/"private" you saw are actually implemented as functions! The parser does not even know them. As the program runs, those functions are called and then you know if there is any errors.

Lexer state

2005-10-10T09:24:00.000-07:00

Lexer state cannot be avoided when implementing a lexer for Perl and Ruby.

In Perl/Ruby, regular expression starts and ends with '/', for example:


/perl|ruby/

As we all know, '/' is the divide operator.

So when the lexer meets a '/' character, it can not tell whether it is the start of a regular expression or a divide operator without the help of the context.

Same thing for "<<", it can be the left shift operator, or the start of "here doc":


<<EOF
whatever
EOF

The Ruby phenomenon

2005-10-04T13:32:00.000-07:00

The recent Ruby phenomenon is very interesting. Lost of people (including me) are first attracted by a killer application (e.g., rails and watir), then start to learn this foreign language (considering the majority programmers are only familiar with "curly braces" languages, like C++/C#/Java).

Both rails and watir are in a very competitive market. To develop web applications, there are big players like J2EE and ASP.net, even PHP has a larger user base than ruby. To do web application testing, javascript and vbscript have been used for years. Believe me, no matter how good ruby is, most people would like to stay with the language they know. Lots of credits should go to rails and watir.

So what makes rails and watir successful? One philosophy: make simple things simple and complex things possible. Everyone knows it, but it is quiet hard to achieve.

We can build very complex web applications with J2EE and .NET, while they have simplified the job greatly, comparing with developing in portable assembly(I am talking C :)), there are still lots of tedious works. On the other hand, there are some RAD tools which can help us to build an eCommerce site without a single line of coding, adding additional features soon becomes painful.

Rails and waitr do lots of things behind the scene, making good choice themselves without bothering us. Meanwhile the simple and powerful ruby language enable us to do complex things without too much works. Now that's a real beauty!

Sniffer and Reactor/Proactor

2005-10-04T12:25:00.000-07:00

While I have used pcap and winpcap for years, I am not big fan of the current programming model. I would wish to have an event driven model(reactor) or asynchronous IO model(procator). In other words, treat PCAP handler just like socket, have someone (select() for example) notify me when something interesting happens.

But maybe it is not pcap to blame, most of time it is just implemented as an user level library, and the underlying operating system may not provide a way to do IO multiplexing for such low level functions.

To implement a sniffer on windows, most of people will go for winpcap (btw, after a long waiting(two years!), winpcap 3.1 is finally out to replace the very buggy 3.0 version). Actually winsock has native sniffing support via raw socket(with SIO_RCVALL option), and it works well with reactor and proactor!

Here is the code to demonstrate the functionality (I used ACE. I know most of people would prefer plain old socket programming, but it is too hard for me to go back the dark days):


//Sniffer and Reactor Example
#include <ace/Log_Msg.h>
#include <ace/Reactor.h>
#include <ace/INET_Addr.h>
#include <ace/OS.h>
#include <ace/WFMO_Reactor.h>

#include <mstcpip.h>//SIO_RCVALL

//ACE_NDEBUG will be defined if you define _DEBUG on windows.
#ifdef ACE_NDEBUG
#define ACE_VERIFY(X) ((void)(X))
#else
#define ACE_VERIFY(X) ACE_ASSERT(X)
#endif //ACE_NDEBUG

class Sniffer : public ACE_Event_Handler
{
public:
 Sniffer(ACE_Reactor& reactor)
  :socket_(ACE_INVALID_HANDLE)
 {
  this->reactor(&reactor);
 }

 ACE_HANDLE get_handle() const
 {
  return socket_;
 }

 ~Sniffer()
 {
  if (ACE_INVALID_HANDLE != socket_)
  {
   ACE_VERIFY(ACE_OS::closesocket(socket_) == 0);
  }
 }

 /*
 MSDN On SIO_RCVALL:
 Enables a socket to receive all IP packets on the network. The socket handle passed to the
 WSAIoctl function must be of AF_INET address family, SOCK_RAW socket type, and IPPROTO_IP
 protocol. The socket also must be bound to an explicit local interface, which means that you cannot
 bind to INADDR_ANY. 
 Once the socket is bound and the ioctl set, calls to the WSARecv or recv functions return IP
 datagrams passing through the given interface. Note that you must supply a sufficiently large
 buffer. Setting this ioctl requires Administrator privilege on the local computer. SIO_RCVALL is
 available in Windows 2000 and later versions of Windows.
 */
 int open (const ACE_INET_Addr& listen_addr)
 {
  socket_ = ACE_OS::socket(AF_INET, SOCK_RAW, IPPROTO_IP);
  if (ACE_INVALID_HANDLE == socket_)
  {
   ACE_ERROR((LM_ERROR, ACE_TEXT("%p\n"), ACE_TEXT("socket() failed in Sniffer::open()")));
   return -1;
  }

  sockaddr* laddr = reinterpret_cast<sockaddr*>(listen_addr.get_addr());
  size_t size = listen_addr.get_size();
  ACE_VERIFY(ACE_OS::bind(socket_, laddr, size) == 0);

  DWORD YES = 1;
  ACE_VERIFY(ACE_OS::setsockopt(socket_, IPPROTO_IP, IP_HDRINCL, reinterpret_cast<const char*>(&YES), sizeof(YES)) == 0);

  DWORD dwBytesRet = 0 ;
  if (WSAIoctl((SOCKET)socket_, SIO_RCVALL, &YES, sizeof(YES), NULL, 0, &dwBytesRet, NULL, NULL) != 0)
  {
   ACE_ERROR((LM_ERROR, ACE_TEXT("%p\n"), ACE_TEXT("WSAIoctl() failed in Sniffer::open()")));
   return -1;
  }

  DWORD rcvtimeo = 1000 ; // 1 sec instead of 45(default)
  ACE_VERIFY(ACE_OS::setsockopt(socket_, SOL_SOCKET, SO_RCVTIMEO, reinterpret_cast<const char*>(&rcvtimeo) , sizeof(rcvtimeo)) == 0);

  // register the event handle
  if( this->reactor()->register_handler(this, ACE_Event_Handler::READ_MASK) < 0)
  {
   ACE_ERROR((LM_ERROR, ACE_TEXT ("%p\n"),
     ACE_TEXT("register_handler")));
   return -1;
  }

  ACE_DEBUG((LM_DEBUG, ACE_TEXT("Sniffer::open\n")));
  return 0;
 }

 int handle_input(ACE_HANDLE fd = ACE_INVALID_HANDLE)
 {
  ACE_ASSERT(fd == socket_);

  char recvbuf[65536] = {0};
  ssize_t nrecv = ACE_OS::recv(socket_, recvbuf, sizeof(recvbuf) - 1);
  if (nrecv > 0)
  {
   ACE_HEX_DUMP((LM_NOTICE, recvbuf, nrecv));
  }
  else
  {
   ACE_ERROR((LM_ERROR, ACE_TEXT("handle_input returns %d: %m\n"), nrecv));
  }

  return 0;
 }

private:
 ACE_HANDLE socket_;
};

int ACE_TMAIN(int /*argc*/, ACE_TCHAR** /*argv*/)
{
 ACE_WFMO_Reactor wfmo_reactor;
 ACE_Reactor reactor (&wfmo_reactor);

 //TODO bind to your local address!!!
 //Do not work with loopback address (127.0.0.1)
 ACE_INET_Addr addr("192.168.0.100:0");

 Sniffer sniffer(reactor);
 if (sniffer.open(addr) == -1)
 {
  ACE_ERROR((LM_ERROR, ACE_TEXT("Can not open to listen: %m\n")));
  return -1;
 }

 //Run event loop
 ACE_DEBUG((LM_NOTICE, ACE_TEXT("Sniffer started\n")));
 reactor.run_reactor_event_loop();
 return 0;
}

--------------------------------


//Sniffer and Proactor example
#include <ace/Log_Msg.h>
#include <ace/INET_Addr.h>
#include <ace/OS.h>
#include <ace/Proactor.h>
#include <ace/WIN32_Proactor.h>

#include <mstcpip.h>//SIO_RCVALL

//ACE_NDEBUG will be defined if you define _DEBUG on windows.
#ifdef ACE_NDEBUG
#define ACE_VERIFY(X) ((void)(X))
#else
#define ACE_VERIFY(X) ACE_ASSERT(X)
#endif //ACE_NDEBUG

class Sniffer : public ACE_Handler
{
 static const size_t PACKET_SIZE = 65526;
public:
 Sniffer(ACE_Proactor& proactor)
  :socket_(ACE_INVALID_HANDLE)
  ,ACE_Handler(&proactor)
  ,mblk_(PACKET_SIZE)
 {
 }

 ACE_HANDLE handle() const
 {
  return socket_;
 }

 ~Sniffer()
 {
  if (ACE_INVALID_HANDLE != socket_)
  {
   ACE_VERIFY(ACE_OS::closesocket(socket_) == 0);
  }
 }

 /*
 MSDN On SIO_RCVALL:
 Enables a socket to receive all IP packets on the network. The socket handle passed to the
 WSAIoctl function must be of AF_INET address family, SOCK_RAW socket type, and IPPROTO_IP
 protocol. The socket also must be bound to an explicit local interface, which means that you cannot
 bind to INADDR_ANY. 
 Once the socket is bound and the ioctl set, calls to the WSARecv or recv functions return IP
 datagrams passing through the given interface. Note that you must supply a sufficiently large
 buffer. Setting this ioctl requires Administrator privilege on the local computer. SIO_RCVALL is
 available in Windows 2000 and later versions of Windows.
 */
 int open (const ACE_INET_Addr& listen_addr)
 {
  if (create_sniffer_socket(listen_addr) < 0)
  {
   return -1;
  }

  // register the event handle
  if (reader_.open(*this) < 0)
  {
   ACE_ERROR((LM_ERROR, ACE_TEXT("open: %m\n")));
   return -1;
  }

  ACE_DEBUG((LM_DEBUG, ACE_TEXT("Sniffer::open\n")));

  //initiate our first asynchronous read
  reader_.read(mblk_, PACKET_SIZE);

  return 0;
 }

 void handle_read_stream(const ACE_Asynch_Read_Stream::Result &result)
 {
  if (!result.success() || result.bytes_transferred() == 0)
  {
   ACE_ERROR((LM_ERROR, ACE_TEXT("read_streamhandle_read_stream error\n")));
   return;
  }

  ACE_ASSERT(&mblk_ == &result.message_block());

  ACE_HEX_DUMP((LM_NOTICE, mblk_.rd_ptr(), result.bytes_transferred()));

  //Reset the write pointer and read again
  mblk_.wr_ptr(mblk_.rd_ptr());
  reader_.read(mblk_, PACKET_SIZE);
 }

private:
 ACE_Asynch_Read_Stream reader_;
 ACE_HANDLE socket_;
 ACE_Message_Block mblk_;

 int create_sniffer_socket(const ACE_INET_Addr& listen_addr)
 {
  socket_ = ACE_OS::socket(AF_INET, SOCK_RAW, IPPROTO_IP);

  if (ACE_INVALID_HANDLE == socket_)
  {
   ACE_ERROR((LM_ERROR, ACE_TEXT("%p\n"), ACE_TEXT("socket() failed in Sniffer::open()")));
   return -1;
  }

  sockaddr* laddr = reinterpret_cast<sockaddr*>(listen_addr.get_addr());
  size_t size = listen_addr.get_size();
  ACE_VERIFY(ACE_OS::bind(socket_, laddr, size) == 0);

  DWORD YES = 1;
  ACE_VERIFY(ACE_OS::setsockopt(socket_, IPPROTO_IP, IP_HDRINCL, reinterpret_cast<const char*>(&YES), sizeof(YES)) == 0);

  DWORD dwBytesRet = 0 ;
  if (WSAIoctl((SOCKET)socket_, SIO_RCVALL, &YES, sizeof(YES), NULL, 0, &dwBytesRet, NULL, NULL) != 0)
  {
   ACE_ERROR((LM_ERROR, ACE_TEXT("%p\n"), ACE_TEXT("WSAIoctl() failed in Sniffer::open()")));
   return -1;
  }

  DWORD rcvtimeo = 1000 ; // 1 sec instead of 45(default)
  ACE_VERIFY(ACE_OS::setsockopt(socket_, SOL_SOCKET, SO_RCVTIMEO, reinterpret_cast<const char*>(&rcvtimeo) , sizeof(rcvtimeo)) == 0);

  return 0;
 }
};

int ACE_TMAIN(int /*argc*/, ACE_TCHAR** /*argv*/)
{
 ACE_WIN32_Proactor win32_proactor;
 ACE_Proactor proactor(&win32_proactor);

 //TODO bind to your local address!!!
 //Do not work with loopback address (127.0.0.1)
 ACE_INET_Addr addr("192.168.0.100:0");

 Sniffer sniffer(proactor);
 if (sniffer.open(addr) == -1)
 {
  ACE_ERROR((LM_ERROR, ACE_TEXT("Can not open to listen: %m\n")));
  return -1;
 }

 //Run event loop
 ACE_DEBUG((LM_NOTICE, ACE_TEXT("Sniffer started\n")));
 proactor.proactor_run_event_loop();
 return 0;
}

Reference:
[0] Raw Sockets in Windows 2000/XP" by ziss

Ruby and Ada do look alike

2005-09-28T13:03:00.000-07:00

I once thought ruby is mostly inspired by perl, since Yukihiro Matsumoto mentioned ruby inherited the Perl philosophy. But it is hard to explain where a few "wired" syntax come from (considering my C/C++/C#/Java background), for example:


Ruby          C/C++/C#/Java
---------------------------
begin/end         {}
rescue            try/catch
raise             throw
elsif             else if
case/when         switch/case
nil               NULL/null

Actually it is all from Ada. Definitely this is not a news, I failed to pay enough attention since I know nothing about Ada.

You can check out syntax across languages per category to compare the two.

Dynamic typing and error checking

2005-09-28T11:41:00.000-07:00

For people who get used to Eclipse or other great IDE, coding in ruby has some dark sides. For example, there is no compilation, so error checking only happens at runtime. Normally after you write a program, you run it, find a bug, fix it and repeat the process again and again. Since the program always stops at the first error, it is very inefficient. Most of the people would prefer the IDE to detect errors (like typo) while they are typing.

Some people believe it is the fault of "dynamic typing". Actually it is not, the poor IDE is the one to blame. As long as the language is strong typed, a compiler(or part of an IDE) is able to do lots of type inference, then validation. You do not have to wait it until runtime. Static typing make type checking easier, in some case even better. But dynamic typing language itself does not prevent you to do it. Unfortunally there is no ruby IDE doing that.

UPDATE(10/11/2005): Eric Lippert has a new post on this topic today[2]. MS's script engine(JScript and VBScript) can check syntax errors without actually running the code, but unfortunately it always stops at the first syntax error.

References:
[0] Compiling in Vain
[1] Typing: Strong vs. Weak, Static vs. Dynamic by Aahz
[2] Checking For Script Syntax Errors by Eric Lippert

How text editor stores text

2005-09-10T22:09:00.000-07:00

Text editors need to store text in an efficient way, so that common operations like insert/delete/replace have minimized overhead. How do they do it?

Well, the simplest way is to put anything into an array(string), but clearly this is not very efficient. If an user is inserting/deleting text at beginning of a file, each keystroke will result in copying the entire buffer.

There is an obvious trick to improve the performance : since most editing activities tend to occur frequently in a local area , we can allocate more space (often called "Gap Buffer") in advance so that further insertions/replacements do not involve copying. Now the text is represented by two contiguous blocks of data, and a gap buffer between them. Whenever an editing action takes place, we first move the gap to the location. Now the design becomes a little more complex (have to make the gap invisible to the user), but it is much more efficient (less copying and still enjoy caching benefits of locality). Gap Buffer is used in SharpDevelop(see SharpDevelop\src\Libraries\ICSharpCode.TextEditor\src\Document\TextBufferStrategy\GapTextBufferStrategy.cs) and Eclipse(see eclipse\plugins\org.eclipse.rcp.source.win32.win32.x86_3.1.0\src\org.eclipse.swt.win32.win32.x86_3.1.0\src.zip\org\eclipse\swt\custom\DefaultContent.java).

While gap buffer might be good enough for small size documents, the copying overhead will soon become unacceptable for larger files. Professional text editors (like MS Word, AbiWord etc) use a method called "piece table". If you know Composite Design Pattern, it is very easy to figure it out all by yourself. In this approach, you start with a buffer containing the original text read from the file (this buffer won't be touched anymore). Instead, the new text is kept in a new append-only block, and during editing, the simple buffer changes into a composite structure. You can think the structure as a linked list, and each item is the "piece". A delete is handled by splitting a piece into two pieces, and an insert is handled by splitting the piece into three pieces (the middle one is the inserted content).

References:

[1] Data Structures for Text Sequences by Charles Crowley

[2] The Craft of Text Editing by Craig A. Finseth

[3] Dissecting a C# Application: Inside SharpDevelop by SharpDevelop Team

[4] MS Word Piece Table by Tomas Frydrych

[5] Performance Improvements by Msevior

[6] Programmer Myopia by Wesner Moise

On Hungarian Notation

2005-09-10T12:50:00.000-07:00

Recently Micah Martin complained about Hungarian Notation on his blog. He is definitely not the first person to bring this issue on, the criticism might start as soon as the Notation was born. The complain is valid: if you need to change a type, in Micah Martin's case, from an interface(ICommand) to abstract class(Command), Hungarian Notation is in your way. You have to delete the capital "I" prefix in lots of places, which is not very "agile" for some people.

But I do not want to ignore the advantage of Hungarian Notation: with its help, you can quickly tell the nature of a type without consulting the documents. If you are using a third party library/framework with stable APIs, you are very likely to appreciate if the vendor uses the Hungarian Notation. And for vendors still doing Big Upfront Design (like MS), changing a type does not happen very frequently.

In fact Micah Martin's example is not very good as well, he'd better to create an abstract class which simply implements the interface, and let the subclasses extend from it (to enjoy benefit of template method). The clients of the interface do not require any change, they still consume the interface.

Thoughts on parser generators

2005-09-06T10:04:00.000-07:00

Recently I have been evaluating several parser generators and eventually picked antlr in my project. Among all the parser generators in the world, yacc, boost.spirit, antlr and Perl 6 represent the current state of art and each has unique characters worth talking about.

Since most languages do not have parsing features built-in (Perl 6 will be the pioneer), almost all of the current parser generators use code generation. A common problem is: the generated code may become very foreign to you, and look totally cryptic in your favorite debugger.

Yacc suffers most in this perspective, since it uses LALR parsing. Lots of people like LL(k) based antlr since it produces human readable code, which I find extremely useful when working on a new/unfamiliar grammar. For more detailed comparison you can read Ian Kaplan's article.

Boost.Spirit is quite unique (and amazing!). With the help of powerful C++ template and operator overloading, it mimics EBNF in C++. Although it looks like perl 6's approach, it actually generates code through template (compile time), and the generated code is not as readable as antlr's output. Even worse, since C++ compiler won't take special care for boost.spirit, a simple grammar error may appears to be multiple pages of impenetrable garbage. Dave Handley has a nice introduction about Boost.Spirit on CodeProject.

Comega(Cω) made an interesting move by integrating popular "foreign language"(XML, SQL) to the core of the host language(C#). Perl 6's built-in parsing support did the similar thing (brings EBNF to perl). This can help to provide ealier and more friendly error reporting, which is a mjor problem for Boost.Spirit.

So far, antlr is sufficient for me. It can generate parser in several languages (Java, C#, C++, Python), reports errors nicely, and the code is readable and debugger friendly.

Happy 10-year Birthday, Opera!

2005-08-30T09:01:00.000-07:00

There are lots of people in the world do not know opera web browser, and may never consider using an IE alternative. But for me, opera has been my browser of choice for 7 years.

The first version of opera I used is 3.x. At that time, IE 4 started to take over the browser market and lots of people were very disappointed about the sluggish Netscape 4. I found opera from Gao Chunhui's personal webstite, and became an fan since then.

The best way to describe opera is: fastest and most innovative web browser. On the office website, they also claim opera has better security track record than others, but I wish they did not say that. Maybe opera's security history looks great on paper, but I believe opera's underdog image helps a lot. I still experience a few crashes with the latest 8.0.2 version, maybe the bugs are hard to be exploited, but they mean something. Firefox once bragged about its security, but you know...

Now lots of people switched to firefox for better user experience (like tabed browsing, easy font size controlling etc), and I know why: it has opera features! And actually it makes me a little bit uncomfortable for firefox not acknowledging its competitor for those features.

Well, there are still lots of things for opera to improve (a few of them are even showstoppers for some people), I am happy with the way opera is. Happy birthday, Opera!

Why IIS is more secure since 6.0?

2005-08-25T10:35:00.000-07:00

If you read Michael Howard's blog, you may remember his article titled"IIS6 vs Apache2 Security Defects". Base on third party's data, IIS 6 shows a better security track record than Apache 2.

So, why IIS 6 becomes much safer? Especially given the notorious security history of IIS 5?

In his new book 19 Deadly Sins of Software Security, Michael Howard tells an interesting story:

(Page 3)"Internet information Server(IIS) 6.0 web server switched entirly to a C++ string class for handling input, and one brave developer claimed he'd amputate his little finger if any buffer overruns were found in his code. As of this writing, the developer still has his finger and no security bulletins have been issued against the web server in the nearly two years since its release."

Using ACE_SPIPE_Acceptor with ACE_Reactor

2005-08-22T14:25:00.000-07:00

Using ACE_SPIPE_Acceptor with ACE_Reactor is not very straightforward. There are three things to be noticed:
1. Use ACE_WFMO_Reactor. ACE_Select_Reactor does not work with windows named pipe. Named pipe handle is not socket, so you can not use select() on it. ACE_WFMO_Reactor works because it uses WaitForMultipleObjects().
2. ACE_SPIPE_Acceptor should be registered with the ACE_Reactor::register_handler (ACE_Event_Handler *, ACE_HANDLE) method, since the handle to demultiplex on is an event handle.
3. handle_signal() will be called instead of handle_input() and handle_output(). WaitForMultipleObjects() can not tell whether read or write condition is happening, so ACE_WFMO_Reactor has to can WSAEventSelect() again to check, which means handle_input()/handle_output() only happens for sockets.

As you can see even if they work together, it is not exactly the same "Reactor" model we used to program sockets (no handle_input and handle_output). Proactor model works better with windows named pipe.

Here is an example:


class ClientAcceptor : public ACE_Event_Handler
{
    public:
    ClientAcceptor(ACE_Reactor& reactor)
    {
        this->reactor(&reactor);
    }
    
    int open (const ACE_SPIPE_Addr &listen_addr);
    
    int handle_signal(int signum, siginfo_t* = 0, ucontext_t* = 0);
    
    protected:
    ACE_SPIPE_Acceptor acceptor_;
};

int ClientAcceptor::open (const ACE_SPIPE_Addr &listen_addr)
{
    if (this->acceptor_.open (listen_addr, 1) == -1)
    {
        ACE_ERROR_RETURN ((LM_ERROR, ACE_TEXT ("%p\n"),
        ACE_TEXT ("acceptor.open")),
        -1);
    }
    
    //register the event handle
    if( this->reactor()->register_handler(this, this->acceptor_.get_handle()) <
    0)
    {
        ACE_ERROR_RETURN ((LM_ERROR, ACE_TEXT ("%p\n"),
        ACE_TEXT ("register_handler")),
        -1);
    }
    
    ACE_DEBUG((LM_DEBUG, ACE_TEXT("ClientAcceptor::open\n")));
    return 0;
}

int ClientAcceptor::handle_signal(int signum, siginfo_t*, ucontext_t*)
{
    ACE_DEBUG((LM_DEBUG, ACE_TEXT("ClientAcceptor::handle_signal\n")));
    
    ACE_SPIPE_Stream stm;
    if (this->acceptor_.accept (stm) == -1)
    {
        ACE_ERROR_RETURN ((LM_ERROR,
        ACE_TEXT ("(%P|%t) %p\n"),
        ACE_TEXT ("Failed to accept ")
        ACE_TEXT ("client connection")),
        -1);
    }
    
    //If you have only one client to deal with, now you can simply play with
    stm.
    //
    //To handle multiple clients at the same time, you can either:
    //a. create a new thread for every connection
    //b. use asynchronous IO (shift to Proactor model)
    
    return 0;
}

int ACE_TMAIN(int argc, ACE_TCHAR*argv[])
{
    ACE_WFMO_Reactor wfmo_reactor;
    ACE_Reactor reactor (&wfmo_reactor);
    
    ACE_SPIPE_Addr addr("mynamedpipe");
    
    ClientAcceptor acceptor(reactor);
    if (acceptor.open(addr) == -1)
    {
        ACE_ERROR_RETURN((LM_ERROR, ACE_TEXT("Can not open to listen:
        %m\n")), -1);
    }
    
    //Run event loop
    ACE_DEBUG((LM_NOTICE, ACE_TEXT("Server started\n")));
    reactor.run_reactor_event_loop();
    return 0;
}

CruiseControl.NET, CVS and SSH password

2005-07-29T10:44:00.000-07:00

I tried to setup up a build server using CruiseControl.NET (CCNET for short) and am not happy about CCNET's documentation. My CVS server is only accessible via SSH and it's password protected. I need to give CCNET a password so that it can login itself (btw, using public/private key provides few security improvement here, you better have a limited account for read only CVS access so that a tool like CCNET can use). I started to read the documentation, but soon lost on this page(Using CruiseControl.NET with CVS). It simply mentiones:
"An example file is 'plinkwithpw.bat' in the tools sub-directory."

But guess where is the tools sub-directory? I searched my installation dir and found nothing. And it's turned out to be in their CVS! Since The script is so short, why not just put it in the documentation?

The documentation page does no allow me to edit (not even to add a comment), so I just post the script here:

0. I use Tortoise CVS in this example, so cvs.exe and plink.exe (renamed as TortoisePlink.exe) are already on my computer(C:\Program Files\TortoiseCVS). If you do not have one you need to download them.

1. Create plinkwithpw.bat like this:
@"C:\Program Files\TortoiseCVS\TortoisePlink.exe" -pw yourpassword %*

2. Create cvswithplinkrsh.bat like this:
@echo off
set CVS_RSH="C:\Program Files\TortoiseCVS\plinkwithpw.bat"
"c:\Program Files\TortoiseCVS\cvs.exe" %*

3. Edit ccnet.config to have something like this:

<sourcecontrol type="cvs">
<executable>C:\Program Files\TortoiseCVS\cvswithplinkrsh.bat</executable>
<workingDirectory>C:\projects\YourProject</workingDirectory>
</sourcecontrol>

Force windbg to load symbols

2005-07-20T08:44:00.000-07:00

I needed to analyze a crash dump yesterday but could not find the associated .pdb symbol file. Usually I check in all the binaries and symbols for public releases, but unfortunately there is a typo in my build script so that the symbol I am looking for is missing.

I do have the source code on my version control system, so it is quite easy for me to rebuild the symbol file. It should match the binary, but winbdg refused to load it since the timestamp differs:

*** WARNING: Unable to verify checksum for Your.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for Your.dll

Modifying the timestamp does not look easy, so I decided to force windbg to load the symbol. while impossible in VS.net, it is quiet easy with windbg, all you need is type in:

.symopt+0x40

This command indead enables SYMOPT_LOAD_ANYTHING option.

Now a complete stack trace showed up, nice...

Scripting ActiveX control in WSH

2005-06-22T11:07:00.000-07:00

Last week is quite frustrating for me. I have a ActiveX control created by MFC ActiveX wizard. The control has been used in web browser (scripted by javascript) and runs very well. Recently I need to script it in WSH, but every method invoked failed with 'catastrophic failure' error message.

The worse part is that it took me significant amount of time to find the answer from google group. I picked wrong words at the beginning which unfortunately generated lots of noise.

The lesson: think harder about the search string instead wasting time on irrelevant results ("catastrophic failure activex wsh" is the best for this issue). And it's better if I go directly to the related news groups/maillists (microsoft.public.scripting.wsh in this case).

Here I just quote the answer from Kyle Alons:

"First, a little background: ActiveX controls are normally persisted to/from some type of permanent storage, and they do this by implementing one or more IPersist* interfaces. Containers are supposed to call one of the IPersist methods to initialize or load the control upon creation.

COleControl, which all MFC controls derive from, enforces that this initialization has taken place before allowing any calls to be made to the control (COleDispatchImpl::Invoke, line 1209 of OLEDISP1.CPP calls IsInvokeAllowed, which checks a protected m_bInitialized member variable for TRUE; this variable is set in the IPersist* implementations in CTLPSTM.CPP).

Since WSH never calls the IPersist* methods, the above test fails, and the result is the 'catastrophic failure' error message. The test control container (and VB6, which I also tested), are smart enough to detect when a control class has been instantiated and call an IPersist*->InitNew method to initialize the control.

One workaround is to add the statement m_bInitialized = TRUE; in your control's constructor. This may be an acceptable workaround in this case, because the control isn't truly being hosted and its state isn't being loaded from anywhere. This seems like an oversight by the WSH team, since most other environments don't require the workaround; on the outside chance that the MS developers are actually reading this, maybe they can address it.

Kyle"

Unit Testing for Network Programming

2005-06-20T08:20:00.000-07:00

Network program is error-prone to write and hard to test, how can we make our life easier with the help of unit testing? Let's start with short examples:

Which function is easier to test?


int funcion1(const char* filename)
{
...open and process file...
}

int function2(std::istream& stm)
{
...process file...
}

The answer is function2: it avoids the dependency on external resource, so you can create a string and pass it in (see more on C++ coding standard item 63).

This is a basic technique that every unit testing book will teach. To apply it to network programming, we need to choose a stubable high level abstraction (instead of the low level socket as integer).

I use ACE as network library. While it provides very nice high level abstractions like ACE_SOCK_Stream, those classes are not designed to be base class (members are not virtual). This is understandable since lots of network programmers prefer to get the best performance they can.

So we have two choices here:

1. Use C++ template, and make the substitution happen at compile time.

Or

2. Create a wrapper to provide dynamic polymorphism (C++ coding style item 64).

I choose the later since it provides more flexibility.

Here is what the code looks like:


class ISockStream
{
public:
 virtual ~ISockStream(){}

 virtual ssize_t send (const void *buf,
     size_t n,
     int flags = 0,
     const ACE_Time_Value *timeout = NULL) = 0;

 virtual ssize_t recv (void *buf,
     size_t n,
     int flags = 0,
     const ACE_Time_Value *timeout = NULL) = 0;
};

template <class _ACE_PEER_STREAM>
class StreamAdapter : public ISockStream
{
protected:
 _ACE_PEER_STREAM stream_;
public:
 ssize_t send (const void *buf,
     size_t n,
     int flags = 0,
     const ACE_Time_Value *timeout = NULL)
 {
  return stream_.send(buf, n, flags, timeout);
 }

 ssize_t recv (void *buf,
     size_t n,
     int flags = 0,
     const ACE_Time_Value *timeout = NULL)
 {
  return stream_.recv(buf, n, flags, timeout);
 }
};

class MockStream : public ISockStream 
{
 ssize_t send (const void *buf,
     size_t len,
     int flags = 0,
     const ACE_Time_Value *timeout = NULL)
 {
  //save the data somewhere (std::string for example)
  //so that we can test the result later.
 }

 ssize_t recv (void *buf,
   size_t len,
   int flags = 0,
   const ACE_Time_Value *timeout = NULL)
 {
  //generate fake data,
  //we can even trigger failure (return -1).
 }

};

In real product, we are going to use things like StreamAdapter<ACE_SOCK_Stream>. While in unit testing, MockStream will be sufficient.

Parser generator used for popular languages

2005-06-15T10:03:00.000-07:00

Recently I need a parser generator in my new project, so I spent a while to check what other people are using. Here is the result:

Jython: JavaCC (with asdl)
Python: YACC
IronPython: Hand made recursive descent parser

Rhino/SpiderMonkey: Hand made recursive descent parser

Beanshell: JavaCC (with JJTree)
Groovy: Antlr

Boo: Antlr

Ruby: YACC
JRuby: Jay

Mono C#: Jay

Perl: YACC

Eclipse JDT: Jikes Parser generator

CFront: YACC, but Stroustrup actaully likes to have a hand made recursive descent parser, see [5]

References:
[1] coyote: Syntax coloring
[2] Pydev parser... the future??!?!?!
[3] Jan Arne Petersen on jruby-devel
[4] Java Abstract Syntax
[5] Stroustrup, The Design and Evolution of C++
[6] JDT Core HowTo: Generate the Parser

To be continued...

"-ko" in tortoise cvs

2005-06-09T12:26:00.000-07:00

When managing third party libraries, I like to create a repository in house (for C++, it includes headers and compiled libs). Pragmatic Version Control Using CVS also recommends this approach.

One of the annoying thing you may encounter is the keyword expansion feature of CVS (I do not like it, so do the authors of the above cvs book). It means when importing/adding files, you have to specify "-ko"(keyword off) in command line. For tortoise CVS, you need to:

1. Select all the files you want to add;
2. Right Click, find "keywords" option, set it to "disabled(0)".

code or data?

2005-06-09T12:06:00.000-07:00

I am thinking the two most common security problems - buffer overflow for binary program and cross site scripting for website - share the same nature: instead of treating user's input as pure data, the system/runtime may execute them as code, which in turn grant the user the ability to 'customize' some of the behavior of the program.

Even though the modern compilers(like java and C#) fight hard to make buffer overrun disappear, those "executable data" will not go away (sql injection for example), and the restriction of sandbox can not help too much.

In Cω, researches start to add build-in support for data access and concurrency. Security may be on the list somedays, although now I have no idea how it fit in.

Suppressing splash screen of source insight

2005-06-03T12:39:00.000-07:00

Source insight is my editor of choice for 6 years. The reason? It is super fast, especially when your are searching symbols in monster projects, like linux kernel, ACE and TAO.

One small issue I feel annoying is the splash screen. The manual did a poor job here, it only tells me how to suppress the splash screen if I launch the program from command line (that is what I do most of the time), but occasionly I just want to launch it from windows explorer (by clicking an associated file), the splash screen just on my way.

I sent their support an email, and their support(Ray Gram) emailed me back the solution in ten minutes! Well, I really fell shame for not asking the question ealier:)

Since I can not find it via google (it is not in the current manual either) , let's share it:


You can suppress the splash screen with the -s command line option.
For example:

Insight3 -s

You can also do it by adding a registry value. Under the key:

HKEY_CURRENT_USER\Software\Source Dynamics\Source Insight\3.0\Options

Create a text value named "NoSplash" with a value of "1".

Ray Gram
Source Dynamics, Inc.

Thank you, Ray!

Memory footprint

2005-06-02T14:39:00.000-07:00

That was two months ago, I need to reduce the memory footprint of one of my application. I checked the "Mem Usage" column in windows task manager, the result is surprsing: the process uses more than 10M just after it hits main().

The good news is it did not take me long to figure out that "Mem Usage" means the size of the current working set, which includes the shared data, dynamic libaries (DLL) for example.

Even the minimal hello world program requires lots of system dlls to run, so it takes about 1,000K after startup. If you add winsock, it soon becomes 1,500K.

The article of Joseph M. Newcomer explained the topic very well. And this one touched .NET.

Trusted and un-trusted vulnerability assessment

2005-05-27T09:04:00.000-07:00

I read a article titled From SATAN to OVAL: The Evolution of Vulnerability Assessment today, and I really like the term "Trusted" and "un-trusted" vulnerability assessment.

As Dr. Gerhard Eschelbeck put in his article:
"Un-trusted vulnerability assessments simulate the scenario of an attacker without prior knowledge about the target system, while trusted assessments leverage credentials to log into the target systems for auditing configuration and patch information."

Trusted vulnerability assessment is kind of the trend that scanner vendors are doing. One example is MS baseline scanner, another is nessus (its plugins number growed from less then 2000 to more than 8000 in 2 years, most of the new ones are "local checks"). The reason behind it is obvious: this is way easier to implement and it works well in enterprise environment (also make it less likly to be used a hacking tool).

Security and Testing Conferences

2005-05-27T08:58:00.000-07:00

Secure Programming Group at University of Oulu (Finland) maintained a very nice list of Security and Testing Conferences. I had tried to do same something on my little personal website when I was in school, but lost interest to update it after graduation:(

http://www.ee.oulu.fi/research/ouspg/sage/conferences/index.html

Building native C++ projects with NAnt

2005-05-20T14:29:00.000-07:00

I have not tried nant for a long time because I thought(wrongly) it is just for .NET, and my daily job involves only native C++.

My current build process use MPC, devenv command line and batch script. I like MPC a lot(especially the idea of inheritance used the build process). MPC's template approach can help me to create .sln and .vcproj with very little effort.

Devenv.com is helpful as well, although it is a little bit slow. vcbuild.exe is faster, but there are some small issues I do not like (the devil is always in the details).

I uses batch script(.bat) to glue the build process together, but I really hate to program in this old awkward langage. I resisted to put more error handling because it can easily mess up the script.

So I decided to switch to nant after I found it can handle c++ project as well. Even though it takes me a while to figure it out, using nant is quiet easy. The solution task is great idea (know how to live peacefully with others is important:)): no need to write verbose build script and no need to wot worry about keeping it in sync with visual studio.

If all the environment variables are setup correctly (I will take about the tricks later), the build file can be as simple as this(and it is faster than devenv):

<?xml version="1.0" ?>
<project name="myproject" default="all" xmlns="http://nant.sf.net/schemas/nant.xsd">
<target name="all" depends="debug, release" />

<target name="debug">
<solution solutionfile="myproject.sln" configuration="Debug" />
</target>

<target name="release">
<solution solutionfile="myproject.sln" configuration="Release" />
</target>
</project>

Unless you run nant from Visual Stuio 2003 Command Prompt, you may see lots of erros about missing headers and libs. This is because you need a few more environment varibles(PATH, INCLUDE and LIB). Visual studio will set them up for itself (see vcvars32.bat) and nant are not aware of them (it knows only system level environment varibles).

Nant supports setenv task so it looks like good place to go. But it will bloat the build file. Why not just use vcvars32.bat? If you are following the installation procedure of nant, you can just add a line into your nant.bat:

@echo off
@call "C:\Program Files\Microsoft Visual Studio .NET 2003\Common7\Tools\vsvars32.bat"
"C:\tools\nant-0.85-rc3\bin\NAnt.exe" %*

You can also use "HKEY_CURRENT_USER\Software\Microsoft\Command Processor\AutoRun" so that vcvars32.bat is launched everytime cmd.exe is created. For cygwin users you know what to do:)

There is a small issue in the current release (0.85 Release Candidate 3): rc files can include c++ header files. Resource.h is one of them. And this practice uses it as well. But in the source code(nant-0.85-rc3\src\NAnt.VisualCpp\Tasks\RcTask.cs::NeedsCompiling), the date of the header files are not checked. So if you modified the include file, the resource will be rebuilt.

Btw, nant uses its own algorithm to calculate the dependencies between files, so you may see different compilation sequence between nant and devenv, it's even true for C# projetcs.

windbg beated VS.NET debugger

2005-05-20T12:17:00.000-07:00

Our QA guy sent me a minidump file of crashed service and asked my help. The developer who wrote the program forgot to check in the full debug symbol file, but we do have the stripped .pdb file since we always ship it along with our product (following the advise of John Robbins).

I opened the minidump file with Visual Studio .NET debugger, but this time it did a terrible job reconstructing stack trace (later I find out the information it gave me is misleading, which is worse than useless). Well, I soon switched to windbg, and it showed me the correct result. The bug was found(printf again!) and fixed in less than 10 minutes, thanks to windbg.

Last time windbg beated VS.NET debugger on my system is a program deadlocked in the system space and VS.NET debugger can not break into it.

Use devenv in cygwin

2005-05-20T09:25:00.000-07:00

I use cygwin as a better shell, not a development environment. Cygwin can do an excellent job as a replacement of cmd.exe(of course it can do much much more), and I like it very much.

There is a few corner cases where command behaves differently in cygwin and cmd.exe, for example:

When you type 'devenv' in cmd.exe, it will bring up devenv.com, which is expected since Windows prefers .com to .exe if both exist in the user's PATH (see the article in C++ QA column of the February 2004 issue of MSDN Magazine). But when you type 'devenv' in cygwin, it will launch devenv.exe instead, the project will get compiled anyway but you can no longer see the output of the compilation process.

"Files and Folders" View of InstallShield

2005-05-18T14:26:00.000-07:00

InstallShield gives you a nice "Files and Folders" view(Application Data) and lets you add new components as drag and drop files. It is very easy for a developer to jumpstart an installation project without reading any documents. But it does not mean you can totally forget the existence of component table.

Recently I find out there are lots of duplicated components in my project, they have the same name but different suffix, for example: mylib.dll1, mylib.dll2, mylib.dll3. It confused me for a while because I am not sure if I should delete some of them(as a developer, removing duplication becomes my habit:)).

Until recently I find out that if you delete files from "Files and Folders" view, they won't be removed from the component table. And after you add the same file again, InstallShield will create a new component named yourfile.dll1(.dll2 if .dll1 exists already). By the way, they will be considered to be different components since the component code changed.

Maybe InstallShield thinks it's nice to keep them for people don't use version control:)

Give Up Microsoft Visual C++ Toolkit 2003

2005-05-13T11:21:00.000-07:00

I tried to get some free lunch from Microsoft Visual C++ Toolkit 2003, but finally gives up. There are lots of missing pieces, most of them can be fixed by downloading other packages, but it is tedious and may not be allowed.

There are a few problems I encountered:

1. Only static-link C/C++ libraries are available, so you can not use /MD, /MDD by default. this link introduced a hack but as mentioned in the same article it has its limits.

2. Do not have a build system. It's easy to generate solution(.sln) and project (.vcproj) with MPC, and we can use vcbuild.exe to compile them. But soon you will find out vcbuild requires several dlls from VS C++ 2003, even through you can copy some dlls from there to make it work(see this article). So you have to purchase a copy of VS C++ anyway.

When URL moniker meets security problems

2005-04-27T13:07:00.000-07:00

URL moniker is a very useful. For example, URLDownloadToFile allows you to download files using IE. While it's easy to write code to do that job yourself, the users may be appreciated if they do not have to config it for their proxy.

Three months ago I met some problems with URLDownloadToFile when dealing with https url. If IE is used, it will pop up an error saying the certificate is not trusted, obviously the owner of the site does not want to spend money to get their certificate signed. URLDownloadToFile will simply fail in this case, it won't show you the dialog as IE does.

To see the dialog, we need to implement a callback object and pass it to URLDownloadToFile. MSDN simply said the callback should implement BindStatusCallback, but here, we need IHttpSecurity as well. And most important method is GetWindow() method. URLDownloadToFile need this to report error.



class Callback : public IBindStatusCallback, IHttpSecurity
{
...

 STDMETHOD(GetWindow)(REFGUID rguidReason, HWND *phwnd)
 {
  if (NULL != phwnd_)
  {
   (*phwnd) = (*phwnd_);
  }
  else
  {
   (*phwnd) = GetDesktopWindow();
  }
  return S_OK;
 }

...
};

CPPUNIT: constructor/destructor VS setUp/tearDown

2005-04-27T12:38:00.000-07:00

In C++, we used to use constructor and destructor to manage resources.
In XUNIT, setUp() and tearDown() are favored instead. One of the reasons
is that some langurages like Java have only undetermined garbage collector,
while we need a "determined" way to clean up the state after each test.

But in C++, we can use stack. As mentioned in
Herb Sutter's OOPSLA keynote, this approach has its own
advantage. So when writing tests with CPPUNIT, I still prefer constructor/destructor over setUp/tearDown, and it works.

There is a disadvantage, althrough it should be a implementation issue
of CPPUNIT: TestFixtures' Constructor will be called before main() to register
themself. So if your constructor fails, the whole program won't run at all.
In this case, you have to call setUp().

resize web browser control

2005-04-13T13:12:00.000-07:00

I have a dialog based MFC application using Microsoft web browser control(IWebBrowser2). From the first version, I disabled the maximize box and used non-resizable border to avoid the painful layout of GUI elements. Now I want to make it resizable, so I edited the properties in resources: now the dialog itself is resizable, but the Active X control doesn't.

I truned to google and hope it can be sovled in 5 minutes, but the search is
not very successful. After about an hour I solved the problem myself, even though it is very simple, I decided to write it down so that another poor man can get helped.

The solution is to use the OnSize() Event to size our control:



BEGIN_MESSAGE_MAP(CWebBrwoserDlg, CDialog)
    ON_WM_SYSCOMMAND()
    ON_WM_PAINT()
    ON_WM_SIZE()
    ON_WM_QUERYDRAGICON()
    //}}AFX_MSG_MAP
END_MESSAGE_MAP()

void CWebBrwoserDlg::OnSize(UINT nType, int cx, int cy)
{
    CDialog::OnSize(nType, cx, cy);//Call parent's OnSize first

    if (m_browser.GetSafeHwnd() != NULL))//Make sure the windows is created.
    {
        m_browser.MoveWindow(0, 0, cx, cy);//resize browser control
    }
}

Faked port 25 from symantec antivirus software

2005-04-06T12:37:00.000-07:00

Symantec antivirus software gives me lots of trouble recently. To check if an email contains virus, it somehow creates a fake port 25 to intercept the traffic. So if you have Symentec antivirus software installed on your desktop, whenever you telnet or call connect() to a port 25, it will always succeed regardless the state of the remote host, even if the host is down.

So port scan using tcp connect will give lots of false positives unless you stop symentec antivirus service. To avoid that you have to bypass compromised tcp/ip stack. For example, synscan with pcap can give you better result, since there is no actual SYN/ACK back from the faked port.

Bug in Netscape Internet Service software

2005-04-06T11:47:00.000-07:00

I spent almost a day on a "bug" appeared in my program, which is a very simple portscanner using non-blocking tcp connect. I closed all the sockets after they are used, but boundchecker still complains about "Outstanding Allocations", and the call path is "My Program->ws2_32.dll->sliplsp.dll". Actually I should pay more attention to sliplsp.dll, but at that moment, ws2_32.dll convinced me that the bug is in my program.

I opened windows task manager and noticed that the handle count keeps increasing. To see the details about the handle I launched Process Explorer, and found out there are lots of file handles named "\Device\WS2IFSL\NifsSct". What the hell is that?

Even google can not help, there is no useful discussion of NifsSct at all.

My program is quiet simple, but to be 100% sure, I started to use simple code to test, and found something interesting:
1. After socket() succeed, a new handle called "\Device\Afd\Endpoint" will be created.
2. After connect() succeed, a new handle called "\Device\Tcp" will be created.
3. After closesocket(), all of them will be closed.

But the problem is, "\Device\WS2IFSL\NifsSct" will be created along with the other two, and I have no idea what they are. Finally I had to give up after several hours of pointless efforts.

Then another day came. I want to solve another myth I saw: after reading tons of good words about Ruby on Rails, I want to give it a try myself, but the installation of rails("gem install rails") always deadlock ruby.exe. I was very disappointed about ruby at that moment, since there are source code at hand, I decided to find the "bug" and fix it.

Then my eyes opened: the debugger shows ruby.exe loads "C:\Programs Files\Netscape Internet Service\sliplsp.dll". Hah? I turned back to boundcheck, and enabled the full path in the view, it is the same dll!

I have had some bad experience before with programs hooked up with standard network stack. So I uninstalled Netscape Internet Service, deleted its folder after reboot, then everything goes fine. There is no more leaking \Device\WS2IFSL\NifsSct and I finally get rails installed.

There are not too much users in the world use Netscape Internet Service. It explains why there are no discussion about it at all.