ngen

As mentioned in Reid Wilkes's article on MSDN Magazine (April 2005), Microsoft has always used NGen on its own core Framework assemblies at .NET Framework installation time.
Here is screen capture during my installtion of .NET 2.0 (the installer spent quite a lot time doing ngen):

Do not disable "DCOM Server Process Launcher" service

DCOM Server Process Launcher is a new service introduced in windows XP SP2. It plays a very important role on the system, and will start other "manual" services when necessary. Since it is relatively new and the documentation about it is poor, some people believe this service is useless and disabled it.

I have experienced several weird problems when the service is not started:

1. Watir scripts won't run, Ruby reports "Failed to create WIN32OLE object from `InternetExplorer.Application' (WIN32OLERuntimeError)
HRESULT error code:0x800706ba The RPC server is unavailable.".

2. Symantec Antivirus LiveUpdate fails with LU1803 error. All the workarounds on the web do not help.

3. Most installers won't run with "RPC server is unavailable" error message. Although you can fix that by starting "Windows Installer" service manually.

If you have the same problems, first check if its startup type is "Automatic".

Let watir do the job

AutoLinking is a very convenience feature for almost every blog writer. For example, when you are writing about antlr, you may want to give a link to www.antlr.org, and a smart blog authoring tool should be able to do that for you. But as Wesner Moise mentioned on his blog , currently no blog-posting tool provides that service.

So I wrote a watir script(Watir is a great web application testing framework in Ruby) to help me. First it uses regular expression to do autolinking, then go to the website and publish the post (so I do not have to click all the links and wait for a reply).

Here is the code:

require 'watir'
include Watir 

#Start your writing here
title = "Let watir do the job"
content = <<EOF
I love watir !
EOF

#Polish and Autolinking etc
content = content.gsub(/\n/, ' ') #replace linebreak with whitespace 
content = content.gsub(/\bwatir\b/, '<a href="http://wtr.rubyforge.org">watir</a>')

#Go to the website
ie = IE.new
ie.goto("http://www.blogger.com/start") 

#Login
ie.text_field(:name, "username").set("yourusrename")
ie.text_field(:name, "password").set("yourpassword")
ie.link(:text, "Sign in").click

#Open blog
ie.link(:text, "your blog title").click

#Create new post
ie.link(:text, "Create new post").click

#Publish it ("value=" is faster then "set")
ie.text_field(:name, "title").value=(title)
ie.text_field(:name, "postBody").value=(content) 
ie.button(:value, "Publish Post").click

A simple port scanner using ACE proactor

Today I wrote a simple port scanner using ACE proactor. Even though the code itself is useless, the process helped me to know ACE proctor's implementation details much better.

Proactor is a very good model for high performance IO, but you can not implement an efficient port scanner with it.

First, since most platforms do not have native capability for asynchronously connecting sockets, ACE may choos to emulate asynchronous connections with an instance of ACE_Select_reactor managed privately by the framework (for example, on windows). Btw, the description on C++NP v2 page 283, sidebar 57 is inaccurate: even though Windows XP/2003 has that capability(via ConnectEX() API), ACE is not using it now (to support older platform like windows 2000). You can check out the source code of ACE_WIN32_Asynch_Connect::connect() to see yourself. So in this case you are using a Reactor in deed, and since the framework need to spawn thread to do the emulation, it is less efficient than using Reactor directly.

Second, since each ACE_Handler can only be bundled with one handle/socket, you need to create one ACE_Handler for each connection, which definitely has a few more extra overhead than just creating a plain old socket handle.

Here is the source:

#include <ace/Log_Msg.h>
#include <ace/INET_Addr.h>
#include <ace/OS.h>
#include <ace/Proactor.h>
#include <ace/WIN32_Proactor.h>

class PortScanner : public ACE_Handler
{
public:

 PortScanner()
 {
 }

 int open(ACE_Proactor& proactor, const ACE_INET_Addr& remote_addr)
 {
  this->proactor(&proactor);
  this->remote_addr_.set(remote_addr);

  if (connect_.open(*this, ACE_INVALID_HANDLE, 0, this->proactor()) < 0)
  {
   return -1;
  }

  return connect_.connect(ACE_INVALID_HANDLE, remote_addr_, ACE_Addr::sap_any, 1);
 }

 void handle_connect(const ACE_Asynch_Connect::Result &result)
 {
  //This handler will be called if connection is established or RST is received,
  //You may no get any response in a timely manner even if there is no firewall.

  ACE_ASSERT(result.connect_handle () != ACE_INVALID_HANDLE);

  if (result.success())
  {
   //Connection is established
   ACE_DEBUG((LM_NOTICE, "%s:%d is open\n",
        remote_addr_.get_host_addr(),
        remote_addr_.get_port_number()));
  }
  else
  {
   //Got a RST
  }
 }

private:
 ACE_INET_Addr remote_addr_;
 ACE_Asynch_Connect connect_;
};

int ACE_TMAIN(int /*argc*/, ACE_TCHAR** /*argv*/)
{
 ACE_Proactor proactor;
 ACE_INET_Addr scan_target("127.0.0.1:0");

 //Asynchronous connection are simulated on Both Windows and Unix
 //(C++NP v2 page 283, sidebar 57 is inaccurate).
 //Since ACE_Select_reactor is used internally, the handles limit
 //applies. You can either increase the limit(1024 by default), or
 //simply do not register too much handles simultaneously.
 unsigned short start_port  = 1;
 unsigned short end_port = 1024;

 ACE_ASSERT(end_port > start_port);
 const size_t size = end_port - start_port + 1;
 //'size' is not const expression so we can not create the array on stack
 PortScanner* portscanners = new PortScanner[size];

 for (unsigned short i = start_port; i < end_port; ++i)
 {
  scan_target.set_port_number(i);
  portscanners[i].open(proactor, scan_target);
 }

 //Run event loop for 10 seconds
 ACE_DEBUG((LM_NOTICE, ACE_TEXT("Portscan started\n")));
 ACE_Time_Value timeout(30);
 proactor.proactor_run_event_loop(timeout);

 delete[] portscanners;

 return 0;
}

Access Control for Ruby Class

The following words are from Programming Ruby: The Pragmatic Programmers' Guide:

"Ruby differs from other OO languages in another important way. Access control is determined dynamically, as the program runs, not statically. You will get an access violation only when the code attempts to execute the restricted method."

That's because those "public"/"protected"/"private" you saw are actually implemented as functions! The parser does not even know them. As the program runs, those functions are called and then you know if there is any errors.

Lexer state

Lexer state cannot be avoided when implementing a lexer for Perl and Ruby.

In Perl/Ruby, regular expression starts and ends with '/', for example:

/perl|ruby/

As we all know, '/' is the divide operator.

So when the lexer meets a '/' character, it can not tell whether it is the start of a regular expression or a divide operator without the help of the context.

Same thing for "<<", it can be the left shift operator, or the start of "here doc":

<<EOF
whatever
EOF

The Ruby phenomenon

The recent Ruby phenomenon is very interesting. Lost of people (including me) are first attracted by a killer application (e.g., rails and watir), then start to learn this foreign language (considering the majority programmers are only familiar with "curly braces" languages, like C++/C#/Java).

Both rails and watir are in a very competitive market. To develop web applications, there are big players like J2EE and ASP.net, even PHP has a larger user base than ruby. To do web application testing, javascript and vbscript have been used for years. Believe me, no matter how good ruby is, most people would like to stay with the language they know. Lots of credits should go to rails and watir.

So what makes rails and watir successful? One philosophy: make simple things simple and complex things possible. Everyone knows it, but it is quiet hard to achieve.

We can build very complex web applications with J2EE and .NET, while they have simplified the job greatly, comparing with developing in portable assembly(I am talking C :)), there are still lots of tedious works. On the other hand, there are some RAD tools which can help us to build an eCommerce site without a single line of coding, adding additional features soon becomes painful.

Rails and waitr do lots of things behind the scene, making good choice themselves without bothering us. Meanwhile the simple and powerful ruby language enable us to do complex things without too much works. Now that's a real beauty!

Sniffer and Reactor/Proactor

While I have used pcap and winpcap for years, I am not big fan of the current programming model. I would wish to have an event driven model(reactor) or asynchronous IO model(procator). In other words, treat PCAP handler just like socket, have someone (select() for example) notify me when something interesting happens.

But maybe it is not pcap to blame, most of time it is just implemented as an user level library, and the underlying operating system may not provide a way to do IO multiplexing for such low level functions.

To implement a sniffer on windows, most of people will go for winpcap (btw, after a long waiting(two years!), winpcap 3.1 is finally out to replace the very buggy 3.0 version). Actually winsock has native sniffing support via raw socket(with SIO_RCVALL option), and it works well with reactor and proactor!

Here is the code to demonstrate the functionality (I used ACE. I know most of people would prefer plain old socket programming, but it is too hard for me to go back the dark days):

//Sniffer and Reactor Example
#include <ace/Log_Msg.h>
#include <ace/Reactor.h>
#include <ace/INET_Addr.h>
#include <ace/OS.h>
#include <ace/WFMO_Reactor.h>

#include <mstcpip.h>//SIO_RCVALL

//ACE_NDEBUG will be defined if you define _DEBUG on windows.
#ifdef ACE_NDEBUG
#define ACE_VERIFY(X) ((void)(X))
#else
#define ACE_VERIFY(X) ACE_ASSERT(X)
#endif //ACE_NDEBUG

class Sniffer : public ACE_Event_Handler
{
public:
 Sniffer(ACE_Reactor& reactor)
  :socket_(ACE_INVALID_HANDLE)
 {
  this->reactor(&reactor);
 }

 ACE_HANDLE get_handle() const
 {
  return socket_;
 }

 ~Sniffer()
 {
  if (ACE_INVALID_HANDLE != socket_)
  {
   ACE_VERIFY(ACE_OS::closesocket(socket_) == 0);
  }
 }

 /*
 MSDN On SIO_RCVALL:
 Enables a socket to receive all IP packets on the network. The socket handle passed to the
 WSAIoctl function must be of AF_INET address family, SOCK_RAW socket type, and IPPROTO_IP
 protocol. The socket also must be bound to an explicit local interface, which means that you cannot
 bind to INADDR_ANY. 
 Once the socket is bound and the ioctl set, calls to the WSARecv or recv functions return IP
 datagrams passing through the given interface. Note that you must supply a sufficiently large
 buffer. Setting this ioctl requires Administrator privilege on the local computer. SIO_RCVALL is
 available in Windows 2000 and later versions of Windows.
 */
 int open (const ACE_INET_Addr& listen_addr)
 {
  socket_ = ACE_OS::socket(AF_INET, SOCK_RAW, IPPROTO_IP);
  if (ACE_INVALID_HANDLE == socket_)
  {
   ACE_ERROR((LM_ERROR, ACE_TEXT("%p\n"), ACE_TEXT("socket() failed in Sniffer::open()")));
   return -1;
  }

  sockaddr* laddr = reinterpret_cast<sockaddr*>(listen_addr.get_addr());
  size_t size = listen_addr.get_size();
  ACE_VERIFY(ACE_OS::bind(socket_, laddr, size) == 0);

  DWORD YES = 1;
  ACE_VERIFY(ACE_OS::setsockopt(socket_, IPPROTO_IP, IP_HDRINCL, reinterpret_cast<const char*>(&YES), sizeof(YES)) == 0);

  DWORD dwBytesRet = 0 ;
  if (WSAIoctl((SOCKET)socket_, SIO_RCVALL, &YES, sizeof(YES), NULL, 0, &dwBytesRet, NULL, NULL) != 0)
  {
   ACE_ERROR((LM_ERROR, ACE_TEXT("%p\n"), ACE_TEXT("WSAIoctl() failed in Sniffer::open()")));
   return -1;
  }

  DWORD rcvtimeo = 1000 ; // 1 sec instead of 45(default)
  ACE_VERIFY(ACE_OS::setsockopt(socket_, SOL_SOCKET, SO_RCVTIMEO, reinterpret_cast<const char*>(&rcvtimeo) , sizeof(rcvtimeo)) == 0);

  // register the event handle
  if( this->reactor()->register_handler(this, ACE_Event_Handler::READ_MASK) < 0)
  {
   ACE_ERROR((LM_ERROR, ACE_TEXT ("%p\n"),
     ACE_TEXT("register_handler")));
   return -1;
  }

  ACE_DEBUG((LM_DEBUG, ACE_TEXT("Sniffer::open\n")));
  return 0;
 }

 int handle_input(ACE_HANDLE fd = ACE_INVALID_HANDLE)
 {
  ACE_ASSERT(fd == socket_);

  char recvbuf[65536] = {0};
  ssize_t nrecv = ACE_OS::recv(socket_, recvbuf, sizeof(recvbuf) - 1);
  if (nrecv > 0)
  {
   ACE_HEX_DUMP((LM_NOTICE, recvbuf, nrecv));
  }
  else
  {
   ACE_ERROR((LM_ERROR, ACE_TEXT("handle_input returns %d: %m\n"), nrecv));
  }

  return 0;
 }

private:
 ACE_HANDLE socket_;
};

int ACE_TMAIN(int /*argc*/, ACE_TCHAR** /*argv*/)
{
 ACE_WFMO_Reactor wfmo_reactor;
 ACE_Reactor reactor (&wfmo_reactor);

 //TODO bind to your local address!!!
 //Do not work with loopback address (127.0.0.1)
 ACE_INET_Addr addr("192.168.0.100:0");

 Sniffer sniffer(reactor);
 if (sniffer.open(addr) == -1)
 {
  ACE_ERROR((LM_ERROR, ACE_TEXT("Can not open to listen: %m\n")));
  return -1;
 }

 //Run event loop
 ACE_DEBUG((LM_NOTICE, ACE_TEXT("Sniffer started\n")));
 reactor.run_reactor_event_loop();
 return 0;
}

--------------------------------

//Sniffer and Proactor example
#include <ace/Log_Msg.h>
#include <ace/INET_Addr.h>
#include <ace/OS.h>
#include <ace/Proactor.h>
#include <ace/WIN32_Proactor.h>

#include <mstcpip.h>//SIO_RCVALL

//ACE_NDEBUG will be defined if you define _DEBUG on windows.
#ifdef ACE_NDEBUG
#define ACE_VERIFY(X) ((void)(X))
#else
#define ACE_VERIFY(X) ACE_ASSERT(X)
#endif //ACE_NDEBUG

class Sniffer : public ACE_Handler
{
 static const size_t PACKET_SIZE = 65526;
public:
 Sniffer(ACE_Proactor& proactor)
  :socket_(ACE_INVALID_HANDLE)
  ,ACE_Handler(&proactor)
  ,mblk_(PACKET_SIZE)
 {
 }

 ACE_HANDLE handle() const
 {
  return socket_;
 }

 ~Sniffer()
 {
  if (ACE_INVALID_HANDLE != socket_)
  {
   ACE_VERIFY(ACE_OS::closesocket(socket_) == 0);
  }
 }

 /*
 MSDN On SIO_RCVALL:
 Enables a socket to receive all IP packets on the network. The socket handle passed to the
 WSAIoctl function must be of AF_INET address family, SOCK_RAW socket type, and IPPROTO_IP
 protocol. The socket also must be bound to an explicit local interface, which means that you cannot
 bind to INADDR_ANY. 
 Once the socket is bound and the ioctl set, calls to the WSARecv or recv functions return IP
 datagrams passing through the given interface. Note that you must supply a sufficiently large
 buffer. Setting this ioctl requires Administrator privilege on the local computer. SIO_RCVALL is
 available in Windows 2000 and later versions of Windows.
 */
 int open (const ACE_INET_Addr& listen_addr)
 {
  if (create_sniffer_socket(listen_addr) < 0)
  {
   return -1;
  }

  // register the event handle
  if (reader_.open(*this) < 0)
  {
   ACE_ERROR((LM_ERROR, ACE_TEXT("open: %m\n")));
   return -1;
  }

  ACE_DEBUG((LM_DEBUG, ACE_TEXT("Sniffer::open\n")));

  //initiate our first asynchronous read
  reader_.read(mblk_, PACKET_SIZE);

  return 0;
 }

 void handle_read_stream(const ACE_Asynch_Read_Stream::Result &result)
 {
  if (!result.success() || result.bytes_transferred() == 0)
  {
   ACE_ERROR((LM_ERROR, ACE_TEXT("read_streamhandle_read_stream error\n")));
   return;
  }

  ACE_ASSERT(&mblk_ == &result.message_block());

  ACE_HEX_DUMP((LM_NOTICE, mblk_.rd_ptr(), result.bytes_transferred()));

  //Reset the write pointer and read again
  mblk_.wr_ptr(mblk_.rd_ptr());
  reader_.read(mblk_, PACKET_SIZE);
 }

private:
 ACE_Asynch_Read_Stream reader_;
 ACE_HANDLE socket_;
 ACE_Message_Block mblk_;

 int create_sniffer_socket(const ACE_INET_Addr& listen_addr)
 {
  socket_ = ACE_OS::socket(AF_INET, SOCK_RAW, IPPROTO_IP);

  if (ACE_INVALID_HANDLE == socket_)
  {
   ACE_ERROR((LM_ERROR, ACE_TEXT("%p\n"), ACE_TEXT("socket() failed in Sniffer::open()")));
   return -1;
  }

  sockaddr* laddr = reinterpret_cast<sockaddr*>(listen_addr.get_addr());
  size_t size = listen_addr.get_size();
  ACE_VERIFY(ACE_OS::bind(socket_, laddr, size) == 0);

  DWORD YES = 1;
  ACE_VERIFY(ACE_OS::setsockopt(socket_, IPPROTO_IP, IP_HDRINCL, reinterpret_cast<const char*>(&YES), sizeof(YES)) == 0);

  DWORD dwBytesRet = 0 ;
  if (WSAIoctl((SOCKET)socket_, SIO_RCVALL, &YES, sizeof(YES), NULL, 0, &dwBytesRet, NULL, NULL) != 0)
  {
   ACE_ERROR((LM_ERROR, ACE_TEXT("%p\n"), ACE_TEXT("WSAIoctl() failed in Sniffer::open()")));
   return -1;
  }

  DWORD rcvtimeo = 1000 ; // 1 sec instead of 45(default)
  ACE_VERIFY(ACE_OS::setsockopt(socket_, SOL_SOCKET, SO_RCVTIMEO, reinterpret_cast<const char*>(&rcvtimeo) , sizeof(rcvtimeo)) == 0);

  return 0;
 }
};

int ACE_TMAIN(int /*argc*/, ACE_TCHAR** /*argv*/)
{
 ACE_WIN32_Proactor win32_proactor;
 ACE_Proactor proactor(&win32_proactor);

 //TODO bind to your local address!!!
 //Do not work with loopback address (127.0.0.1)
 ACE_INET_Addr addr("192.168.0.100:0");

 Sniffer sniffer(proactor);
 if (sniffer.open(addr) == -1)
 {
  ACE_ERROR((LM_ERROR, ACE_TEXT("Can not open to listen: %m\n")));
  return -1;
 }

 //Run event loop
 ACE_DEBUG((LM_NOTICE, ACE_TEXT("Sniffer started\n")));
 proactor.proactor_run_event_loop();
 return 0;
}

Reference:
[0] Raw Sockets in Windows 2000/XP" by ziss