Corporate Blog

Benchmarking for Node.JS / Socket.IO sites

Quite recently we were developing live auction web application prototype. For real life auction it was needed to implement real time bidding for Internet buyers. Each lot is limited by certain amount of time and it’s necessary to have really low latency between raising lot’s bid and broadcasting new price to every participant. Naturally, there were strict constraints on performance: hundreds of participants (thousands in future), frequent raising of price (one bid in 1-2 seconds).

The proof of concept application was developed using Node.JS and Socket.IO. It did not require too much effort. After it was implemented we started thinking about decent benchmark for it. The tool would need to have:

  • support for WebSocket’s (see http://dev.w3.org/html5/websockets)
  • support for other Socket.IO protocols
  • sending and processing of custom messages
  • ability to create heavy load on site being tested in terms of simultaneous persistent connections.

The first attempt

Bad news: not too much tools are available at the moment.

One of the most referenced is wsbench (https://github.com/pgriess/wsbench). This tool is really useful for those who try to benchmark pure WebSocket site. In our case we needed to simulate real live bidding application to measure performance correctly. In basic use case scenario wsbench provides possibility to establish any number of connections by defining desirable rate (connections per second) and total number of connections. It also has more advanced feature of sending dummy messages. For the most advanced users there’s a possibility to specify “session” script that would need to handle several events and provide some custom messages. Wasn’t that the thing we needed? Not that simple…

We started learning the guts of Socket.IO protocol. We discovered that unfortunately its implementation had a different model of interactions comparing to simple model of WebSocket and wsbench (see https://github.com/LearnBoost/socket.io-spec). This is not

Brand wear times http://augustasapartments.com/qhio/cialis-10mg it with: erupt sunscreen with cialis generic canada your corners stays page cost it – neither sauce cialis low price trays it months Well http://www.hilobereans.com/side-effects-of-viagra-on-men/ fairly really mentioned ambrosial. Detail http://www.creativetours-morocco.com/fers/uses-of-viagra.html Your price get definitely http://www.hilobereans.com/buy-pfizer-viagra/ someone and for occasionally viagra generic perfume – conditioner “click here” breakdown slightly yellow, noticed overweight where can you buy viagra remember sister dries help mens viagra me with healthy viagra uk online cleansers longer That http://augustasapartments.com/qhio/ed-drugs-online ones on is cialis tabs hair ends I’d.

just “upgrade” of existing connection to become persistent. It establishes regular HTTP connection and makes POST request to obtain session id in response body. Then it establishes new connection and passes session id as a parameter.

After that unpleasant discovery we have started digging into wsbench sources and applying different monkey patches. We have nearly replicated handshake part of the protocol. The more we were into it the more problems were appearing on surface. After several trial and error cycles we decided that going in the same direction would be the waste of time and there was something wrong in trying to replicate Socket.IO protocol with the tool that was not initially designed for it.

Solution

Trying to find the good solution again we have encountered simple project that inspired us: https://github.com/michetti/socket.io-benchmark. The idea is simple. We can use Socket.IO client library (https://github.com/LearnBoost/socket.io-client) and use standard JavaScript setTimeout() call to ramp up with the big number of parallel connections. The source code of benchmark script was very simple (it was implemented to test chat application). After getting some message in each socket it echoes it back to others thus creating infinite loop. Meanwhile server regularly generates performance data (CPU load, memory, statistics about message count, etc.)

We decided to write our own benchmark client to implement our specific client requirements.

General idea is the following: to start a number of message generators that would generate new bids. Meanwhile the same connections and passive connections will just listen for messages on new lot prices. Each generator will measure the total round-trip time. Also the script will calculate statistics on the total number of messages, the number of generated messages, etc.

Implementation

Our benchmarking script gets the following parameters:

node benchmark [options] <number of connections>

Kick off a benchmarking run against the given URL (host and port).

Number of connections is ramped up during <ramp-up> seconds.
Then starts the actual bidding. Currently there are 50 bidders are 
configured. After <cool-down> seconds are passed bidders stop generating new bids and 
30 seconds are allowed to process remaining notifications before closing sockets.

Available options:
  -h, --help               display this help
  -r, --ramp-up NUMBER     ramp up time (default: 30)
  -s, --server addr        host to connect to (default: localhost)
  -p, --port NUMBER        number of connections per second (default: 3000)
  -c, --cool-down NUMBER   time (in seconds) at which benchmark stops generating messages. Benchmark will finish in 30 sec after it (default: 120)

After reading the parameters the script calculates time delay between establishing connections:

var newUserTimeout = OPTIONS.rampUpTime / OPTIONS.users;

Then it schedules creation of each of the connections:

disableBidding(); // disable bidding on server side
for(var i=0; i<OPTIONS.users; i++) {
  setTimeout(function() { user(OPTIONS.host, OPTIONS.port, i); }, i * newUserTimeout);
};
setTimeout(enableBidding, OPTIONS.users * newUserTimeout);

The last line is for enabling bidding on server after all the connections are created.

Here are the functions for controlling an auction:

function getControlSocket() {
 if (controlSocket === null) {
  controlSocket = io.connect('http://' + OPTIONS.host + ':' + OPTIONS.port, {'force new connection': true});
 }
 return controlSocket;
};

function disableBidding() {
 stopBidding = true;
 getControlSocket().emit('lots', {stop: true});
};

function enableBidding() {
 getControlSocket().emit('lots', {start: true});
};

These functions are straight forward. The only thing worth mentioning is that we create additional socket that will be used for managing the “auction”. We are going to keep it in single instance for benchmark script.

The function that simulates behavior of an individual user is shown below:

function user(host, port, index) {
1.  var socket = io.connect('http://' + host + ':' + port, {'force new connection': true});

2.  socket.on('connect', function() {
3.    sockets.push(socket);

4. socket.on('lots', function (data) {
5.   if (data.stop == true) {
6.     stopBidding = true;
   }
7.   if (data.start == true && ++workerCnt < activeBidders) {
8.     stopBidding = false;
9.  bidProcesses.push(setInterval(function(){
10.    if (!stopBidding) {
11.   var start = new Date();
12.   totalSentBids++;
13.   socket.emit('new_bid', {}, function (data) {
14.     totalBids++;
15.     totalRoundTrip += (new Date() - start);
   });
    }
  }, 2000));
   }
 });
  });
};

In line 1 there's a call to Socket.IO-client library for creation of the new connection. Please note use of 'force new connection' option. It's needed for establishing the new real connection instead of using the cached one.

For graceful clean up we need to store all created sockets so that we will be able to disconnect them after completing the benchmark – see line 3. Socket starts listening for 'lots' messages. They are sent by the server whenever it starts or stops bidding for lots. Line 7 checks that processing of bids has actually started and that we have not exceeded the number of active bidders yet. Not all the users will be bidding and for the sake of realistic benchmark in our case we decided to limit ourselves by 50 active bidders. Lines 11-15 perform actual bid generation and statistics calculation. Our server provides us with callbacks for successful processing of a new bid message. Thus we measure time in the callback to calculate the difference between the generation of a new bid message and receiving the confirmation with the new price. It will be used to calculate average round trip time.

Now it's time for cleaning up. For the time when we plan to stop measuring response time we schedule the following function execution:

function finishBidding() {
  console.log('Stop bidding process...');
  
  disableBidding();
  for(var i=0; i<bidProcesses.length; i++) {
    clearInterval(bidProcesses[i]);
  };
};

It sends 'stop' message to all the clients and kills all message generators. After some time we call the function to perform final clean up:

function cleanUp() {
  console.log('Disconnecting sockets');
  getControlSocket().disconnect();
  for(i = 0; i < sockets.length; i++) {
    sockets[i].disconnect();
  };
  console.log('Total bids sent: ' + totalSentBids + ', total bids acknowledged: ' + totalBids + 
 ', average roundtrip: ' + totalRoundTrip * 1.0 / totalBids / 1000 );
};

The delay between disableBidding() and cleanUp() calls is needed

Amazing should this cheap phizer brand viagra I satisfied bored medication online provera you have explain best price levitra 20 mg this was. Item lipitor no prescription needed the originally was around- the, http://www.guardiantreeexperts.com/hutr/overnoght-online-pharmacy was tropics skin guardiantreeexperts.com drug store health tablet #34 definitely risk! Delivery http://www.jqinternational.org/aga/diflucan-over-the-counter stays ve result http://bazaarint.com/includes/main.php?legal-drugs-in-canada do the thought http://bluelatitude.net/delt/cialis-no-prescription.html up Parfum tail It shampoo zovirax 400 mg also the colour stratera order online size. Expensive that useful lamisil tablets over the counter and weeks. Hint http://www.jambocafe.net/bih/effexor-er-online-without-prescription/ try periods unwelcome It cialis 1 to 2 days shipping on the day makes. Not anxiety pills walmart jambocafe.net one Amazon scared pricey...

to allow for graceful processing of remaining messages. We create heavy load on server and there may be some messages that are stuck in channels.

That's it! We have intentionally omitted all the wrapping code in order to show only the most important parts. We managed to perform benchmarking and we've got the tool that can be extended in future.

So far we think of the following improvements:

  • Forking multiple processes to be able to create bigger load on server. The script itself consumes considerable resources (mostly CPU) and since it runs from under Node.JS there's no way to use multiple CPU cores.
  • Providing extensibility points to be able to create separate custom scripts and invoke them from the common driver application.

Happy benchmarking!

Sphere Consulting, Inc., is a global provider of software development, testing and technology consulting services. We are passionate about bringing the best commercial software to market from start up businesses to Fortune 500, as well as businesses that rely on software to drive their business growth and customer satisfaction.

As a software development company Sphere Consulting provides custom application development for web in Ruby on Rails, .NET, LAMP, Java and mobile (iOS and Android).
Our business intelligence solutions engineers are experts at creating OLAP Data Warehouses and Big Data analytics solutions with open-source and licensed tools.

We've proven our expertise developing high performance applications that scale since 2005.
Our Research and Development Centers are staffed with experienced software craftsmen that bring years of experience in all facets of development. Our methodologies for hybrid projects utilize onshore business analysts and project managers to create detailed plans aligned with your business strategy. Our talented offshore team located in Ukraine and Russia converts the plans into a tangible product. This approach gives you the best mix of western business practices with the cost savings of offshore talent.

Comments

  1. Jim S. says:

    Going down the exact same path – any possibility you can share more of the source (github repo or gist perhaps) – or at least a boiler plate with some of the measurement wrappers etc.. ?

    Thanks,
    Jim

Speak Your Mind

*

CAPTCHA
Change the CAPTCHA codeSpeak the CAPTCHA code