
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Vjeux &#187; optimization</title>
	<atom:link href="http://blog.vjeux.com/tag/optimization/feed" rel="self" type="application/rss+xml" />
	<link>http://blog.vjeux.com</link>
	<description>French Web Developer</description>
	<lastBuildDate>Wed, 21 Jul 2010 01:30:44 +0000</lastBuildDate>
	<generator>http://wordpress.org/?v=2.9</generator>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
			<item>
		<title>Javascript &#8211; Sorting Table</title>
		<link>http://blog.vjeux.com/2010/javascript/javascript-sorting-table.html</link>
		<comments>http://blog.vjeux.com/2010/javascript/javascript-sorting-table.html#comments</comments>
		<pubDate>Fri, 08 Jan 2010 12:16:58 +0000</pubDate>
		<dc:creator>vjeux</dc:creator>
				<category><![CDATA[Javascript]]></category>
		<category><![CDATA[optimization]]></category>
		<category><![CDATA[sort]]></category>

		<guid isPermaLink="false">http://blog.vjeux.com/?p=737</guid>
		<description><![CDATA[For my new project on World of Raids I have to implement a table sorting. The browser not stable sorting and the faster sorting trick add difficulty to the task.
String Comparison
As mentionned in the Speed Up Javascript Sort() article, using a string as a key to represent each element is faster than using a custom [...]]]></description>
			<content:encoded><![CDATA[<p>For my new project on World of Raids I have to implement a table sorting. The browser not stable sorting and the faster sorting trick add difficulty to the task.</p>
<h3>String Comparison</h3>
<p>As mentionned in the <a href="/2009/javascript/speed-up-javascript-sort.html">Speed Up Javascript Sort()</a> article, using a string as a key to represent each element is faster than using a custom sort function in most browsers.</p>
<p>Tables are commonly made of two data types: Numbers and Strings. Strings are the final representation so no work is involved there so our work will be focused on Numbers. What we want is to fit a number into a string. A string is a succession of characters, each one able to hold 256 values. So, we can see the problem as encoding the number into a base 256 which is trivial.</p>
<h4>Padding</h4>
<p>String comparison does not work exactly like we want it. It reads all the characters one by one and if they mismatch then it returns who is the highest. So for example <code>"10" < "5"</code> because </code><code>'1' < '5'</code>. We have to pad every numbers with </code><code>0</code> at the beginning and become <code>"10" > "05"</code>.</p>
<p>In this case we are adding one <code>0</code> because 10 is 2 digits and 5 is only one. We won't know the values of other elements when making a comparison, so two solutions apply: either we iterate over all the elements and retrieve the highest, or we set an arbitrary maximum.</p>
<table>
<thead>
<tr>
<th style="text-align: center;">Characters</th>
<th style="text-align: center;">Maximum</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">1</td>
<td style="text-align: right;">256</td>
</tr>
<tr>
<td style="text-align: center;">2</td>
<td style="text-align: right;">65 536</td>
</tr>
<tr>
<td style="text-align: center;">3</td>
<td style="text-align: right;">23 330 816</td>
</tr>
<tr>
<td style="text-align: center;">4</td>
<td style="text-align: right;">5 972 688 896</td>
</tr>
</tbody>
</table>
<p>With that table in mind, it is obvious that it would be a waste of time to inspect every element just to find the highest value. 1 or 2 characters will probably fit most uses but if you want to be safe, you will always be able to store your number into 4 characters which is acceptable.</p>
<p>Here is the code to do the conversion:</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;"><span style="color: #003366; font-weight: bold;">function</span> compareAsc<span style="color: #009900;">&#40;</span>num<span style="color: #339933;">,</span> digits<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
  <span style="color: #003366; font-weight: bold;">var</span> s <span style="color: #339933;">=</span> <span style="color: #3366CC;">&quot;&quot;</span><span style="color: #339933;">;</span>
  <span style="color: #006600; font-style: italic;">// Encode num in base 256</span>
  <span style="color: #000066; font-weight: bold;">while</span> <span style="color: #009900;">&#40;</span>num <span style="color: #339933;">!==</span> <span style="color: #CC0000;">0</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    s <span style="color: #339933;">=</span> String.<span style="color: #660066;">fromCharCode</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#40;</span>num <span style="color: #339933;">%</span> <span style="color: #CC0000;">256</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">|</span> <span style="color: #CC0000;">0</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">+</span> s<span style="color: #339933;">;</span>
    num <span style="color: #339933;">=</span> <span style="color: #009900;">&#40;</span>num <span style="color: #339933;">/</span> <span style="color: #CC0000;">256</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">|</span> <span style="color: #CC0000;">0</span><span style="color: #339933;">;</span>
    digits <span style="color: #339933;">-=</span> <span style="color: #CC0000;">1</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
  <span style="color: #006600; font-style: italic;">// Fill with 0</span>
  <span style="color: #000066; font-weight: bold;">while</span> <span style="color: #009900;">&#40;</span>digits <span style="color: #339933;">&gt;</span> <span style="color: #CC0000;">0</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    s <span style="color: #339933;">=</span> String.<span style="color: #660066;">fromCharCode</span><span style="color: #009900;">&#40;</span><span style="color: #CC0000;">0</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">+</span> s<span style="color: #339933;">;</span>
    digits <span style="color: #339933;">-=</span> <span style="color: #CC0000;">1</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
  <span style="color: #000066; font-weight: bold;">return</span> s<span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span></pre></div></div>

<p>For the sake of simplicity, this code is using string concatenation which requires to build up new strings and make copy several times. Since this is for small strings (up to 4 characters!) this is probably alright but one looking for performances could bench this with the use of an <code>Array</code> with a <code>.concat()</code> at the end to build the string.</p>
<p>Also note that numbers are stored as float in Javascript, so we are rounding (flooring to be exact) all the values using the <code>|0</code> trick.</p>
<h4>Descending Order</h4>
<p>We have made the code for an ascending sort but we have to handle the other case. We cannot just revert the order of the sorting or adding a minus sign before the number. What we have to do is to do a 256-complement of the final result, in other terms: digit = 255 - digit. Let see an example for a base 10.</p>
<pre>00 -> 99
01 -> 98
...
54 -> 45
55 -> 44
56 -> 43
...
98 -> 01
99 -> 00
</pre>
<p>You can see that all the numbers are now sorted in the opposite order.</p>
<p>Here is the associated code :</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;"><span style="color: #003366; font-weight: bold;">function</span> compareDesc<span style="color: #009900;">&#40;</span>num<span style="color: #339933;">,</span> digits<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
  <span style="color: #003366; font-weight: bold;">var</span> s <span style="color: #339933;">=</span> <span style="color: #3366CC;">&quot;&quot;</span><span style="color: #339933;">;</span>
  <span style="color: #006600; font-style: italic;">// Encode num in base 256 and do the 256-complement</span>
  <span style="color: #000066; font-weight: bold;">while</span> <span style="color: #009900;">&#40;</span>num <span style="color: #339933;">!==</span> <span style="color: #CC0000;">0</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    s <span style="color: #339933;">=</span> String.<span style="color: #660066;">fromCharCode</span><span style="color: #009900;">&#40;</span><span style="color: #CC0000;">255</span> <span style="color: #339933;">-</span> <span style="color: #009900;">&#40;</span>num <span style="color: #339933;">%</span> <span style="color: #CC0000;">256</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">|</span> <span style="color: #CC0000;">0</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">+</span> s<span style="color: #339933;">;</span>
    num <span style="color: #339933;">=</span> <span style="color: #009900;">&#40;</span>num <span style="color: #339933;">/</span> <span style="color: #CC0000;">256</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">|</span> <span style="color: #CC0000;">0</span><span style="color: #339933;">;</span>
    digits <span style="color: #339933;">-=</span> <span style="color: #CC0000;">1</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
  <span style="color: #000066; font-weight: bold;">while</span> <span style="color: #009900;">&#40;</span>digits <span style="color: #339933;">&gt;=</span> <span style="color: #CC0000;">0</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    s <span style="color: #339933;">=</span> String.<span style="color: #660066;">fromCharCode</span><span style="color: #009900;">&#40;</span><span style="color: #CC0000;">255</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">+</span> s<span style="color: #339933;">;</span>
    digits <span style="color: #339933;">-=</span> <span style="color: #CC0000;">1</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
  <span style="color: #000066; font-weight: bold;">return</span> s<span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span></pre></div></div>

<h4>Floats and Negatives</h4>
<p>We have handled all the positive integers but they are not alone. In order to deal with the negative numbers you can apply the <a href="http://en.wikipedia.org/wiki/Two%27s_complement">2-complement</a> to the number. This is how it is done in computer arithmetic. </p>
<p>To handle floats, it is possible to treat them as integers by multiplying by 10^(number of decimal displayed). So <code>12.34</code> is translated to <code>1234</code> and <code>11</code> to <code>1100</code> and it's working just fine. You will probably need more than 4 characters if you are using huge numbers though.</p>
<h3>Stable Sorting</h3>
<p>Implementing a table sorting requires the sorting algorithm to be stable (values that have the same key keep their original order). This property allows people to sort by multiple columns, the last one having the highest weight.</p>
<p>However, browser implementations of the <code>Array.sort()</code> are not stable. From there we have two solutions, implementing a stable sort in javascript or find a way to emulate that behaviour. Since Javascript is slow in many browsers, implementing such a computation heavy algorithm in Javascript is going to slow down things and requires more efforts than I am willing to spend on this project.</p>
<p>The solution instead is to understand what the stable property means and find a way to code it.</p>
<blockquote><p>Stable sorting algorithms maintain the relative order of records with equal keys. [...] Whenever there are two records (let's say R and S) with the same key, and R appears before S in the original list, then R will always appear before S in the sorted list.</p>
<div style="text-align: right; margin-top: 5px;"><a href="http://en.wikipedia.org/wiki/Sorting_algorithm">Sorting Algorithm - Wikipedia</a></div>
</blockquote>
<p>We can easily transpose that sentence into the sorting function:</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;"><span style="color: #000066; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>R.<span style="color: #660066;">key</span> <span style="color: #339933;">==</span> S.<span style="color: #660066;">key</span><span style="color: #009900;">&#41;</span>
  <span style="color: #000066; font-weight: bold;">return</span> compare<span style="color: #009900;">&#40;</span>R.<span style="color: #660066;">position</span><span style="color: #339933;">,</span> S.<span style="color: #660066;">position</span><span style="color: #009900;">&#41;</span></pre></div></div>

<p>We need to know one more thing to do the computation: the position of each element in the list before sorting. This is straightforward to do, you have to iterate over all the elements and update their position field. This requires <i>n</i> steps which is acceptable.</p>
<p>Here is the full Javascript code</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;"><span style="color: #003366; font-weight: bold;">var</span> sort <span style="color: #339933;">=</span> <span style="color: #003366; font-weight: bold;">function</span> <span style="color: #009900;">&#40;</span>a<span style="color: #339933;">,</span> b<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
  <span style="color: #000066; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>a.<span style="color: #660066;">key</span> <span style="color: #339933;">===</span> b.<span style="color: #660066;">key</span><span style="color: #009900;">&#41;</span>
    <span style="color: #000066; font-weight: bold;">return</span> a.<span style="color: #660066;">position</span> <span style="color: #339933;">-</span> b.<span style="color: #660066;">position</span><span style="color: #339933;">;</span>
  <span style="color: #000066; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span>a.<span style="color: #660066;">key</span> <span style="color: #339933;">&lt;</span> b.<span style="color: #660066;">key</span><span style="color: #009900;">&#41;</span>
    <span style="color: #000066; font-weight: bold;">return</span> <span style="color: #339933;">-</span><span style="color: #CC0000;">1</span><span style="color: #339933;">;</span>
  <span style="color: #000066; font-weight: bold;">return</span> <span style="color: #CC0000;">1</span><span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span><span style="color: #339933;">;</span></pre></div></div>

<h4>String comparison</h4>
<p>To apply this concept to the string comparison, you can append the position at the end of the string. <code>"Key"</code> now being <code>"Key1"</code>, <code>"Key2"</code> and so on. When the keys are equal, the position is used for the comparison instead. We can reuse the <code>compareAsc()</code> function to encode the position using the minimal size.</p>
<p>This works well for fixed-size strings like the representation of numbers. However common strings do not share that property and a problem arise when two strings share a common base. </p>
<p>Take "Method" and "Methodology". It is obvious that <code>"Method" &lt; "Methodology"</code>. Now append the position: <code>"Method" + chr(250) &gt; "Method<strong>ology</strong>" + chr(251)</code> the comparison changed because <code>chr(250) > 'o'</code>. (Where <code>chr = String.fromCharCode'</code>).</p>
<p>The way to fix this problem is to add a <code>chr(0)</code> before the position. Considering that a normal string never contains a <code>chr(0)</code>, the <code>chr(0)</code> will always be lesser than any character of the string and therefore stop the comparison. The longer string always wins!</pre>
]]></content:encoded>
			<wfw:commentRss>http://blog.vjeux.com/2010/javascript/javascript-sorting-table.html/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Bistromathique &#8211; Optimized Arbitrary Precision Calculator</title>
		<link>http://blog.vjeux.com/2010/epita/bistromathique-optimized-arbitrary-precision-calculator.html</link>
		<comments>http://blog.vjeux.com/2010/epita/bistromathique-optimized-arbitrary-precision-calculator.html#comments</comments>
		<pubDate>Sun, 03 Jan 2010 17:48:29 +0000</pubDate>
		<dc:creator>vjeux</dc:creator>
				<category><![CDATA[EPITA]]></category>
		<category><![CDATA[optimization]]></category>

		<guid isPermaLink="false">http://blog.vjeux.com/?p=714</guid>
		<description><![CDATA[The Bistromathique is an Arbitrary Precision Calculator EPITA project where the main focus is optimization. The input can be in any base (up to 250) and the following operations have to be performed: Addition, Subtraction, Multiplication, Division, Modulo.
Base Representation
Going back and forth a textual representation to a computable representation is the first part of the [...]]]></description>
			<content:encoded><![CDATA[<p>The Bistromathique is an Arbitrary Precision Calculator EPITA project where the main focus is optimization. The input can be in any base (up to 250) and the following operations have to be performed: Addition, Subtraction, Multiplication, Division, Modulo.</p>
<h3>Base Representation</h3>
<p>Going back and forth a textual representation to a computable representation is the first part of the project and several optimizations are possible at this step.</p>
<h4>Binary Representation</h4>
<p>Encoding the numbers as binary allows you to make the best use of the processor calculation unit. There is no space lost and the number being represented as an uninterrupted series of 0 and 1 can be added and subtracted directly with the hardware.</p>
<p>However, in order to convert the number for a base B representation to a base 2 requires roughly <i>n</i> divisions by B. And to reconvert it back requires another <i>n</i> multiplications. This can be affordable for applications that uses the same numbers again and again but in our case, this cost is prohibitive.</p>
<p>The other major drawback is that you are required to have the multiplications and divisions (and therefore addition and subtraction) in order to have the numbers in the right form to start working on the operations ... In summary, you have to code all these operations before being able to test them!</p>
<h4><code>char</code> Representation</h4>
<p>The binary representation not being possible, the solution that comes in mind is to put each digit into a <code>char</code> (8 bits). Since we are limited up to a base of 250, any base will fit. This comes at a price, there is a lot of space wasted, and it increases as the base size decreases.</p>
<p>We are in a 1 to 1 relation between the input and the represented number, this way we can store the represented number inside the input. This prevents from allocating memory at this step. The conversion can be efficiently done, each character goes through vector of 256 slots storing its representation and a magic value used as a non valid input. One final trick is to add a non number at the end of the input to remove a end of input check in the loop.</p>
<h4>Filling more</h4>
<p>A <code>char</code> can store values up to 256. If we are working on base 16 we could store two digits into 1 <code>char</code> and thus divide by 2 the number of digits. Additions are therefore 2 times faster and even more for <i>n log n</i> operations such as multiplication.</p>
<p>Calculating the number of digits to fill into a char is quite straightforward however some caution has to be taken during the transition. We have to know in advance the length of the input to decide how to start filling the digits. If you can fit 2 digits per char, if the number is even then it goes as expected, but if the number is odd you have to place 0 at the beginning to fill the gap.</p>
<p><strong>Even</strong>: <code>[1] [2] [3] [4] -> [12] [34]</code><br />
<strong>Odd (Naive)</strong>: <code>[2] [3] [4] -> [23] [4<strong>?</strong>]</code><br />
<strong>Odd (Smart)</strong>: <code>[2] [3] [4] -> [<strong>0</strong>2] [34]</code></p>
<p>This method works well but requires 2 passes for each number and a special case at the beginning of each number. Not enough to stop us from using it.</p>
<h4>Using a larger representation</h4>
<p>Now that we filled the char at its maximum, why stick with it? This should ring a bell since you know that common architecture is 32-bit, and from now we are talking about a 8-bit representation. We would be tempted to change the <code>char</code> to an <code>int</code> (32-bit) or even a <code>long</code> (64-bit) that could benefit from register optimizations.</p>
<p>Moving away from a <code>char</code>, we lose the assumption that the output being smaller (or equal) than the input. Without it, this is no longer possible to work in place. We now have to compare the cost of a reduction in the number of operations and the cost of the memory allocation.</p>
<p>The overhead happens for values that are represented in less than 4 characters for an int representation. This is a maximum 3-characters overhead possible for each number, which is negligible. We could eat the operation character but we are left with 2 characters. I haven't found a solution to stay in place yet and this minor problem prevents from accessing a great speed up.</p>
<p><strong>To be continued ...</strong></p>
]]></content:encoded>
			<wfw:commentRss>http://blog.vjeux.com/2010/epita/bistromathique-optimized-arbitrary-precision-calculator.html/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Guild Recruitement &#8211; Search Optimizations</title>
		<link>http://blog.vjeux.com/2009/javascript/guild-recruitement-search-optimizations.html</link>
		<comments>http://blog.vjeux.com/2009/javascript/guild-recruitement-search-optimizations.html#comments</comments>
		<pubDate>Thu, 24 Dec 2009 00:03:50 +0000</pubDate>
		<dc:creator>vjeux</dc:creator>
				<category><![CDATA[Javascript]]></category>
		<category><![CDATA[optimization]]></category>
		<category><![CDATA[recruitment]]></category>

		<guid isPermaLink="false">http://blog.vjeux.com/?p=661</guid>
		<description><![CDATA[In World of Raids Guild Recruitment Tool, we wanted the search to be instant! We achieved this goal by not having a single page refresh or ajax call while updating the settings. Since it runs in the browser, performance was a real concern. We will see in this article what were the tricks used to [...]]]></description>
			<content:encoded><![CDATA[<p>In <a href="http://www.worldofraids.com/recruitment/">World of Raids Guild Recruitment Tool</a>, we wanted the search to be instant! We achieved this goal by not having a single page refresh or ajax call while updating the settings. Since it runs in the browser, performance was a real concern. We will see in this article what were the tricks used to make this real.</p>
<h3>Sorting</h3>
<p>In order to sort the guilds, each guild is given a matching score (between 0 and 65535) based on the current filters. At start, each guild has the maximum points. Then, each filter is giving a malus. For example when a specialization doesn't match, the guild loses 10%.</p>
<p>This means that everytime you change a filter, you have to recompute that matching score for all the guilds. Instead of recomputing the whole matching score, the guild holds the result of each filter. Everytime a filter is updated, the value of this filter is upated on every guild. The global matching score is now recalculated by summing all the partial precomputed scores.</p>
<p>Every bit of code in the matching function has been optimized, no more jquery code, loops are unrolled and great use of lazy boolean expression evaluation. The sort function itself also got boosted thanks to a trick I commented on my previous article <a href="http://blog.vjeux.com/2009/javascript/speed-up-javascript-sort.html">Speed Up Javascript Sort()</a>.</p>
<div style="text-align: center; margin: 20px 0;"><a href="http://www.worldofraids.com/recruitment/"><img src="http://blog.vjeux.com/wp-content/uploads/2009/12/wor-recruit-search-filters.png" alt="Guild Recruitment Search Filters" title="wor-recruit-search-filters" width="550" class="size-full wp-image-691" /></a></div>
<h3>Display</h3>
<p>Once we have the data sorted, it has to be displayed! And this is another big time eater. When it takes 50ms to sort the data, it takes about 150ms to display 10 guilds!</p>
<p>Since we wanted performances, DOM elements are created at page loading, they are then only being updated as the guilds change. And still, only the required changes are made, for example tooltips are created as the user trigger the mouseover event.</p>
<p>The main problem with this is that we are not able to make bulk changes. Everytime you make the slightest change, the whole UI has to be redrawn which is a waste of time. I hope there will be such API in the future.</p>
<div style="text-align: center; margin: 20px 0;"><a href="http://www.worldofraids.com/recruitment/"><img src="http://blog.vjeux.com/wp-content/uploads/2009/12/wor-recruit-guild.png" alt="Guild Display" title="wor-recruit-guild" width="550" class="size-full wp-image-687" /></a></div>
<p>Since it's a web application, we had to maintain the url up to date, this is done using <a href="http://blog.vjeux.com/2009/javascript/smallhash-information-compression.html">SmallHash</a>, a small library of mine that compresses filter data into the smallest possible hash.</p>
<h3>Scaling</h3>
<p>Having all the data being sent at page loading is a pain, it makes it load slowly. One way to avoid this problem would be to use the localstorage to keep these data accross sessions and only send the updated guilds. However at the time I started developping the application there was no released browser supporting HTML5.</p>
<p>However, the application is focused on World of Warcraft guilds. This is a limited audience and there is no need on heavy scaling there. So we decided that it was a good option to go.</p>
<p>Up to 3000 guilds, this is pretty much instant in every browser in my MacBook Pro. When increasing that number to 10 000 it is taking more than a second to update on IE8. It requires 50 000 guilds for Chrome 3 to get slow, which is a pretty good score. It means that it is possible to do heavy processing, the bandwith now being the limiting factor.</p>
]]></content:encoded>
			<wfw:commentRss>http://blog.vjeux.com/2009/javascript/guild-recruitement-search-optimizations.html/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Javascript &#8211; Dynamic Query Throttling</title>
		<link>http://blog.vjeux.com/2009/javascript/javascript-dynamic-query-throttling.html</link>
		<comments>http://blog.vjeux.com/2009/javascript/javascript-dynamic-query-throttling.html#comments</comments>
		<pubDate>Fri, 30 Oct 2009 13:52:46 +0000</pubDate>
		<dc:creator>vjeux</dc:creator>
				<category><![CDATA[Javascript]]></category>
		<category><![CDATA[optimization]]></category>

		<guid isPermaLink="false">http://blog.vjeux.com/?p=544</guid>
		<description><![CDATA[Working on the World of Raids Recruitment Tool we wanted automatic saving while editing. There are basically two ways of handling the problem.

Send update everytime something changes. The main advantage of this technique is that you are always up to date. However, it is spamming the server on fields that are constantly being updated (text [...]]]></description>
			<content:encoded><![CDATA[<p>Working on the <a href="http://www.worldofraids.com/recruitment/guild.html#VJX">World of Raids Recruitment Tool</a> we wanted automatic saving while editing. There are basically two ways of handling the problem.</p>
<ul>
<li><strong>Send update everytime something changes.</strong> The main advantage of this technique is that you are always up to date. However, it is spamming the server on fields that are constantly being updated (text fields for example).</li>
<li><strong>Send update every X seconds.</strong> There is no longer query spike but there are two main disadvantages.
<p>The delay the data are sent after a modification can vary from nothing to the timer length for no apparent reason. This is a really bad feeling for the user. He will think that the application is buggy.</p>
<p>The application is going to send queries all the time, even if there are no changes. We can safely assume that people are interacting a lot less than viewing so this is a real waste of resources.
</li>
</ul>
<p>We are interested in the first one, we want instant update but removing the ability to send too much data at once. What we want is to send data right after the user is done typing instead of everytime a character is typed. There is an easy way to do it in Javascript. </p>
<p>When you type the first character, it will start a timer of let say 100ms. Then everytime you type another character it resets the timer to 100ms. When it triggers, it will send data to the server. </p>
<p>This way, data will only be sent when the user interacts and with no risk of spamming the server.</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;"><span style="color: #003366; font-weight: bold;">var</span> Object <span style="color: #339933;">=</span> <span style="color: #009900;">&#123;</span>
  <span style="color: #009966; font-style: italic;">/* Object Code ... */</span>
&nbsp;
  timer<span style="color: #339933;">:</span> <span style="color: #003366; font-weight: bold;">null</span><span style="color: #339933;">,</span>
  sendChanges<span style="color: #339933;">:</span> <span style="color: #003366; font-weight: bold;">function</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #000066; font-weight: bold;">if</span> <span style="color: #009900;">&#40;</span><span style="color: #000066; font-weight: bold;">this</span>.<span style="color: #660066;">timer</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      clearTimeout<span style="color: #009900;">&#40;</span>timer<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      <span style="color: #000066; font-weight: bold;">this</span>.<span style="color: #660066;">timer</span> <span style="color: #339933;">=</span> <span style="color: #003366; font-weight: bold;">null</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #000066; font-weight: bold;">this</span>.<span style="color: #660066;">timer</span> <span style="color: #339933;">=</span> setTimeout<span style="color: #009900;">&#40;</span>
      <span style="color: #003366; font-weight: bold;">function</span> <span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span> <span style="color: #009966; font-style: italic;">/* Send Changes */</span> <span style="color: #009900;">&#125;</span><span style="color: #339933;">,</span>
      <span style="color: #CC0000;">200</span> <span style="color: #009966; font-style: italic;">/* Timeout in ms */</span>
    <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#125;</span><span style="color: #339933;">;</span></pre></div></div>

]]></content:encoded>
			<wfw:commentRss>http://blog.vjeux.com/2009/javascript/javascript-dynamic-query-throttling.html/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Speed Up Javascript Sort()</title>
		<link>http://blog.vjeux.com/2009/javascript/speed-up-javascript-sort.html</link>
		<comments>http://blog.vjeux.com/2009/javascript/speed-up-javascript-sort.html#comments</comments>
		<pubDate>Mon, 10 Aug 2009 17:39:39 +0000</pubDate>
		<dc:creator>vjeux</dc:creator>
				<category><![CDATA[Javascript]]></category>
		<category><![CDATA[optimization]]></category>
		<category><![CDATA[sort]]></category>

		<guid isPermaLink="false">http://blog.vjeux.com/?p=46</guid>
		<description><![CDATA[By overriding the toString Object prototype, it is possible to speed up by 5x the sort function. This is an easy to implement trick that gives astonishing results
I wanted to know if there were ways to speed up the Javascript Sort function. I came across an interesting article (Yet another faster Javascript Sorting) that presents [...]]]></description>
			<content:encoded><![CDATA[<p><i>By overriding the toString Object prototype, it is possible to speed up by 5x the sort function. This is an easy to implement trick that gives astonishing results</i></p>
<p>I wanted to know if there were ways to speed up the Javascript Sort function. I came across an interesting article (<a href="http://blog.piterpen.net/2007/01/08/yet-another-faster-javascript-sorting/">Yet another faster Javascript Sorting</a>) that presents a way to boost the builtin sort function. However, the link with the detailed explanation is dead, so i make you a summary here.</p>
<p>To sort some data, you are likely to do something that looks like that:</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;">data.<span style="color: #660066;">sort</span><span style="color: #009900;">&#40;</span><span style="color: #003366; font-weight: bold;">function</span> <span style="color: #009900;">&#40;</span>a<span style="color: #339933;">,</span> b<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span> <span style="color: #000066; font-weight: bold;">return</span> b.<span style="color: #660066;">key</span> <span style="color: #339933;">-</span> a.<span style="color: #660066;">key</span><span style="color: #339933;">;</span> <span style="color: #009900;">&#125;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span></pre></div></div>

<p>The comparison function is being called <i>n log n</i> times. Since it's a javascript function, it is slow. sort() with no parameters will first convert all elements into strings and then use native (therefore faster) string comparison.</p>
<p>To make this work, we just have to override the toString method of the Object prototype to return the key.</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;"><span style="color: #003366; font-weight: bold;">var</span> save <span style="color: #339933;">=</span> Object.<span style="color: #660066;">prototype</span>.<span style="color: #660066;">toString</span><span style="color: #339933;">;</span>
Object.<span style="color: #660066;">prototype</span>.<span style="color: #660066;">toString</span> <span style="color: #339933;">=</span> <span style="color: #003366; font-weight: bold;">function</span> <span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span> <span style="color: #000066; font-weight: bold;">return</span> <span style="color: #000066; font-weight: bold;">this</span>.<span style="color: #660066;">key</span><span style="color: #339933;">;</span> <span style="color: #009900;">&#125;</span><span style="color: #339933;">;</span>
&nbsp;
data.<span style="color: #660066;">sort</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
Object.<span style="color: #660066;">prototype</span>.<span style="color: #660066;">toString</span> <span style="color: #339933;">=</span> save<span style="color: #339933;">;</span></pre></div></div>

<p>You have to make sure that the key variable is a string. In my application, the key range is [0, 100] so the it is written as String.fromCharCode(key). If you have to deal with larger key range, the best solution is to convert the number into base 256. Make sure the number is padded with 0 because of the string comparison.</p>
<p>I made a little <a href="http://blog.vjeux.com/wp-content/uploads/2009/08/javascript.sort.html">benchmark</a> of the implementation to see how well it performs</p>
<style>
.sort_results { border-collapse: collapse; margin-left: 2px; margin-bottom: 0; }
.sort_results td { text-align: center; width: 100px; border: 1px solid #ccc; padding: 0 5px; }
.sort_results td.right { text-align: right; width: 200px; }
</style>
<table class="sort_results">
<thead>
<tr>
<td><a href="http://blog.vjeux.com/wp-content/uploads/2009/08/javascript.sort.html">toString Sort Benchmark</a></td>
<td><b>Firefox<br />3.5.2</b></td>
<td><b>IE<br />8</b></td>
<td><b>Safari<br />4.528</b></td>
<td><b>Chrome<br />3.0.197</b></td>
</tr>
</thead>
<tbody>
<tr>
<td class="right">Normal - 10 000</td>
<td>135ms</td>
<td>188ms</td>
<td>45ms</td>
<td>16ms</td>
</tr>
<tr>
<td class="right">Fast - 10 000</td>
<td>10ms</td>
<td>31ms</td>
<td>14ms</td>
<td>68ms</td>
</tr>
<tr>
<td class="right">Improvement - 10 000</td>
<td><strong>x13.5</strong></td>
<td><strong>x6.1</strong></td>
<td><strong>x3.2</strong></td>
<td><strong>/4.3</strong></td>
</tr>
<tr>
<td class="right">Normal - 100 000</td>
<td>695ms</td>
<td>2125ms</td>
<td>200ms</td>
<td>128ms</td>
</tr>
<tr>
<td class="right">Fast - 100 000</td>
<td>101ms</td>
<td>437ms</td>
<td>46ms</td>
<td>326ms</td>
</tr>
<tr>
<td class="right">Improvement - 100 000</td>
<td><strong>x6.9</strong></td>
<td><strong>x4.9</strong></td>
<td><strong>x4.3</strong></td>
<td><strong>/2.5</strong></td>
</tr>
<tr>
<td class="right">Normal - 1 000 000</td>
<td>10102ms</td>
<td>*</td>
<td>2736ms</td>
<td>970ms</td>
</tr>
<tr>
<td class="right">Fast - 1 000 000</td>
<td>1158ms</td>
<td>6828ms</td>
<td>482ms</td>
<td>2593ms</td>
</tr>
<tr>
<td class="right">Improvement - 1 000 000</td>
<td><strong>x8.7</strong></td>
<td><strong></strong></td>
<td><strong>x5.7</strong></td>
<td><strong>/2.7</strong></td>
</tr>
</tbody>
</table>
<p><small style="float: right; margin-top: -10px;">*: Script time limit has been exceeded</small></p>
<p>It gives about a 5x increase of all the browsers I have tested with except in Chrome with a 3x decrease.</p>
<p>Since Chrome is already times faster than all the browsers, it doesn't look slowed by this feature. However it gives a real boost to all other browsers.</p>
<p><strong>Update (24 December 2009)</strong>: <a href="http://www.google.com/codesearch/p?hl=fr#W9JxUuHYyMg/trunk/src/array.js&#038;q=ArraySort%20package:http://v8%5C.googlecode%5C.com&#038;sa=N&#038;cd=1&#038;ct=rc&#038;l=616">Chrome Array.sort()</a> function is written directly in javascript and calls the ToString function everytime when a comparison is needed. Therefore, it is making 2 function calls (ToString(x), ToString(y) instead of one (compare(x, y)).</p>
<p>In order to check if that optimization will indeed give an actual boost, we can count the number of time the ToString method is being executed for 3 values. 3 times means that it is executed <i>n</i> time and more means that it is executed <i>n log n</i> times.</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;"><span style="color: #003366; font-weight: bold;">var</span> need_custom_sort <span style="color: #339933;">=</span> <span style="color: #009900;">&#40;</span><span style="color: #003366; font-weight: bold;">function</span> <span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
  <span style="color: #006600; font-style: italic;">// Fill the array with 3 values</span>
  <span style="color: #003366; font-weight: bold;">var</span> array <span style="color: #339933;">=</span> <span style="color: #003366; font-weight: bold;">new</span> Array<span style="color: #009900;">&#40;</span><span style="color: #CC0000;">3</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #000066; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span><span style="color: #003366; font-weight: bold;">var</span> i <span style="color: #339933;">=</span> <span style="color: #CC0000;">0</span><span style="color: #339933;">;</span> i <span style="color: #339933;">&lt;</span> <span style="color: #CC0000;">3</span><span style="color: #339933;">;</span> <span style="color: #339933;">++</span>i<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    array<span style="color: #009900;">&#91;</span>i<span style="color: #009900;">&#93;</span> <span style="color: #339933;">=</span> <span style="color: #003366; font-weight: bold;">new</span> Object<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
&nbsp;
  <span style="color: #006600; font-style: italic;">// Override the toString method that counts how many times it is being called</span>
  <span style="color: #003366; font-weight: bold;">var</span> count <span style="color: #339933;">=</span> <span style="color: #CC0000;">0</span><span style="color: #339933;">;</span>
  <span style="color: #003366; font-weight: bold;">var</span> save <span style="color: #339933;">=</span> Object.<span style="color: #660066;">prototype</span>.<span style="color: #660066;">toString</span><span style="color: #339933;">;</span>
  Object.<span style="color: #660066;">prototype</span>.<span style="color: #660066;">toString</span> <span style="color: #339933;">=</span> <span style="color: #003366; font-weight: bold;">function</span> <span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span> count <span style="color: #339933;">+=</span> <span style="color: #CC0000;">1</span><span style="color: #339933;">;</span> <span style="color: #000066; font-weight: bold;">return</span> <span style="color: #3366CC;">&quot;&quot;</span><span style="color: #339933;">;</span> <span style="color: #009900;">&#125;</span><span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #006600; font-style: italic;">// Sort</span>
  array.<span style="color: #660066;">sort</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  Object.<span style="color: #660066;">prototype</span>.<span style="color: #660066;">toString</span> <span style="color: #339933;">=</span> save<span style="color: #339933;">;</span>
&nbsp;
  <span style="color: #006600; font-style: italic;">// 3 times is good, more is bad!</span>
  <span style="color: #000066; font-weight: bold;">return</span> <span style="color: #009900;">&#40;</span>count <span style="color: #339933;">===</span> <span style="color: #CC0000;">3</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span></pre></div></div>

</pre>
]]></content:encoded>
			<wfw:commentRss>http://blog.vjeux.com/2009/javascript/speed-up-javascript-sort.html/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>SmallHash &#8211; Information Compression</title>
		<link>http://blog.vjeux.com/2009/javascript/smallhash-information-compression.html</link>
		<comments>http://blog.vjeux.com/2009/javascript/smallhash-information-compression.html#comments</comments>
		<pubDate>Fri, 07 Aug 2009 13:08:22 +0000</pubDate>
		<dc:creator>vjeux</dc:creator>
				<category><![CDATA[Javascript]]></category>
		<category><![CDATA[optimization]]></category>

		<guid isPermaLink="false">http://blog.vjeux.com/?p=3</guid>
		<description><![CDATA[SmallHash encodes any range of integers into the smallest possible string. This way, you can use the hash part of your url with efficiency.]]></description>
			<content:encoded><![CDATA[<p><i>SmallHash encodes any range of integers into the smallest possible string. This way, you can use the hash part of your url with efficiency.</i></p>
<p>My problem is having these options stored in the minimum characters as possible.</p>
<ul>
<li><strong>Faction</strong>: Alliance, Horde</li>
<li><strong>Region</strong>: US, Europe</li>
<li><strong>Type</strong>: PvE, PvP, RP</li>
<li><strong>Lang</strong>: EN, FR, ES, DE, RU</li>
</ul>
<p>The two faction and region could be stored in base 2 with no problem. However, if we wanted to store the others in base 2, there would have been space left. So i started digging up into the base conversion.</p>
<p>Here is the code to do a base2 to base10 conversion.</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;">base10 <span style="color: #339933;">=</span> <span style="color: #CC0000;">0</span>
foreach <span style="color: #009900;">&#40;</span>bit <span style="color: #000066; font-weight: bold;">in</span> base2<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
  base10 <span style="color: #339933;">*=</span> <span style="color: #CC0000;">2</span>
  base10 <span style="color: #339933;">+=</span> bit
<span style="color: #009900;">&#125;</span></pre></div></div>

<p>As you can see, we multiply the final number by 2, which is the number of possibilities. So, instead of multiplying by 2, we multiply by the number of possible options and it works! The decoding process is using the same technique by changing the divisor.</p>
<p>To get back to our example. [Alliance, US, PvP, DE] can be expressed as [0,0,1,3] over [2,2,3,5]. It will be encoded and decoded easily with the SmallHash library:</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;"><span style="color: #003366; font-weight: bold;">var</span> input <span style="color: #339933;">=</span> <span style="color: #009900;">&#91;</span><span style="color: #CC0000;">0</span><span style="color: #339933;">,</span><span style="color: #CC0000;">0</span><span style="color: #339933;">,</span><span style="color: #CC0000;">1</span><span style="color: #339933;">,</span><span style="color: #CC0000;">3</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
<span style="color: #003366; font-weight: bold;">var</span> encoded <span style="color: #339933;">=</span> SmallHash.<span style="color: #660066;">encode</span><span style="color: #009900;">&#40;</span>input<span style="color: #339933;">,</span> <span style="color: #009900;">&#91;</span><span style="color: #CC0000;">2</span><span style="color: #339933;">,</span><span style="color: #CC0000;">2</span><span style="color: #339933;">,</span><span style="color: #CC0000;">3</span><span style="color: #339933;">,</span><span style="color: #CC0000;">5</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">'abcdefghijklmnopqrstuvwxyz'</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #003366; font-weight: bold;">var</span> decoded <span style="color: #339933;">=</span> SmallHash.<span style="color: #660066;">decode</span><span style="color: #009900;">&#40;</span>encoded<span style="color: #339933;">,</span> <span style="color: #009900;">&#91;</span><span style="color: #CC0000;">2</span><span style="color: #339933;">,</span><span style="color: #CC0000;">2</span><span style="color: #339933;">,</span><span style="color: #CC0000;">3</span><span style="color: #339933;">,</span><span style="color: #CC0000;">5</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">'abcdefghijklmnopqrstuvwxyz'</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
console.<span style="color: #660066;">log</span><span style="color: #009900;">&#40;</span>input<span style="color: #339933;">,</span> encoded<span style="color: #339933;">,</span> decoded<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #006600; font-style: italic;">// Result: [0, 0, 1, 3], &quot;bo&quot;, [0, 0, 1, 3]</span></pre></div></div>

<p>As you can see, it fits into 2 characters instead of 4 with the easy way. The gain increases with the number of data you have to encode. This can also be improved by enlarging the base characters (uppercase letter, digits and special characters).</p>
<p>The algorithm is fairly easy, it is the same one explain before but using the range instead of 2 (when converting in base 2). This is the pseudo-code version.</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;">&nbsp;
SmallHash <span style="color: #339933;">=</span> <span style="color: #009900;">&#123;</span>
  <span style="color: #006600; font-style: italic;">// encode( [2, 4], [10, 15], '0123456789' ) : '42'</span>
  encode<span style="color: #339933;">:</span> <span style="color: #003366; font-weight: bold;">function</span> <span style="color: #009900;">&#40;</span>input<span style="color: #339933;">,</span> ranges<span style="color: #339933;">,</span> base<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #003366; font-weight: bold;">var</span> result <span style="color: #339933;">=</span> <span style="color: #CC0000;">0</span>
    <span style="color: #000066; font-weight: bold;">for</span> offset <span style="color: #339933;">=</span> ranges.<span style="color: #660066;">length</span> <span style="color: #339933;">-</span> <span style="color: #CC0000;">1</span> downto <span style="color: #CC0000;">0</span>
      result <span style="color: #339933;">=</span> result <span style="color: #339933;">*</span> ranges<span style="color: #009900;">&#91;</span>offset<span style="color: #009900;">&#93;</span>
      result <span style="color: #339933;">=</span> result <span style="color: #339933;">+</span> input<span style="color: #009900;">&#91;</span>offset<span style="color: #009900;">&#93;</span>
&nbsp;
    <span style="color: #000066; font-weight: bold;">return</span> int2str<span style="color: #009900;">&#40;</span>result<span style="color: #339933;">,</span> base<span style="color: #009900;">&#41;</span>
  <span style="color: #009900;">&#125;</span><span style="color: #339933;">,</span>
&nbsp;
  <span style="color: #006600; font-style: italic;">// decode( '42', [10, 15], '0123456789' ) : [2, 4]</span>
  decode<span style="color: #339933;">:</span> <span style="color: #003366; font-weight: bold;">function</span> <span style="color: #009900;">&#40;</span>input<span style="color: #339933;">,</span> ranges<span style="color: #339933;">,</span> base<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    input <span style="color: #339933;">=</span> str2int<span style="color: #009900;">&#40;</span>input<span style="color: #339933;">,</span> base<span style="color: #009900;">&#41;</span>
    <span style="color: #003366; font-weight: bold;">var</span> result <span style="color: #339933;">=</span> <span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span>
&nbsp;
    <span style="color: #000066; font-weight: bold;">for</span> offset <span style="color: #339933;">=</span> <span style="color: #CC0000;">0</span> to ranges <span style="color: #339933;">-</span> <span style="color: #CC0000;">1</span>
      result<span style="color: #009900;">&#91;</span>offset<span style="color: #009900;">&#93;</span> <span style="color: #339933;">=</span> inputs <span style="color: #339933;">%</span> ranges<span style="color: #009900;">&#91;</span>offset<span style="color: #009900;">&#93;</span>
      inputs <span style="color: #339933;">=</span> inputs <span style="color: #339933;">/</span> ranges<span style="color: #009900;">&#91;</span>offset<span style="color: #009900;">&#93;</span>
&nbsp;
    <span style="color: #000066; font-weight: bold;">return</span> result<span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#125;</span><span style="color: #339933;">;</span></pre></div></div>

<p>Here is the full source code. This is the same code but being less readable due to the use of BigInt and the need of managing the allocation size.</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;"><span style="color: #006600; font-style: italic;">// Requires BigInt.js ( http://blog.vjeux.com/wp-content/uploads/2009/08/BigInt.js )</span>
SmallHash <span style="color: #339933;">=</span> <span style="color: #009900;">&#123;</span>
  encode<span style="color: #339933;">:</span> <span style="color: #003366; font-weight: bold;">function</span> <span style="color: #009900;">&#40;</span>input<span style="color: #339933;">,</span> ranges<span style="color: #339933;">,</span> base<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #006600; font-style: italic;">// Rough majoration of the final result size</span>
    <span style="color: #006600; font-style: italic;">// It makes the sum of all the minimum of bits required for each range</span>
    <span style="color: #003366; font-weight: bold;">var</span> size <span style="color: #339933;">=</span> <span style="color: #CC0000;">0</span><span style="color: #339933;">;</span>
    <span style="color: #000066; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span><span style="color: #003366; font-weight: bold;">var</span> i <span style="color: #339933;">=</span> <span style="color: #CC0000;">0</span><span style="color: #339933;">,</span> len <span style="color: #339933;">=</span> ranges.<span style="color: #660066;">length</span><span style="color: #339933;">;</span> i <span style="color: #339933;">&lt;</span> len<span style="color: #339933;">;</span> i <span style="color: #339933;">=</span> i <span style="color: #339933;">+</span> <span style="color: #CC0000;">1</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      size <span style="color: #339933;">+=</span> Math.<span style="color: #660066;">ceil</span><span style="color: #009900;">&#40;</span>Math.<span style="color: #660066;">log</span><span style="color: #009900;">&#40;</span>ranges<span style="color: #009900;">&#91;</span>i<span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span> <span style="color: #339933;">/</span> Math.<span style="color: #660066;">LN2</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #003366; font-weight: bold;">var</span> result <span style="color: #339933;">=</span> bigInt.<span style="color: #660066;">int2bigInt</span><span style="color: #009900;">&#40;</span><span style="color: #CC0000;">0</span><span style="color: #339933;">,</span> size<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #000066; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span><span style="color: #003366; font-weight: bold;">var</span> bit <span style="color: #339933;">=</span> ranges.<span style="color: #660066;">length</span> <span style="color: #339933;">-</span> <span style="color: #CC0000;">1</span><span style="color: #339933;">,</span> pos <span style="color: #339933;">=</span> <span style="color: #CC0000;">0</span><span style="color: #339933;">;</span> bit <span style="color: #339933;">&gt;=</span> <span style="color: #CC0000;">0</span><span style="color: #339933;">;</span> bit <span style="color: #339933;">=</span> bit <span style="color: #339933;">-</span> <span style="color: #CC0000;">1</span><span style="color: #339933;">,</span> pos <span style="color: #339933;">=</span> pos <span style="color: #339933;">+</span> <span style="color: #CC0000;">1</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      <span style="color: #006600; font-style: italic;">// If the value is higher than the expected range, the value is maximized</span>
      <span style="color: #006600; font-style: italic;">// Therefore the result is always valid, even if the input is not</span>
      <span style="color: #003366; font-weight: bold;">var</span> parsed_bit <span style="color: #339933;">=</span> Math.<span style="color: #660066;">min</span><span style="color: #009900;">&#40;</span>Math.<span style="color: #660066;">abs</span><span style="color: #009900;">&#40;</span>Math.<span style="color: #660066;">floor</span><span style="color: #009900;">&#40;</span>input<span style="color: #009900;">&#91;</span>bit<span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">,</span> ranges<span style="color: #009900;">&#91;</span>bit<span style="color: #009900;">&#93;</span> <span style="color: #339933;">-</span> <span style="color: #CC0000;">1</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      bigInt.<span style="color: #660066;">mult_</span><span style="color: #009900;">&#40;</span>result<span style="color: #339933;">,</span> bigInt.<span style="color: #660066;">int2bigInt</span><span style="color: #009900;">&#40;</span>ranges<span style="color: #009900;">&#91;</span>bit<span style="color: #009900;">&#93;</span><span style="color: #339933;">,</span> <span style="color: #CC0000;">32</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      bigInt.<span style="color: #660066;">add_</span><span style="color: #009900;">&#40;</span>result<span style="color: #339933;">,</span> bigInt.<span style="color: #660066;">int2bigInt</span><span style="color: #009900;">&#40;</span>parsed_bit<span style="color: #339933;">,</span> <span style="color: #CC0000;">32</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000066; font-weight: bold;">return</span> bigInt.<span style="color: #660066;">bigInt2str</span><span style="color: #009900;">&#40;</span>result<span style="color: #339933;">,</span> base.<span style="color: #660066;">length</span><span style="color: #339933;">,</span> base<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span><span style="color: #339933;">,</span>
  decode<span style="color: #339933;">:</span> <span style="color: #003366; font-weight: bold;">function</span> <span style="color: #009900;">&#40;</span>input<span style="color: #339933;">,</span> ranges<span style="color: #339933;">,</span> base<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    input <span style="color: #339933;">=</span> bigInt.<span style="color: #660066;">str2bigInt</span><span style="color: #009900;">&#40;</span>input<span style="color: #339933;">,</span> base.<span style="color: #660066;">length</span><span style="color: #339933;">,</span> base<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #003366; font-weight: bold;">var</span> remainder <span style="color: #339933;">=</span> bigInt.<span style="color: #660066;">dup</span><span style="color: #009900;">&#40;</span>input<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span> <span style="color: #006600; font-style: italic;">// Allocates enough room for the remainder</span>
    <span style="color: #003366; font-weight: bold;">var</span> result <span style="color: #339933;">=</span> <span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
    <span style="color: #000066; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span><span style="color: #003366; font-weight: bold;">var</span> pos <span style="color: #339933;">=</span> <span style="color: #CC0000;">0</span><span style="color: #339933;">,</span> len <span style="color: #339933;">=</span> ranges.<span style="color: #660066;">length</span><span style="color: #339933;">;</span> pos <span style="color: #339933;">&lt;</span> len<span style="color: #339933;">;</span> pos <span style="color: #339933;">=</span> pos <span style="color: #339933;">+</span> <span style="color: #CC0000;">1</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
      bigInt.<span style="color: #660066;">divide_</span><span style="color: #009900;">&#40;</span>input<span style="color: #339933;">,</span> bigInt.<span style="color: #660066;">int2bigInt</span><span style="color: #009900;">&#40;</span>ranges<span style="color: #009900;">&#91;</span>pos<span style="color: #009900;">&#93;</span><span style="color: #339933;">,</span> <span style="color: #CC0000;">32</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">,</span> input<span style="color: #339933;">,</span> remainder<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
      result<span style="color: #009900;">&#91;</span>pos<span style="color: #009900;">&#93;</span> <span style="color: #339933;">=</span> Number<span style="color: #009900;">&#40;</span>bigInt.<span style="color: #660066;">bigInt2str</span><span style="color: #009900;">&#40;</span>remainder<span style="color: #339933;">,</span> <span style="color: #CC0000;">10</span><span style="color: #339933;">,</span> <span style="color: #3366CC;">'0123456789'</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000066; font-weight: bold;">return</span> result<span style="color: #339933;">;</span>
  <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#125;</span><span style="color: #339933;">;</span></pre></div></div>

<p>This script is using the <a href="http://www.leemon.com/crypto/BigInt.html">BigInt library</a> from Leemon Baird. I made some changes in order not to pollute the global namespace and added the possibility to modify the base string.</pre>
]]></content:encoded>
			<wfw:commentRss>http://blog.vjeux.com/2009/javascript/smallhash-information-compression.html/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Project &#8211; MMO-Champion Optimization</title>
		<link>http://blog.vjeux.com/2009/project/project-mmo-champion-optimization.html</link>
		<comments>http://blog.vjeux.com/2009/project/project-mmo-champion-optimization.html#comments</comments>
		<pubDate>Tue, 04 Aug 2009 16:30:59 +0000</pubDate>
		<dc:creator>vjeux</dc:creator>
				<category><![CDATA[Project]]></category>
		<category><![CDATA[optimization]]></category>

		<guid isPermaLink="false">http://blog.vjeux.com/?p=484</guid>
		<description><![CDATA[MMO-Champion is the biggest news website of World of Warcraft. The main page is viewed millions times a month and was done with old school tables. As a result, it was really slow to load but worse, all the content had to be loaded before being displayed.
The first thing I did was to rewrite the [...]]]></description>
			<content:encoded><![CDATA[<div id="attachment_485" class="wp-caption alignright" style="width: 310px"><a href="http://www.fooo.fr/~vjeux/boub/mmoc/"><img src="http://blog.vjeux.com/wp-content/uploads/2009/09/mmoc.png" alt="MMO-Champion" title="MMO-Champion" width="300" height="206" class="size-full wp-image-485" /></a><p class="wp-caption-text"><a href='http://www.fooo.fr/~vjeux/boub/mmoc/'>MMO-Champion</a></p></div>
<p><a href="http://www.mmo-champion.com/">MMO-Champion</a> is the biggest news website of World of Warcraft. The main page is viewed millions times a month and was done with old school tables. As a result, it was really slow to load but worse, all the content had to be loaded before being displayed.</p>
<p>The first thing I did was to rewrite the whole main page template using clean and valid HTML + CSS. The goal was to make it compatible up to IE6. The rendering was so much pixel perfect that nobody noticed a change when we pushed it live.</p>
<p>The main challenge was to rewrite the menu. Previously, the menu was using several images that were cut in order to make it easy to implement it in CSS. However, it was a torture to add another menu. The new one is now using a single image that is basically a screenshot of the rendered menu.</p>
<p>In order to save bandwidth, if the browser is supporting HTML5, instead of storing the menu state in a cookie, it is saved under the new localStorage feature.</p>
]]></content:encoded>
			<wfw:commentRss>http://blog.vjeux.com/2009/project/project-mmo-champion-optimization.html/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Project &#8211; Shortest Path</title>
		<link>http://blog.vjeux.com/2009/project/project-shortest-path.html</link>
		<comments>http://blog.vjeux.com/2009/project/project-shortest-path.html#comments</comments>
		<pubDate>Tue, 04 Aug 2009 09:56:52 +0000</pubDate>
		<dc:creator>vjeux</dc:creator>
				<category><![CDATA[Project]]></category>
		<category><![CDATA[optimization]]></category>

		<guid isPermaLink="false">http://blog.vjeux.com/?p=422</guid>
		<description><![CDATA[A school project was to find the shortest path in a dungeon graph. You start with an amount of hit points, and each edge gives or removes hit points. You have to find the path from two points going through the minimum of edges (no matter their value) alive (hp > 0 all along the [...]]]></description>
			<content:encoded><![CDATA[<p>A school project was to find the shortest path in a dungeon graph. You start with an amount of hit points, and each edge gives or removes hit points. You have to find the path from two points going through the minimum of edges (no matter their value) alive (hp > 0 all along the path). The difficulty here is that you have to go a finite amount of time through absorbing cycle to complete interesting graphs. Here is the <a href="/wp-content/uploads/2009/09/miniprojC.htm">full subject</a>.</p>
<div id="attachment_427" class="wp-caption aligncenter" style="width: 517px"><img src="http://blog.vjeux.com/wp-content/uploads/2009/09/graph.png" alt="Example dungeon graph" title="Example: dungeon graph" width="507" height="177" class="size-full wp-image-427" /><p class="wp-caption-text">Example: dungeon graph</p></div>
<p><strong>Example</strong>: You start with 10 hp. The shortest path is <cite>[1, 2], [3, 4, 2] * 492, [3, 5]</cite>. You notice that you have to loop a huge number of time on the <cite>[3, 4, 2]</cite> cycle that gives you 1 hp per run.</p>
<p>The common solutions were either a custom Breadth First Search or Dijkstra algorithm along with a heuristic. It is easy to write, always gives the shortest path and have <a href="http://wiki-prog.kh405.net/index.php/MPC2009:Correction">space for improvement</a>. However, it has a major flaw: it doesn't scale well with high values. Instead of 500, put 50 000 000 and it will run forever.</p>
<p>My approach is to detect the cycles and find the best cycle organization. In this example, you would notice that <cite>[3, 4, 2]</cite> is a cycle that takes 3 edges and gives 1 hp. You come to this cycle with 9 hp, you need to pass with 501 hp so you need to take this cycle (501 - 9) = 492 times.</p>
<h3>Finding Atomic Paths &#038; Cycles</h3>
<h4>Definition</h4>
<p>What we want to do is find all the possible paths going from point A to point B. Since there are cycles involved, you can't just go through and enumerate them all. Instead, you will have to find atomic path that doesn't loop and the smallest possible cycles (you don't want your cycle to repeat itself). </p>
<p>The first definition I took of an atomic path is a path that does not go through the same node twice. However, I found out that is was not taking all possibilities. After some reflexion, I figured out that nodes aren't important, however edges are! So <i>an atomic path is a path that does not go through the same edge twice</i>.</p>
<p>This definition is handy, it also works for cycles: <i>an atomic cycle of point A is an atomic path that goes from point A and ends to point A</i>.</p>
<h4>Implementation</h4>
<h5>Atomic Paths <i>A -> B</i></h5>
<p>In order to get all the path starting from point A, we are going to traverse the graph recursively from the point A. While going through a child, we are going to make a link <i>child -> parent</i> in order to know all the edges we have already crossed. Before we go to that child, we must traverse that linked list and make sure the specified edge has not been already walked through.</p>
<p>When we arrive to the destination point, we can store the path we found.</p>
<h5>Freeing the list</h5>
<p>A problem occurs when you want to free the linked list. It is basically a tree chained in the reverse order. A solution would be to double-link that list and when all the atomic paths been found, free the tree from the starting point. </p>
<p>But a clever solution is to use a reference counting (inspired from Garbage Collection). Each time you add a link to a parent you adds one to its reference count. Then, when you arrive at the end of a path, you go backward and free while the reference count equals to 1. If it is higher, you just remove one and stop.</p>
<h5>Atomic Cycle A</h5>
<p>Looking for the atomic cycle of A is the same as looking for the atomic path from A to A. However there are several optimizations we can do. First, when we arrive at the destination point, we want to save the path only if the sum of the edges cost is negative: we only want to go through absorbing cycles.</p>
<p>As you have seen previously, the whole graph is being traversed when looking for an atomic path. Instead, we can limit the search area to the strongly connected component containing A. Finding these components requires a simple traverse of the graph with <a href="http://en.wikipedia.org/wiki/Tarjan%27s_strongly_connected_components_algorithm">Tarjan's algorithm</a>.</p>
<h3>Combining Atomic Paths and Cycles</h3>
<p>At this point, we have all the atomic paths that goes from A to B and all the atomic cycles of each node, left to us to organize everything to get the shortest path. From now on we are going to study how to find the best combination of atomic cycles in an atomic path.</p>
<p>In order to see the problem, we can use a digression. Imagine a plane that wants to cross a mountain. The only way to get up is to stay on hot air spot for a while. He has to chose the best combination of spots in order to go over each peak loosing as little time as possible.</p>
<h4>Atomic Cycle Characteristics</h4>
<p>Absorbing cycles can only be unlocked when moving through the path. In the example, the cycle <cite>[3, 4, 2]</cite> is only available after being on the node 2. So you have to have 1 hp to get to node 2 and 1 more hp to get to node 3 and then be able to get the -3 bonus. This highlights an important characteristic: minimum hp. Each cycle and node have a minimum hp to be visited.</p>
<p>We are now able to fully describe an atomic cycle with these four characteristics:</p>
<ul>
<li>Entry point</li>
<li>Edge count</li>
<li>Cost (negative)</li>
<li>Mininum hp</li>
</ul>
<h4>Nodes to States</h4>
<p>Atomic cycles have an inner minimum hp, hence, this is not possible to merge them with their entry point. We have to make a virtual state representing the possibility to walk that cycle. The condition of that state is having at least <i>node minimum hp + cycle minimum hp</i>. This way we can define a state minimum hp.</p>
<div id="attachment_468" class="wp-caption aligncenter" style="width: 315px"><img src="http://blog.vjeux.com/wp-content/uploads/2009/08/mod11.png" alt="Example: Node to State" title="Example: Node to State" width="305" height="430" class="size-full wp-image-468" /><p class="wp-caption-text">Example: Node to State</p></div>
<p>In the example, the third node has a minimum hp of <i>1 + 2</i> and the yellow cycle has a minimum hp of 1, so the resulting state has a minimum hp of 4. </p>
<p><strong>Warning:</strong> This technique is subject to <a href="http://blog.vjeux.com/wp-content/uploads/2009/08/mod21.png">overlapping</a>. I did not find an efficient way to handle this case so it can generate wrong result.</p>
<h4>Greedy Combination</h4>
<p>What we have now is a list of states, each one unlocking one or more cycles. We have to find the best combination of cycles.</p>
<p>We have first to define what is the best cycle. There can be two definitions based on the ratio hp / edge.</p>
<ul>
<li><strong>Global ratio</strong>: The one that gives the most hp per edge.
</li>
<li><strong>Local ratio</strong>: The one that goes from A to B with the less number of edges</li>
</ul>
<p>We could be tempted to use the global ratio but this is not a good idea. Imagine you need to get 2 hp to go to the end. You don't want to use the cycle that gives you 1 billion hp for only 100 edges when there is a cycle that gives you 2 hp for 10 edges.</p>
<p>A simple algorithm to find a good solution is to take the cycle with the best local ratio available at state A enough times to go to state B, then repeat the same process with cycles available at state B to go to state C and so on. </p>
<p>This works great but is too much subject to local errors. Let say you have the two cycles of the previous example available at state A and you want to go to state Z that is 200 hp away with small steps between. You will always choose the cycle that gives you 2 hp for 10 edges and end up with 200 edges when you could have used the 1 billion hp for 100 edges in the first time.</p>
<p>A solution is to sophisticate the algorithm a little. Instead of doing <i>[A -> B -> C -> D]</i>, we are going to do <br /><i>[A -> B -> C -> D]</i>, <i>[A -> C -> D]</i>, <i>[A -> B -> D]</i> and <i>[A -> D]</i> and see what's best. This is more processing but gives much better results.</p>
<p><strong>Warning:</strong> This method does not give the best possible combination, only an approximation. Also, it does not handle cases were a combination of multiple cycles is better than a single cycle to go from one state to another.</p>
<h4>Optimizations</h4>
<p>Before attempting to compute an atomic path, take the cycle with the best global ratio available and see how many edges would be necessary to get to the end with that ratio. If that's superior to the already found shortest path, it is worthless to do it. This simple test should allow you to skip the majority of the paths.</p>
<p>When you are generating the states, you do not have to take care of cycles when there is a better one available before. A better cycle is a cycle that has less edges for the same amount of hp. This reduces the number of states, therefore by a great factor the number of combinations needed in the sophisticated algorithm.</p>
<h3>Conclusion</h3>
<h4>Advantages</h4>
<h5>Complexity edge value independent</h5>
<p>The main advantage of this method is that the complexity depends of the number of nodes and the density of the graph but is not affected by edge values. If you set the critical edge to be 50 or 50 000 000, this will give you the result in the same time. However, the time required increases faster with the number of nodes than the basic method.</p>
<h5>Parallelizable code</h5>
<p>To compensate, the code becomes parallelizable. During the first step, you can give each atomic path and cycle detection to different computers at the same time. Then, each path can be computed no matter the order.</p>
<h5>Low memory output</h5>
<p>The main problem with the basic method is memory allocation. For each node of the final and potential (to some extent) paths a memory chunk is allocated. When you are testing you usually do it with long paths and memory becomes really fast the limiting factor.</p>
<h4>Disadvantages</h4>
<h5>Much more code</h5>
<p>The resulting code is much bigger than the basic method. This means a longer development time, more bugs.</p>
<h5>Not perfect result</h5>
<p>The real problem are the edge cases when it does not return the shortest path or worse, when it gives an invalid path. The three problems are the overlapping issue when converting nodes to states, the greedy algorithm to find combinations and finally the single-only cycle to go from one state to another.</p>
<p>I don't know if all of them could be fixed without seriously affecting the complexity of the algorithm. </p>
<h5>Is strongly depending of the size of the graph</h5>
<p>Since the algorithm does a brute force of all possibilities, the complexity goes exponentially with the size of the graph. However, in the case of the project, we were told that only small (no more than 50 nodes) would be tested.</p>
]]></content:encoded>
			<wfw:commentRss>http://blog.vjeux.com/2009/project/project-shortest-path.html/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>
