Javascript – Sorting Table

For my new project on World of Raids I have to implement a table sorting. The browser not stable sorting and the faster sorting trick add difficulty to the task.

String Comparison

As mentionned in the Speed Up Javascript Sort() article, using a string as a key to represent each element is faster than using a custom sort function in most browsers.

Tables are commonly made of two data types: Numbers and Strings. Strings are the final representation so no work is involved there so our work will be focused on Numbers. What we want is to fit a number into a string. A string is a succession of characters, each one able to hold 256 values. So, we can see the problem as encoding the number into a base 256 which is trivial.

Padding

String comparison does not work exactly like we want it. It reads all the characters one by one and if they mismatch then it returns who is the highest. So for example "10" < "5" because '1' < '5'. We have to pad every numbers with 0 at the beginning and become "10" > "05".

In this case we are adding one 0 because 10 is 2 digits and 5 is only one. We won’t know the values of other elements when making a comparison, so two solutions apply: either we iterate over all the elements and retrieve the highest, or we set an arbitrary maximum.

Characters Maximum
1 256
2 65 536
3 16 777 216
4 4 294 967 296

With that table in mind, it is obvious that it would be a waste of time to inspect every element just to find the highest value. 1 or 2 characters will probably fit most uses but if you want to be safe, you will always be able to store your number into 4 characters which is acceptable.

Here is the code to do the conversion:

function compareAsc(num, digits) {
  var s = "";
  // Encode num in base 256
  while (num !== 0) {
    s = String.fromCharCode((num % 256) | 0) + s;
    num = (num / 256) | 0;
    digits -= 1;
  }
  // Fill with 0
  while (digits > 0) {
    s = String.fromCharCode(0) + s;
    digits -= 1;
  }
  return s;
}

For the sake of simplicity, this code is using string concatenation which requires to build up new strings and make copy several times. Since this is for small strings (up to 4 characters!) this is probably alright but one looking for performances could bench this with the use of an Array with a .concat() at the end to build the string.

Also note that numbers are stored as float in Javascript, so we are rounding (flooring to be exact) all the values using the |0 trick.

Descending Order

We have made the code for an ascending sort but we have to handle the other case. We cannot just revert the order of the sorting or adding a minus sign before the number. What we have to do is to do a 256-complement of the final result, in other terms: digit = 255 – digit. Let see an example for a base 10.

00 -> 99
01 -> 98
...
54 -> 45
55 -> 44
56 -> 43
...
98 -> 01
99 -> 00

You can see that all the numbers are now sorted in the opposite order.

Here is the associated code :

function compareDesc(num, digits) {
  var s = "";
  // Encode num in base 256 and do the 256-complement
  while (num !== 0) {
    s = String.fromCharCode(255 - (num % 256) | 0) + s;
    num = (num / 256) | 0;
    digits -= 1;
  }
  while (digits >= 0) {
    s = String.fromCharCode(255) + s;
    digits -= 1;
  }
  return s;
}

Floats and Negatives

We have handled all the positive integers but they are not alone. In order to deal with the negative numbers you can apply the 2-complement to the number. This is how it is done in computer arithmetic.

To handle floats, it is possible to treat them as integers by multiplying by 10^(number of decimal displayed). So 12.34 is translated to 1234 and 11 to 1100 and it’s working just fine. You will probably need more than 4 characters if you are using huge numbers though.

Stable Sorting

Implementing a table sorting requires the sorting algorithm to be stable (values that have the same key keep their original order). This property allows people to sort by multiple columns, the last one having the highest weight.

However, browser implementations of the Array.sort() are not stable. From there we have two solutions, implementing a stable sort in javascript or find a way to emulate that behaviour. Since Javascript is slow in many browsers, implementing such a computation heavy algorithm in Javascript is going to slow down things and requires more efforts than I am willing to spend on this project.

The solution instead is to understand what the stable property means and find a way to code it.

Stable sorting algorithms maintain the relative order of records with equal keys. […] Whenever there are two records (let’s say R and S) with the same key, and R appears before S in the original list, then R will always appear before S in the sorted list.

We can easily transpose that sentence into the sorting function:

if (R.key == S.key)
  return compare(R.position, S.position)

We need to know one more thing to do the computation: the position of each element in the list before sorting. This is straightforward to do, you have to iterate over all the elements and update their position field. This requires n steps which is acceptable.

Here is the full Javascript code

var sort = function (a, b) {
  if (a.key === b.key)
    return a.position - b.position;
  if (a.key < b.key)
    return -1;
  return 1;
};

String comparison

To apply this concept to the string comparison, you can append the position at the end of the string. "Key" now being "Key1", "Key2" and so on. When the keys are equal, the position is used for the comparison instead. We can reuse the compareAsc() function to encode the position using the minimal size.

This works well for fixed-size strings like the representation of numbers. However common strings do not share that property and a problem arise when two strings share a common base.

Take “Method” and “Methodology”. It is obvious that "Method" < "Methodology". Now append the position: "Method" + chr(250) > "Methodology" + chr(251) the comparison changed because chr(250) > 'o'. (Where chr = String.fromCharCode').

The way to fix this problem is to add a chr(0) before the position. Considering that a normal string never contains a chr(0), the chr(0) will always be lesser than any character of the string and therefore stop the comparison. The longer string always wins!

The Bistromathique is an Arbitrary Precision Calculator EPITA project where the main focus is optimization. The input can be in any base (up to 250) and the following operations have to be performed: Addition, Subtraction, Multiplication, Division, Modulo.

Base Representation

Going back and forth a textual representation to a computable representation is the first part of the project and several optimizations are possible at this step.

Binary Representation

Encoding the numbers as binary allows you to make the best use of the processor calculation unit. There is no space lost and the number being represented as an uninterrupted series of 0 and 1 can be added and subtracted directly with the hardware.

However, in order to convert the number for a base B representation to a base 2 requires roughly n divisions by B. And to reconvert it back requires another n multiplications. This can be affordable for applications that uses the same numbers again and again but in our case, this cost is prohibitive.

The other major drawback is that you are required to have the multiplications and divisions (and therefore addition and subtraction) in order to have the numbers in the right form to start working on the operations … In summary, you have to code all these operations before being able to test them!

char Representation

The binary representation not being possible, the solution that comes in mind is to put each digit into a char (8 bits). Since we are limited up to a base of 250, any base will fit. This comes at a price, there is a lot of space wasted, and it increases as the base size decreases.

We are in a 1 to 1 relation between the input and the represented number, this way we can store the represented number inside the input. This prevents from allocating memory at this step. The conversion can be efficiently done, each character goes through vector of 256 slots storing its representation and a magic value used as a non valid input. One final trick is to add a non number at the end of the input to remove a end of input check in the loop.

Filling more

A char can store values up to 256. If we are working on base 16 we could store two digits into 1 char and thus divide by 2 the number of digits. Additions are therefore 2 times faster and even more for n log n operations such as multiplication.

Calculating the number of digits to fill into a char is quite straightforward however some caution has to be taken during the transition. We have to know in advance the length of the input to decide how to start filling the digits. If you can fit 2 digits per char, if the number is even then it goes as expected, but if the number is odd you have to place 0 at the beginning to fill the gap.

Even: [1] [2] [3] [4] -> [12] [34]
Odd (Naive): [2] [3] [4] -> [23] [4?]
Odd (Smart): [2] [3] [4] -> [02] [34]

This method works well but requires 2 passes for each number and a special case at the beginning of each number. Not enough to stop us from using it.

Using a larger representation

Now that we filled the char at its maximum, why stick with it? This should ring a bell since you know that common architecture is 32-bit, and from now we are talking about a 8-bit representation. We would be tempted to change the char to an int (32-bit) or even a long (64-bit) that could benefit from register optimizations.

Moving away from a char, we lose the assumption that the output being smaller (or equal) than the input. Without it, this is no longer possible to work in place. We now have to compare the cost of a reduction in the number of operations and the cost of the memory allocation.

The overhead happens for values that are represented in less than 4 characters for an int representation. This is a maximum 3-characters overhead possible for each number, which is negligible. We could eat the operation character but we are left with 2 characters. I haven’t found a solution to stay in place yet and this minor problem prevents from accessing a great speed up.

To be continued …

In World of Raids Guild Recruitment Tool, we wanted the search to be instant! We achieved this goal by not having a single page refresh or ajax call while updating the settings. Since it runs in the browser, performance was a real concern. We will see in this article what were the tricks used to make this real.

Sorting

In order to sort the guilds, each guild is given a matching score (between 0 and 65535) based on the current filters. At start, each guild has the maximum points. Then, each filter is giving a malus. For example when a specialization doesn’t match, the guild loses 10%.

This means that everytime you change a filter, you have to recompute that matching score for all the guilds. Instead of recomputing the whole matching score, the guild holds the result of each filter. Everytime a filter is updated, the value of this filter is upated on every guild. The global matching score is now recalculated by summing all the partial precomputed scores.

Every bit of code in the matching function has been optimized, no more jquery code, loops are unrolled and great use of lazy boolean expression evaluation. The sort function itself also got boosted thanks to a trick I commented on my previous article Speed Up Javascript Sort().

Guild Recruitment Search Filters

Display

Once we have the data sorted, it has to be displayed! And this is another big time eater. When it takes 50ms to sort the data, it takes about 150ms to display 10 guilds!

Since we wanted performances, DOM elements are created at page loading, they are then only being updated as the guilds change. And still, only the required changes are made, for example tooltips are created as the user trigger the mouseover event.

The main problem with this is that we are not able to make bulk changes. Everytime you make the slightest change, the whole UI has to be redrawn which is a waste of time. I hope there will be such API in the future.

Guild Display

Since it’s a web application, we had to maintain the url up to date, this is done using SmallHash, a small library of mine that compresses filter data into the smallest possible hash.

Scaling

Having all the data being sent at page loading is a pain, it makes it load slowly. One way to avoid this problem would be to use the localstorage to keep these data accross sessions and only send the updated guilds. However at the time I started developping the application there was no released browser supporting HTML5.

However, the application is focused on World of Warcraft guilds. This is a limited audience and there is no need on heavy scaling there. So we decided that it was a good option to go.

Up to 3000 guilds, this is pretty much instant in every browser in my MacBook Pro. When increasing that number to 10 000 it is taking more than a second to update on IE8. It requires 50 000 guilds for Chrome 3 to get slow, which is a pretty good score. It means that it is possible to do heavy processing, the bandwith now being the limiting factor.

Project – WoR Guild Recruitment

World of Raids Guild Recruitment

Guild recruitment is a recurrent problem in World of Warcraft, many attempt have been made but none succedeed so far. After a brainstorming we decided that the following points were crucial.

  • The guild recruiter has to spend less time as possible to set-up a guild and maintain it.
  • The guild search must be easy and focus on what people expect from their guild.

With these points in mind, we had to find technical responses in order to make the World of Raids Guild Recruitment Tool.

  • What You See Is What You Get: The guild management interface is the final display, there is no intermediate step.
  • Tailored Widgets: About every widget has been heavily customized to fit the user need.
  • Automatic Save: Every time you make a change, it is automatically published and available to anyone!
  • Javascript Search: Having all the guilds fetched during the loading allows to have a complex filtering system instantly updated.

Guild Search

Looking for a guild should no more be a pain! You just have to tweak the filters and result appear sorted as you edit them! No more long page reload or even ajax requests. It is instant!

We focused hard on making filters one click away. We also took great care of the raid time filter, it is an important aspect of the guild choice that is too often avoided because of its complexity.

For more details on the optimizations made, see the post Guild Recruitement – Search Optimizations.

Guild Management

Our original goal was to be able to create a guild in less than a minute (yes, 60 seconds!) and we pretty much did it. The proof in the following video.

In order to achieve this, we opted for a Wysiwyg approach to remove the need of making two interfaces for both display and edition. Every widget has been tailored to fit the user needs. For example, as soon as you enter your name and realm, it automatically gathers your progression from the official website.

Having to deal with dependencies in Makefile is a real pain, there are a lot of examples of way to deal with it on the web but none of them is satisfying.

For example using gcc -MM does not work with subfolders, a depend rule requires the user to use it everytimes he adds new files …

Here is what is wanted:

  • Completly automatic: No user interaction is required when adding new files
  • Works with an arbitrary amount of files
  • Works with an arbitrary amount of level of folders
  • Is not recalculated when nothing changed
  • Use only one file to store dependencies
  • Do not depend on complicated regular expressions

Here is the result:

BINARY  = project.exe
CC      = gcc
CFLAGS  = 
FILES   = $(shell find src/ -name "*.c")
HEADERS = $(shell find src/ -name "*.h")
OBJS    = $(FILES:.c=.o)
 
all: $(BINARY)
 
-include Makefile.deps
 
$(BINARY): Makefile.deps $(OBJS)
        $(CC) $(CFLAGS) $(OBJS) -o $(BINARY)
 
Makefile.deps: $(FILES) $(HEADERS)
        makedepend -- $(CFLAGS) -- $(FILES) -f- > Makefile.deps

all: $(BINARY)

-include Makefile.deps

$(BINARY): Makefile.deps $(OBJS)
$(CC) $(CFLAGS) $(OBJS) -o $(BINARY)

Makefile.deps: $(FILES) $(HEADERS)
makedepend — $(CFLAGS) — $(FILES) -f- > Makefile.deps

This is in fact really easy. In your $(BINARY) rule, you add Makefile.deps as a prerequisite.

In order to generate the Makefile.deps you mark all $(FILES) and $(HEADERS) as prerequisite, so every time you change a file or header it will recompile the list.
We use makedepend to generate the dependencies list. It works like gcc -MM except that it outputs the correct file path when used with folders.

Then all is required is to include the Makefile.deps. We include it with -include so it does work the first time you compile.

Thanks to Lemoine Gauthier who helped me to discover this technique.

Light & Mirror Programmation

Prime number recognition is a very hard problem and yet no good enough solution has been found using classical algorithms. There are two ways to get around those limitations: find an algorithm with a better complexity or find a way to compute faster. The first one has already been researched by a large amount of people so I decided to give a shot to the second one.

We are limited by the speed of the electricity, so my idea is to use speed of light to improve speed by a great factor. I’ve come up with a simple example of how to use it to compute prime numbers.

We set up 2 mirrors with light sources that are directed.

Caption

We first set the sensor to the position lower 2

2 had no light so we create a new one that is going to en-light all the multiples of 2

Caption

3 had no light so we create a new one that is going to en-light all the multiples of 3

4 is already in the light, we skip it

4 is already in the light, we skip it

Here is the pseudo code of the prime calculation with Light & Mirror.

hasLight(n):
// Sensor placed at lower n that tells weither if it is enlightened or not
 
newLight(n):
// Creates a light that starts from lower n, targeted at the upper n + n/2,
// that first reflects to lower 2n and then to all the multiples of n.
 
isPrime(n):
for i = 2 to sqrt(n)
  if not hasLight(i)
    newLight(i)
 
return not hasLight(n)

newLight(n):
// Creates a light that starts from lower n, targeted at the upper n + n/2,
// that first reflects to lower 2n and then to all the multiples of n.

isPrime(n):
for i = 2 to sqrt(n)
if not hasLight(i)
newLight(i)

return not hasLight(n)

This method allows us to recognize prime numbers with light sources, mirrors, sensors and a bit of programming.

Recognizing if a number is prime or not requires to have a lot of lasers and sensors. I don’t think that is a viable process, however this could be interesting to do multiplications. You want to fire 2 lights and then see in what point they cross. However there are 2 main problems:

  • It requires to have 2 really long mirrors possibly infinite which is not possible in practice. It may be possible to add a vertical mirror in order to work in a bounded area. Yet to know how many times the light hit that mirror.
  • We are required either to have an infinite number of sensors. We could have only one sensor that checks all positions sequentially but then we loose the speed of light! What is wanted instead are mirrors in each position that would redirect light to a unique sensor capable of knowing where it came from.

This is an experiment and it is not at the moment near to be faster than actual methods but this is a new way to think about programming. I don’t know if anything useful is going to be deviated from this but it shows how to use the light to compute things.

Working on the World of Raids Recruitment Tool we wanted automatic saving while editing. There are basically two ways of handling the problem.

  • Send update everytime something changes. The main advantage of this technique is that you are always up to date. However, it is spamming the server on fields that are constantly being updated (text fields for example).
  • Send update every X seconds. There is no longer query spike but there are two main disadvantages.

    The delay the data are sent after a modification can vary from nothing to the timer length for no apparent reason. This is a really bad feeling for the user. He will think that the application is buggy.

    The application is going to send queries all the time, even if there are no changes. We can safely assume that people are interacting a lot less than viewing so this is a real waste of resources.

We are interested in the first one, we want instant update but removing the ability to send too much data at once. What we want is to send data right after the user is done typing instead of everytime a character is typed. There is an easy way to do it in Javascript.

When you type the first character, it will start a timer of let say 100ms. Then everytime you type another character it resets the timer to 100ms. When it triggers, it will send data to the server.

This way, data will only be sent when the user interacts and with no risk of spamming the server.

var Object = {
  /* Object Code ... */
 
  timer: null,
  sendChanges: function() {
    if (this.timer) {
      clearTimeout(timer);
      this.timer = null;
    }
 
    this.timer = setTimeout(
      function () { /* Send Changes */ },
      200 /* Timeout in ms */
    );
  }
};

timer: null,
sendChanges: function() {
if (this.timer) {
clearTimeout(timer);
this.timer = null;
}

this.timer = setTimeout(
function () { /* Send Changes */ },
200 /* Timeout in ms */
);
}
};

MySQL – Select Previous/Next Entries

In SQL, this is a common issue to query the previous and next entries of the database. In order to achieve this, we are going to use a table that is sorted by the `id` field. To get the previous, the technique consists in sorting all the fields that are lower that the current one and taking the first one.

SELECT *
FROM `table`
WHERE `id` < $current_id
ORDER BY `id` DESC
LIMIT 1

Let’s take an example with the values [8, 3, 4, 9, 6, 1]. We want to get the previous of 8.

  • WHERE `id` < $current_id
    We first take all the values strictly lower than 8
    Result: [3, 4, 1, 6]
  • ORDER BY `id` DESC
    We then sort them in descending order
    Result: [6, 4, 3, 1]
  • LIMIT 1
    We take the first one
    Result: 6

You can apply the same method for the next entry:

SELECT *
FROM `table`
WHERE `id` > $current_id
ORDER BY `id` ASC
LIMIT 1

EPITA – Project Recap

It has been one month that the third year (called Ing1) of EPITA started and this is a recap of the interesting projects that I developed.

malloc (1 week)
The goal of this mini-project is to improve our knowledge of memory management by implementing the C standard library memory allocator (malloc, free, realloc). Implementation using the Binary Buddy technique:

  • List of free blocks of size 2^k.
  • Split and Merge using adjacent block of the same size called buddy.
  • Insertion and Deletion in O(log n).
  • 32 & 64 bit ready.

find (1 week)
find is a very well known UNIX command used to satisfy research upon files and directories.

  • File listing and options parsing with error handling.
  • -name: Filtering using globbing.
  • -newer, -atime, -size, -inum, -type, -links: Filtering off information sources. Handles -/+ for “less/more than”.
  • -perm: Filtering based on permission expressed in octal (0777) or symbolic (u+rwx,go=r) notation.

fnmatch (10 hours)
fnmatch is a function that implements globbing. It returns whether the string matches the pattern. This is mainly used to filter out files (eg: *.txt) or text search in databases (eg: LIKE ‘%Text%’).

  • *: any character 0 or more times.
  • ?: any character 1 time.
  • [abcA-Z]: any character from the class.
  • *: escaping.

Libstream (10 hours)
I/O functions are slow hence they should be called few time as possible. This is why they are typically buffered. The project is to recode all the I/O buffered functions of the C standard library as they are defined by the Single UNIX Specifications V3.

  • fopen, fclose, fgetc, fputc, fflush: Core functions to handle buffered I/O at a character level.
  • fseek, ftell, frewind: File positioning functions.
  • fread, fwrite, fgets, fputs, fgetdelim: Helper functions to use buffered I/O at a string level.
  • tinyPrintf: Basic string formatting (%d, %u, %s, %c, %o, %x without modifiers).

Javascript – Type Casting

Comparison

In Javascript there are 3 types we are often comparing: String, Number and Boolean. After digging through the ECMA-262 specifications, here is the behaviour of the == operator (11.9.3) on these types:

  • Number == String
    Typecasted as follow: Number == Number(String)
  • Number == Boolean
    Typecasted as follow: Number == Number(Boolean)
  • String == Boolean
    Typecasted as follow: Number(String) == Number(Boolean)

This means that when comparing data of two different types, they will first be converted to Number.

Note: The order of the equality is not important: A == B is the same as B == A (except the order of evalution of A and B).

You can force comparison of a and b with the type you want:

  • String: "" + a == "" + b
  • Number: +a == +b
  • Integer: a | 0 == b | 0
  • Boolean: !!a == !!b

Addition / Concatenation

The binary + operator follows a simple rule (11.6.1):

  • if one of the operands is a String, both operands are converted to String and the + is a concatenation.
  • Else, both operands are converted to Number and the + is an addition.

Note: Some Objects are considered as Strings like Arrays. See 8.6.2 for more informations.

Note: The operator is binary so 1 + 1 + 'a' will be executed as (1 + 1) + 'a' and therefore be equal to "2a" and not "11a".