No-database for real-time applications (better than No-Sql)

Written by philip on November 20, 2013

Recently my thinking has tended towards having a database entirely in RAM since the price of RAM is so low for so much, as compared to the past. By having the database all in RAM the queries can be much faster and unusual indexes can be created in code, there is also no mismatch between code and database since all data is in RAM at all times. I don’t need to worry so much about efficiency as CPU and RAM is so fast, the bottleneck would then be bandwidth.

From the programming perspective I would use Scala and the Pickling library to serialize data to and from memory to the disk, when the software starts up it would load its database from a file int memory objects as are used in Scala, then memory would change from time to time and then serialize back to a file now and then and when the computer shut’s down.

The practical problem to this is that buying a server or renting a server with a lot of RAM is costly, although the price of RAM overall is low, the cost of server RAM is still high, well, higher than the cost of it on a general home PC. Interestingly here in Hong Kong the cost of home internet is low and the bandwidth is high, all relative to other countries of course.

So then I have some unusual thinking about this, what if the database is not stored on the server but on a home PC with an always-on internet connection. Then the flow of communications would go from someones computer to my server, then my server would contact my home PC and the home PC would use its in memory database to fetch the results. The results would go back to my server and then back to the customer. Since the server and my home PC would be located in Hong Kong, the round-trip time should be in the order of 10 milliseconds overhead. I would have to measure the over-head in communications roundtrip time between the server and my home PC.

The two computers could be connected via a secure VPN.

In this case, my home PC would keep a large database all in memory which is the customers database. It could have 64 Gigabytes of RAM, around $800 USD or $6000 HKD, more than enough Ram for most customers databases. Also I could have more than one home computer, perhaps 5 or so and maybe at different locations in Hong Kong, this solution could scale out if I just added more home computers. That is a sort of cloud computer network.

Some problems with this are firstly that if the home computer dies or disconnects then the customers database is not available. This might invalidate the home internet connection agreement with the service provider since its for business use and not for personal use, in this case they might disconnect or be upset about the usage. Although, people use VPN’s and no-one knows what they are using it for. The round-trip time from the server to the home computer is a slight worry, 10 milliseconds or 20 milliseconds? Even if it was 50 milliseconds, its better than querying a slow database on a server hard disk.

Think of how the web is moving forward, we are going into real-time communications and away from static web pages, fast data movement and a dynamic environment is the future. Eventually the cost of server RAM will come down, so its better to work in an environment which will appear eventually given enough time.

Ultimately, this is worthwhile, not having a database means not having to worry about efficient queries (in general, but not always), also not having to worry about database/code mis-match. Code can be changed easily and re-started without a need to change the database. Development cost is therefore lower. Hmm interesting.

Scala redesign part 2

Written by philip on November 10, 2013

Pyret is a language with nice syntax, some of the syntax could be applied to my ideas about re-designing Scala syntax.

I like the word fun rather than def for function definition.

 fun square(n :: Number) -> Number:
  n * n

In Scala I would wish this to appear as:

fun square(n: Number): Number
  n * n

Data keyword can be used to define case classes.

data BinTree
  node(v, l, r)

Would then be in Scala:

case class BinTree
case class Leaf: BinTree
case class Node(v, l: BinTree, r: BinTree): BinTree

Hmm not sure about it, get back to this post later

Scala redesign part 1

Written by philip on September 29, 2013

I like Scala, its powerful and useful and worthwhile programming in. I program faster in it than in other languages. However, I dislike the syntax, so here are some adjustments I wish to make. The following normal Scala code looks as follows:

val copyright: String = dateP1 findFirstIn "Date of this document: 2011-07-15" match {
 case Some(dateP1(year, month, day)) => "Copyright "+year
 case None => "No copyright"

I would prefer it to look as follows.

val copyright = dateP1.find first in "Date of this document: 2011-07-15"
 | Some /dateP1 /year, month, day => "Copyright " + year
 | None => "No copyright"

The features are:

Two spaces indicate indentation.
Function calls can be written with spaces between words, find first in becomes findFirstIn.
The | symbol acts as a match … case.
The / symbol means that the function will have brackets. This comes from Pogoscript, in Pogoscript it is written with @ and reduces the number of brackets.

How would you go about converting my syntax into Scala syntax?

Indents are blocks, also we need to note that this block had all | symbols on its lines. So.

val copyright = dateP1.find first in "Date of this document: 2011-07-15" {
 Some /dateP1 /year, month, day => "Copyright " + year
 None => "No copyright"

The | symbol mean’t that it was a match case block.

val copyright = dateP1.find first in "Date of this document: 2011-07-15" match {
 case Some /dateP1 /year, month, day => "Copyright " + year
 case None => "No copyright"

The / symbol against a word or string or number means its a start bracket.

val copyright = dateP1.find first in "Date of this document: 2011-07-15" match {
 case Some (dateP1 (year, month, day)) => "Copyright " + year
 case None => "No copyright"

Are we there yet? collapse the find first in.

val copyright = dateP1.findFirstIn "Date of this document: 2011-07-15" match {
 case Some (dateP1 (year, month, day)) => "Copyright " + year
 case None => "No copyright"


Why I don’t need Google Glasses

Written by philip on April 28, 2013

Most of my time working is in front of the computer, I already have a google glass, that is my computer. The only other thing I do when I’m working is talk to people. The times I am not working, I am either relaxing, eating or sleeping, in any of these cases I do not want Google Glasses.

I’m not sure that Google gets that, the Google glasses are a toy just like the Segway, I don’t need it to get around, in fact the Segway would make getting around more difficult as if I ride it from A to B where do I put it when I get to B? I put it in my office? difficult. The Google glasses also have no place in my life, Google I needed a search engine yes but I don’t need the glasses. A nice toy for the rich but not actually useful in day to day life for me.

Object Oriented API’s are difficult

Written by philip on April 21, 2013

I always find OO API’s difficult to get started with and difficult to learn and remember. The first problem is where do we start, what class should I start to create to use the API, consider the CinderLib API. We can say there is a long list of classes there, but where do I start, do I start to create one of these classes? Which classes can I then use after creating one of these classes? If I’m just limiting myself to look at the API here on the page with all the classes, its really difficult to know how to get started programming with this. Ok lets say I make some good guesses and manage to get started, its not at all clear how to do certain things, the grammer of the relationships between classes, functions and objects are not at all clear. So then I must resort to different methods.

Usually to use an OO API I look at example code and modify the example code to fit my need as I don’t know the magical incantations to get things started and cannot remember all the classes involved. For the magical incantations, what classes should I start with and then what can I do from there, its rarely clear or obvious. For the person who designed the API it is clear and simple, they know the grammar of their objects but how long do they expect me to read documentation to “get” their grammer. For memory, preferably I do not wish to remember all the classes, how can I remember all the classes for all the API’s I use, its really impossible, there’s so many API’s, JQuery, HTML5, NodeJS, … and all the forgotten ones I no longer use such as ActiveX, MFC.

So there is a problem of what is the grammar “In linguistics, grammar is the set of structural rules that governs the composition of clauses, phrases, and words in any given natural language”, what are the relationships between classes and how do I use them to accomplish a goal.

Then I can argue that a flat functional API is easier to understand and use. JQuery is such an API, JQuery doesn’t rely on many classes, there is just one large page of the API. Lets say I want to hide something on the page, all I need to do is search this page for the word “hide” and I find .fadeOut() which can be applied to a JQuery DOM node. Now if this was a Object Oriented API there would be many classes and I would have to work out which class to use and which function to use with what other classes.

Perhaps the problem is that OO API’s hide information within them, they encapsulate their own API within many different classes, its difficult to work out which class should work with what class and where to start. I would like to suggest that functions and global state are easier to understand and develop with than OO API’s when the API’s become large.

Getting started with Foundation 4 and Yeoman

Written by philip on April 13, 2013

Yeoman is a tool for managing a web development project, it allows for the installation of client side CSS and Javascript libraries, basically an easy way to get started building a website and tools to manage the dependancies of javascript, css, and any libraries.

So this Yeoman npm package works just as you would expect, it installs Foundation 4 into a Yeoman project. The interesting thing is it uses Stylus CSS to allow modification of the CSS. It has a settings.styl which has many overrideable styles, for example.

// We use these to control header font styles
// $header-font-family ?= "Helvetica Neue", "Helvetica", Helvetica, Arial, sans-serif;
// $header-font-weight ?= bold;
// $header-font-style ?= normal;
// $header-font-color ?= #222;
// $header-line-height ?= 1.4;
// $header-top-margin ?= .2em;
// $header-bottom-margin ?= .5em;
// $header-text-rendering ?= optimizeLegibility;

By uncommenting and modifying these styles the foundation styles can be overridden easily. One problem people complained about with boostrap was that it was difficult to override the default styles, although this is not bootstrap, its good to know it has support for easy adjustment of styles.

Idris programming: 1 Getting Started

Written by philip on March 21, 2013

Idris is a relatively new computer language which uses dependant types, I don’t actually know what that means at the moment except that it potentially allows for more errors to be detected at compile time rather than at run time. Its good to find errors before a program runs and if the dependant types makes it easier and does not make programming much harder then I’m for that.

Idris can also compile to Javascript or C language which means it can work in the browser or on the PC. To install Idris you first need to install Haskell then to following the installation instructions on the Idris page. The reason the Haskell platform is involved is that the Idris language is implemented in Haskell.

Here I show how to use Idris from a very simple basic programmers point of view, not showing off the parts of Idris which involve complexity as I do not understand those parts yet. So taking it step by step.

Printing strings on the console in a main method.

module Main

main : IO ()
main = putStrLn "Hello world"

To run this, place the above text into a file called test.idr do the following on the command line.

$ idris test.idr -o test
$ ./test
Hello world

As you can see it outputs Hello world as you would expect.

The following is a simple example of appending strings.

"Hello " ++ "there"

The next example is a main program outputting string and inputting string and concating strings, place it in the file test.idr, compile and run.

module Main

main : IO ()
main = do putStr "What is your name? "
          name <- getLine           
          putStrLn ("Hello " ++ name)

To print a List of integers, the line inserted is putStrLn (show testIt)

module Main

testIt : List Int
testIt = [1,2,3]

main : IO ()
main = do putStr "The list contains "
          putStrLn (show testIt)
          putStrLn "Your name?"
          name <- getLine
          putStrLn ("Hello " ++ name)

Ok so we can do some basic operations that you would expect with a programming language, inputting, outputting strings and printing a data structure. Next post we do some basic operations on lists.


Written by philip on March 19, 2013

Lo-Dash is worthwhile, at 19kb on the disk when minified, that is reasonable.

So your going to write a program in Javascript and you want to keep it small I suggest to use LiveScript with Prelude, Lo-Dash rather than Underscore.js, BackboneJS, well I’m saying you want to keep it small, I’m not saying use this if you want to have the best environment to program in. You may want to keep it small to work well with mobile devices with slower download rates.

If you don’t care about the size of the Javascript then I suggest go with something else.