Understanding Monty Hall, and learning Python

Finally back with a post after a long hiatus!

I've finished the Masters that has been sucking the hours out of my life for the last 2 years and suddenly realized that I needed to add some better programming skills to my CV. I started doing the excellent R and interactive Python courses on Coursera. They give a good introduction and provide challenging assignments to help students build their knowledge.

I decided to code up some of the algorithms and concepts I've learned over the last 2 years to  improve my understanding of them and to develop my skills at coding. The first concept I decided to tackle was that of variable change, best captured in the Monty Hall problem.

The Monty Hall problem is a great example of where intuition is at odds with statistics. The player has to guess which of 3 doors of a prize is behind. Upon selection the game show host removes or opens a door that you haven't selected. He then offers you the choice to stick with the door you chose originally or switch to the remaining door.

The most common response is that it shouldn't make any difference; You chose 1 door out of 3 (33% chance) and now there are only 2 doors left so the odds are now 50/50. You may even feel that your odds have improved. So sticking with your initial choice should be as good a choice as switching to the other door.
This is actually wrong! It took me a long time to understand it and the reason is that we get very tied up thinking about winning. Instead lets think about losing. When the game started you had a 2 in 3 chance of picking the wrong door, which means you probably did. Since the other losing door has now been removed it makes the most sense to switch away from the initial (and likely wrong) door you chose in the first place.

Statistical evidence shows that contestants that switch from their initial choice win 2 out of every 3 times. This is what I wanted to code up and demonstrate with python. The table below demonstrates the results I collected over several runs with the python code I developed.

I've included the code snipped below which you can paste into Python or save as a .py and execute.
Appreciate any feedback either if the post was helpful to you, or if you have any improvements to suggest.

<CODE>
import random
import math

def setPrize():


    tmp=[0,0,0]
    door=random.randrange(0,3)

    tmp[door] = 1
    tmp[((door+1) % 3)]=0
    tmp[((door+2) % 3)]=0
       
    return tmp
prize=[0,0,0]
finalDoorz = [0,0]
n=100
   


switchnwin=0.000000
switchnlose=0.000000
sticknwin=0.000000
sticknlose=0.000000


prize = setPrize()

for i in range(0,n):
    prize = setPrize()
    #Player chooses their first door
    choice = random.randint(0,2)
   

    #We put the players choice in position 0
    if prize[choice]==1:
       finalDoorz[0]=1  
       finalDoorz[1]=0        
    elif prize[choice]==0:
       finalDoorz[0]=0  
       finalDoorz[1]=1


    #Second part; stick or switch; 0 stays with the users choice, 1 switches it.
    choiceb= random.randint(0,1)

    if finalDoorz[choiceb]==1:
        status="WIN"
    else:
        status="LOSE"

    if choiceb==0:
        switched="NO"
    else:
        switched="YES"

    if finalDoorz[choiceb]==1 and choiceb==0:
        sticknwin +=1
    elif finalDoorz[choiceb]==1 and choiceb==1:
        switchnwin +=1
    elif finalDoorz[choiceb]==0 and choiceb==1:
        switchnlose +=1
    elif finalDoorz[choiceb]==0 and choiceb==0:
        sticknlose +=1

print "Number of Observations: " + str(n)
print "Switch and Win: " + str(switchnwin/n)
print "Stick and Win: " + str(sticknwin/n)
print "Switch and Lose: " + str(switchnlose/n)
print "Stick and lose: " + str(sticknlose/n)
</CODE>

Revenues Property Evaulations

Its been a while since I've posted anything as life has been very quiet because of college, However I just had to get up on my high horse about Revenues new Local Property Tax tool and property evaulations

The Tool
First of all I want to divorce the tool from the content and say that it is great to see them using maps and visualization to get their message across. This sort of thing has been common in the mash up community for some time; overlaying statistics or information on a searchable map. I would question their choice of colours as it makes it very difficult to easily associate a particular hue with its corresponding band, but there is an option to click on a zone and get the relevant information.

The Content
I really am jumping on the bandwagon with the criticism of the valuation process but when I look at the area I grew up in it seems to have a blanket band of €350,000, which is actually Boom prices, not the reality we have 5 years on.
With resources like the property price register (which is a publicly available legal record of the price a house is transacted at over the last 3 years) I'm surprised that they didn't use this information to get a better sense of likely values. In addition there is a wide degree of variation within the dwelling types.

From a quick check on the PPR I see prices of closer to the €250,000 mark being transacted in the recent past. I also notice that the bound seems to defined by the voting district, which to my mind doesn't adequately describe or cluster similar properties together. I suspect if this was done we would wind up with many more, smaller zones.

What could be done to improve the valuations.
Given that we know the closing prices on recently transacted properties and there should be some record somewhere (land commission, planning permission, county council) of each property. This could be cross referenced with some key variables of property type like number of rooms, stories, Square footage to construct a more granular model of the accommodation stock. A clustering algorithm could break down areas into smaller more similar subgroups, which would make valuations more relevant.

What homeowners can do to argue their case
If your house was transacted in the last 2-3 years then you have a very strong basis on which to argue your band, unless you feel that there has been some serious negative equity.
For others the most obvious thing is to look at the property price register over the last 3 years and spot properties that are nearby. It would also be helpful to work out how those houses are the similar or different to your own.
In addition, checking out the prices on DAFT for nearby properties will also help you work out a fairer market price.
There seems to be significant scope for people to argue the valuation of the revenue commissioners  but this also means a lot of work for the commissioners to investigate and respond.
Hopefully they will publish an updated set of valuations before next year so that we can see a more realistic picture of property price declines.

Breaking the ice with Twitter

I was at a tech meet up recently and being my first time there, I didn't know anyone.
I'm not great at networking and I was sitting in a pub, not knowing where the group was.
I sent up some tweets with right hash tags, and voila; several people responded almost immediately!
Twitter as an icebreaker for shy people!

The Facebook IPO

After all the hype Facebook IPO'd yesterday at $38 a share.

Despite the lackluster price increase, this was technically a successful IPO. In fact it might have been one of the best run IPO's in a long time. This sounds like a strange statement and to be honest I had thought "what a wash out!" but after reading an article on cnet this morning it reminded me that the purpose of an IPO is not to immediately raise the wealth of the people that are buying into the company. It is to raise capital for the company.

The company wants to raise as much money as possible by offering its share. Thus it need to offer them at a price that investors think is reasonable. If the price is too low then people will buy the IPO shares and then sell them very shortly afterwards at a profit.

If the price is too high investors will be reluctant to buy at all, which means the company won't raise the capital it needs; and the share price on the day will collapse below its ipo price - quite embarrassing!

Facebook launched at $38 and closed at $38.23, which means that the company foretasted the amount it could raise to about 99% accuracy. Thus this was a success for them.

The next few days and weeks will tell the true story as to whether the IPO price was good for investors, and this could have repercussions for the reputation of Facebook as a vehicle for investment. If the price begins to sag then people will feel duped and may try to sell off their shares to avoid further losses, further depressing the price for the stock. 

The cnet article also mentions that underwriters intervened to keep the stock at $38, however their pockets aren't limitless and as soon as they stop supporting Facebook we could see a rapid decline in price.

My guess (and this is not based on any maths or deep investment analysis) is that the underwriters will stop supporting late next week. The price will tip forward and back between 37 and 39 over the week and then begin a gradual decline to some lower equilibrium price (probably somewhere in the mid 25's) until the first set of quarterly trading statements. At that point we'll start to get a picture of what is really going on.

For me I don't really see the value. 1bn users who don't pay a thing for the base product.

I heard before (source unknown) that they reckoned that each user was worth $6 annually, but that means each user has to generate $6 of revenue each year.

I can't imagine that clicking on one ad generates $6 in one go so they would probably have to click on at least 10 ads. I've never clicked on one ad.

Another source of revenue might be sharing user data to analytics companies but this is fraught with privacy issues. Upsetting your users might drive them away, or certainly reduce the amount of data that they share.

Bottom line: Facebook are going to have to do a lot to justify that $38 a share.

Exams - That Electric Feeling!

Well I'm back! My exams are over so hopefully I'll be doing more posting!!

One odd thing I have noticed. In the two weeks running up to the exams I was getting regular static shocks; When I stood up, when I got into or out of the car, when I kissed the LAK (to her annoyance!) But all of a sudden they have just disappeared!

Anyone any ideas?

Colour Blindness strikes again!!

So part of data mining is clustering, and one of the ways to visualize clusters in the application (Rapid Miner) is through color coding as in the below:

To me, Clusters 0, 4 and 6 appear the same. On closer inspection cluster_0 is different but I have to zoom in.
I really hope that the application has some way of showing these as symbols!

Welfords One Pass Agorithm

Howdy!
I've continued to be quiet recently as I am back at college so a lot of my time is taken trying to understand the material.

This semester I am taking Simulation and Data Mining. The maths is a lot heavier that last semester. Most of it is covered in the form of proofs in the Simulation course. In addition I am re-learning Java programming, pretty much from scratch so when as I understand the maths bits I then try to code it up in Java or implement it in an assignment solution.

We learned about Welford's One Pass Algorithm during the week and it turns out that it should be implemented in an assignment.The algorithm is an efficient way of calculating mean, variance and standard deviation for a sample on the fly.Normally if we were working on paper or in excel we would store the data first and then run back through it, calculating the individual parts of the equation first and then adding those up to get the final answers. This algorithm allows us to compute mean and variance as we iterate through a loop.
The real value from a computer resource point of view is that we don't have the memory overhead of storing the individual bits of the formula first for calculation after.

 I wasted a lot of time searching the net yesterday in a hungover state trying to find a way to code it up, as I thought that one of the variables was dependent on the the other two and so would not compute properly. Hopefully this blog entry will help anyone searching for it in plain English.

Standard equations
The fomula for the mean is:

The sum of the individual observations divided by n, the number of observations.

The formula for the variance is:

Standard deviation is simply the square root of s above.


As you can imagine in the above you would have to store values first, then send them through a loop to store the individual differences, and then finally do aggregated calculation. Very costly in terms of memory and code.

Welfords One Pass Algorithm
Welfords algorithm gets around this by calculating the minor changes on the fly. This removes the need to store data and simply looks at the delta's.

The algorithm is below; I have used a general notation which you can apply to whatever language you need.
int n = 0;
double xbar = 0.0; //mean
double var = 0.0; //Variance
double x = 0.0; //Next Observation
double d = 0.0; // Difference

For every observation:
{
x = getNextData();
n++;
d = x - xbar;
v = v + (d^2);
xbar = xbar + d/n;
}

v = v/n ***

*** I made this correction after trying to run this algorithm a second time on a different machine.

One problem I had was that if you are working through an array, the first array index is 0, which caused the entire equation to return NaN; but when I added 1 to it to change the indexing it seemed to work.


You can also calculate the covariance quite easily as well
Covariance is simply (the sum of (x*y) - (the sum of x times the sum of y) / n)/n-1
You can collect x*y as you iterate through the loop
sum of X times sum y can be calculated after the fact!