Leadership

Leadership - yet another commonly used  but poorly understood concept despite the vast amount of literature available on this topic. The one definition that I follow is what Partha Ghosh , visiting lecturer at MIT taught me - True leaders are the servants of a cause. There cannot be true leadership without a worthwhile cause. Once you dedicate yourself to a worthwhile cause , leadership pivots around "enlightenment" , not "entitlement" . True leaders are often silent - they bring about transformation without much fanfare or hoopla .  

Espoused process vs Process in use

Managers beware . Processes that are espoused may not be the  processes in use.
Several  academics have mentioned this phenomenon.  Simply put, what is actually   practised may  be different from  what is believed to be getting practised. Most common cause of this is  processes evolve since their  initial rollout  but the evolution is rarely documented. Structural changes such as reorgs , attrition , new hiring also contribute to the widening of  the gap. Nothing beats experience - going through the process hands on is the best  in order to understand what really is going on . Next best is  first hand  observation of the process  in use.

Predictive Analytics

Predictive analytics is  hot.  Advances in hardware,statistics and business intelligence software  have made it usable and performant. Predictive analytics , as the name suggests helps gain business intelligence from data using various data mining, pattern recognition and probablistic algorithms. Consider the following example - Given the history of orders for a product , how likely is a customer belonging to a certain age group to buy a certain product ? The answer can be computed in several ways.

1.Find out clusters of customers who bought the product. Find out the distance of the cluster from this customer to indicate the likelihood of buying.
2.Use regression analysis.   Y = a1*X1 + a2*X2 .....
where Y  represents  revenue for a product and X1,X2,X3  could represent causal factors such as age,geography etc.a1,a2,a3 are the coefficients. If age is statistically signigicant, the coefficient will have a significant value. Once age is confirmed to be statistically significant, we could have multiple causal variables
for each age bracket and then find out which among those is the most significant.

Oracle offers predictive analytics at several layers. PL/SQL/JAVA  comes with an API for predictive analytics called  DBMS_PREDICTIVE_ANALYTICS . Oracle also has a product called RTD - Real time decision making that is bundled with OBIEE - Oracle business intelligence enterprise edition.
Other tools like crystall ball, excel add on for predictive analytics  and  Oracle data mining are some other tools in Oracle's arsenal . Oracle Demantra  provide predictive analytics related to forecasting and demand management. With IBM having acquired SPSS  , the industry's  landscape has become interesting.

Academic publications like "Competing on Analytics" by Harvard Press  and the recent survey published in

MIT's Sloan Management Review on BI trends have contributed to the hieghtened   interest and investment in this upcoming discipline. Companies like Netflix have grown to a billion dollar by predicting  consumer buying patterns based on  "clicks".

Correlation vs Causality

I was in Chennai a few years ago at a conference sitting atop the terrace of a restaurant with a few colleagues. The sales of cold drinks were at an all time high with everybody ordering coke ,beer etc.
At the same time, I noticed that there was a huge influx of patients at the hospital  near the restaurant .
So there must have been a positive correlation between the data for the sale of cold drinks and the data for the inflow of patients to the hospital , meaning as one went up or down the other went up or down too.
However does it mean that the  cold drinks caused people to get hospitalized ? Or vice versa - did people
drink because someone got hospitalized ?
None of the above was entirely correct in this situation. The reality was , the influx of people to the hospital and the sales of cold drinks were  caused by the sweltering heat of Chennai.. So if we were to forecast the sale of cold drinks ,  the causal factor would be  "temperature"  and not the "number of people admitted to the nearby hospital" . 
And this precisely  is one of the key  things to watch out for in analyzing the results of  regression analysis. While regression will give you correlation between 2 variables  , it may require an expert to confirm  if there is causality between the two.

The dangers of serial thinking

I bought a horse for 10$ and sold it to a friend for 20$. I bought the same horse from the same person for 30$ and sold it to the same person for 40$ . What was my profit ?

If you came up with 10$ as the answer  (20-10) + (20-30) + (40-30)  , you fell into the trap of serial thinking.

If you came up with 20$ as the answer , you are right because these two are separate transactions.

Transaction 1 - You bought and sold the horse. Profit = 20 - 10 = 10$
Transaction 2 - You bought and sold the horse. Profit = 40-30 = 10$

Total profit = 10 + 10 = 20$

If  you are financially savvy  , you would do

profit = total cash inflow - total cash outflow
         = (40 + 20) - ( 30 + 10)
        = 60 - 40
        = 20$

Beware  the dangers of serial thinking .
Start  thinking laterally.

The three best practices in software development

Michael Cusumano has documented the three most effective practices in software development  in his

book  "The Business of Software". These three practices have resulted in significant defect reductions in large companies.While  Six Sigma, Lean, CMMI and other methodologies can be applied to software and have been effective with varying degrees  , the following three practices have to be   implemented. All other methodologies and practices would be add-ons. So,  what are the three key practices ?.

1.Early prototyping -

Do a proof of concept by prototyping early and show it to the users.

Seeing is believing. When the users see something, they can spot obvious flaws and limitations.  

2.Reviews,reviews,reviews

We need reviews at each stage - High level design reviews ,detailed design review, code reviews, unit test cases reviews, system test reviews , project plan reviews etc. Review provides a negative feedback mechanism and hence stabilizes the output, to give an analogy of a closed look negative feedback control system.  In a control system with a negative feedback, the output Y = X * (G/1 + GH) .

X is the input, G is the gain/amplification/distortion  and H is the amount of feedback

In a  large project , G can be assumed to  be large and H close to 1 . Hence  GH  can be assumed to be much greater 

than 1. Thus Y = X * (G/GH) = X/H

Now if the review is 100% perfect , H =1 . Therefore,  Y =X .

Thus the output of the software stage equals the inputs which is the   specification to that stage.

In essence the output of the software development   will follow the specifications accurately if there are

rigorous reviews.

Linus Torvalds , the inventor of Linux said " Every bug appears shallow if a large number of eyes look at the code". How true ! No wonder why Linux is such a robust operating system despite being open source.

Millions of eyes look at each line of code each day  and suggest improvements.  So the next time you want to review code, call all developers in a room  and project their code . Let them all look at it and have fun dissecting it in a cordial  and friendly environment.

3.Daily regression tests and builds.

The code must be compiled and linked daily.

Automated regression tests must be run daily - the more the better and emails send to all concerned including

senior management when RT's fail. This ensures that as new features are added, existing features continue to work.

Bayes theorem

Bayes theorem is the foundation of Bayesian statistics and the new and emerging discipline of  predictive analytics. Reverend Bayes , a 17th century British mathematician wrote the Bayes theorem as a special case of probability theory. He did not disclose the theorem  fearing that it might not pass rigorous scientific scruitiny. The theorem was recovered after his death .
Here's the theorem and  a  few practical applications.

P(Ri/E) = p(E/Ri) p(Ri) / sum [p(E/Ri) * p(Ri)] i = 1 to n

p(Ri) is called anterior probability  of event Ri. It represents what we already know from past history about event Ri..E is the fresh new evidence that has arrived that would influence the event Ri.

p(Ri/E) is the probability of event Ri given the evidence E.
p(E/Ri) is the likelihood of the evidence itself.

As can be seen above as new evidence surfaces, the theorem lets us update our knowledge about the probability of occurrence of the event Ri. p(Ri) was our knowledge of the event based on its history,
while p(Ri/E) is our updated knowledge after taking the evidence E into consideration.

These probabilities can be deterministic or may represent a certain distribution.
Here are a few practical examples  where the theroem could be applied.
(An example similar to the first example below was  cited in a recent issue of Sloan Management Review).

1.I know the history of rainfall in my region for the past 10 years. I now have the evidence that this year
the temperatures were higher than normal. We know that higher temperatures correspond to higher rainfall
and follows a certain distribution. Using this evidence, we can predict the probability of rains this year  .
If I calculated probability based on history alone (which is what frequentists would do)  , I would be ignorning the key evidence that surfaced this year.

2. We know the history of deliveries of a certain supplier. A new evidence has arrived that the supplier's capacity is full due to a new contract they have signed with some customer. We know the relationship between the supplier's capacity and on time delivery . Given the evidence, we can find out the probability of on time delivery this month. If we had not used Bayes theorem, (and instead relied  on history alone as a frequentist would do)  the probability would have been based solely on the supplier's  historical performance and hence would have ignored the critical new evidence.

Bayesian theorem does have certain limitations/gotchas  in practical life as would be seen in future blogs.