Financial Crisis and Big Data

The progression of Cathy O’Neil from academia to WMD disillusionment was no quick event. Beginning with her time at the hedgefund D.E. Shaw, O’Neil was able to get hands on with the mathematical models which run the world’s financial markets. While knowing mathematics was at work behind the scenes, she didn’t quite realize the ways in which it was applied. While working at Shaw, O’Neil began to notice issues with the risk assessment of mortgage backed securities. The mathematical model would clump various mortgages into packages which could be slapped with a single risk assessment and sold. She noticed the models themselves seem to neglect the individual people they were designed to represent. Alongside this was a trend for an event never before experienced. Having models trained to make predictions based on past data would prove inaccurate when encountering unexpected data. This trend coupled with negligence on the side of companies in regards to acknowledging and handling the risks associated with the mortgage backed securities led to the instability and eventual collapse of the financial market. Many other markets had based their foundation on the housing market assuming the companies who owned those investments had taken the proper avenues and accurately assessed their risk. It is no surprise then that when the housing market fell, the markets which had built their foundation on top of it collapsed as well. It was then that O’Neil would move on to Big Data and find the same recurring pattern of inaccurate use of algorithms leading to her disillusionment and desire to investigate the topic. While reading this chapter, I realized how much I had underestimated the responsibility taken by these companies to ensure they are using their algorithms ethically. These types of algorithms are capable of great power and they should be maintained and monitored to ensure use that is beneficial to humanity and not the pocketbooks of those who seek to profit.