Exploring Pearson Correlation in the World of Finance: A Deeper Understanding

 

Hello there!

 

As I embark on this blogging journey, I've decided to go with the flow and see where it takes me. Today, it struck me that a great idea for my first post could be about correlation - it's relatively simple, frequently used, and, I believe, often overinterpreted.

 Let's start from scratch, avoiding getting bogged down in details or formulas. We'll begin with some code action. Let's examine the correlations of a few popular financial instruments.

 I selected several instruments and downloaded their data from the past three years using yfinance. After transforming this data into a DataFrame (DF), everything seems set for correlation analysis. But have we chosen the right data to start these calculations? At first, it seems obvious - we're looking at closing prices. Let's generate some results.

code in which we take data , create df and present correlation matrix

results from the presentation of the first five rows of our df and correlation matrix

heat map of our correlation matrix

Correlation: Simple, But Is It Clear?

We started with a straightforward question: how are the closing prices of different assets related to each other? Seems simple, right? But as we dig deeper, the issue of time series stationarity emerges. Are our data stable over time, or do their characteristics change?

 Stationarity refers to the constancy of statistics such as mean, variance, and covariance in a time series. If a series is non-stationary, we might fall into the trap of interpreting correlation as a meaningful relationship when, in fact, it is coincidental.

ADF test

The Augmented Dickey-Fuller (ADF) Test: The Key to Stationarity

To tackle this, we employed the ADF test. This test checks whether a time series is non-stationary by looking for a unit root. Practically speaking, if the test result (result[0]) is low and the p-value (result[1]) is below 0.05, we can consider the series to be stationary.

 Case in Point: Gold

ADF test for gold

Let's see how this works in practice. For gold (Gold):

 ADF Statistic: -1.871027

p-value: 0.345811

Critical Values: -3.439 (1%), -2.865 (5%), -2.569 (10%)

The interpretation is straightforward: gold appears to be non-stationary, which means we need to be cautious in interpreting its correlation with other assets.

 Differencing: Our Plan B

When we find a time series to be non-stationary, we can apply differencing, i.e., calculate the percentage changes in values. This often helps to stabilize the series, making it more suitable for correlation analysis.

code for percentage change correlation matrix

results of correlation matrix from closing prices and percentage changes

You can already see that the change we have made has brought a change in results .

heat map of our correlation matrix

Pearson Correlation: Not Just Numbers

We calculated the Pearson correlation coefficients, which measure the strength and direction of the linear relationship between two variables. Interestingly, Nasdaq and S&P show a strong correlation. This makes sense as both indices often respond to similar market factors.

 What Next?

Remember, correlation does not imply causation. A high correlation between two assets doesn't necessarily mean one influences the other. It’s an important lesson on our path to deeper financial understanding.

 Wrapping Up

In this post, we dived into the intriguing world of correlation analysis, paying attention to the importance of stationarity and methods of interpreting results. This is just the beginning of our journey. More explorations, questions, and discoveries await.

 Thank you for joining me in this episode. I invite you to engage in the discussion, ask questions, and share your own experiences!