Using NLP to extract sentiment from financial news and predict stock returns
This project applies FinBERT — a BERT model fine-tuned on financial text — to analyze sentiment in 20,550 news articles (2017–2020) and investigate its predictive power for stock returns. A fine-tuned neural network is compared against FinBERT's pretrained sentiment scores.
Both the pretrained FinBERT and the fine-tuned model achieve roughly 50% accuracy on binary return prediction — consistent with the efficient market hypothesis. FinBERT's monthly sentiment shows a negative correlation (ρ = −0.32) with next-month S&P 500 returns, though statistically insignificant (p = 0.31).
The pipeline has three stages: (1) FinBERT scores each article as positive/negative/neutral; (2) BERT base embeddings (768-d CLS vectors) feed a 3-layer neural network trained to predict next-day returns; (3) monthly aggregated sentiment is regressed against S&P 500 returns via OLS.
Data is split chronologically: training on pre-2019 articles, validation on 2019, and testing on 2020+, avoiding look-ahead bias.