Inspiration
The financial markets are heavily influenced by not just numerical data but also investor sentiment, news, and other unstructured signals. By combining deep-learning with sentiment analysis, we aim to bridge the gap between quantitative time-series modelling and the qualitative side of market behaviour — turning the “noise” of news & social sentiment into actionable forecasting insight.
What it does
Sentiment Forecast (Sentiment-Aware Stock Forecasting) is a full-stack AI application that:
Predicts next-day stock returns by combining historical market data with sentiment signals (e.g., news, social media).
Uses state-of-the-art models (e.g., Transformer architectures + domain-specific sentiment analysis via models like FinBERT).
Provides an interactive UI built with Dash for visualising predictions, back-testing metrics (e.g., CAGR, Sharpe ratio, MSE) and model explainability (e.g., via SHAP).
Is wrapped into a deployable architecture (for example via FastAPI for API endpoints) and hosted on services like Render for full-stack deployment.
How we built it
Data ingestion & preprocessing: Collected historical stock data (open/close/high/low/volume), computed technical indicators, and extracted sentiment features using FinBERT (or another financial-language sentiment model) from news and social text.
Model architecture: Built a Transformer-based model that ingests both the time-series features and the sentiment embeddings to predict next-day returns.
Back-testing & metrics: Created a back-testing module to compute standard performance metrics like CAGR (Compound Annual Growth Rate), Sharpe ratio, and MSE (Mean Squared Error) for model outputs versus actual returns.
Explainability layer: Applied SHAP to get feature-attribution insights (e.g., how much did sentiment contribute to a specific prediction) and surface them in the UI.
UI/UX layer: Developed a Dash dashboard for interactive plots: e.g., predicted vs actual returns, cumulative return curves, feature importance plots.
Deployment: Wrapped the predictive model and dashboard in a FastAPI server, containerised (or otherwise packaged) and deployed via Render (or similar cloud platform).
CI/CD & monitoring: (Optional) Set up monitoring for data drift, model performance decay, and periodic retraining pipelines.
Challenges we ran into
Data alignment & latency: Aligning sentiment features (which could arrive at different time-points) with market data for next-day prediction was tricky (e.g., news at 3 pm vs market close at 4 pm).
Feature engineering for sentiment: Raw sentiment scores (even from FinBERT) don’t always correlate cleanly with returns, so deriving meaningful sentiment features (e.g., sentiment momentum, cross-company spillovers) required experimentation.
Overfitting & market non-stationarity: Stock markets are non-stationary; what worked historically may not work tomorrow. Ensuring generalisation (vs simply fitting the training period) was a challenge.
Explainability vs complexity: Deep models (Transformers) can be black boxes; balancing performance with interpretability (via SHAP) took extra engineering.
Deployment & scaling: Running heavy models on cloud resources while ensuring low latency and cost-effectiveness (especially for real-time inference) required trade-offs (e.g., model size, batching).
Back-testing pitfalls: Ensuring realistic trading assumptions (e.g., transaction costs, slippage, look-ahead bias) was important for credible performance metrics.
Accomplishments that we’re proud of
Built a fully end-to-end system from data ingestion → modelling → dashboard → deployment.
Achieved meaningful predictive performance (for example: improved MSE or Sharpe ratio relative to baseline models that ignore sentiment).
Incorporated explainability: users can view why the model made a given prediction (e.g., “Sentiment spike for Company X contributed +0.3% expected return”).
Created a clean, interactive UI in Dash that non-technical stakeholders (traders, analysts) can explore.
Deployed to a live environment (Render) so the solution is accessible and scalable.
What we learned
Sentiment is not a silver bullet: while it adds value, the signal is noisy and requires careful feature engineering (e.g., lagging, smoothing, context filtering).
Model robustness is paramount: given the instability of financial markets, models must be stress-tested for regime changes and periods of high volatility.
The user interface matters: presenting predictions is one thing; presenting them in a way that builds trust (via explainability, transparency) is equally important.
Deployment & maintenance are as important as modelling: building a model is the start — operationalising it (monitoring, retraining, cost-control) is a major part of the work.
Interdisciplinary integration (finance + NLP + ML + devops) is key — you can’t treat each piece in isolation.
What’s next
Expand feature universe: Incorporate additional sentiment sources (e.g., Reddit, Twitter, alternative data), or peer-company relationships (company A’s news affecting company B) to capture spillovers.
Real-time streaming: Move from daily predictions to intraday (or hourly) predictions, where latency becomes more critical.
Portfolio construction: Extend from predicting single-stock returns to constructing an adaptive portfolio based on model signals (e.g., ranking stocks by expected return, adjusting weights dynamically).
Risk modelling: Add risk-adjusted decision layers (e.g., drawdown constraints, tail-risk hedging) and scenario-analysis (what if a major news event hits?).
Model ensemble & meta-learning: Combine the Transformer + sentiment model with other model types (e.g., Random Forest, TCN) or use meta-learners that decide which model to trust under which market regime.
Production-grade monitoring & retraining: Implement data-drift detection, model-drift alerts, auto-retraining pipelines, and dashboards for operational metrics (latency, throughput, cost).
User interface enhancements: Add features like “what-if” simulation (if sentiment for company X changes by +0.5 standard deviations what is the predicted return?), alerts/notifications for large sentiment deviations, and mobile-friendly UI.
Explainability upgrades: Explore more advanced explanation systems (e.g., counterfactual explanations: “what minimum change in sentiment would flip the prediction?”).
Live pilot or A/B testing: If possible, deploy a live pilot with real capital (or paper trading) to validate model predictions in a live market environment and refine based on real-world outcomes.
Log in or sign up for Devpost to join the conversation.