The financial industry has undergone a major transformation with the growth of digital data, computing power, and machine learning. Traditional investment decisions were largely based on financial statements, economic indicators, analyst research, company reports, and historical market information. Today, investors can access a much broader range of information generated through smartphones, websites, social media, commercial transactions, satellites, sensors, and other digital systems.
“Big Data and AI Strategies: Machine Learning and Alternative Data Approach to Investing” is a comprehensive 2017 research report from J.P. Morgan's Quantitative and Derivatives Strategy team, authored by Marko Kolanovic and Rajesh T. Krishnamachari, with additional contributors. The report examines how Big Data, alternative data, Machine Learning, and Artificial Intelligence can be incorporated into investment research and quantitative strategies.
The report is particularly interesting because it does not discuss machine learning only as a technology. Instead, it examines how data and machine-learning techniques can potentially create new information advantages for investors.
The Rise of Big Data in Investing
One of the central ideas of the report is that the investment industry is moving toward a world where enormous amounts of information are generated digitally.
Traditional economic and financial information is often released at specific intervals. For example, investors may receive economic statistics monthly or company results quarterly.
Digital data can provide information much more frequently.
Examples discussed in the report include:
Online product prices
Consumer activity
Social-media information
Commercial transactions
Satellite imagery
Mobile-phone data
Shipping information
Web-based information
Sensor-generated data
This creates the possibility of observing economic activity much closer to the time it actually happens.
Download the PDF for free:
https://cpb-us-e2.wpmucdn.com/faculty.sites.uci.edu/dist/2/51/files/2018/05/JPM-2017-MachineLearningInvestments.pdf
What Is Alternative Data?
Alternative data refers broadly to information outside the traditional datasets normally used by investors.
Instead of relying only on company reports and conventional economic statistics, investors can examine information generated by digital activities and real-world systems.
The report organizes alternative data into several broad categories.
Major categories include:
Data generated by individuals
Data generated by businesses
Data generated by machines and sensors
Data aggregators
Technology providers
This classification is important because different datasets can provide different types of investment information.
For example, social-media activity may provide insight into consumer sentiment, while satellite imagery may provide information about physical economic activity.
Data Generated by Individuals
People generate enormous quantities of digital information through their everyday activities.
Examples include:
Social-media activity
Mobile-phone activity
Online searches
Reviews
Web browsing
Consumer behavior
Location-related information
For investors, these datasets can potentially provide information about consumer preferences, sentiment, demand, and behavior.
The important idea is that individual activity can become an economic signal when aggregated and analyzed appropriately.
Data Generated by Business Processes
Businesses also produce large amounts of information as part of their normal operations.
Examples include:
Commercial transactions
Credit-card activity
Retail information
Online sales
Supply-chain information
Shipping activity
Corporate operational data
Such information can sometimes provide a more timely view of business activity than traditional financial reporting.
For example, transaction information could potentially provide an indication of changes in consumer spending before those changes appear in conventional financial reports.
Data Generated by Machines and Sensors
Modern machines continuously generate information.
Satellites, cameras, industrial sensors, connected devices, vehicles, and other systems can generate large quantities of data.
The report highlights satellite imagery as one example of how machine-generated data can be applied to investment research. Satellite observations could potentially provide information about areas such as:
Agricultural activity
Industrial facilities
Oil infrastructure
Shipping
Construction
Physical economic activity
This demonstrates an important shift in investment research: investors can increasingly analyze the physical world through digital information.
Why Alternative Data Can Be Valuable
Alternative data is valuable when it provides information that is:
Relevant
Timely
Difficult to obtain
Difficult to replicate
Predictive
Cost-effective
However, simply having a large dataset does not automatically create an investment advantage.
The data must contain useful information, and investors must be able to process it correctly.
The report emphasizes that the potential value of alternative datasets must be considered alongside the cost of acquiring and implementing them.
Machine Learning as a Tool for Investors
Large datasets are often too complex to analyze effectively using traditional manual approaches.
This is where Machine Learning becomes important.
Machine-learning systems can process large datasets and identify patterns that may be difficult for humans to discover manually.
The report examines several categories of machine-learning techniques, including supervised learning, unsupervised learning, deep learning, and reinforcement learning.
Supervised Machine Learning
Supervised learning is based on historical examples where the desired outcome is known.
The system learns relationships between available information and an outcome of interest.
In investing, supervised learning can be used for tasks such as:
Prediction
Classification
Signal generation
Risk analysis
Financial forecasting
Pattern recognition
The report discusses regression and classification as major supervised-learning approaches.
The advantage is that the model can learn from historical relationships and use those relationships to make predictions on new observations.
Regression-Based Approaches
Regression is one of the traditional statistical techniques that can be used for prediction.
In an investment context, regression-based approaches can help analyze relationships between financial variables and potential outcomes.
They can be used for:
Forecasting
Identifying relationships
Estimating financial variables
Building predictive signals
Studying economic relationships
The report places regression within the broader family of supervised machine-learning methods and compares it with other approaches.
Classification in Investment Research
Classification approaches are useful when the desired result belongs to a category.
For example, an investment system could attempt to classify situations into categories such as:
Positive or negative market conditions
High or low risk
Improving or deteriorating business activity
Different market regimes
Classification can be especially useful when the objective is not to predict an exact numerical value but to determine which category an observation belongs to.
Unsupervised Machine Learning
Unsupervised learning takes a different approach.
Instead of providing the model with predefined outcomes, the system attempts to discover structures and relationships within the data.
The report discusses techniques such as:
Clustering
Factor analysis
Pattern discovery
Data grouping
This can be useful when investors do not know in advance what patterns exist in a dataset.
For example, clustering can help identify groups of assets or observations that behave similarly.
Clustering and Investment Analysis
Clustering groups observations based on similarities.
In finance, this can potentially be used to identify:
Similar companies
Similar securities
Market regimes
Behavioral patterns
Groups of economic indicators
Related investment signals
The important benefit is that clustering can reveal structures that may not be obvious from traditional analysis.
It allows investors to explore datasets without first imposing a predefined classification.
Factor Analysis
Factor analysis attempts to identify underlying factors that help explain relationships within a dataset.
Factor-based thinking has a long history in quantitative investing.
Machine-learning approaches can extend this idea by allowing investors to analyze larger and more complex collections of variables.
This creates an interesting connection between traditional quantitative finance and modern machine learning.
Deep Learning in Finance
The report also discusses Deep Learning, which uses multilayer neural networks to analyze complex patterns.
Deep learning became increasingly important because of improvements in:
Computing power
Data availability
Storage capacity
Machine-learning techniques
Deep-learning approaches can process complex and high-dimensional information and are particularly relevant to areas such as:
Image analysis
Text analysis
Pattern recognition
Natural-language processing
Complex prediction problems
The report explores the potential application of deep learning to investment-related problems.
Reinforcement Learning
Reinforcement learning is another approach discussed in the report.
Instead of learning only from labeled examples, reinforcement-learning systems learn through interaction and feedback.
An algorithm can explore different actions and learn from the results associated with those actions.
In an investment context, reinforcement learning is interesting because financial decision-making can involve sequential choices.
Potential areas of application include:
Trading strategies
Portfolio decisions
Dynamic allocation
Strategy optimization
Sequential decision-making
However, financial markets introduce significant complexity, uncertainty, and changing conditions, making this an especially challenging application.
Big Data and the Search for Investment Advantage
One of the major themes of the report is the search for new sources of investment advantage.
Traditional investment strategies can become crowded as more participants discover and use similar information.
Alternative data provides the possibility of finding information that is less widely used.
Machine learning can then help analyze that information at scale.
This creates a broader investment workflow:
New Data → Data Processing → Pattern Discovery → Signal Generation → Investment Decision
The report describes this movement as part of a broader transformation toward quantitative and data-driven investing.
From Fundamental Investing to Quantitative Investing
Traditional fundamental investing often involves studying companies, industries, management teams, financial statements, and economic conditions.
Quantitative investing approaches these questions more systematically through data and statistical methods.
Big Data and Machine Learning can push this transformation further by allowing investors to process information that would be difficult to evaluate manually.
This does not necessarily mean that fundamental analysis disappears.
Instead, the report discusses the increasing combination of fundamental and quantitative approaches.
The Importance of Data Quality
More data does not necessarily mean better investment decisions.
A large dataset may contain:
Noise
Errors
Missing information
Duplicates
Bias
Irrelevant variables
Changing relationships
Therefore, data preparation becomes a critical part of the investment process.
Before machine learning can produce useful insights, investors need to understand where the data comes from, how it was collected, how reliable it is, and whether it actually represents the phenomenon being studied.
Data Collection and Web-Based Information
The report also includes material on techniques for collecting data from websites.
This reflects an important aspect of the Big Data ecosystem: much of the information potentially useful for investment research exists in digital form.
However, collecting data is only the beginning.
A complete process may involve:
Finding relevant sources
Collecting information
Cleaning the data
Organizing datasets
Extracting useful features
Applying machine-learning methods
Testing results
Monitoring performance
This makes data engineering an important component of modern quantitative investment research.
Challenges of Machine Learning in Investing
Machine learning can be powerful, but applying it to financial markets is not straightforward.
Financial data presents several unique challenges.
Important challenges include:
Market conditions change over time
Historical relationships may disappear
Financial data can contain substantial noise
Models can overfit historical observations
Trading costs can reduce theoretical returns
Data acquisition can be expensive
Signals can become crowded
Some datasets may have limited historical coverage
Model performance can deteriorate after deployment
These challenges mean that a model that performs well in historical testing is not automatically a successful investment strategy.
Overfitting and Model Reliability
One of the most important concerns in machine-learning-based investing is overfitting.
Overfitting occurs when a model learns historical patterns too closely and fails to generalize to new situations.
This is particularly dangerous in financial research because researchers can test many possible variables, datasets, and strategies.
A model may appear highly successful simply because it has accidentally captured historical noise.
Therefore, robust testing and careful validation are essential.
The Cost of Alternative Data
Alternative datasets can vary significantly in cost.
Some datasets may be inexpensive, while comprehensive and specialized datasets can be extremely expensive.
The report emphasizes that investors should evaluate the potential usefulness of a dataset relative to the cost of acquiring and implementing it.
This leads to an important business question:
Does the information provided by the dataset justify its cost?
A technically impressive dataset is not necessarily a commercially valuable one.
The Big Data Ecosystem
The report also describes a growing ecosystem around Big Data and Artificial Intelligence.
This ecosystem includes:
Data providers
Data aggregators
Technology companies
Analytics platforms
Investment firms
Quantitative researchers
Machine-learning specialists
The report contains a handbook covering more than 500 alternative-data and technology providers, illustrating how large the ecosystem had already become by 2017.
The Role of Computing Power
The growth of Big Data would not have been possible without advances in computing.
Modern computing systems make it possible to:
Store enormous datasets
Process information quickly
Train complex models
Analyze large numbers of variables
Automate data-processing workflows
The report identifies increasing computing power and declining costs of computing and storage as important factors behind the Big Data transformation.
Big Data, AI, and the Future of Investing
The report presents Big Data and Machine Learning as technologies capable of significantly influencing investment management.
As more investors adopt these approaches, the investment industry can become increasingly data-driven.
This creates both opportunities and challenges.
Investors who successfully identify useful data and build reliable analytical systems may gain an advantage.
At the same time, widespread adoption can reduce the uniqueness of commonly used signals.
Therefore, the competitive advantage may increasingly come from:
Finding unique datasets
Processing data efficiently
Developing better models
Combining different information sources
Building robust investment systems
Continuously evaluating model performance
Why This Report Is Important for Data Science
Although the report is focused on investing, its concepts are highly relevant to data science.
It demonstrates a complete real-world application of data science:
Data Collection → Data Cleaning → Feature Development → Machine Learning → Prediction → Decision Making
This makes the report useful for people studying:
Data Science
Machine Learning
Artificial Intelligence
Quantitative Finance
Financial Analytics
Big Data
Alternative Data
Algorithmic Trading
It shows how theoretical machine-learning techniques can be connected to an actual industry problem.
Key Takeaways
1. Data Is Becoming a Competitive Asset
Modern organizations can generate enormous quantities of information. The ability to transform this information into useful insights can become a competitive advantage.
2. Alternative Data Expands Investment Research
Information from social media, transactions, satellites, mobile devices, and sensors can complement traditional financial datasets.
3. Machine Learning Helps Analyze Complexity
Machine learning allows investors to process large and complicated datasets and search for patterns systematically.
4. Different Problems Require Different Methods
Regression, classification, clustering, deep learning, and reinforcement learning have different purposes and strengths.
5. More Data Does Not Guarantee Better Results
Data quality, relevance, cost, and predictive value are more important than simply collecting huge quantities of information.
6. Financial Machine Learning Is Challenging
Changing markets, noise, overfitting, transaction costs, and competition can make financial prediction significantly harder than many standard machine-learning applications.
7. Human Judgment Still Matters
Machine learning can support investment research, but interpreting results, evaluating risks, understanding market conditions, and designing robust strategies remain important.
Who Should Read This Report?
This report is particularly valuable for:
Data science students
Machine-learning learners
Quantitative finance students
AI researchers
Financial analysts
Investment professionals
Algorithmic-trading enthusiasts
Python and machine-learning developers
Researchers interested in alternative data
It can also serve as a bridge between data science and finance, showing how machine-learning concepts can be applied to a complex real-world domain.
Download the PDF for free:
https://cpb-us-e2.wpmucdn.com/faculty.sites.uci.edu/dist/2/51/files/2018/05/JPM-2017-MachineLearningInvestments.pdf
Conclusion
Big Data and AI Strategies: Machine Learning and Alternative Data Approach to Investing provides a detailed look at how the combination of Big Data and Machine Learning was beginning to reshape investment research.
The central message is simple but powerful: modern investors have access to far more information than traditional financial datasets alone can provide. The challenge is not merely collecting this information, but determining which data is useful, processing it effectively, discovering meaningful patterns, and converting those insights into reliable decisions.
The report brings together alternative data, quantitative investing, machine learning, deep learning, reinforcement learning, and data technologies into a single investment framework.
Even though the report was published in 2017, its fundamental ideas remain highly relevant to understanding the evolution of data-driven investing. It provides an excellent example of how Big Data and AI can move from theoretical technologies into practical decision-making systems.
.png)
Posts
