Chapter 1
History & Landscape
For most of the 20th century, making predictions from data required a domain expert with intuition built over decades, or a statistician with a specific model already in mind. Neither scaled. The question the last 80 years of AI and data science have been trying to answer is: can machines learn patterns from data on their own, without being told the rules? Before we start to answer that question, we need to orient ourselves. This chapter is our map. It spans time and space: where the field came from, what it looks like today, and how the roles in it fit together. It’s the grounding the rest of this book relies on.
This chapter has no AWS SageMaker Notes sidebar. We’re making a map, not doing the work. The cloud-specific material begins at Chapter 2, once there’s something concrete to deploy.
1.1 Motivation
Data science isn’t a single discipline. It’s the intersection of three older fields: statistics (making inferences from data under uncertainty), computer science (building systems that process information efficiently), and domain expertise (knowing enough about a problem to ask the right questions). None of these are new. Statistics has been around since at least 1662, when John Graunt and Sir William Petty analysed London’s mortality rolls. Computer science arrived in 1936, when Alan Turing introduced the concept of the Turing machine. Domain expertise is as old as humanity. What is new is having all three at scale, at the same time, with cheap compute. Almost every technique in this book has a simpler ancestor. The “new” ideas in deep learning and large language models are extensions of ideas that are decades old.
1.2 Exposition
1.2.1 The Three Eras
Era 1 — Rules and Expert Systems (1950s-1980s). The first AI systems tried to encode human knowledge as rules: if symptom A and symptom B, then diagnosis C. The research focus of the era matched that structure. Engineers spent their time writing the rules themselves, one if/then at a time. They didn’t train a system to find the rules. These worked in narrow domains: chemistry (DENDRAL), medical diagnosis (MYCIN), computer configuration (XCON). But they failed to scale. Writing rules for everything a domain expert knows takes longer than training that expert. The insight that ended this era: the bottleneck isn’t compute or algorithms. It’s knowledge acquisition. Complex real-world domains can’t be hand-coded. They have to be learned from data. Two names got fixed in place during this era. John McCarthy coined the term “artificial intelligence” for the 1956 Dartmouth Conference. That workshop is usually credited with founding AI as a research field. “Data science” arrived later in the same era. Danish computer scientist Peter Naur used it in his 1974 book Concise Survey of Computer Methods, as a synonym for “datalogy,” a name he had proposed in the 1960s as a replacement for “computer science.” This doesn’t mean deep technical work wasn’t happening. The world wasn’t ready yet. The first neural networks were built during this era, on groundwork laid even earlier. Warren McCulloch and Walter Pitts proposed the first mathematical model of a neuron in 1943. Pure theory, no hardware. Frank Rosenblatt turned that idea into something real: the perceptron. He first simulated it in software in 1957, then built it as dedicated hardware, the Mark I Perceptron, demonstrated publicly in 1960. It read images through a 20x20 grid of photocells, making it one of the earliest computer vision systems. It was a single-layer network, not the multi-layer perceptron architecture used today. Stacking layers and training them with backpropagation didn’t arrive until the 1980s. But it’s still part of the architecture inside Large Language Models.
Era 2 — Statistical Machine Learning (1980s-2010s). Instead of encoding rules, the field pivoted to learning patterns from labelled examples. That pivot produced the algorithms that still dominate production ML today. Paul Werbos first described backpropagation in 1974. Rumelhart, Hinton, and Williams rediscovered it in 1986, and that made neural networks trainable. Support Vector Machines and AdaBoost, the first practical boosting method, both arrived in 1995. LSTMs came in 1997: recurrent networks that could learn dependencies across long sequences, which earlier recurrent networks could not. Random Forests and gradient boosting both arrived in 2001. Gradient boosting is the direct ancestor of XGBoost and LightGBM. The research focus moved with these tools, from writing logic to crafting features. That meant deciding which measurements, a pixel intensity, a transaction amount, a word count, predict the outcome, then feeding those into one of the algorithms above. Most practitioners at the time called this work “data mining”: finding patterns already sitting in large databases rather than encoding them by hand. The field’s own name caught up with the work late in this era. Statistician C. F. Jeff Wu argued for renaming statistics to data science in a 1997 lecture. William S. Cleveland made the case formally in a 2001 paper, proposing it as an expanded, independent field built on statistics’ technical core. This era had self-driving cars earlier than most people realise. Carnegie Mellon’s NavLab 1 drove itself down the road in 1986, steering by camera and onboard computer. A few years later, the same lab’s ALVINN system went further. It threw out hand-written steering rules. In their place it used a neural network, trained by backpropagation, that learned to steer straight from camera images. The limitation of this era was data and compute. Neural networks existed. Training them on anything interesting took too long.
Era 3 — Deep Learning and Foundation Models (2012-now). In 2012, a deep convolutional neural network called AlexNet won the ImageNet competition. It won by a margin that shocked the computer vision community. The ingredients had been assembling quietly for years: large labelled datasets, GPU compute, better training techniques. What followed was fast. GANs arrived in 2014. The Transformer architecture arrived in 2017 (“Attention Is All You Need”), replacing recurrence with attention and enabling massive training parallelism. BERT followed in 2018. GPT-3 landed in 2020 at 175 billion parameters, with emergent few-shot learning. ChatGPT arrived in 2022, combining instruction fine-tuning with RLHF. From 2023 onward came multimodal models, long context windows, agentic systems, and open-source models competitive with proprietary ones. The research focus shifted again. Instead of hand-crafting features, the model learns its own internal representations end-to-end, straight from raw pixels or tokens. What’s left for a human to do is prepare and curate the data it learns from at scale. Feature engineering moved up, from the individual column to the whole dataset. The key insight of this era: If in doubt, do as Texas does, and make it bigger. Models trained on enough data with enough compute develop capabilities nobody explicitly trained them for. Nobody fully understands why.
1.2.2 The Landscape Today
Data science today is not a bunch of data scientists sitting around doing research. It’s a cluster of specialisations with a shared toolbox:
| Role | Core focus | Where they live |
|---|---|---|
| Data Analyst | Understanding what the data says now | BI tools, SQL, visualisation |
| Data Engineer | Building pipelines that move and store data | Spark, Kafka, dbt, cloud warehouses |
| Data Scientist | Building models that predict or explain | sklearn, XGBoost, PyTorch |
| ML Engineer | Making models work reliably in production | MLOps, serving, monitoring |
| Research Scientist | Advancing the state of the art | Papers, novel architectures |
| AI/ML Solutions Architect | Designing systems that use all of the above | You, after this book |
As a Solutions Architect, your goal isn’t to be the best at any one of these. You’re a generalist. Your goal is to understand each area well enough to design systems that connect them. You also need to hold a credible conversation with each type of specialist. The AI stack most enterprise projects sit inside runs, roughly bottom to top, through six layers: raw, cleaned, labelled, and embedded data; then data engineering (pipelines, warehouses); then classical ML (XGBoost, sklearn, forecasting); then foundation models (GPT-4, Claude, Llama); then fine-tuned or prompted foundation models; then applications (chatbots, copilots, agents). Foundation models are increasingly a commodity. The differentiation is in data quality, fine-tuning, and the operational layer around the model. That operational layer is most of this book.
The table above is the technical side of the landscape. Every project also has a business side: account teams, strategists, product managers, and executive sponsors. They’re the people asking what problem is worth solving and whether it’s worth the budget, regardless of which company or industry it happens in. As a Solutions Architect, that’s part of your job description too. You translate between the technical roles above and this business side, in both directions. You own the feasibility conversation when the two disagree. A data scientist’s responsibility usually stops at translating a business ask into a measurable modelling problem. The ongoing relationship with the business side, the trade-offs, the expectation-setting, is not typically theirs to own. It is yours.
1.3 Trade-offs and Limits
Deep learning is better at unstructured data: images, text, audio, video. It’s also better for tasks where the relevant features aren’t known in advance, and for tasks with massive amounts of training data. Classical ML still wins on tabular data with clear features, most enterprise datasets. It wins in low-data regimes too, hundreds or thousands of examples rather than millions. It wins in situations requiring interpretability: regulated industries, medical, financial. And it wins on speed and cost. A gradient-boosted tree runs on a laptop. Kimi K3 does not.
Each approach has its own ceiling, not just a preference. Deep learning’s limit is compute and data. Without enough of both, a large network won’t beat a simpler model. It often won’t even train stably. Classical ML’s limit is the human doing the feature engineering. It can only be as good as the features fed into it. And hand-crafting those features doesn’t scale to raw images, audio, or free text the way deep learning does. Foundation models’ limit is their own generality. Trained to do everything reasonably well, they need fine-tuning, prompting, or retrieval on top before they do one specific enterprise task well. That adaptation step carries its own cost.
One of the most common mistakes in enterprise AI today: reaching for the biggest, most complex solution when a simpler one works. A foundation model, when a well-tuned XGBoost model would be cheaper, faster, more accurate, and more explainable. A multi-agent, LLM-orchestrated mess, when a static, deterministic pipeline is all that’s needed. Developing the judgement for which situation you’re in is a running thread through this entire book.
1.4 Procedures
Before any of the ideas above become useful, we need to add tools to our toolbox. The procedures below are the first-steps recipes this book returns to constantly. Every later chapter’s Worked Example assumes them.
Procedure 1.1 (Load a Dataset and Do a First Inspection). When to use: every time a new dataset is encountered. Never skip this.
import pandas as pd
df = pd.read_csv('data.csv')
print(df.shape) # (rows, columns)
df.head() # first 5 rows
df.info() # column names, types, non-null counts
df.describe() # summary stats for numeric columnsWhat to look for: column types that don’t match expectations, like a date column loaded as object. Missing values (df.isnull().sum()). Numeric ranges that don’t look sensible. Wildly large or small values often mean an encoding error, not a real observation. Investigate a person with a negative age, or someone born in 1792.
Procedure 1.2 (Select and Filter Rows and Columns). When to use: narrowing data down to the subset being analysed.
# one column, as a Series
df['column_name']
# multiple columns
df[['col_a', 'col_b']]
# filter by condition
df[df['year'] > 2010]
# AND — use & and |
df[(df['year'] > 2010) & (df['subfield'] == 'deep_learning')]Procedure 1.3 (Compute Grouped Summary Statistics). When to use: comparing a metric across categories.
df.groupby('category')['value'].mean()
df.groupby('year').agg(
total=('papers', 'sum'),
average=('papers', 'mean'),
count=('papers', 'count'),
)Procedure 1.4 (Create a Derived Column). When to use: feature engineering, or calculating a derived metric for a chart.
df['ratio'] = df['col_a'] / df['col_b']
df['yoy_change'] = df['value'].pct_change() * 100
df['label'] = df['value'].apply(
lambda x: 'high' if x > 100 else 'low'
)Common mistake: pct_change() returns NaN for the first row (there’s no prior value to compare against). That’s expected, not a bug. Drop it with .dropna() or fill with .fillna(0), depending on what the chart needs.
Procedure 1.5 (Plot and Annotate a Line Chart). When to use: showing how a value changes over an ordered variable, usually time, and marking a specific event on it.
import matplotlib.pyplot as plt
plt.figure(figsize=(10, 5))
plt.plot(
df['year'], df['value'],
label='My Series', color='#4E79A7', linewidth=2,
)
plt.axvline(x=2017, color='grey', linestyle='--', linewidth=1.5)
plt.text(
2017.2, plt.ylim()[1] * 0.9,
'Key event', fontsize=9, color='grey',
)
plt.xlabel('Year')
plt.ylabel('Value')
plt.legend()
plt.tight_layout()
# savefig BEFORE show()
plt.savefig(
'output.png', dpi=150, bbox_inches='tight',
)
plt.show()1.5 Worked Example
The tutorial notebook behind this chapter builds two things: a visual timeline of the milestones above, and a real trend analysis of machine learning research growth. Both use nothing but Procedures 1.1-1.5.
The timeline starts as data the reader defines by hand. That’s practice building structured records in Python before loading anything external:
RULES = "Rules & Expert Systems"
STATS_ML = "Statistical ML"
DEEP_LEARNING = "Deep Learning & Foundation Models"
milestones = [
{"year": 1950, "label": "Turing Test proposed", "era": RULES},
{"year": 1956, "label": "Dartmouth Conference", "era": RULES},
{"year": 1986, "label": "Backpropagation rediscovered", "era": STATS_ML},
{"year": 2001, "label": "Random Forests", "era": STATS_ML},
{"year": 2012, "label": "AlexNet wins ImageNet", "era": DEEP_LEARNING},
{
"year": 2017,
"label": "Transformer architecture ('Attention Is All You Need')",
"era": DEEP_LEARNING,
},
{"year": 2022, "label": "ChatGPT", "era": DEEP_LEARNING},
]
df_milestones = pd.DataFrame(milestones)
n_eras = df_milestones['era'].nunique()
print(f"{len(df_milestones)} milestones across {n_eras} eras")Procedure 1.5 plots this as points on a timeline, with era-coloured backgrounds and per-point annotation. It becomes a single figure spanning the same three eras this chapter opened with. It’s the same structured-data-then-visualise pattern used for feature engineering and distribution plots throughout the book.
The second half loads real arXiv paper counts by subfield and year, 1993 to 2024, pulled from the arXiv API. arXiv has no “deep learning” category, so each count is the number of papers that year whose abstract mentions one of the subfield’s key phrases, such as “deep learning” or “random forest”. A paper can count in more than one subfield. Then it asks a question instead of plotting it: what year did deep learning paper volume first exceed classical ML?
df['dl_vs_classical'] = df['deep_learning'] - df['classical_ml']
# .iloc[0] is the earliest year only if rows are in year order
df = df.sort_values('year')
crossover = df[df['dl_vs_classical'] > 0].iloc[0]
year = int(crossover['year'])
classical = crossover['classical_ml']
deep = crossover['deep_learning']
print(f"Deep learning overtook classical ML in: {year}")
print(f" Classical ML papers that year: {classical:,}")
print(f" Deep learning papers that year: {deep:,}")The crossover falls in 2015 (classical ML at 794 papers, deep learning at 933). Three years after AlexNet. Deep learning then grew fast, to more than 5 papers for every classical ML paper by 2020. The ratio has held near 5 since: 4.8 to 1 in 2024. Classical ML kept growing steadily the whole time. The newest surge is language models. Papers mentioning them more than doubled in both 2023 and 2024, and overtook deep learning in 2024.
One gotcha worth keeping: computing year-on-year growth with pct_change() (Procedure 1.4) can blow up. If a series starts at exactly zero, the first nonzero value produces an infinite percentage. That’s a real inf in the output. Not a bug. It’s the kind of artefact nlargest() will happily rank first if it isn’t excluded by hand. Once that artefact is excluded, the NLP series’ largest growth year is 2000 (350%), a low-base effect: 6 papers to 27. The first big jump on a large base is 2023 (169%, from 4,253 papers to 11,447), the year after ChatGPT’s release. 2024 follows at 112%.
1.6 History and Further Reading
Burkov, Ch. 1, gives a crisp introduction to what machine learning is and isn’t (Burkov 2019). Murphy’s Introduction, Ch. 1, gives the formal framing of supervised, unsupervised, and reinforcement learning (Murphy 2022). “Attention Is All You Need” started the transformer era. It’s worth skimming even on a first read of this book (Vaswani et al. 2017). Domingos’ “A Few Useful Things to Know About Machine Learning” is a classic essay. It’s still one of the best single pieces on ML intuition (Domingos 2012). For a broader frame around everything in this book, look at Sayama’s Introduction to the Modeling and Analysis of Complex Systems. It situates machine learning within the wider study of complex systems, rather than treating it as a standalone discipline. That’s a framing worth having before the field-specific chapters that follow narrow the focus (Sayama 2024).
1.7 How This Connects
This chapter builds on the systems thinking a Solutions Architect already has. It enables everything that follows. We’ve created the map the rest of the book explores. One common mistake: treating AI, machine learning, and deep learning as synonyms. They’re nested subsets. AI is the broadest term. Machine learning is a subset of it. Deep learning is a subset of machine learning that specifically uses neural networks.
1.8 Exercises
[W] Name the era (Rules & Expert Systems / Statistical ML / Deep Learning & Foundation Models) each of the following belongs to, and give one reason it belongs there: (a) Random Forests; (b) BERT; (c) MYCIN.
[W] Explain in one or two sentences why “AI,” “machine learning,” and “deep learning” are not synonyms, using the nesting relationship from this chapter.
[A] Using Listing 1.6 as a starting point, add three milestones of your choosing from outside this chapter’s list (they can be from any era). Then, using Procedure 1.2, determine how many of the resulting milestones fall after the year 2017.
[A] Reproduce Listing 1.7’s logic to find the year reinforcement learning paper volume first exceeded classical ML paper volume. Report the ratio between them in 2024.
[E] A customer’s data science team has 3,000 labelled examples of a tabular fraud-detection problem and is proposing to fine-tune a foundation model for it. Using this chapter’s Trade-offs section, write the two or three sentences you’d say to them. Include what you’d suggest they try first.
[E] The chapter notes that
pct_change()on a zero-to-nonzero transition produces an infinite percentage. Propose two different ways to handle this before ranking growth years withnlargest(). State a situation where each would be the wrong choice.[R] Read the Domingos essay cited above. Pick one “useful thing” from it that this chapter did not mention. Explain how it would have changed the way Listing 1.7’s finding was interpreted if applied to it.
1.9 Mental Model
Data science is applied statistics with better compute and worse theory. Machine learning is a way of fitting functions to data. Deep learning is what happens when those functions get very large and are trained on huge amounts of data. A foundation model is deep learning at a further extreme: even bigger networks, trained on even more data. It’s general enough to be pointed at many different tasks instead of one. That’s also why it sometimes does things nobody explicitly trained it to do. The rest is engineering. Keep this deflationary framing. It prevents being intimidated by complexity that is just implementation detail dressed up in jargon.