Libraries & Ecosystem

Understanding the Python data science ecosystem — not just how to call functions, but why each library exists and how they relate — distinguishes a practitioner from someone who has memorised syntax.

---

Q1: Why is NumPy faster than plain Python lists for numerical computation?

Show answer

Three reasons: contiguous memory, C-level loops, and vectorised operations.

Python lists store pointers to arbitrary Python objects. Each object carries type information and reference counts. Iterating a list means following pointers and dispatching Python operations for each element.

NumPy arrays store raw numeric values in a single contiguous block of typed memory (e.g., float64 — 8 bytes per element, nothing else). Operations are dispatched to pre-compiled C (or Fortran) routines that loop over that memory directly, bypassing Python's interpreter overhead entirely.

import numpy as np
import time

n = 1_000_000

# Python list
py_list = list(range(n))
start = time.perf_counter()
result = sum(x**2 for x in py_list)
py_time = time.perf_counter() - start

# NumPy array
np_array = np.arange(n)
start = time.perf_counter()
result = np.sum(np_array**2)
np_time = time.perf_counter() - start

print(f"Python: {py_time:.3f}s")
print(f"NumPy:  {np_time:.3f}s")
# NumPy is typically 20–100x faster

Additional benefit: NumPy releases the GIL during most operations, enabling true parallelism when used with threads.

The tradeoff: NumPy arrays are typed and fixed-shape. They cannot hold heterogeneous data (a list can hold [1, "text", None]; a NumPy array cannot).

---

Q2: What is the difference between a Pandas Series and a DataFrame?

Show answer

A Series is a one-dimensional labelled array. Think of it as a NumPy array with an index.

A DataFrame is a two-dimensional table of columns, where each column is a Series sharing a common row index.

import pandas as pd
import numpy as np

# Series
ages = pd.Series([25, 32, 41], index=["Alice", "Bob", "Carol"])
print(ages)
# Alice    25
# Bob      32
# Carol    41
# dtype: int64

print(ages["Bob"])    # 32
print(type(ages))     # <class 'pandas.core.series.Series'>

# DataFrame
df = pd.DataFrame({
    "name": ["Alice", "Bob", "Carol"],
    "age": [25, 32, 41],
    "score": [92.5, 88.0, 95.3]
})
print(df.dtypes)
# name      object
# age        int64
# score    float64

# Each column is a Series
print(type(df["age"]))   # <class 'pandas.core.series.Series'>

Key point: a DataFrame is conceptually a dict of Series objects sharing the same index. Operations like df["age"] + 1 operate element-wise, returning a new Series.

---

Q3: When should you use Pandas versus NumPy directly?

Show answer

Use NumPy when: - Data is purely numerical and homogeneous - You need broadcasting, linear algebra (np.dot, np.linalg), or FFT - You are writing performance-critical inner loops - You are passing data to ML libraries (scikit-learn expects NumPy arrays)

Use Pandas when: - Data is tabular with mixed types (strings + numbers + dates) - You need labelled axes — column names, named indices - You are doing data cleaning, reshaping, groupby, merge, or time series operations

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "age": [25, 32, 41, 28],
    "salary": [55000, 72000, 89000, 61000]
})

# Use NumPy when you need raw array computation
correlation_matrix = np.corrcoef(df["age"], df["salary"])
print(correlation_matrix)

# Use Pandas when labels and grouping matter
print(df.groupby(df["age"] > 30)["salary"].mean())
# age
# False    58000.0
# True     80500.0

In practice, you move data between them constantly: df.values or df.to_numpy() converts a DataFrame to a NumPy array; pd.DataFrame(array, columns=names) wraps an array in a DataFrame.

---

Q4: What is scikit-learn's design pattern and why is it powerful?

Show answer

Scikit-learn's design is built around three consistent interfaces:

• fit(X, y) — learn parameters from training data
• transform(X) — apply a learned transformation (preprocessing objects)
• predict(X) — output predictions (models)

Objects that implement both fit and transform also expose fit_transform(X) as a convenience.

from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
import numpy as np

X_train = np.array([[1, 200], [2, 150], [3, 300], [4, 250]])
y_train = np.array([0, 0, 1, 1])
X_test = np.array([[2.5, 225]])

# Pipeline chains fit/transform/predict automatically
pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression())
])

pipeline.fit(X_train, y_train)
print(pipeline.predict(X_test))          # [0] or [1]
print(pipeline.predict_proba(X_test))    # probabilities

Why this matters: - Any estimator can slot into Pipeline, GridSearchCV, or cross_val_score without special cases - Fitted parameters are stored as attributes ending in _ (e.g., scaler.mean_, model.coef_) — the underscore convention signals "set by fit, not by user" - You can write custom transformers by subclassing BaseEstimator and TransformerMixin and implementing fit + transform, and they instantly become compatible with the entire ecosystem

---

Q5: What is the difference between Matplotlib and Seaborn? When do you reach for each?

Show answer

Matplotlib is the foundation layer — low-level, highly configurable, but verbose. Everything in the Python visualisation ecosystem builds on it.

Seaborn is a high-level statistical visualisation library built on top of Matplotlib. It provides beautiful defaults and high-level functions for common statistical plots.

import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np

np.random.seed(42)
data = np.random.normal(loc=0, scale=1, size=200)

# Matplotlib — explicit control
fig, ax = plt.subplots()
ax.hist(data, bins=20, color="steelblue", edgecolor="white")
ax.set_title("Distribution")
ax.set_xlabel("Value")
plt.tight_layout()

# Seaborn — same result, much less code, better defaults
sns.histplot(data, bins=20, kde=True)
plt.title("Distribution with KDE")

Use Matplotlib when: - You need fine-grained control over every element (subplots, axes positions, custom annotations) - You are building publication figures with precise formatting requirements - You are animating plots

Use Seaborn when: - You want quick, attractive statistical visualisations (violin plots, pair plots, heatmaps, regression plots) - You are doing exploratory data analysis - You want to facet plots by categorical variables with minimal code

import seaborn as sns
tips = sns.load_dataset("tips")

# Faceted scatter with regression lines — 2 lines vs ~20 in Matplotlib
sns.lmplot(data=tips, x="total_bill", y="tip", hue="smoker", col="time")

The two are complementary. A common workflow: Seaborn for exploration, Matplotlib for final polish.

---

Q6: What is a virtual environment and why should you always use one?

Show answer

A virtual environment is an isolated Python installation with its own set of packages, separate from the system Python and from other projects.

# Create a virtual environment
python -m venv .venv

# Activate it (Windows)
.venv\Scripts\activate

# Activate it (macOS/Linux)
source .venv/bin/activate

# Install packages — go into the venv, not the system
pip install numpy pandas scikit-learn

# Capture the environment
pip freeze > requirements.txt

Why it matters: - Isolation — project A can use scikit-learn==1.2 and project B can use scikit-learn==1.5 without conflict - Reproducibility — requirements.txt or environment.yml captures exact versions so teammates get the same environment - Safety — you cannot accidentally break system packages

A common mistake: installing data science packages globally. Six months later, an upgrade for Project B breaks Project A silently because they share the same interpreter.

# Check you are in the right environment
import sys
print(sys.executable)   # should point to your .venv, not system Python

---

Q7: What is the difference between pip and conda?

Show answer

pip is Python's standard package installer. It installs Python packages from PyPI. It is fast and ubiquitous.

conda is a package and environment manager that handles Python packages and non-Python dependencies (C libraries, CUDA, MKL). It installs packages from conda channels (primarily conda-forge and defaults).

	pip	conda
Package source	PyPI	conda channels
Non-Python deps	No	Yes
Environment management	venv (separate tool)	Built-in
Speed	Faster	Slower (dependency solver)
Scientific C libs (BLAS, MKL)	Difficult	Straightforward

Use pip when: - Your project lives entirely in Python - You are deploying to production (Docker, cloud) - You need the latest version of a package (PyPI is more up to date)

Use conda when: - You need GPU libraries (CUDA, cuDNN) or native scientific libraries (BLAS/LAPACK) - You are working on Windows where C extensions are painful to compile via pip - You want a self-contained environment that includes the Python interpreter itself

You can mix them: conda for the base environment and GPU libs, pip for pure-Python packages not on conda. But manage conda environments first — pip inside conda can sometimes break solver state.

---

Q8: What are the standard import conventions for data science libraries?

Show answer

These aliases are universal conventions. Deviating from them makes your code harder for other practitioners to read at a glance.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import scipy.stats as stats

from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingRegressor
from sklearn.metrics import accuracy_score, mean_squared_error, roc_auc_score

import tensorflow as tf
from tensorflow import keras

import torch
import torch.nn as nn

The pattern is: import <library> as <short_alias> for top-level packages, and from <package> import <specific_class> for commonly used individual classes.

Avoid star imports (from numpy import *) — they pollute the namespace and make it impossible to tell where a function came from when reading the code later.

---

Q9: When should you use NumPy operations directly versus Pandas methods for the same task?

Show answer

For purely numeric tabular data, the answer usually comes down to readability vs performance vs whether you need labels.

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "a": [1.0, 2.0, 3.0, np.nan],
    "b": [4.0, 5.0, 6.0, 7.0]
})

# Pandas — reads naturally, handles NaN automatically
col_means = df.mean()            # NaN ignored by default
print(col_means)
# a    2.0
# b    5.5

# NumPy — faster, but does NOT ignore NaN by default
arr = df.to_numpy()
print(np.mean(arr, axis=0))      # [nan, 5.5] — NaN propagates
print(np.nanmean(arr, axis=0))   # [2.0, 5.5] — explicit NaN handling

Prefer Pandas when: - Data has missing values (Pandas handles NaN gracefully) - You need column labels in the result - You are doing groupby, merge, reshape, or time series operations

Prefer NumPy when: - Data is clean and numeric, no NaN - You need linear algebra (np.linalg.inv, np.dot) - Performance is critical (loop over millions of values) - You are passing data to a library that expects np.ndarray

A common pattern: use Pandas for data loading and cleaning, then call .to_numpy() before training an ML model.

---

Q10: How do you read library source code and documentation effectively?

Show answer

Knowing how to navigate unfamiliar library code is as important as memorising common functions.

In a notebook or REPL:

import pandas as pd
import inspect

# View the docstring
help(pd.DataFrame.merge)

# View the source code
print(inspect.getsource(pd.DataFrame.groupby))

# View method signature
import inspect
sig = inspect.signature(pd.read_csv)
print(sig)

In an IDE: hold Ctrl (or Cmd) and click on a function name to jump to its definition.

When reading docs, start with: 1. The function signature — what does it accept and return? 2. The parameters table — what does each argument do? 3. The examples section — these are usually the fastest way to understand behaviour 4. Notes/Warnings — edge cases the authors considered important enough to call out

When debugging unexpected behaviour: - Check the version: pd.__version__, sklearn.__version__. API changes between minor versions. - Read the changelog for your version range. - Use dir(obj) to list all methods on an object when you do not know what is available.

# Discover what methods a Series has
s = pd.Series([1, 2, 3])
public_methods = [m for m in dir(s) if not m.startswith("_")]
print(public_methods[:10])

The ability to navigate source code confidently is what separates a practitioner from someone dependent on Stack Overflow for every task.

---

Q11: What is the relationship between Pandas and NumPy under the hood?

Show answer

Pandas is built on NumPy. Under the hood, each column of a DataFrame is stored as a NumPy array (or a Pandas ExtensionArray for newer types like pd.NA nullable integers and categoricals).

import pandas as pd
import numpy as np

df = pd.DataFrame({"x": [1, 2, 3], "y": [4.0, 5.0, 6.0]})

# Each column's underlying array
print(df["x"].values)          # array([1, 2, 3])
print(type(df["x"].values))    # <class 'numpy.ndarray'>

# Full DataFrame as NumPy array
arr = df.to_numpy()
print(arr)
# [[1.  4.]
#  [2.  5.]
#  [3.  6.]]

Because of this relationship: - NumPy functions work directly on Series and DataFrame columns: np.log(df["y"]) returns a Series - Broadcasting rules from NumPy apply to Pandas arithmetic - Calling .to_numpy() is nearly free — it exposes the underlying buffer, not a copy

However, Pandas has moved toward its own Extension Array system for types like pd.Categorical, pd.StringDtype, and nullable integer types (Int64 with capital I). These do not expose a plain NumPy array via .values, which is why .to_numpy() is preferred over .values for compatibility.