Functional Programming & Iterators¶
Functional patterns make data transformation code more predictable, testable, and composable. Interviewers ask these questions to see whether you can write clean data pipelines and whether you understand Python's lazy evaluation model.
Q1: What is a pure function and why does it matter for data science code?¶
Show answer
A pure function produces the same output for the same input and has no side effects — it does not modify external state, write to disk, or mutate its arguments.
# Pure — same input always gives same output, no side effects
def scale(values, factor):
return [v * factor for v in values]
data = [1, 2, 3]
print(scale(data, 2)) # [2, 4, 6]
print(data) # [1, 2, 3] — unchanged
# Impure — modifies input in place
def scale_inplace(values, factor):
for i in range(len(values)):
values[i] *= factor
scale_inplace(data, 2)
print(data) # [2, 4, 6] — original mutated
Pure functions matter in data science because: - Reproducibility — re-running the same transformation always gives the same result - Testability — no setup or teardown, no mocking of global state - Composability — pure functions chain safely because they do not interfere with each other - Parallelism — pure functions can run in parallel without locks
A preprocessing pipeline of pure functions is easy to debug: you can test each step in isolation, cache intermediate results, and replay the pipeline on any subset of data.
Q2: How do map() and filter() work? When would you use them over a comprehension?¶
Show answer
map(func, iterable) applies a function to every element and returns a lazy iterator.
filter(func, iterable) keeps only elements where the function returns truthy, also as a lazy iterator.
numbers = [1, 2, 3, 4, 5, 6]
# map
doubled = list(map(lambda x: x * 2, numbers))
print(doubled) # [2, 4, 6, 8, 10, 12]
# filter
evens = list(filter(lambda x: x % 2 == 0, numbers))
print(evens) # [2, 4, 6]
# Chained
even_doubled = list(map(lambda x: x * 2, filter(lambda x: x % 2 == 0, numbers)))
print(even_doubled) # [4, 8, 12]
Equivalent comprehensions:
doubled = [x * 2 for x in numbers]
evens = [x for x in numbers if x % 2 == 0]
even_doubled = [x * 2 for x in numbers if x % 2 == 0]
When to prefer map/filter:
- You already have a named function and passing it avoids a lambda
- You need a lazy iterator for memory efficiency (no list() wrapping)
When to prefer comprehensions: - The logic is more readable inline - Most Python practitioners find comprehensions clearer
In practice, list comprehensions are more idiomatic Python. map and filter shine when composing with functools.reduce or when working with very large data streams.
Q3: What is functools.reduce and when is it the right tool?¶
Show answer
reduce(func, iterable) applies a two-argument function cumulatively to reduce a sequence to a single value.
from functools import reduce
# Sum — for demonstration (use built-in sum() in production)
numbers = [1, 2, 3, 4, 5]
total = reduce(lambda acc, x: acc + x, numbers)
print(total) # 15
# Product
product = reduce(lambda acc, x: acc * x, numbers)
print(product) # 120
# Flatten a list of lists
nested = [[1, 2], [3, 4], [5, 6]]
flat = reduce(lambda acc, x: acc + x, nested)
print(flat) # [1, 2, 3, 4, 5, 6]
You can provide an initial value as a third argument:
When reduce is the right tool:
- You need to collapse a collection to a single value using a custom accumulation logic
- The operation does not have a built-in equivalent (sum, max, min already exist)
- You are building a pipeline of transformations programmatically:
Q4: When should you use a lambda function and when should you avoid it?¶
Show answer
lambda creates an anonymous single-expression function. It is syntactic sugar — anything written as a lambda can be written as a def.
# lambda
double = lambda x: x * 2
print(double(5)) # 10
# equivalent def
def double(x):
return x * 2
Good uses of lambda:
- As the key argument to sorted, min, max:
records = [("Alice", 95), ("Bob", 87), ("Carol", 92)]
sorted_records = sorted(records, key=lambda r: r[1], reverse=True)
print(sorted_records)
# [('Alice', 95), ('Carol', 92), ('Bob', 87)]
- Short, throwaway functions inside
maporfiltercalls
Avoid lambda when:
- The function is used more than once — give it a name with def
- The expression is complex enough to need a variable or conditional
- You are assigning the lambda to a name (as in the double = lambda example above) — that is always better written as def
Flake8/pylint will warn on lambda assigned to a variable for this reason. The name aids debugging because tracebacks show the function name.
Q5: How does sorted() with a key function work? What makes an effective key?¶
Show answer
sorted() applies the key function to each element once, sorts by the resulting key values, and returns the original elements in sorted order.
words = ["banana", "apple", "kiwi", "cherry"]
# Sort by length
print(sorted(words, key=len))
# ['kiwi', 'apple', 'banana', 'cherry']
# Sort by last character
print(sorted(words, key=lambda w: w[-1]))
# ['banana', 'apple', 'kiwi', 'cherry']
# Sort by multiple criteria: primary = length, secondary = alphabetical
print(sorted(words, key=lambda w: (len(w), w)))
# ['kiwi', 'apple', 'banana', 'cherry']
operator.itemgetter and operator.attrgetter are faster than lambda for attribute/index access:
from operator import itemgetter, attrgetter
records = [{"name": "Alice", "score": 95}, {"name": "Bob", "score": 87}]
print(sorted(records, key=itemgetter("score"), reverse=True))
# [{'name': 'Alice', 'score': 95}, {'name': 'Bob', 'score': 87}]
The key function is called exactly once per element — Python's "Schwartzian transform" under the hood. This is more efficient than using cmp (which was removed in Python 3).
Q6: What is a generator and how does yield work?¶
Show answer
A generator is a function that produces values one at a time using yield, pausing execution between each value. It is a memory-efficient alternative to building a full list.
def count_up(start, stop):
current = start
while current < stop:
yield current
current += 1
gen = count_up(0, 5)
print(next(gen)) # 0
print(next(gen)) # 1
for value in count_up(0, 5):
print(value, end=" ")
# 0 1 2 3 4
When yield executes, the function's local state is frozen and control returns to the caller. The next call to next() resumes from where it left off.
Memory comparison:
import sys
# List — all values computed and stored immediately
lst = [x**2 for x in range(1_000_000)]
print(sys.getsizeof(lst)) # ~8 MB
# Generator — produces values on demand
gen = (x**2 for x in range(1_000_000))
print(sys.getsizeof(gen)) # ~112 bytes
A generator can only be iterated once. After it is exhausted, it yields nothing:
Q7: What is the difference between a generator function and a generator expression?¶
Show answer
Both produce lazy iterators. The difference is syntax and complexity.
Generator expression — inline, single expression:
Generator function — full function with yield, handles multi-step logic:
def read_batches(data, batch_size):
for start in range(0, len(data), batch_size):
yield data[start : start + batch_size]
data = list(range(10))
for batch in read_batches(data, 3):
print(batch)
# [0, 1, 2]
# [3, 4, 5]
# [6, 7, 8]
# [9]
Use a generator expression when the logic fits on one line. Use a generator function when you need loops, conditionals, or multiple yield points.
Generator functions can also use yield from to delegate to a sub-iterator:
Q8: How do zip() and enumerate() work, and when should you use them?¶
Show answer
enumerate(iterable, start=0) adds a counter to an iterable, yielding (index, value) pairs.
features = ["age", "income", "score"]
# Instead of range(len(...))
for i, name in enumerate(features):
print(f"Feature {i}: {name}")
# Feature 0: age
# Feature 1: income
# Feature 2: score
zip(*iterables) pairs elements from multiple iterables together, stopping at the shortest:
names = ["Alice", "Bob", "Carol"]
scores = [95, 87, 92]
for name, score in zip(names, scores):
print(f"{name}: {score}")
# Alice: 95
# Bob: 87
# Carol: 92
# Transpose a matrix (list of rows → list of columns)
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
transposed = list(zip(*matrix))
print(transposed)
# [(1, 4, 7), (2, 5, 8), (3, 6, 9)]
Both are lazy iterators in Python 3. Use itertools.zip_longest when you need to pad shorter iterables instead of stopping.
Q9: What is itertools.chain and when is it useful?¶
Show answer
itertools.chain(*iterables) concatenates multiple iterables into a single lazy iterator without creating a new list.
import itertools
train = [1, 2, 3]
val = [4, 5]
test = [6, 7, 8]
# Memory-efficient concatenation
all_data = itertools.chain(train, val, test)
print(list(all_data)) # [1, 2, 3, 4, 5, 6, 7, 8]
# chain.from_iterable for nested iterables
batches = [[1, 2], [3, 4], [5, 6]]
flat = list(itertools.chain.from_iterable(batches))
print(flat) # [1, 2, 3, 4, 5, 6]
Practical use case — streaming large file processing without loading everything into memory:
Q10: What is functools.partial and when is it useful?¶
Show answer
functools.partial(func, *args, **kwargs) creates a new callable with some arguments pre-filled — a partial application of the function.
from functools import partial
def power(base, exponent):
return base ** exponent
square = partial(power, exponent=2)
cube = partial(power, exponent=3)
print(square(5)) # 25
print(cube(3)) # 27
This is useful when you have a general function and want specialised versions without writing new wrapper functions.
Data science use case — configuring a metric for use with cross-validation:
from functools import partial
from sklearn.metrics import fbeta_score
# sklearn's cross_val_score expects scorer(estimator, X, y)
# f2_scorer pre-fills beta=2
f2_scorer = partial(fbeta_score, beta=2, average="binary")
Another common pattern — pre-configuring print for debugging in a pipeline:
Q11: What is functools.lru_cache and when does it provide a meaningful speedup?¶
Show answer
@lru_cache (Least Recently Used cache) memoises a function — it stores the result of each unique input combination and returns the cached result on repeated calls, skipping recomputation.
from functools import lru_cache
@lru_cache(maxsize=128)
def fibonacci(n):
if n < 2:
return n
return fibonacci(n - 1) + fibonacci(n - 2)
print(fibonacci(50)) # 12586269025 — instant, no stack overflow
Without the cache, naive recursive Fibonacci has O(2^n) time complexity. With the cache, it is O(n) because each unique n is computed once.
When it helps: - Pure functions with expensive computation called repeatedly with the same arguments - Recursive algorithms (DP problems) - Feature engineering functions called on the same input values many times
Requirements: - The function must be pure — same input → same output - All arguments must be hashable (no lists, dicts)
# Use tuple instead of list to make arguments hashable
@lru_cache(maxsize=None)
def compute_features(row_tuple):
return sum(row_tuple), max(row_tuple)
result = compute_features((1, 2, 3))
maxsize=None makes the cache unbounded — use with care in long-running processes.
Q12: What is a decorator and how do you write one?¶
Show answer
A decorator is a function that takes another function as input, wraps it with additional behaviour, and returns the wrapped version. The @ syntax is shorthand for func = decorator(func).
import time
from functools import wraps
def timer(func):
@wraps(func) # preserves the original function's name and docstring
def wrapper(*args, **kwargs):
start = time.perf_counter()
result = func(*args, **kwargs)
elapsed = time.perf_counter() - start
print(f"{func.__name__} took {elapsed:.4f}s")
return result
return wrapper
@timer
def train_model(n_samples):
data = list(range(n_samples))
return sum(data)
result = train_model(1_000_000)
# train_model took 0.0312s
Decorators with arguments require an extra layer of nesting:
def retry(n_attempts):
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
for attempt in range(n_attempts):
try:
return func(*args, **kwargs)
except Exception as e:
if attempt == n_attempts - 1:
raise
print(f"Attempt {attempt + 1} failed: {e}")
return wrapper
return decorator
@retry(n_attempts=3)
def fetch_data(url):
pass # might raise on network error
Always use @functools.wraps(func) inside your wrapper — without it, the wrapped function loses its __name__, __doc__, and __module__, which breaks debugging and documentation tools.